Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
---

(Updated Nov. 27, 2015, 5:05 a.m.)


Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.


Changes
---

2 Chnages--

1. Patch Appy failed . Merged with latest code 
2. Updated Test case TestQueryConstraints to use multiple drivers support . 
Also fixed a bug in this test case which was causing intermittent failure due 
to assertValidity() logic.
org.apache.lens.server.query.TestQueryConstraints.assertValidity()
if (count.running == 4) {
assertEquals(count.queued, 0);
  } else {
assertEquals(count.waiting, 0);
  }
Assuming … all candidate queries have moved to waiting from queued and 4 
queries are running. Lets say the 4th query finishes before we call 
queryService.getQueryCountSnapshot() from the test case assertValidity(), so 
running queries =3.  Now we call   assertEquals(count.waiting, 0), but the 
query service thread has still not pushed a waiting query to launched . In this 
case we will see waiting queries >0 even though running queries != 4


Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123


Repository: lens


Description
---

Ability to load multiple drivers on lens server. 

As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
loaded by lens server. Hence lens can not support for example, two jdbc 
instances, one for MySql and one for Vertica or just two different MySQl 
deployments. This can be a big limitation for some deployments.  

Made below changes in fix 
1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
structure was picked.  (Please refer to the discussion on JIRA if required - 
last few comments )
-conf
--drivers 
---hive 
driver1
-hivedriver-site.xml
driver2
-hivedriver-site.xml
---jdbc
driver1
-jdbcdriver-site.xml
driver2
-jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf 
location. Conf loaction is set while starting the server as -Dconfig.location = 
"" 

2. Added an abstract class  
"org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
implemenation for getting lens driver's fully qualified name and some methods 
for getting driver specific resource paths. All existing driver implementations 
extend this class. In future, we can also add common functionality here(if 
required) without affecting existing driver implementations that extend this 
abstract class.

3. Updated driver interface 
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely. 

4. Value of lens.server.drivers property in lens-site.xml (and 
default-lens-site) updated to include driver type and driver class name 
(earlier only driver class name was present).
hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver

5. Updated LensServerDAO to use driver qualified name instead of driver class 
name. The table column name is chnaged to drivername form driverclass 

6. The driver instance was earlier referred to by its class name 
(Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
now changed to driver's fully qualified name which has the driver type and 
driver name (Example :hive/hive1)

7. All metrics were also relying on driver class name. This is also changed to 
driver's fully qualified name.

8. Updated driver folder struture for test cases and lens server build (as per 
point1).

9. Query execution service will fail to start if no drivers are found. (This is 
a change from previous flow . We can discuss this if req.)  


Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation

Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
spark use different versions of Hive. This will not work as of now as we have 
common Hive jars as part of war classpath. We need to have driver specific jars 
which can have different version. For this, the jars and other hive version 
specific resources should be picked from driver's folder by a separate class 
loader).


Diffs (updated)
-

  lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
9595ce9 
  lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
  lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
83b919b 
  lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
  lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  

Re: Review Request 39842: Ability to load different instances of same driver class

[Rajat Khandelwal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review108191
---



lens-server/src/test/resources/drivers/mockHive/mockHive1/hivedriver-site.xml 
(lines 75 - 78)


Do we need this for this test?



lens-server/src/test/resources/drivers/mockHive/mockHive1/hivedriver-site.xml 
(lines 80 - 83)


Isn't name auto-inferred?


- Rajat Khandelwal


On Nov. 27, 2015, 10:35 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 27, 2015, 10:35 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 
> separate class loader).
> 
> 
> Diffs
> -
> 
>   lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
> 9595ce9 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
>   lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
> 83b919b 
>   lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
>   lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   
> 

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
---

(Updated Nov. 26, 2015, 8:33 a.m.)


Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.


Changes
---

Fixed intermittent test failure in 
org.apache.lens.server.TestServerRestart.testHiveServerRestart()
- Updated thread safe type for LensSessionImpl.ResourceEntry.restoreCount
- Updated Test case
- Added Driver Qualified name to some some logs in HiveDriver and 
QueryExecutionServiceImpl


Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123


Repository: lens


Description
---

Ability to load multiple drivers on lens server. 

As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
loaded by lens server. Hence lens can not support for example, two jdbc 
instances, one for MySql and one for Vertica or just two different MySQl 
deployments. This can be a big limitation for some deployments.  

Made below changes in fix 
1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
structure was picked.  (Please refer to the discussion on JIRA if required - 
last few comments )
-conf
--drivers 
---hive 
driver1
-hivedriver-site.xml
driver2
-hivedriver-site.xml
---jdbc
driver1
-jdbcdriver-site.xml
driver2
-jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf 
location. Conf loaction is set while starting the server as -Dconfig.location = 
"" 

2. Added an abstract class  
"org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
implemenation for getting lens driver's fully qualified name and some methods 
for getting driver specific resource paths. All existing driver implementations 
extend this class. In future, we can also add common functionality here(if 
required) without affecting existing driver implementations that extend this 
abstract class.

3. Updated driver interface 
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely. 

4. Value of lens.server.drivers property in lens-site.xml (and 
default-lens-site) updated to include driver type and driver class name 
(earlier only driver class name was present).
hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver

5. Updated LensServerDAO to use driver qualified name instead of driver class 
name. The table column name is chnaged to drivername form driverclass 

6. The driver instance was earlier referred to by its class name 
(Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
now changed to driver's fully qualified name which has the driver type and 
driver name (Example :hive/hive1)

7. All metrics were also relying on driver class name. This is also changed to 
driver's fully qualified name.

8. Updated driver folder struture for test cases and lens server build (as per 
point1).

9. Query execution service will fail to start if no drivers are found. (This is 
a change from previous flow . We can discuss this if req.)  


Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation

Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
spark use different versions of Hive. This will not work as of now as we have 
common Hive jars as part of war classpath. We need to have driver specific jars 
which can have different version. For this, the jars and other hive version 
specific resources should be picked from driver's folder by a separate class 
loader).


Diffs (updated)
-

  lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
9595ce9 
  lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
  lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
83b919b 
  lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
  lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 
72f1497 
  lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99 
  lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java 
f453416 
  lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 
c96ef20 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java 
fc57c94 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java
 98edc28 
  lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  

Re: Review Request 39842: Ability to load different instances of same driver class

[Amareshwari Sriramadasu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review108098
---

Ship it!


Ship It!

- Amareshwari Sriramadasu


On Nov. 26, 2015, 8:33 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 26, 2015, 8:33 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 
> separate class loader).
> 
> 
> Diffs
> -
> 
>   lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
> 9595ce9 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
>   lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
> 83b919b 
>   lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
>   lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 
> 72f1497 
>   lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 
> 14d9f99 
>   lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java 
> f453416 
>   lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 
> c96ef20 
>   
> 

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
---

(Updated Nov. 24, 2015, 9:39 a.m.)


Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.


Changes
---

- Updated test case to check for driver specific configration 
(lens.driver.test.drivername) also.
- Executed All test cases and tested lens examples


Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123


Repository: lens


Description
---

Ability to load multiple drivers on lens server. 

As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
loaded by lens server. Hence lens can not support for example, two jdbc 
instances, one for MySql and one for Vertica or just two different MySQl 
deployments. This can be a big limitation for some deployments.  

Made below changes in fix 
1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
structure was picked.  (Please refer to the discussion on JIRA if required - 
last few comments )
-conf
--drivers 
---hive 
driver1
-hivedriver-site.xml
driver2
-hivedriver-site.xml
---jdbc
driver1
-jdbcdriver-site.xml
driver2
-jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf 
location. Conf loaction is set while starting the server as -Dconfig.location = 
"" 

2. Added an abstract class  
"org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
implemenation for getting lens driver's fully qualified name and some methods 
for getting driver specific resource paths. All existing driver implementations 
extend this class. In future, we can also add common functionality here(if 
required) without affecting existing driver implementations that extend this 
abstract class.

3. Updated driver interface 
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely. 

4. Value of lens.server.drivers property in lens-site.xml (and 
default-lens-site) updated to include driver type and driver class name 
(earlier only driver class name was present).
hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver

5. Updated LensServerDAO to use driver qualified name instead of driver class 
name. The table column name is chnaged to drivername form driverclass 

6. The driver instance was earlier referred to by its class name 
(Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
now changed to driver's fully qualified name which has the driver type and 
driver name (Example :hive/hive1)

7. All metrics were also relying on driver class name. This is also changed to 
driver's fully qualified name.

8. Updated driver folder struture for test cases and lens server build (as per 
point1).

9. Query execution service will fail to start if no drivers are found. (This is 
a change from previous flow . We can discuss this if req.)  


Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation

Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
spark use different versions of Hive. This will not work as of now as we have 
common Hive jars as part of war classpath. We need to have driver specific jars 
which can have different version. For this, the jars and other hive version 
specific resources should be picked from driver's folder by a separate class 
loader).


Diffs (updated)
-

  lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
9595ce9 
  lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
  lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
83b919b 
  lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
  lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 
72f1497 
  lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99 
  lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java 
f453416 
  lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 
c96ef20 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java 
fc57c94 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java
 98edc28 
  lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d 
  lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 
a8b980f 
  

Re: Review Request 39842: Ability to load different instances of same driver class

[Amareshwari Sriramadasu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107744
---


Seems http://lens.apache.org/lenshome/install-and-run.html requires update.

- Amareshwari Sriramadasu


On Nov. 24, 2015, 9:39 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 24, 2015, 9:39 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 
> separate class loader).
> 
> 
> Diffs
> -
> 
>   lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
> 9595ce9 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
>   lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
> 83b919b 
>   lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
>   lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 
> 72f1497 
>   lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 
> 14d9f99 
>   lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java 
> f453416 
>   lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 
> c96ef20 
>   

Re: Review Request 39842: Ability to load different instances of same driver class

[Amareshwari Sriramadasu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107756
---



src/site/apt/admin/config-server.apt (line 42)


Conf/ should have been conf/


- Amareshwari Sriramadasu


On Nov. 24, 2015, 10:46 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 24, 2015, 10:46 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 
> separate class loader).
> 
> 
> Diffs
> -
> 
>   lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
> 9595ce9 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
>   lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
> 83b919b 
>   lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
>   lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 
> 72f1497 
>   lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 
> 14d9f99 
>   lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java 
> f453416 
>   

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
---

(Updated Nov. 24, 2015, 10:46 a.m.)


Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.


Changes
---

updated documenatation 
-install-and-run.apt
-config-server.apt


Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123


Repository: lens


Description
---

Ability to load multiple drivers on lens server. 

As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
loaded by lens server. Hence lens can not support for example, two jdbc 
instances, one for MySql and one for Vertica or just two different MySQl 
deployments. This can be a big limitation for some deployments.  

Made below changes in fix 
1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
structure was picked.  (Please refer to the discussion on JIRA if required - 
last few comments )
-conf
--drivers 
---hive 
driver1
-hivedriver-site.xml
driver2
-hivedriver-site.xml
---jdbc
driver1
-jdbcdriver-site.xml
driver2
-jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf 
location. Conf loaction is set while starting the server as -Dconfig.location = 
"" 

2. Added an abstract class  
"org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
implemenation for getting lens driver's fully qualified name and some methods 
for getting driver specific resource paths. All existing driver implementations 
extend this class. In future, we can also add common functionality here(if 
required) without affecting existing driver implementations that extend this 
abstract class.

3. Updated driver interface 
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely. 

4. Value of lens.server.drivers property in lens-site.xml (and 
default-lens-site) updated to include driver type and driver class name 
(earlier only driver class name was present).
hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver

5. Updated LensServerDAO to use driver qualified name instead of driver class 
name. The table column name is chnaged to drivername form driverclass 

6. The driver instance was earlier referred to by its class name 
(Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
now changed to driver's fully qualified name which has the driver type and 
driver name (Example :hive/hive1)

7. All metrics were also relying on driver class name. This is also changed to 
driver's fully qualified name.

8. Updated driver folder struture for test cases and lens server build (as per 
point1).

9. Query execution service will fail to start if no drivers are found. (This is 
a change from previous flow . We can discuss this if req.)  


Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation

Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
spark use different versions of Hive. This will not work as of now as we have 
common Hive jars as part of war classpath. We need to have driver specific jars 
which can have different version. For this, the jars and other hive version 
specific resources should be picked from driver's folder by a separate class 
loader).


Diffs (updated)
-

  lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
9595ce9 
  lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
  lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
83b919b 
  lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
  lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 
72f1497 
  lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99 
  lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java 
f453416 
  lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 
c96ef20 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java 
fc57c94 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java
 98edc28 
  lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d 
  lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 
a8b980f 
  
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java
 7772d16 

Re: Review Request 39842: Ability to load different instances of same driver class

[Amareshwari Sriramadasu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107753
---

Ship it!


Ship It!

- Amareshwari Sriramadasu


On Nov. 24, 2015, 10:46 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 24, 2015, 10:46 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 
> separate class loader).
> 
> 
> Diffs
> -
> 
>   lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
> 9595ce9 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
>   lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
> 83b919b 
>   lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
>   lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 
> 72f1497 
>   lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 
> 14d9f99 
>   lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java 
> f453416 
>   lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 
> c96ef20 
>   
> 

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
---

(Updated Nov. 24, 2015, 11:28 a.m.)


Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.


Changes
---

"Conf" changed to "conf" in config-server.apt.


Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123


Repository: lens


Description
---

Ability to load multiple drivers on lens server. 

As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
loaded by lens server. Hence lens can not support for example, two jdbc 
instances, one for MySql and one for Vertica or just two different MySQl 
deployments. This can be a big limitation for some deployments.  

Made below changes in fix 
1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
structure was picked.  (Please refer to the discussion on JIRA if required - 
last few comments )
-conf
--drivers 
---hive 
driver1
-hivedriver-site.xml
driver2
-hivedriver-site.xml
---jdbc
driver1
-jdbcdriver-site.xml
driver2
-jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf 
location. Conf loaction is set while starting the server as -Dconfig.location = 
"" 

2. Added an abstract class  
"org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
implemenation for getting lens driver's fully qualified name and some methods 
for getting driver specific resource paths. All existing driver implementations 
extend this class. In future, we can also add common functionality here(if 
required) without affecting existing driver implementations that extend this 
abstract class.

3. Updated driver interface 
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely. 

4. Value of lens.server.drivers property in lens-site.xml (and 
default-lens-site) updated to include driver type and driver class name 
(earlier only driver class name was present).
hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver

5. Updated LensServerDAO to use driver qualified name instead of driver class 
name. The table column name is chnaged to drivername form driverclass 

6. The driver instance was earlier referred to by its class name 
(Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
now changed to driver's fully qualified name which has the driver type and 
driver name (Example :hive/hive1)

7. All metrics were also relying on driver class name. This is also changed to 
driver's fully qualified name.

8. Updated driver folder struture for test cases and lens server build (as per 
point1).

9. Query execution service will fail to start if no drivers are found. (This is 
a change from previous flow . We can discuss this if req.)  


Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation

Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
spark use different versions of Hive. This will not work as of now as we have 
common Hive jars as part of war classpath. We need to have driver specific jars 
which can have different version. For this, the jars and other hive version 
specific resources should be picked from driver's folder by a separate class 
loader).


Diffs (updated)
-

  lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
9595ce9 
  lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
  lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
83b919b 
  lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
  lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 
72f1497 
  lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99 
  lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java 
f453416 
  lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 
c96ef20 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java 
fc57c94 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java
 98edc28 
  lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d 
  lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 
a8b980f 
  
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java
 7772d16 
  

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
---

(Updated Nov. 23, 2015, 11:40 a.m.)


Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.


Changes
---

-Fixed Comments and updated documentation
-Added Driver Qualified name to logs
-Build and Test case sanity
-Started Server and Tested with Len-Examples

[INFO] Reactor Summary:
[INFO] 
[INFO] Lens Checkstyle Rules . SUCCESS [2.214s]
[INFO] Lens .. SUCCESS [3.272s]
[INFO] Lens API .. SUCCESS [24.859s]
[INFO] Lens API for server and extensions  SUCCESS [22.549s]
[INFO] Lens Cube . SUCCESS [5:34.207s]
[INFO] Lens DB storage ... SUCCESS [19.570s]
[INFO] Lens Query Library  SUCCESS [15.746s]
[INFO] Lens Hive Driver .. SUCCESS [2:55.797s]
[INFO] Lens Driver for JDBC .. SUCCESS [39.876s]
[INFO] Lens Elastic Search Driver  SUCCESS [16.485s]
[INFO] Lens Server ... SUCCESS [8:14.493s]
[INFO] Lens client ... SUCCESS [33.955s]
[INFO] Lens CLI .. SUCCESS [53.948s]
[INFO] Lens Examples . SUCCESS [8.196s]
[INFO] Lens Ship Jars to Distributed Cache ... SUCCESS [1.791s]
[INFO] Lens Distribution . SUCCESS [9.214s]
[INFO] Lens ML Lib ... SUCCESS [1:20.010s]
[INFO] Lens ML Ext Distribution .. SUCCESS [2.412s]
[INFO] Lens Regression ... SUCCESS [13.762s]
[INFO] Lens UI ... SUCCESS [27.177s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 23:00.464s


Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123


Repository: lens


Description
---

Ability to load multiple drivers on lens server. 

As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
loaded by lens server. Hence lens can not support for example, two jdbc 
instances, one for MySql and one for Vertica or just two different MySQl 
deployments. This can be a big limitation for some deployments.  

Made below changes in fix 
1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
structure was picked.  (Please refer to the discussion on JIRA if required - 
last few comments )
-conf
--drivers 
---hive 
driver1
-hivedriver-site.xml
driver2
-hivedriver-site.xml
---jdbc
driver1
-jdbcdriver-site.xml
driver2
-jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf 
location. Conf loaction is set while starting the server as -Dconfig.location = 
"" 

2. Added an abstract class  
"org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
implemenation for getting lens driver's fully qualified name and some methods 
for getting driver specific resource paths. All existing driver implementations 
extend this class. In future, we can also add common functionality here(if 
required) without affecting existing driver implementations that extend this 
abstract class.

3. Updated driver interface 
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely. 

4. Value of lens.server.drivers property in lens-site.xml (and 
default-lens-site) updated to include driver type and driver class name 
(earlier only driver class name was present).
hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver

5. Updated LensServerDAO to use driver qualified name instead of driver class 
name. The table column name is chnaged to drivername form driverclass 

6. The driver instance was earlier referred to by its class name 
(Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
now changed to driver's fully qualified name which has the driver type and 
driver name (Example :hive/hive1)

7. All metrics were also relying on driver class name. This is also changed to 
driver's fully qualified name.

8. Updated driver folder struture for test cases and lens server build (as per 
point1).

9. Query execution service will fail to start if no drivers are found. (This is 
a change from previous flow . We can discuss this if req.)  


Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating 

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

> On Nov. 20, 2015, 7:52 a.m., Rajat Khandelwal wrote:
> > I see some instances where both the drivers are using same initialization 
> > but since property name is different(`lens.driver.hive.x` vs 
> > `lens.driver.jdbc.x`), the same code is repeated. Such code can go in the 
> > base class. example of such code:
> > 
> > ```
> > this.queryConstraints = 
> > getImplementations(QUERY_LAUNCHING_CONSTRAINT_FACTORIES_KEY, this.conf);
> > this.selectionPolicies = 
> > getImplementations(WAITING_QUERIES_SELECTION_POLICY_FACTORIES_KEY, 
> > this.conf);
> > ```
> > 
> > Secondly, the property names -- if unchanged -- will contain driver type 
> > now. I think that after this change, we can give a thought to property 
> > names. We can have a method in base class for looking up a property in conf:
> > 
> > 
> > ```
> > getProperty(conf, key){
> >   return conf.get('lens.driver.name.x', conf.get('lens.driver.type.x', 
> > conf.get('x', DEFAULT)))
> > }
> > ```
> > 
> > This will be backward compatible and allow us to move to a new convention 
> > of smaller property names. 
> > 
> > Thoughts?
> 
> Amareshwari Sriramadasu wrote:
> If we have common properties they can simply be lens.driver. 
> (with out type) - each driver can give its own default value or override in 
> site.

We can take this up as part of another JIRA, if really needed.


- Puneet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107323
---


On Nov. 23, 2015, 11:40 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 23, 2015, 11:40 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be 

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

> On Nov. 23, 2015, 1:05 p.m., Amareshwari Sriramadasu wrote:
> > tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml, line 31
> > 
> >
> > Shall we add lens server db url with one more jdbc driver, by default?

This should be done in production envt where we decide on the lens server DB ( 
hsql or my sql /etc) ?


- Puneet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107582
---


On Nov. 23, 2015, 11:40 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 23, 2015, 11:40 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 
> separate class loader).
> 
> 
> Diffs
> -
> 
>   lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
> 9595ce9 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
>   lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
> 83b919b 
>   lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
>   lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   
> 

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

> On Nov. 23, 2015, 10:44 a.m., Rajat Khandelwal wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, 
> > line 340
> > 
> >
> > I'm wondering whether we should move away from file names like 
> > `hivedriver-site.xml` and move to generic names like `driver-site.xml`. A 
> > driver's final conf will be an overlay of `drivers/lensdriver-site.xml`, 
> > `drivers/type/typedriver-site.xml`, `drivers/type/name/name-site.xml`. The 
> > conf initialization can be moved to `AbstractLensDriver` like with other 
> > common code. I'd tried to start a discussion on this in an earlier review.
> 
> Amareshwari Sriramadasu wrote:
> Minimal changes would be always better. Current layout and changes look 
> good to me. What do others think?

Common properties can be part of lens-site.xml for now and can be handled in 
AbstractLensDriver. If really required , later we can have a separate 
lensdriver-site.xml


- Puneet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107553
---


On Nov. 23, 2015, 11:40 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 23, 2015, 11:40 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

> On Nov. 23, 2015, 1:05 p.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java,
> >  line 231
> > 
> >
> > Can we have one configuration to be different in all drivers and 
> > validate the conf value for each driver?

Added a test key "lens.driver.test.key2" . The value will be driver's qualified 
name . Validating the same in test case


- Puneet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107582
---


On Nov. 23, 2015, 11:40 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 23, 2015, 11:40 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 
> separate class loader).
> 
> 
> Diffs
> -
> 
>   lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
> 9595ce9 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
>   lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
> 83b919b 
>   lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
>   

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

> On Nov. 20, 2015, 12:06 p.m., Amareshwari Sriramadasu wrote:
> > lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java,
> >  line 438
> > 
> >
> > Why is this test changed? Shouldnt have been.

The Driver config path was not set properly and the regular expression was not 
getting picked up which converts weekofyear to week.
Test Case Fixed ..


> On Nov. 20, 2015, 12:06 p.m., Amareshwari Sriramadasu wrote:
> > src/site/apt/admin/config-server.apt, line 45
> > 
> >
> > Can you illustrate the conf directory layout here, as done review board 
> > description? You can put it as block instead of paragraph

Updated


- Puneet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107352
---


On Nov. 20, 2015, 11:43 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 20, 2015, 11:43 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 
> separate class loader).
> 
> 
> Diffs
> -
> 
>   lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
> 9595ce9 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

> On Nov. 2, 2015, 10:35 a.m., Rajat Khandelwal wrote:
> > lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java, 
> > line 100
> > 
> >
> > Verify that no other occurences exist by doing `git grep 
> > selectedDriverClassName`

Found one instance in a comment.


- Puneet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review104688
---


On Nov. 20, 2015, 11:43 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 20, 2015, 11:43 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 
> separate class loader).
> 
> 
> Diffs
> -
> 
>   lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
> 9595ce9 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
>   lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
> 83b919b 
>   lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
>   lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 
> 

Re: Review Request 39842: Ability to load different instances of same driver class

[Amareshwari Sriramadasu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107352
---


Can you make sure you are able to bring up server in local setup and run 
examples?


lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java
 (line 438)


Why is this test changed? Shouldnt have been.



src/site/apt/admin/config-server.apt (line 45)


Can you illustrate the conf directory layout here, as done review board 
description? You can put it as block instead of paragraph


- Amareshwari Sriramadasu


On Nov. 20, 2015, 11:43 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 20, 2015, 11:43 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 
> separate class loader).
> 
> 
> Diffs
> -
> 
>   lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
> 9595ce9 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
>   lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
> 83b919b 
>   lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
> 

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
---

(Updated Nov. 20, 2015, 11:27 a.m.)


Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.


Changes
---

Added Documentation and test case. Removed redundant file (.gitignore)


Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123


Repository: lens


Description
---

Ability to load multiple drivers on lens server. 

As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
loaded by lens server. Hence lens can not support for example, two jdbc 
instances, one for MySql and one for Vertica or just two different MySQl 
deployments. This can be a big limitation for some deployments.  

Made below changes in fix 
1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
structure was picked.  (Please refer to the discussion on JIRA if required - 
last few comments )
-conf
--drivers 
---hive 
driver1
-hivedriver-site.xml
driver2
-hivedriver-site.xml
---jdbc
driver1
-jdbcdriver-site.xml
driver2
-jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf 
location. Conf loaction is set while starting the server as -Dconfig.location = 
"" 

2. Added an abstract class  
"org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
implemenation for getting lens driver's fully qualified name and some methods 
for getting driver specific resource paths. All existing driver implementations 
extend this class. In future, we can also add common functionality here(if 
required) without affecting existing driver implementations that extend this 
abstract class.

3. Updated driver interface 
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely. 

4. Value of lens.server.drivers property in lens-site.xml (and 
default-lens-site) updated to include driver type and driver class name 
(earlier only driver class name was present).
hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver

5. Updated LensServerDAO to use driver qualified name instead of driver class 
name. The table column name is chnaged to drivername form driverclass 

6. The driver instance was earlier referred to by its class name 
(Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
now changed to driver's fully qualified name which has the driver type and 
driver name (Example :hive/hive1)

7. All metrics were also relying on driver class name. This is also changed to 
driver's fully qualified name.

8. Updated driver folder struture for test cases and lens server build (as per 
point1).

9. Query execution service will fail to start if no drivers are found. (This is 
a change from previous flow . We can discuss this if req.)  


Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation

Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
spark use different versions of Hive. This will not work as of now as we have 
common Hive jars as part of war classpath. We need to have driver specific jars 
which can have different version. For this, the jars and other hive version 
specific resources should be picked from driver's folder by a separate class 
loader).


Diffs (updated)
-

  lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
9595ce9 
  lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
  lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
83b919b 
  lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
  lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 
72f1497 
  lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99 
  lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java 
f453416 
  lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 
c96ef20 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java 
fc57c94 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java
 98edc28 
  lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d 
  lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 
a8b980f 
  
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java
 

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
---

(Updated Nov. 20, 2015, 11:18 a.m.)


Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.


Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123


Repository: lens


Description
---

Ability to load multiple drivers on lens server. 

As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
loaded by lens server. Hence lens can not support for example, two jdbc 
instances, one for MySql and one for Vertica or just two different MySQl 
deployments. This can be a big limitation for some deployments.  

Made below changes in fix 
1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
structure was picked.  (Please refer to the discussion on JIRA if required - 
last few comments )
-conf
--drivers 
---hive 
driver1
-hivedriver-site.xml
driver2
-hivedriver-site.xml
---jdbc
driver1
-jdbcdriver-site.xml
driver2
-jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf 
location. Conf loaction is set while starting the server as -Dconfig.location = 
"" 

2. Added an abstract class  
"org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
implemenation for getting lens driver's fully qualified name and some methods 
for getting driver specific resource paths. All existing driver implementations 
extend this class. In future, we can also add common functionality here(if 
required) without affecting existing driver implementations that extend this 
abstract class.

3. Updated driver interface 
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely. 

4. Value of lens.server.drivers property in lens-site.xml (and 
default-lens-site) updated to include driver type and driver class name 
(earlier only driver class name was present).
hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver

5. Updated LensServerDAO to use driver qualified name instead of driver class 
name. The table column name is chnaged to drivername form driverclass 

6. The driver instance was earlier referred to by its class name 
(Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
now changed to driver's fully qualified name which has the driver type and 
driver name (Example :hive/hive1)

7. All metrics were also relying on driver class name. This is also changed to 
driver's fully qualified name.

8. Updated driver folder struture for test cases and lens server build (as per 
point1).

9. Query execution service will fail to start if no drivers are found. (This is 
a change from previous flow . We can discuss this if req.)  


Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation

Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
spark use different versions of Hive. This will not work as of now as we have 
common Hive jars as part of war classpath. We need to have driver specific jars 
which can have different version. For this, the jars and other hive version 
specific resources should be picked from driver's folder by a separate class 
loader).


Diffs (updated)
-

  lens-api/.gitignore PRE-CREATION 
  lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
9595ce9 
  lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
  lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
83b919b 
  lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
  lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 
72f1497 
  lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99 
  lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java 
f453416 
  lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 
c96ef20 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java 
fc57c94 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java
 98edc28 
  lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d 
  lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 
a8b980f 
  
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java
 7772d16 
  

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
---

(Updated Nov. 20, 2015, 11:43 a.m.)


Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.


Changes
---

There was an issue with preious two revisions . Please ignore them . 
Updated Documentation, few comments and test case


Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123


Repository: lens


Description
---

Ability to load multiple drivers on lens server. 

As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
loaded by lens server. Hence lens can not support for example, two jdbc 
instances, one for MySql and one for Vertica or just two different MySQl 
deployments. This can be a big limitation for some deployments.  

Made below changes in fix 
1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
structure was picked.  (Please refer to the discussion on JIRA if required - 
last few comments )
-conf
--drivers 
---hive 
driver1
-hivedriver-site.xml
driver2
-hivedriver-site.xml
---jdbc
driver1
-jdbcdriver-site.xml
driver2
-jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf 
location. Conf loaction is set while starting the server as -Dconfig.location = 
"" 

2. Added an abstract class  
"org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
implemenation for getting lens driver's fully qualified name and some methods 
for getting driver specific resource paths. All existing driver implementations 
extend this class. In future, we can also add common functionality here(if 
required) without affecting existing driver implementations that extend this 
abstract class.

3. Updated driver interface 
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely. 

4. Value of lens.server.drivers property in lens-site.xml (and 
default-lens-site) updated to include driver type and driver class name 
(earlier only driver class name was present).
hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver

5. Updated LensServerDAO to use driver qualified name instead of driver class 
name. The table column name is chnaged to drivername form driverclass 

6. The driver instance was earlier referred to by its class name 
(Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
now changed to driver's fully qualified name which has the driver type and 
driver name (Example :hive/hive1)

7. All metrics were also relying on driver class name. This is also changed to 
driver's fully qualified name.

8. Updated driver folder struture for test cases and lens server build (as per 
point1).

9. Query execution service will fail to start if no drivers are found. (This is 
a change from previous flow . We can discuss this if req.)  


Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation

Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
spark use different versions of Hive. This will not work as of now as we have 
common Hive jars as part of war classpath. We need to have driver specific jars 
which can have different version. For this, the jars and other hive version 
specific resources should be picked from driver's folder by a separate class 
loader).


Diffs (updated)
-

  lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
9595ce9 
  lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
  lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
83b919b 
  lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
  lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 
72f1497 
  lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99 
  lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java 
f453416 
  lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 
c96ef20 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java 
fc57c94 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java
 98edc28 
  lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
PRE-CREATION 
  lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d 
  lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 
a8b980f 
  

Re: Review Request 39842: Ability to load different instances of same driver class

[Amareshwari Sriramadasu]

> On Nov. 20, 2015, 7:52 a.m., Rajat Khandelwal wrote:
> > I see some instances where both the drivers are using same initialization 
> > but since property name is different(`lens.driver.hive.x` vs 
> > `lens.driver.jdbc.x`), the same code is repeated. Such code can go in the 
> > base class. example of such code:
> > 
> > ```
> > this.queryConstraints = 
> > getImplementations(QUERY_LAUNCHING_CONSTRAINT_FACTORIES_KEY, this.conf);
> > this.selectionPolicies = 
> > getImplementations(WAITING_QUERIES_SELECTION_POLICY_FACTORIES_KEY, 
> > this.conf);
> > ```
> > 
> > Secondly, the property names -- if unchanged -- will contain driver type 
> > now. I think that after this change, we can give a thought to property 
> > names. We can have a method in base class for looking up a property in conf:
> > 
> > 
> > ```
> > getProperty(conf, key){
> >   return conf.get('lens.driver.name.x', conf.get('lens.driver.type.x', 
> > conf.get('x', DEFAULT)))
> > }
> > ```
> > 
> > This will be backward compatible and allow us to move to a new convention 
> > of smaller property names. 
> > 
> > Thoughts?

If we have common properties they can simply be lens.driver. (with 
out type) - each driver can give its own default value or override in site.


- Amareshwari


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107323
---


On Nov. 2, 2015, 4:08 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 2, 2015, 4:08 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 

Re: Review Request 39842: Ability to load different instances of same driver class

[Rajat Khandelwal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107323
---


I see some instances where both the drivers are using same initialization but 
since property name is different(`lens.driver.hive.x` vs `lens.driver.jdbc.x`), 
the same code is repeated. Such code can go in the base class. example of such 
code:

```
this.queryConstraints = 
getImplementations(QUERY_LAUNCHING_CONSTRAINT_FACTORIES_KEY, this.conf);
this.selectionPolicies = 
getImplementations(WAITING_QUERIES_SELECTION_POLICY_FACTORIES_KEY, this.conf);
```

Secondly, the property names -- if unchanged -- will contain driver type now. I 
think that after this change, we can give a thought to property names. We can 
have a method in base class for looking up a property in conf:


```
getProperty(conf, key){
  return conf.get('lens.driver.name.x', conf.get('lens.driver.type.x', 
conf.get('x', DEFAULT)))
}
```

This will be backward compatible and allow us to move to a new convention of 
smaller property names. 

Thoughts?

- Rajat Khandelwal


On Nov. 2, 2015, 9:38 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 2, 2015, 9:38 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 
> separate class loader).
> 
> 
> Diffs
> -
> 
>   

Re: Review Request 39842: Ability to load different instances of same driver class

[Amareshwari Sriramadasu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review105437
---


Overall changes looked clean and nice. I really enjoyed reviewing the changes :)
Looking forward for the updated patch with documentation and testcases added.


lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java (line 
38)


Will it be possible to define driver types as constant strings or enum?


- Amareshwari Sriramadasu


On Nov. 2, 2015, 4:08 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 2, 2015, 4:08 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 
> separate class loader).
> 
> 
> Diffs
> -
> 
>   lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
> 9595ce9 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
>   lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
> 096fd7a 
>   lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0 
>   lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   
> 

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

> On Nov. 2, 2015, 10:35 a.m., Rajat Khandelwal wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java, 
> > line 195
> > 
> >
> > `driverName` or `drivername`?
> 
> Puneet Gupta wrote:
> +1. 
> A test case should have pointed this too. Not sure why it didn't. Will 
> change to lowercase.

Most DBs(incl.. MySQL) do not require strict case for column names. Thats why 
the test cases are passing, I guess. 
Will change to lower case though.


- Puneet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review104688
---


On Nov. 2, 2015, 4:08 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 2, 2015, 4:08 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 
> separate class loader).
> 
> 
> Diffs
> -
> 
>   lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
> 9595ce9 
>   lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee 
>   lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 
> 096fd7a 
>   lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
> PRE-CREATION 
>   

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review104876
---



lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
(line 953)


This is an existing property which is set in lens-ctl. I added a constant 
to refer to it in code.



lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java (line 
195)


+1. 
A test case should have pointed this too. Not sure why it didn't. Will 
change to lowercase.



lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
 (line 423)


The condition which I followed for failing is when not even one driver is 
found. In other cases the query service can start. 

Further check that created driver(s) can actaually execute queries (may be 
the database its pointing to is down or some configration parameter is wrong 
like user/pwd,etc )is not part of this change. It is kept same as before.


- Puneet Gupta


On Nov. 2, 2015, 4:08 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 2, 2015, 4:08 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 

Re: Review Request 39842: Ability to load different instances of same driver class

[Puneet Gupta]

> On Nov. 2, 2015, 10:35 a.m., Rajat Khandelwal wrote:
> > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java,
> >  line 953
> > 
> >
> > Should this be lens.server.config.location?

This is an existing property which is set in lens-ctl. I added a constant to 
refer to it in code.


> On Nov. 2, 2015, 10:35 a.m., Rajat Khandelwal wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java, 
> > line 195
> > 
> >
> > `driverName` or `drivername`?

+1. 
A test case should have pointed this too. Not sure why it didn't. Will change 
to lowercase.


> On Nov. 2, 2015, 10:35 a.m., Rajat Khandelwal wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java,
> >  line 439
> > 
> >
> > Should we throw an error here? It'll be better to fail fast.

The condition which I followed for failing is when not even one driver is 
found. In other cases (provided driver configartion is correct - as in 
configuration is syntactically correct, drivers folder exists , driver class 
instance creation is successful) the query service can start. 

Further check that created driver(s) can actaually execute queries (may be the 
database its pointing to is down or some configration parameter is wrong like 
user/pwd,etc )is not part of this change. It is kept same as before.


- Puneet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review104688
---


On Nov. 2, 2015, 4:08 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 2, 2015, 4:08 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if 

Re: Review Request 39842: Ability to load different instances of same driver class

[Rajat Khandelwal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review104688
---



lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java (line 
100)


Verify that no other occurences exist by doing `git grep 
selectedDriverClassName`



lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
(line 953)


Should this be lens.server.config.location?



lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java (line 
195)


`driverName` or `drivername`?



lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
 (line 423)


Should we throw an error here? It'll be better to fail fast.


- Rajat Khandelwal


On Nov. 2, 2015, 9:38 a.m., Puneet Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> ---
> 
> (Updated Nov. 2, 2015, 9:38 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
> 
> 
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Ability to load multiple drivers on lens server. 
> 
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be 
> loaded by lens server. Hence lens can not support for example, two jdbc 
> instances, one for MySql and one for Vertica or just two different MySQl 
> deployments. This can be a big limitation for some deployments.  
> 
> Made below changes in fix 
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder 
> structure was picked.  (Please refer to the discussion on JIRA if required - 
> last few comments )
> -conf
> --drivers 
> ---hive 
> driver1
> -hivedriver-site.xml
> driver2
> -hivedriver-site.xml
> ---jdbc
> driver1
> -jdbcdriver-site.xml
> driver2
> -jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf 
> location. Conf loaction is set while starting the server as -Dconfig.location 
> = "" 
> 
> 2. Added an abstract class  
> "org.apache.lens.server.api.driver.AbstractLensDriver". This has common 
> implemenation for getting lens driver's fully qualified name and some methods 
> for getting driver specific resource paths. All existing driver 
> implementations extend this class. In future, we can also add common 
> functionality here(if required) without affecting existing driver 
> implementations that extend this abstract class.
> 
> 3. Updated driver interface 
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely. 
> 
> 4. Value of lens.server.drivers property in lens-site.xml (and 
> default-lens-site) updated to include driver type and driver class name 
> (earlier only driver class name was present).
> hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver
> 
> 5. Updated LensServerDAO to use driver qualified name instead of driver class 
> name. The table column name is chnaged to drivername form driverclass 
> 
> 6. The driver instance was earlier referred to by its class name 
> (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is 
> now changed to driver's fully qualified name which has the driver type and 
> driver name (Example :hive/hive1)
> 
> 7. All metrics were also relying on driver class name. This is also changed 
> to driver's fully qualified name.
> 
> 8. Updated driver folder struture for test cases and lens server build (as 
> per point1).
> 
> 9. Query execution service will fail to start if no drivers are found. (This 
> is a change from previous flow . We can discuss this if req.)  
> 
> 
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
> 
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on 
> spark use different versions of Hive. This will not work as of now as we have 
> common Hive jars as part of war classpath. We need to have driver specific 
> jars which can have different version. For this, the jars and other hive 
> version specific resources should be picked from driver's folder by a 
> separate class loader).
> 
> 
> Diffs
> -
> 
>   lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 
> 9595ce9 
>