Re: Review Request 61881: Lens-1461:Lens services state are not persisted correctly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61881/ --- (Updated Oct. 17, 2017, 12:07 p.m.) Review request for lens and Puneet Gupta. Bugs: LENS-1461 https://issues.apache.org/jira/browse/LENS-1461 Repository: lens Description --- Persist process for Session service should not fail if any session in invalid. Persist service should ignore these session and continue persisting state of other session instead of failing. Diffs (updated) - lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java b480d140 Diff: https://reviews.apache.org/r/61881/diff/2/ Changes: https://reviews.apache.org/r/61881/diff/1-2/ Testing --- Thanks, Ankit Kailaswar
Review Request 61881: Lens-1461:Lens services state are not persisted correctly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61881/ --- Review request for lens and Puneet Gupta. Bugs: LENS-1461 https://issues.apache.org/jira/browse/LENS-1461 Repository: lens Description --- Persist process for Session service should not fail if any session in invalid. Persist service should ignore these session and continue persisting state of other session instead of failing. Diffs - lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java b480d14 Diff: https://reviews.apache.org/r/61881/diff/1/ Testing --- Thanks, Ankit Kailaswar
Re: Review Request 61881: Lens-1461:Lens services state are not persisted correctly
> On Aug. 29, 2017, 10:19 a.m., Puneet Gupta wrote: > > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java > > Lines 430 (patched) > > <https://reviews.apache.org/r/61881/diff/1/?file=1802852#file1802852line430> > > > > out.writeInt(SESSION_MAP.size()) would be wrong in this case . It > > should be SESSION_MAP.size()-1. How do we handle this ? changed logs to display correct value for number of session persisted. - Ankit --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61881/#review184038 ------- On Oct. 17, 2017, 12:07 p.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61881/ > --- > > (Updated Oct. 17, 2017, 12:07 p.m.) > > > Review request for lens and Puneet Gupta. > > > Bugs: LENS-1461 > https://issues.apache.org/jira/browse/LENS-1461 > > > Repository: lens > > > Description > --- > > Persist process for Session service should not fail if any session in > invalid. Persist service should ignore these session and continue persisting > state of other session instead of failing. > > > Diffs > - > > > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java > b480d140 > > > Diff: https://reviews.apache.org/r/61881/diff/2/ > > > Testing > --- > > > Thanks, > > Ankit Kailaswar > >
Re: Review Request 61881: Lens-1461:Lens services state are not persisted correctly
> On Aug. 28, 2017, 10:36 a.m., Amareshwari Sriramadasu wrote: > > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java > > Lines 428 (patched) > > <https://reviews.apache.org/r/61881/diff/1/?file=1802852#file1802852line428> > > > > ClientError path should not come in server path. Can we avoid this? BaseLensService for getSession is throwing ClientErrorException for, 1. invalid session handle 2. if resource for session is release my HiveSessionManager. Ideally we shoud throw ClientErrorException for invalid session only. Only possible reason for "2" is since we are using Hive session manager in lens to retrieve HiveSession object. We can discuss this later to decide for what could be better approach here. - Ankit --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61881/#review183912 ------- On Oct. 17, 2017, 12:07 p.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61881/ > --- > > (Updated Oct. 17, 2017, 12:07 p.m.) > > > Review request for lens and Puneet Gupta. > > > Bugs: LENS-1461 > https://issues.apache.org/jira/browse/LENS-1461 > > > Repository: lens > > > Description > --- > > Persist process for Session service should not fail if any session in > invalid. Persist service should ignore these session and continue persisting > state of other session instead of failing. > > > Diffs > - > > > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java > b480d140 > > > Diff: https://reviews.apache.org/r/61881/diff/2/ > > > Testing > --- > > > Thanks, > > Ankit Kailaswar > >
Re: Review Request 61881: Lens-1461:Lens services state are not persisted correctly
> On Oct. 25, 2017, 5:32 a.m., Rajat Khandelwal wrote: > > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java > > Lines 432 (patched) > > <https://reviews.apache.org/r/61881/diff/2/?file=1860577#file1860577line432> > > > > Won't this cause issues While reading sessions back? As the count of > > sessions persisted is written as `SESSION_MAP.size()`. It'll try to read > > that many session objects. What will happen if some of the sessions are not > > written at all? fixed it, now storing all sessions in memory before persisting individually. - Ankit --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61881/#review189159 ------- On Oct. 17, 2017, 12:07 p.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61881/ > --- > > (Updated Oct. 17, 2017, 12:07 p.m.) > > > Review request for lens and Puneet Gupta. > > > Bugs: LENS-1461 > https://issues.apache.org/jira/browse/LENS-1461 > > > Repository: lens > > > Description > --- > > Persist process for Session service should not fail if any session in > invalid. Persist service should ignore these session and continue persisting > state of other session instead of failing. > > > Diffs > - > > > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java > b480d140 > > > Diff: https://reviews.apache.org/r/61881/diff/2/ > > > Testing > --- > > > Thanks, > > Ankit Kailaswar > >
Re: Review Request 61881: Lens-1461:Lens services state are not persisted correctly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61881/ --- (Updated Oct. 26, 2017, 8:23 a.m.) Review request for lens and Puneet Gupta. Bugs: LENS-1461 https://issues.apache.org/jira/browse/LENS-1461 Repository: lens Description --- Persist process for Session service should not fail if any session in invalid. Persist service should ignore these session and continue persisting state of other session instead of failing. Diffs (updated) - lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java b480d140 Diff: https://reviews.apache.org/r/61881/diff/3/ Changes: https://reviews.apache.org/r/61881/diff/2-3/ Testing --- Thanks, Ankit Kailaswar
Review Request 67214: git diff origin/master master >> Lens-150
org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:30:09 [Thread-2] INFO org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 18 May 2018 09:33:17 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. Thanks, Ankit Kailaswar
Re: Review Request 67316: For SSL enabled lens client must always use trust manager for root X509 cert
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67316/ --- (Updated May 25, 2018, 9:59 a.m.) Review request for lens and Rajitha R. Bugs: LENS-1515 https://issues.apache.org/jira/browse/LENS-1515 Repository: lens Description --- In case of if cert verification is disabled then we are returning root cert. Also server cert validation will be taken care by default trust manager. Diffs - lens-client/src/main/java/org/apache/lens/client/LensTrustManager.java 4a696178 Diff: https://reviews.apache.org/r/67316/diff/1/ Testing (updated) --- https://build.corp.inmobi.com/view/Platform/job/Platform_Common_Job/2959/console untrusted cert, 25 May 2018 08:37:12 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 25 May 2018 08:37:12 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 25 May 2018 08:37:12 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 25 May 2018 08:37:12 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - Will skip server cert verification 1. 25 May 2018 08:37:12 [Spring Shell] INFO org.apache.lens.client.LensHostnameVerifier - Will skip hostname verification 1. 25 May 2018 08:37:13 [Spring Shell] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 25 May 2018 08:37:13 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - return root X509. 25 May 2018 08:37:14 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 25 May 2018 08:37:14 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - Will skip server cert verification 1. 25 May 2018 08:37:14 [Spring Shell] INFO org.apache.lens.client.LensHostnameVerifier - Will skip hostname verification 1. 25 May 2018 08:37:14 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - return root X509. 25 May 2018 08:37:20 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 25 May 2018 08:37:20 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - Will skip server cert verification 1. 25 May 2018 08:37:20 [Spring Shell] INFO org.apache.lens.client.LensHostnameVerifier - Will skip hostname verification 1. 25 May 2018 08:37:20 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - return root X509. CA signed cert, 25 May 2018 08:34:52 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 25 May 2018 08:34:52 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 25 May 2018 08:34:52 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 25 May 2018 08:34:53 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - Server cert verification is enabled. 25 May 2018 08:34:53 [Spring Shell] INFO org.apache.lens.client.LensHostnameVerifier - Host name verification is enabled. 25 May 2018 08:34:53 [Spring Shell] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 25 May 2018 08:34:53 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - return first CA X509 cert. 25 May 2018 08:34:54 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 25 May 2018 08:34:54 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - Server cert verification is enabled. 25 May 2018 08:34:54 [Spring Shell] INFO org.apache.lens.client.LensHostnameVerifier - Host name verification is enabled. 25 May 2018 08:34:54 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - return first CA X509 cert. Thanks, Ankit Kailaswar
Re: Review Request 67316: For SSL enabled lens client must always use trust manager for root X509 cert
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67316/ --- (Updated May 25, 2018, 10:56 a.m.) Review request for lens and Rajitha R. Bugs: LENS-1515 https://issues.apache.org/jira/browse/LENS-1515 Repository: lens Description (updated) --- In case of if cert verification is disabled then we should use root cert else we should rely on our trust manager which is first found instance of X509TrustManager in jvm TrustManagerFactory. Diffs - lens-client/src/main/java/org/apache/lens/client/LensTrustManager.java 4a696178 Diff: https://reviews.apache.org/r/67316/diff/1/ Testing --- https://build.corp.inmobi.com/view/Platform/job/Platform_Common_Job/2959/console untrusted cert, 25 May 2018 08:37:12 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 25 May 2018 08:37:12 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 25 May 2018 08:37:12 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 25 May 2018 08:37:12 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - Will skip server cert verification 1. 25 May 2018 08:37:12 [Spring Shell] INFO org.apache.lens.client.LensHostnameVerifier - Will skip hostname verification 1. 25 May 2018 08:37:13 [Spring Shell] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 25 May 2018 08:37:13 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - return root X509. 25 May 2018 08:37:14 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 25 May 2018 08:37:14 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - Will skip server cert verification 1. 25 May 2018 08:37:14 [Spring Shell] INFO org.apache.lens.client.LensHostnameVerifier - Will skip hostname verification 1. 25 May 2018 08:37:14 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - return root X509. 25 May 2018 08:37:20 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 25 May 2018 08:37:20 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - Will skip server cert verification 1. 25 May 2018 08:37:20 [Spring Shell] INFO org.apache.lens.client.LensHostnameVerifier - Will skip hostname verification 1. 25 May 2018 08:37:20 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - return root X509. CA signed cert, 25 May 2018 08:34:52 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 25 May 2018 08:34:52 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 25 May 2018 08:34:52 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 25 May 2018 08:34:53 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - Server cert verification is enabled. 25 May 2018 08:34:53 [Spring Shell] INFO org.apache.lens.client.LensHostnameVerifier - Host name verification is enabled. 25 May 2018 08:34:53 [Spring Shell] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 25 May 2018 08:34:53 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - return first CA X509 cert. 25 May 2018 08:34:54 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 25 May 2018 08:34:54 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - Server cert verification is enabled. 25 May 2018 08:34:54 [Spring Shell] INFO org.apache.lens.client.LensHostnameVerifier - Host name verification is enabled. 25 May 2018 08:34:54 [Spring Shell] INFO org.apache.lens.client.LensTrustManager - return first CA X509 cert. Thanks, Ankit Kailaswar
Re: Review Request 67323: SSL changes to ignore cert for python module
ava:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) at hudson.model.Build$BuildExecution.post2(Build.java:186) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) at hudson.model.Run.execute(Run.java:1749) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Caused by: com.sun.mail.smtp.SMTPAddressFailedException: 554 5.7.1 <barunstha...@gmail.com>: Relay access denied at com.sun.mail.smtp.SMTPTransport.rcptTo(SMTPTransport.java:1733) at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:1098) ... 14 more Finished: FAILURE Thanks, Ankit Kailaswar
Review Request 67323: SSL changes to ignore cert for python module
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67323/ --- Review request for lens and Rajitha R. Bugs: LENS-1516 https://issues.apache.org/jira/browse/LENS-1516 Repository: lens Description --- In Python module we need to ignore self signed server cert for each client request. Diffs - contrib/clients/python/lens/client/log.py e427f7fb contrib/clients/python/lens/client/query.py b09ff19d contrib/clients/python/lens/client/session.py 3b527e01 Diff: https://reviews.apache.org/r/67323/diff/1/ Testing --- https://build.corp.inmobi.com/view/Platform/job/Platform_Common_Job/2960/ python client logs /Library/Python/2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) /Library/Python/2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) /Library/Python/2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) /Library/Python/2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) Thanks, Ankit Kailaswar
Re: Review Request 67323: SSL changes to ignore cert for python module
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67323/ --- (Updated May 25, 2018, 2:26 p.m.) Review request for lens and Rajitha R. Bugs: LENS-1516 https://issues.apache.org/jira/browse/LENS-1516 Repository: lens Description --- In Python module we need to ignore self signed server cert for each client request. Diffs (updated) - contrib/clients/python/lens/client/log.py e427f7fb contrib/clients/python/lens/client/query.py b09ff19d contrib/clients/python/lens/client/session.py 3b527e01 Diff: https://reviews.apache.org/r/67323/diff/2/ Changes: https://reviews.apache.org/r/67323/diff/1-2/ Testing --- https://build.corp.inmobi.com/view/Platform/job/Platform_Common_Job/2960/ python client logs /Library/Python/2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) /Library/Python/2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) /Library/Python/2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) /Library/Python/2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) Thanks, Ankit Kailaswar
Re: Review Request 67214: git diff origin/master master >> Lens-150
[Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:30:09 [Thread-2] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:30:09 [Thread-2] INFO org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 18 May 2018 09:33:17 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. Thanks, Ankit Kailaswar
Re: Review Request 67214: git diff origin/master master >> Lens-150
> On May 21, 2018, 6:42 a.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/LensServer.java > > Lines 296 (patched) > > <https://reviews.apache.org/r/67214/diff/2/?file=2026212#file2026212line299> > > > > "null should be allowed here. We should reply on Grizzly server > > creation flow to throw specific Exception or Runtim Exception instead. > > " > > This reply is unclear..if the password is not set, are we throwing an > > exception here ? since I can't see any. Am I missing something? As discuused offline SSLContextConfigurator should handle password instead of we explicitly doing it. - Ankit --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67214/#review203481 ------- On May 21, 2018, 4:46 a.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67214/ > --- > > (Updated May 21, 2018, 4:46 a.m.) > > > Review request for lens. > > > Bugs: LENS-1511 > https://issues.apache.org/jira/browse/LENS-1511 > > > Repository: lens > > > Description > --- > > enable SSL/TLS for lens server and client communication > > > Diffs > - > > lens-client/src/main/java/org/apache/lens/client/LensClientConfig.java > eb12ee30 > lens-client/src/main/java/org/apache/lens/client/LensConnection.java > ab498314 > lens-client/src/main/java/org/apache/lens/client/LensHostnameVerifier.java > PRE-CREATION > lens-client/src/main/java/org/apache/lens/client/LensTrustManager.java > PRE-CREATION > lens-client/src/main/resources/lens-client-default.xml 132e1b14 > > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java > bda995dd > lens-server/src/main/java/org/apache/lens/server/LensServer.java 7f8b3363 > lens-server/src/main/resources/lensserver-default.xml bef82518 > lens-server/src/test/resources/lens-site.xml f5a02376 > src/site/apt/admin/config.apt 8720c822 > src/site/apt/admin/hivedriver-config.apt 221c1662 > src/site/apt/user/olap-query-conf.apt 90d635b8 > > > Diff: https://reviews.apache.org/r/67214/diff/2/ > > > Testing > --- > > build logs : > > INFO] Installing > /Users/ankit.kailaswar/inmobi/apache/lens1/lens/contrib/clients/python/pom.xml > to > /Users/ankit.kailaswar/.m2/repository/org/apache/lens/lens-python-client/2.8.0-SNAPSHOT/lens-python-client-2.8.0-SNAPSHOT.pom > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Lens Checkstyle Rules .. SUCCESS [ 1.732 > s] > [INFO] Lens ... SUCCESS [ 4.058 > s] > [INFO] Lens API ... SUCCESS [ 12.199 > s] > [INFO] Lens API for server and extensions . SUCCESS [ 11.348 > s] > [INFO] Lens Cube .. SUCCESS [ 20.835 > s] > [INFO] Lens DB storage SUCCESS [ 4.510 > s] > [INFO] Lens Query Library . SUCCESS [ 6.098 > s] > [INFO] Lens Hive Driver ... SUCCESS [ 8.414 > s] > [INFO] Lens Driver for JDBC ... SUCCESS [ 9.297 > s] > [INFO] Lens Elastic Search Driver . SUCCESS [ 7.908 > s] > [INFO] Lens Server SUCCESS [ 33.810 > s] > [INFO] Lens client SUCCESS [ 10.876 > s] > [INFO] Lens CLI ... SUCCESS [ 7.791 > s] > [INFO] Lens Examples .. SUCCESS [ 4.987 > s] > [INFO] Lens Ship Jars to Distributed Cache SUCCESS [ 0.771 > s] > [INFO] Lens Distribution .. SUCCESS [ 12.208 > s] > [INFO] Lens ML Lib SUCCESS [ 11.166 > s] > [INFO] Lens ML Ext Distribution ... SUCCESS [ 2.854 > s] > [INFO] Lens Regression SUCCESS [ 7.349 > s] > [INFO] Lens UI SUCCESS [ 2.988 > s] > [INFO] Lens Contrib ... SUCCESS [ 0.203 > s] > [INFO] Lens Con
Re: Review Request 67214: git diff origin/master master >> Lens-150
> On May 21, 2018, 7:35 a.m., Puneet Gupta wrote: > > lens-client/src/main/java/org/apache/lens/client/LensTrustManager.java > > Lines 63 (patched) > > <https://reviews.apache.org/r/67214/diff/2/?file=2026209#file2026209line63> > > > > Should we have separate config for client and server to ignore > > certificate check ? We are providing functionality of one way trust only, client will be verifying server's cert and vice versa is not true. We can have this feature in future with seperate config flag for server and client. > On May 21, 2018, 7:35 a.m., Puneet Gupta wrote: > > lens-client/src/main/java/org/apache/lens/client/LensTrustManager.java > > Lines 87 (patched) > > <https://reviews.apache.org/r/67214/diff/2/?file=2026209#file2026209line87> > > > > Should this not come from trustManager.getAcceptedIssuers() ? yes, updated the same. > On May 21, 2018, 7:35 a.m., Puneet Gupta wrote: > > lens-server/src/main/resources/lensserver-default.xml > > Lines 984 (patched) > > <https://reviews.apache.org/r/67214/diff/2/?file=2026213#file2026213line984> > > > > incomplete description. Please update updated - Ankit --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67214/#review203483 --- On May 21, 2018, 4:46 a.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67214/ > --- > > (Updated May 21, 2018, 4:46 a.m.) > > > Review request for lens. > > > Bugs: LENS-1511 > https://issues.apache.org/jira/browse/LENS-1511 > > > Repository: lens > > > Description > --- > > enable SSL/TLS for lens server and client communication > > > Diffs > - > > lens-client/src/main/java/org/apache/lens/client/LensClientConfig.java > eb12ee30 > lens-client/src/main/java/org/apache/lens/client/LensConnection.java > ab498314 > lens-client/src/main/java/org/apache/lens/client/LensHostnameVerifier.java > PRE-CREATION > lens-client/src/main/java/org/apache/lens/client/LensTrustManager.java > PRE-CREATION > lens-client/src/main/resources/lens-client-default.xml 132e1b14 > > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java > bda995dd > lens-server/src/main/java/org/apache/lens/server/LensServer.java 7f8b3363 > lens-server/src/main/resources/lensserver-default.xml bef82518 > lens-server/src/test/resources/lens-site.xml f5a02376 > src/site/apt/admin/config.apt 8720c822 > src/site/apt/admin/hivedriver-config.apt 221c1662 > src/site/apt/user/olap-query-conf.apt 90d635b8 > > > Diff: https://reviews.apache.org/r/67214/diff/2/ > > > Testing > --- > > build logs : > > INFO] Installing > /Users/ankit.kailaswar/inmobi/apache/lens1/lens/contrib/clients/python/pom.xml > to > /Users/ankit.kailaswar/.m2/repository/org/apache/lens/lens-python-client/2.8.0-SNAPSHOT/lens-python-client-2.8.0-SNAPSHOT.pom > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Lens Checkstyle Rules .. SUCCESS [ 1.732 > s] > [INFO] Lens ... SUCCESS [ 4.058 > s] > [INFO] Lens API ... SUCCESS [ 12.199 > s] > [INFO] Lens API for server and extensions . SUCCESS [ 11.348 > s] > [INFO] Lens Cube .. SUCCESS [ 20.835 > s] > [INFO] Lens DB storage SUCCESS [ 4.510 > s] > [INFO] Lens Query Library . SUCCESS [ 6.098 > s] > [INFO] Lens Hive Driver ... SUCCESS [ 8.414 > s] > [INFO] Lens Driver for JDBC ... SUCCESS [ 9.297 > s] > [INFO] Lens Elastic Search Driver . SUCCESS [ 7.908 > s] > [INFO] Lens Server SUCCESS [ 33.810 > s] > [INFO] Lens client SUCCESS [ 10.876 > s] > [INFO] Lens CLI ... SUCCESS [ 7.791 > s] > [INFO] Lens Examples .. SUCCESS [ 4.987 > s] > [INFO] Lens Ship Jars to Distributed Cache SUCCESS [ 0.771 > s] > [INFO] L
Re: Review Request 67214: enable SSL/TLS for lens server and client communication
2018 09:27:53 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:28:30 [Thread-2] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:28:30 [Thread-2] INFO org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:30:03 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:30:03 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:30:03 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:30:05 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:30:09 [Thread-2] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:30:09 [Thread-2] INFO org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 18 May 2018 09:33:17 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. Thanks, Ankit Kailaswar
Re: Review Request 67214: enable SSL/TLS for lens server and client communication
added requestfilter 18 May 2018 09:30:03 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:30:05 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:30:09 [Thread-2] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:30:09 [Thread-2] INFO org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 18 May 2018 09:33:17 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. Thanks, Ankit Kailaswar
Re: Review Request 67214: enable SSL/TLS for lens server and client communication
org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:30:05 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:30:09 [Thread-2] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:30:09 [Thread-2] INFO org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 18 May 2018 09:33:17 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. Thanks, Ankit Kailaswar
Re: Review Request 67214: git diff origin/master master >> Lens-150
> On May 18, 2018, 12:58 p.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/LensServer.java > > Lines 286 (patched) > > <https://reviews.apache.org/r/67214/diff/1/?file=2025351#file2025351line289> > > > > Is it ok if this is null? null should be allowed here. We should reply on Grizzly server creation flow to throw specific Exception or Runtim Exception instead. > On May 18, 2018, 12:58 p.m., Rajitha R wrote: > > lens-server/src/test/resources/lens-site.xml > > Lines 230 (patched) > > <https://reviews.apache.org/r/67214/diff/1/?file=2025353#file2025353line230> > > > > This might have to be set to something under /tmp like other configs Generally this has to be in install directory with specific privileges. Its better to have above mentioned path as default path for certs. - Ankit --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67214/#review203407 ------- On May 18, 2018, 11:22 a.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67214/ > --- > > (Updated May 18, 2018, 11:22 a.m.) > > > Review request for lens. > > > Repository: lens > > > Description > --- > > enable SSL/TLS for lens server and client communication > > > Diffs > - > > lens-client/src/main/java/org/apache/lens/client/LensClientConfig.java > b703e132 > lens-client/src/main/java/org/apache/lens/client/LensConnection.java > bb15b23c > lens-client/src/main/java/org/apache/lens/client/LensHostnameVerifier.java > PRE-CREATION > lens-client/src/main/java/org/apache/lens/client/LensTrustManager.java > PRE-CREATION > lens-client/src/main/resources/lens-client-default.xml 132e1b14 > > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java > f14ae44a > lens-server/src/main/java/org/apache/lens/server/LensServer.java 7f8b3363 > lens-server/src/main/resources/lensserver-default.xml bef82518 > lens-server/src/test/resources/lens-site.xml f5a02376 > src/site/apt/admin/config.apt 8720c822 > src/site/apt/admin/hivedriver-config.apt 221c1662 > src/site/apt/user/olap-query-conf.apt 90d635b8 > > > Diff: https://reviews.apache.org/r/67214/diff/1/ > > > Testing > --- > > build logs : > > INFO] Installing > /Users/ankit.kailaswar/inmobi/apache/lens1/lens/contrib/clients/python/pom.xml > to > /Users/ankit.kailaswar/.m2/repository/org/apache/lens/lens-python-client/2.8.0-SNAPSHOT/lens-python-client-2.8.0-SNAPSHOT.pom > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Lens Checkstyle Rules .. SUCCESS [ 1.732 > s] > [INFO] Lens ... SUCCESS [ 4.058 > s] > [INFO] Lens API ... SUCCESS [ 12.199 > s] > [INFO] Lens API for server and extensions . SUCCESS [ 11.348 > s] > [INFO] Lens Cube .. SUCCESS [ 20.835 > s] > [INFO] Lens DB storage SUCCESS [ 4.510 > s] > [INFO] Lens Query Library . SUCCESS [ 6.098 > s] > [INFO] Lens Hive Driver ... SUCCESS [ 8.414 > s] > [INFO] Lens Driver for JDBC ... SUCCESS [ 9.297 > s] > [INFO] Lens Elastic Search Driver . SUCCESS [ 7.908 > s] > [INFO] Lens Server SUCCESS [ 33.810 > s] > [INFO] Lens client SUCCESS [ 10.876 > s] > [INFO] Lens CLI ... SUCCESS [ 7.791 > s] > [INFO] Lens Examples .. SUCCESS [ 4.987 > s] > [INFO] Lens Ship Jars to Distributed Cache SUCCESS [ 0.771 > s] > [INFO] Lens Distribution .. SUCCESS [ 12.208 > s] > [INFO] Lens ML Lib SUCCESS [ 11.166 > s] > [INFO] Lens ML Ext Distribution ... SUCCESS [ 2.854 > s] > [INFO] Lens Regression SUCCESS [ 7.349 > s] > [INFO] Lens UI SUCCESS [ 2.988
Re: Review Request 67214: git diff origin/master master >> Lens-150
[Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:30:09 [Thread-2] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:30:09 [Thread-2] INFO org.apache.lens.client.LensConnection - SSL is disabled, Creating http client. 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnectionParams - Request filter added requestfilter 18 May 2018 09:33:15 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. 18 May 2018 09:33:17 [Spring Shell] INFO org.apache.lens.client.LensConnection - SSL is enabled, Creating https client. Thanks, Ankit Kailaswar
Review Request 67432: changes for Python3 compatibility issues with python client and ignore cert fo self signed cert
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67432/ --- Review request for lens. Repository: lens Description --- Fix of urlparse import in auth.py to make compatibiliy with Python 2 and 3. Jira : https://issues.apache.org/jira/browse/LENS-1517 And Fir for ignore cert variable in query.py Jira : https://issues.apache.org/jira/browse/LENS-1518 Diffs - contrib/clients/python/lens/client/auth.py fccc75c9 contrib/clients/python/lens/client/query.py 8b0616b6 contrib/clients/python/setup.py 5c64067e Diff: https://reviews.apache.org/r/67432/diff/1/ Testing --- Tested with Python 2.7.12 and Python 3.5.2 by Michael McCarthy for Lens-1517. Tested with Python 2.7 for Lens-1518 and queried on self signed and ca signed cert. Thanks, Ankit Kailaswar
Re: Review Request 61881: Lens-1461:Lens services state are not persisted correctly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61881/ --- (Updated Oct. 26, 2017, 3:28 p.m.) Review request for lens and Puneet Gupta. Bugs: LENS-1461 https://issues.apache.org/jira/browse/LENS-1461 Repository: lens Description --- Persist process for Session service should not fail if any session in invalid. Persist service should ignore these session and continue persisting state of other session instead of failing. Diffs (updated) - lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java b480d140 Diff: https://reviews.apache.org/r/61881/diff/4/ Changes: https://reviews.apache.org/r/61881/diff/3-4/ Testing --- Thanks, Ankit Kailaswar
Review Request 68267: Lens api to display number of sessions created per user
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68267/ --- Review request for lens and Rajitha R. Bugs: LENS-1526 https://issues.apache.org/jira/browse/LENS-1526 Repository: lens Description --- Exposing lens api to display number of sessions created per user. There is already an API which displays session info for each session. This API will dispplay number of sessions created per user. Diffs - lens-api/src/main/java/org/apache/lens/api/session/SessionPerUserInfo.java PRE-CREATION lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java 20ec6869 lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 006ee887 lens-server/src/main/java/org/apache/lens/server/session/SessionResource.java dfecba54 Diff: https://reviews.apache.org/r/68267/diff/1/ Testing --- sample output, XX5ankit.kailaswar1 Jira, https://issues.apache.org/jira/browse/LENS-1526 Build output, Installing /d0/jenkins/workspace/Platform_Common_Job/lens-ship-jars/target/lens-ship-jars-2.8.0-SNAPSHOT-tests.jar to /home/jenkins/.m2/repository/org/apache/lens/lens-ship-jars/2.8.0-SNAPSHOT/lens-ship-jars-2.8.0-SNAPSHOT-tests.jar [INFO] Installing /d0/jenkins/workspace/Platform_Common_Job/lens-ship-jars/target/lens-ship-jars-2.8.0-SNAPSHOT-sources.jar to /home/jenkins/.m2/repository/org/apache/lens/lens-ship-jars/2.8.0-SNAPSHOT/lens-ship-jars-2.8.0-SNAPSHOT-sources.jar [INFO] [INFO] [INFO] Building Lens Distribution 2.8.0-SNAPSHOT [INFO] [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ lens-dist --- [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ lens-dist --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ lens-dist --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 4 resources to /d0/jenkins/workspace/Platform_Common_Job/lens-dist/target/debian/control [INFO] skip non existing resourceDirectory /d0/jenkins/workspace/Platform_Common_Job/lens-dist/src/main/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ lens-dist --- [INFO] No sources to compile [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ lens-dist --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /d0/jenkins/workspace/Platform_Common_Job/lens-dist/src/test/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ lens-dist --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.15:test (default-test) @ lens-dist --- [INFO] [INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ lens-dist --- [INFO] Building jar: /d0/jenkins/workspace/Platform_Common_Job/lens-dist/target/lens-dist-2.8.0-SNAPSHOT.jar [INFO] [INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ lens-dist --- [INFO] [INFO] --- apache-rat-plugin:0.11:check (rat-check) @ lens-dist --- [INFO] 51 implicit excludes (use -debug for more details). [INFO] Exclude: **/README.* [INFO] Exclude: DEPENDENCIES [INFO] Exclude: **/LICENSE* [INFO] Exclude: .git/** [INFO] Exclude: .reviewboardrc [INFO] Exclude: **/patchprocess/** [INFO] Exclude: .idea/** [INFO] Exclude: **/*.iml [INFO] Exclude: **/.classpath [INFO] Exclude: **/.project [INFO] Exclude: **/node/** [INFO] Exclude: **/node_modules/** [INFO] Exclude: **/.checkstyle [INFO] Exclude: **/.settings/** [INFO] Exclude: **/maven-eclipse.xml [INFO] Exclude: **/.externalToolBuilders/** [INFO] Exclude: **/target/** [INFO] Exclude: **/build/** [INFO] Exclude: **/*.diff [INFO] Exclude: *.patch [INFO] Exclude: **/*.log [INFO] Exclude: **/*.png [INFO] Exclude: **/*.pptx [INFO] Exclude: **/*.data [INFO] Exclude: **/codemirror.min.* [INFO] Exclude: **/*.js [INFO] Exclude: **/*.json [INFO] Exclude: **/.tox/ [INFO] Exclude: **/.cache/ [INFO] Exclude: **/*.egg-info/** [INFO] Exclude: **/*.egg [INFO] 7 resources included (use -debug for more details) [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 7 licence. [INFO] [INFO] --- maven-checkstyle-plugin:2.17:check (checkstyle-check) @ lens-dist --- [INFO] Starting audit... Audit done. [INFO] [INFO] --- maven-assembly-plugin:2.4.1:single (assemble) @ lens-dist --- [INFO] Reading assembly descriptor: src/main/assembly/bin-dist.xml [WARNING] The following patterns were never triggered in this artifact inclusion filter: o 'org.apache.lens:lens-server' [WARNING] The following patterns were never
Re: Review Request 68471: Authorization check to accept session conf
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68471/#review208346 --- Ship it! Ship It! - Ankit Kailaswar On Aug. 22, 2018, 10:27 a.m., Rajitha R wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68471/ > --- > > (Updated Aug. 22, 2018, 10:27 a.m.) > > > Review request for lens, Amareshwari Sriramadasu and Puneet Gupta. > > > Repository: lens > > > Description > --- > > The current isAuthorized call in AuthorizationUtil is checking for session > related configuration from service conf which should rather be session conf. > This leads to incorrect session params being read for different sessions > causing authorization failures > > > Diffs > - > > > lens-cube/src/main/java/org/apache/lens/cube/authorization/AuthorizationUtil.java > ccd46a3 > > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java > e6afcff > > lens-cube/src/main/java/org/apache/lens/cube/parse/QueryAuthorizationResolver.java > 78dd642 > > > Diff: https://reviews.apache.org/r/68471/diff/2/ > > > Testing > --- > > [INFO] > [INFO] Lens Checkstyle Rules .. SUCCESS [ 2.734 > s] > [INFO] Lens ... SUCCESS [ 5.553 > s] > [INFO] Lens API ... SUCCESS [ 28.035 > s] > [INFO] Lens API for server and extensions . SUCCESS [ 25.289 > s] > [INFO] Lens Cube .. SUCCESS [08:06 > min] > [INFO] Lens DB storage SUCCESS [ 33.949 > s] > [INFO] Lens Query Library . SUCCESS [ 16.945 > s] > [INFO] Lens Hive Driver ... SUCCESS [01:57 > min] > [INFO] Lens Driver for JDBC ... SUCCESS [01:11 > min] > [INFO] Lens Elastic Search Driver . SUCCESS [ 40.179 > s] > [INFO] Lens Server SUCCESS [13:02 > min] > [INFO] Lens client SUCCESS [02:09 > min] > [INFO] Lens CLI ... SUCCESS [02:14 > min] > [INFO] Lens Examples .. SUCCESS [ 12.243 > s] > [INFO] Lens Ship Jars to Distributed Cache SUCCESS [ 2.325 > s] > [INFO] Lens Distribution .. SUCCESS [ 22.174 > s] > [INFO] Lens ML Lib SUCCESS [01:52 > min] > [INFO] Lens ML Ext Distribution ... SUCCESS [ 4.972 > s] > [INFO] Lens Regression SUCCESS [ 14.733 > s] > [INFO] Lens UI SUCCESS [ 6.186 > s] > [INFO] Lens Contrib ... SUCCESS [ 0.438 > s] > [INFO] Lens Contributed Clients ... SUCCESS [ 0.347 > s] > [INFO] Lens Python Client . SUCCESS [ 0.543 > s] > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 34:11 min > [INFO] Finished at: 2018-08-22T16:46:43+05:30 > [INFO] Final Memory: 269M/1572M > > > Thanks, > > Rajitha R > >
Re: Review Request 66081: Kerberos authentication in lens
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66081/ --- (Updated April 3, 2018, 12:18 p.m.) Review request for lens, Amareshwari Sriramadasu, Rajat Khandelwal, and Puneet Gupta. Repository: lens Description --- https://issues.apache.org/jira/browse/LENS-1506 This patch contains code changes to enable kerberos authentication for 1. lens to hive 2. lens to metastore 3. lens to hdfs code changes are as follows, 1. new http thrift client for hive driver to support sasl transport for kerberozied hive server. 2. cron to update KDC ticket before it expires. Diffs (updated) - lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 2eb94aa7 lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RemoteThriftConnection.java 54885f77 lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java PRE-CREATION lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java d5273be8 lens-server/src/main/java/org/apache/lens/server/BaseLensService.java c30a2d72 lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 31ac358d lens-server/src/main/resources/lensserver-default.xml 28b1db29 lens-server/src/test/resources/lens-site.xml 7e5f522e tools/scripts/lens-ctl 304b4f5a Diff: https://reviews.apache.org/r/66081/diff/6/ Changes: https://reviews.apache.org/r/66081/diff/5-6/ Testing --- unit testing Thanks, Ankit Kailaswar
Re: Review Request 66081: Kerberos authentication in lens
> On April 3, 2018, 12:06 p.m., Puneet Gupta wrote: > > lens-server/src/main/resources/lensserver-default.xml > > Lines 971 (patched) > > <https://reviews.apache.org/r/66081/diff/5/?file=1991690#file1991690line971> > > > > can you update this changed this to 360 minutes - Ankit --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66081/#review200352 --- On April 3, 2018, 8:14 a.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66081/ > --- > > (Updated April 3, 2018, 8:14 a.m.) > > > Review request for lens, Amareshwari Sriramadasu, Rajat Khandelwal, and > Puneet Gupta. > > > Repository: lens > > > Description > --- > > https://issues.apache.org/jira/browse/LENS-1506 > > This patch contains code changes to enable kerberos authentication for > 1. lens to hive > 2. lens to metastore > 3. lens to hdfs > > code changes are as follows, > 1. new http thrift client for hive driver to support sasl transport for > kerberozied hive server. > 2. cron to update KDC ticket before it expires. > > > Diffs > - > > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java > 2eb94aa7 > > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RemoteThriftConnection.java > 54885f77 > > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java > PRE-CREATION > > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java > d5273be8 > lens-server/src/main/java/org/apache/lens/server/BaseLensService.java > c30a2d72 > > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java > 31ac358d > lens-server/src/main/resources/lensserver-default.xml 28b1db29 > lens-server/src/test/resources/lens-site.xml 7e5f522e > tools/scripts/lens-ctl 304b4f5a > > > Diff: https://reviews.apache.org/r/66081/diff/5/ > > > Testing > --- > > unit testing > > > Thanks, > > Ankit Kailaswar > >
Re: Review Request 66081: Kerberos authentication in lens
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66081/ --- (Updated April 5, 2018, 10:03 a.m.) Review request for lens, Amareshwari Sriramadasu, Rajat Khandelwal, and Puneet Gupta. Repository: lens Description --- https://issues.apache.org/jira/browse/LENS-1506 This patch contains code changes to enable kerberos authentication for 1. lens to hive 2. lens to metastore 3. lens to hdfs code changes are as follows, 1. new http thrift client for hive driver to support sasl transport for kerberozied hive server. 2. cron to update KDC ticket before it expires. Diffs (updated) - lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 2eb94aa7 lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RemoteThriftConnection.java 54885f77 lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java PRE-CREATION lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java d5273be8 lens-server-api/src/main/java/org/apache/lens/server/api/util/LensUtil.java 9d732c1c lens-server/src/main/java/org/apache/lens/server/BaseLensService.java c30a2d72 lens-server/src/main/java/org/apache/lens/server/LensServices.java 903b19fe lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 31ac358d lens-server/src/main/resources/lensserver-default.xml 28b1db29 lens-server/src/test/resources/lens-site.xml 7e5f522e tools/scripts/lens-ctl 304b4f5a Diff: https://reviews.apache.org/r/66081/diff/8/ Changes: https://reviews.apache.org/r/66081/diff/7-8/ Testing --- unit testing Thanks, Ankit Kailaswar
Re: Review Request 66081: Kerberos authentication in lens
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66081/ --- (Updated April 5, 2018, 5:41 a.m.) Review request for lens, Amareshwari Sriramadasu, Rajat Khandelwal, and Puneet Gupta. Repository: lens Description --- https://issues.apache.org/jira/browse/LENS-1506 This patch contains code changes to enable kerberos authentication for 1. lens to hive 2. lens to metastore 3. lens to hdfs code changes are as follows, 1. new http thrift client for hive driver to support sasl transport for kerberozied hive server. 2. cron to update KDC ticket before it expires. Diffs (updated) - lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 2eb94aa7 lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RemoteThriftConnection.java 54885f77 lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java PRE-CREATION lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java d5273be8 lens-server-api/src/main/java/org/apache/lens/server/api/util/LensUtil.java 9d732c1c lens-server/src/main/java/org/apache/lens/server/BaseLensService.java c30a2d72 lens-server/src/main/java/org/apache/lens/server/LensServices.java 903b19fe lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 31ac358d lens-server/src/main/resources/lensserver-default.xml 28b1db29 lens-server/src/test/resources/lens-site.xml 7e5f522e tools/scripts/lens-ctl 304b4f5a Diff: https://reviews.apache.org/r/66081/diff/7/ Changes: https://reviews.apache.org/r/66081/diff/6-7/ Testing --- unit testing Thanks, Ankit Kailaswar
Review Request 66730: hive's server princiapl in lens's hive driver
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66730/ --- Review request for lens and Rajitha R. Repository: lens Description --- since the property name for lens and hive's principal name are conflicting we need explicitly need to mention hive's principal while creating hive driver for lens. (https://issues.apache.org/jira/browse/LENS-1510) Diffs - lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 3fd5d2c6 lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 0e05d28e Diff: https://reviews.apache.org/r/66730/diff/1/ Testing --- Thanks, Ankit Kailaswar
Re: Review Request 66730: hive's server princiapl in lens's hive driver
ARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.server2.authentication.ldap.security.protocol does not exist 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.server.setugi does not exist 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.hive.HiveDriver - set hive's server principal for hive driver as hive/mysql1001.grid.ev1.inmobi@inmobi.com 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.hive.HiveDriver - Hive driver hive/platinum configured successfully 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Driver platinum for type hive is loaded 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.api.query.cost.RangeBasedQueryCostTypeDecider - cost was: HIGH(0.0), decided querytype: HIGH 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.jdbc.JDBCDriver - JDBC Driver jdbc/clarity configured 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Driver clarity for type jdbc is loaded 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.api.query.cost.RangeBasedQueryCostTypeDecider - cost was: HIGH(0.0), decided querytype: HIGH 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.jdbc.JDBCDriver - JDBC Driver jdbc/jdbc1 configured 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Driver jdbc1 for type jdbc is loaded 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.api.query.cost.RangeBasedQueryCostTypeDecider - cost was: HIGH(0.0), decided querytype: HIGH 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.jdbc.JDBCDriver - JDBC Driver jdbc/prod configured 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Driver prod for type jdbc is loaded 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.api.query.cost.RangeBasedQueryCostTypeDecider - cost was: HIGH(-1.0), decided querytype: LOW 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.jdbc.JDBCDriver - JDBC Driver jdbc/memsql configured 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Driver memsql for type jdbc is loaded 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Using driver selector class: org.apache.lens.server.api.driver.MinQueryCostSelector Thanks, Ankit Kailaswar
Re: Review Request 66730: hive's server princiapl in lens's hive driver
d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.server2.authentication.ldap.security.protocol does not exist 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.server.setugi does not exist 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.hive.HiveDriver - set hive's server principal for hive driver as hive/mysql1001.grid.ev1.inmobi@inmobi.com 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.hive.HiveDriver - Hive driver hive/platinum configured successfully 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Driver platinum for type hive is loaded 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.api.query.cost.RangeBasedQueryCostTypeDecider - cost was: HIGH(0.0), decided querytype: HIGH 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.jdbc.JDBCDriver - JDBC Driver jdbc/clarity configured 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Driver clarity for type jdbc is loaded 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.api.query.cost.RangeBasedQueryCostTypeDecider - cost was: HIGH(0.0), decided querytype: HIGH 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.jdbc.JDBCDriver - JDBC Driver jdbc/jdbc1 configured 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Driver jdbc1 for type jdbc is loaded 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.api.query.cost.RangeBasedQueryCostTypeDecider - cost was: HIGH(0.0), decided querytype: HIGH 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.jdbc.JDBCDriver - JDBC Driver jdbc/prod configured 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Driver prod for type jdbc is loaded 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.api.query.cost.RangeBasedQueryCostTypeDecider - cost was: HIGH(-1.0), decided querytype: LOW 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.jdbc.JDBCDriver - JDBC Driver jdbc/memsql configured 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Driver memsql for type jdbc is loaded 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Using driver selector class: org.apache.lens.server.api.driver.MinQueryCostSelector Thanks, Ankit Kailaswar
Re: Review Request 66730: hive's server princiapl in lens's hive driver
d-4ee9-4879-972c-1d8a26d8b425] [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.server2.authentication.ldap.security.protocol does not exist 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.server.setugi does not exist 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.hive.HiveDriver - set hive's server principal for hive driver as hive/mysql1001.grid.ev1.inmobi@inmobi.com 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.hive.HiveDriver - Hive driver hive/platinum configured successfully 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Driver platinum for type hive is loaded 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.api.query.cost.RangeBasedQueryCostTypeDecider - cost was: HIGH(0.0), decided querytype: HIGH 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.jdbc.JDBCDriver - JDBC Driver jdbc/clarity configured 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Driver clarity for type jdbc is loaded 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.api.query.cost.RangeBasedQueryCostTypeDecider - cost was: HIGH(0.0), decided querytype: HIGH 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.jdbc.JDBCDriver - JDBC Driver jdbc/jdbc1 configured 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Driver jdbc1 for type jdbc is loaded 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.api.query.cost.RangeBasedQueryCostTypeDecider - cost was: HIGH(0.0), decided querytype: HIGH 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.jdbc.JDBCDriver - JDBC Driver jdbc/prod configured 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Driver prod for type jdbc is loaded 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.api.query.cost.RangeBasedQueryCostTypeDecider - cost was: HIGH(-1.0), decided querytype: LOW 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.driver.jdbc.JDBCDriver - JDBC Driver jdbc/memsql configured 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Driver memsql for type jdbc is loaded 20 Apr 2018 05:49:56 [0d280b8d-4ee9-4879-972c-1d8a26d8b425] [main] INFO org.apache.lens.server.query.QueryExecutionServiceImpl - Using driver selector class: org.apache.lens.server.api.driver.MinQueryCostSelector Thanks, Ankit Kailaswar
Re: Review Request 66081: Kerberos authentication in lens
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66081/ --- (Updated April 2, 2018, 12:58 p.m.) Review request for lens, Amareshwari Sriramadasu, Rajat Khandelwal, and Puneet Gupta. Repository: lens Description --- https://issues.apache.org/jira/browse/LENS-1506 This patch contains code changes to enable kerberos authentication for 1. lens to hive 2. lens to metastore 3. lens to hdfs code changes are as follows, 1. new http thrift client for hive driver to support sasl transport for kerberozied hive server. 2. cron to update KDC ticket before it expires. Diffs (updated) - lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 2eb94aa706c648b50dce1bbb5360ddd22a242295 lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RemoteThriftConnection.java 54885f77643d5723ab7735f5b666ce6bfa75a782 lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java PRE-CREATION lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java d5273be8b1d0c6c1e63da049c1cba712a5817903 lens-server/src/main/java/org/apache/lens/server/BaseLensService.java c30a2d72d27acdbab4c175e6e723f6369d934de9 lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 31ac358d0a1747b1dbfe1d6dacaf0fb82556475d lens-server/src/main/resources/lensserver-default.xml 28b1db29a8be7dc9762a2ce413a988d3b14b8543 lens-server/src/test/resources/lens-site.xml 7e5f522ea581e83dfd85210f27f10473e93c0097 tools/scripts/lens-ctl 304b4f5a8b82a5d313cdf31852cf502a214ba83b Diff: https://reviews.apache.org/r/66081/diff/3/ Changes: https://reviews.apache.org/r/66081/diff/2-3/ Testing --- unit testing Thanks, Ankit Kailaswar
Re: Review Request 66081: Kerberos authentication in lens
> On March 20, 2018, 6:37 a.m., Rajat Khandelwal wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java
> > Lines 125 (patched)
> > <https://reviews.apache.org/r/66081/diff/1/?file=1976384#file1976384line125>
> >
> > This class is also redundant. Can we reuse
> > `org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.InvocationResult`?
it is protected inner class, usage are scoped within package only. Wont be a
good idea to follow package struture for single class in lens. cant even extend
outside package, no default ctor available for InvocationResult in
RetryingThriftCLIServiceClient.
> On March 20, 2018, 6:37 a.m., Rajat Khandelwal wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java
> > Lines 137-182 (patched)
> > <https://reviews.apache.org/r/66081/diff/1/?file=1976384#file1976384line137>
> >
> > Both these methods can be easily avoided by simply extending
> > `RetryingThriftCLIServiceClient`.
it can't be extended form RetryingThriftCLIServiceClient please read above
reply to puneet's comment.
On March 20, 2018, 6:37 a.m., Ankit Kailaswar wrote:
> > Current design:
> > ```java
> > HiveConf conf;
> > if (conf.auth == Kerberose) {
> > org.apache.lens.driver.hive.RetryingThriftCLIServiceClientSasl.newClient
> > } else {
> >
> > org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.newClient
> > }
> >
> >
> > class RetryingThriftCLIServiceClientSasl {
> > // some reuse from RetryingThriftCLIServiceClient
> > // some code copied from RetryingThriftCLIServiceClient
> > }
> >
> > ```
> >
> > Proposal:
> >
> >
> > ```java
> > HiveConf conf;
> >
> > org.apache.lens.driver.hive.client.thrift.RetryingThriftCLIServiceClient.newClient
> >
> > class RetryingThriftCLIServiceClient extends
> > org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient {
> > @Override
> > protected synchronized TTransport connect(HiveConf conf) throws
> > HiveSQLException, TTransportException {
> > if (this.transport != null && this.transport.isOpen()) {
> > this.transport.close();
> > }
> >
> > String host = conf.getVar(ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST);
> > int port = conf.getIntVar(ConfVars.HIVE_SERVER2_THRIFT_PORT);
> > LOG.info("Connecting to " + host + ":" + port);
> > this.transport = new TSocket(host, port);
> >
> > ((TSocket)this.transport).setTimeout((int)conf.getTimeVar(ConfVars.SERVER_READ_SOCKET_TIMEOUT,
> > TimeUnit.SECONDS) * 1000);
> >
> > try {
> >
> > ((TSocket)this.transport).getSocket().setKeepAlive(conf.getBoolVar(ConfVars.SERVER_TCP_KEEP_ALIVE));
> > } catch (SocketException var8) {
> > LOG.error("Error setting keep alive to " +
> > conf.getBoolVar(ConfVars.SERVER_TCP_KEEP_ALIVE), var8);
> > }
> > /* Move the following code to new method getTransport below in
> > the else part:
> > String userName =
> > conf.getVar(ConfVars.HIVE_SERVER2_THRIFT_CLIENT_USER);
> > String passwd =
> > conf.getVar(ConfVars.HIVE_SERVER2_THRIFT_CLIENT_PASSWORD);
> >
> > try {
> > this.transport = PlainSaslHelper.getPlainTransport(userName,
> > passwd, this.transport);
> > } catch (SaslException var7) {
> > LOG.error("Error creating plain SASL transport", var7);
> > }
> > */
> > this.transport = getTransport(conf);
> > TProtocol protocol = new TBinaryProtocol(this.transport);
> > this.transport.open();
> > this.base = new ThriftCLIServiceClient(new Client(protocol));
> > LOG.info("Connected!");
> > return this.transport;
> > }
> > protected TTransport getTransport(HiveConf conf) {
> > if (conf.auth == Kerberose) {
> > // use KerberoseSaslHelper
> > } else {
> > // use PlainSaslHelper
> > }
> > }
> > }
> >
> > ```
member variables of base class cant be accesses from derived class.
- Ankit
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66081/#review199524
--
Re: Review Request 66081: Kerberos authentication in lens
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66081/ --- (Updated April 3, 2018, 8:14 a.m.) Review request for lens, Amareshwari Sriramadasu, Rajat Khandelwal, and Puneet Gupta. Repository: lens Description --- https://issues.apache.org/jira/browse/LENS-1506 This patch contains code changes to enable kerberos authentication for 1. lens to hive 2. lens to metastore 3. lens to hdfs code changes are as follows, 1. new http thrift client for hive driver to support sasl transport for kerberozied hive server. 2. cron to update KDC ticket before it expires. Diffs (updated) - lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 2eb94aa7 lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RemoteThriftConnection.java 54885f77 lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java PRE-CREATION lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java d5273be8 lens-server/src/main/java/org/apache/lens/server/BaseLensService.java c30a2d72 lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 31ac358d lens-server/src/main/resources/lensserver-default.xml 28b1db29 lens-server/src/test/resources/lens-site.xml 7e5f522e tools/scripts/lens-ctl 304b4f5a Diff: https://reviews.apache.org/r/66081/diff/5/ Changes: https://reviews.apache.org/r/66081/diff/4-5/ Testing --- unit testing Thanks, Ankit Kailaswar
Re: Review Request 66081: Kerberos authentication in lens
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66081/ --- (Updated March 20, 2018, 10:27 a.m.) Review request for lens, Amareshwari Sriramadasu, Rajat Khandelwal, and Puneet Gupta. Repository: lens Description --- https://issues.apache.org/jira/browse/LENS-1506 This patch contains code changes to enable kerberos authentication for 1. lens to hive 2. lens to metastore 3. lens to hdfs code changes are as follows, 1. new http thrift client for hive driver to support sasl transport for kerberozied hive server. 2. cron to update KDC ticket before it expires. Diffs - lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 2eb94aa lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RemoteThriftConnection.java 54885f7 lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java PRE-CREATION lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java d5273be lens-server/src/main/java/org/apache/lens/server/BaseLensService.java b9fcdd8 lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 31ac358 Diff: https://reviews.apache.org/r/66081/diff/1/ Testing --- unit testing Thanks, Ankit Kailaswar
Re: Review Request 66081: Kerberos authentication in lens
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66081/#review199570 --- lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java Lines 34 (patched) <https://reviews.apache.org/r/66081/#comment279889> added. lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java Lines 36 (patched) <https://reviews.apache.org/r/66081/#comment279888> If we try to extend RetryingThriftCLIServiceClient then it forces us to override all member functions since we cant access private data members of base class in derived class. we are doing it now anyhow but invokeInternal in base class refer to inner protected class "InvocationResult" if we try to overide invokeinternal then we are force to use "InvocationResult" in our implementation. since this class is inner protected class it cant be accessed from outside the class. This problem can be solved by extending InvocationResult. This require initialising base class in derived class's constructor causing same issue of not been able to use class outside package. lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java Lines 37-42 (patched) <https://reviews.apache.org/r/66081/#comment279890> added lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java Lines 44 (patched) <https://reviews.apache.org/r/66081/#comment279891> removed lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java Lines 51 (patched) <https://reviews.apache.org/r/66081/#comment279892> made it private lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java Lines 61 (patched) <https://reviews.apache.org/r/66081/#comment279893> added this lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java Lines 68 (patched) <https://reviews.apache.org/r/66081/#comment279894> made it private lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java Lines 88 (patched) <https://reviews.apache.org/r/66081/#comment279895> made it private lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java Lines 116 (patched) <https://reviews.apache.org/r/66081/#comment279896> corrected lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java Lines 120 (patched) <https://reviews.apache.org/r/66081/#comment279897> added lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java Lines 722 (patched) <https://reviews.apache.org/r/66081/#comment279898> corrected lens-server/src/main/java/org/apache/lens/server/BaseLensService.java Lines 177 (patched) <https://reviews.apache.org/r/66081/#comment279878> yes.. lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java Lines 578 (patched) <https://reviews.apache.org/r/66081/#comment279899> added - Ankit Kailaswar On March 20, 2018, 10:27 a.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66081/ > --- > > (Updated March 20, 2018, 10:27 a.m.) > > > Review request for lens, Amareshwari Sriramadasu, Rajat Khandelwal, and > Puneet Gupta. > > > Repository: lens > > > Description > --- > > https://issues.apache.org/jira/browse/LENS-1506 > > This patch contains code changes to enable kerberos authentication for > 1. lens to hive > 2. lens to metastore > 3. lens to hdfs > > code changes are as follows, > 1. new http thrift client for hive driver to support sasl transport for > kerberozied hive server. > 2. cron to update KDC ticket before it expires. > > > Diffs > - > > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java > 2eb94aa > > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RemoteThriftConnection.java > 54885f7 > > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java > PRE-CREATION > > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java > d5273be > lens-server/src/main/java/org/apache/lens/server/BaseLensService.java > b9fcdd8 > > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java > 31ac358 > > > Diff: https://reviews.apache.org/r/66081/diff/1/ > > > Testing > --- > > unit testing > > > Thanks, > > Ankit Kailaswar > >
Re: Review Request 66081: Kerberos authentication in lens
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66081/ --- (Updated March 20, 2018, 8:43 p.m.) Review request for lens, Amareshwari Sriramadasu, Rajat Khandelwal, and Puneet Gupta. Repository: lens Description --- https://issues.apache.org/jira/browse/LENS-1506 This patch contains code changes to enable kerberos authentication for 1. lens to hive 2. lens to metastore 3. lens to hdfs code changes are as follows, 1. new http thrift client for hive driver to support sasl transport for kerberozied hive server. 2. cron to update KDC ticket before it expires. Diffs (updated) - lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 2eb94aa7 lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RemoteThriftConnection.java 54885f77 lens-driver-hive/src/main/java/org/apache/lens/driver/hive/RetryingThriftCLIServiceClientSasl.java PRE-CREATION lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java d5273be8 lens-server/src/main/java/org/apache/lens/server/BaseLensService.java b9fcdd8b lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 31ac358d Diff: https://reviews.apache.org/r/66081/diff/2/ Changes: https://reviews.apache.org/r/66081/diff/1-2/ Testing --- unit testing Thanks, Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
] Lens Contributed Clients .. SUCCESS [0.226s] [INFO] Lens Python Client SUCCESS [0.231s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 30:34.060s [INFO] Finished at: Thu Jan 17 10:47:01 UTC 2019 [INFO] Final Memory: 300M/3097M [INFO] [Platform_Common_Job] $ /bin/sh -xe /tmp/jenkins5693507200862759570.sh + ruby /d0/jenkins/scripts/licenseCheck.rb /d0/jenkins/workspace/Platform_Common_Job 0 Diffs (updated) - lens-client/src/test/java/org/apache/lens/client/TestLensClient.java d9e60fb7 lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098 lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4 lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70 lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java PRE-CREATION lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java 066525b7 Diff: https://reviews.apache.org/r/69554/diff/8/ Changes: https://reviews.apache.org/r/69554/diff/7-8/ Testing --- Unit testing Thanks, Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java > > Lines 590 (patched) > > <https://reviews.apache.org/r/69554/diff/1/?file=2113078#file2113078line590> > > > > Instead of making a db call for every session , we need to have some > > batching mechanism for better performance > > Ankit Kailaswar wrote: > This has to be instantaneous. If we batch then we might end up in > situation where session is closed on one server but it is still present on > another. > > Rajitha R wrote: > By batching I mean to say if we can send a list of ids to delete at once, > than one of them each time. I am assuming here you it is w.r.t. session clenaup thread only. for other scenario it has to be instantaneous. In case of session cleanup If we create a batch it will add bit of complexity here. Consider a scenario where query is failing because it has one or two session which are not in db then none of the session will be cleaned up it means none of the session will be cleaned up from db as well as in memory, we can increment counter though but this will add dependency of adderesing the issue more quickly since session backlog will increase at higher rate. - Ankit --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69554/#review211269 --- On Jan. 22, 2019, 11:08 a.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69554/ > --- > > (Updated Jan. 22, 2019, 11:08 a.m.) > > > Review request for lens, Amareshwari Sriramadasu and Rajitha R. > > > Bugs: LENS-1538 > https://issues.apache.org/jira/browse/LENS-1538 > > > Repository: lens > > > Description > --- > > Implementation > > Session > > We need to persist all client session in database. > This involve persisting client session’s id and configuration in mysql > server. persisting entire session object is not required. We need to do this > as first thing while creating user session. > For each request from client lens server should not validate session id > from its in memory map. It should rather validate it from persisted session > in mysql server. > > if > > session is present in mysql and not present in lens server’s in memory > map(session manager) then it should create a lens session with same session > id and add it to in memory map > > else If > > session is not present in mysql and present in server’s in memory map then it > signifies that user have initiated close session from other host and we will > need to remove this session from current host’s in memory map as well > > else if > > session is not present in mysql and in server’s in memory map then return > “invalid session”. > > Whenever session expired/closed then we need to remove this session from > lens server’s in memory map and mysql. > > > > Query > > For sync queries lens server will close connection as soon as it is > stopped or failed with appropriate failure message. Client will be retrying > the query. > For async-light queries and async-heavy queries lens server takes care of > rescheduling all queries which were in running or queued state at the time of > restart. we can use this to make query ids available with both servers. > For async queries we need to persist all non finished async user queries > in mysql server in new table current_query. > If user executes query on host1 and it goes down and nginx points to > host2 then user should be able to poll on query status. Lens server will > first check query id in its inmemory maps if it is not present then it will > check in “finished_query” table in mysql else in “current_query” table in > mysql. In this case we will continue showing old status of query since it is > scheduled by host1. As soon as host1 comes up it will reschedule these > queries and will change the status. Optionally we can have host2 to move > these queries in “allqueries” map of its query service which will take care > of recovering, in this case we don't need to wait for host1 to come up and > reschedule. > While moving query from ”finished_quey” map to “finished_query” table in > mysql we will need to remove it from “current_query” tables as well. > This flow will remain same for request ids/query ids generated for query
Re: Review Request 69554: Lens HA changes
> On Jan. 7, 2019, 3:47 a.m., Amareshwari Sriramadasu wrote: > > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java > > Lines 1176 (patched) > > <https://reviews.apache.org/r/69554/diff/6/?file=2116214#file2116214line1176> > > > > no counter for failures? added - Ankit --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69554/#review211710 --- On Jan. 22, 2019, 11:08 a.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69554/ > --- > > (Updated Jan. 22, 2019, 11:08 a.m.) > > > Review request for lens, Amareshwari Sriramadasu and Rajitha R. > > > Bugs: LENS-1538 > https://issues.apache.org/jira/browse/LENS-1538 > > > Repository: lens > > > Description > --- > > Implementation > > Session > > We need to persist all client session in database. > This involve persisting client session’s id and configuration in mysql > server. persisting entire session object is not required. We need to do this > as first thing while creating user session. > For each request from client lens server should not validate session id > from its in memory map. It should rather validate it from persisted session > in mysql server. > > if > > session is present in mysql and not present in lens server’s in memory > map(session manager) then it should create a lens session with same session > id and add it to in memory map > > else If > > session is not present in mysql and present in server’s in memory map then it > signifies that user have initiated close session from other host and we will > need to remove this session from current host’s in memory map as well > > else if > > session is not present in mysql and in server’s in memory map then return > “invalid session”. > > Whenever session expired/closed then we need to remove this session from > lens server’s in memory map and mysql. > > > > Query > > For sync queries lens server will close connection as soon as it is > stopped or failed with appropriate failure message. Client will be retrying > the query. > For async-light queries and async-heavy queries lens server takes care of > rescheduling all queries which were in running or queued state at the time of > restart. we can use this to make query ids available with both servers. > For async queries we need to persist all non finished async user queries > in mysql server in new table current_query. > If user executes query on host1 and it goes down and nginx points to > host2 then user should be able to poll on query status. Lens server will > first check query id in its inmemory maps if it is not present then it will > check in “finished_query” table in mysql else in “current_query” table in > mysql. In this case we will continue showing old status of query since it is > scheduled by host1. As soon as host1 comes up it will reschedule these > queries and will change the status. Optionally we can have host2 to move > these queries in “allqueries” map of its query service which will take care > of recovering, in this case we don't need to wait for host1 to come up and > reschedule. > While moving query from ”finished_quey” map to “finished_query” table in > mysql we will need to remove it from “current_query” tables as well. > This flow will remain same for request ids/query ids generated for query > plan/estimate. > > > > Issue addressed > > Lens downtime will be zero in case of deployment and if primary fail. > User will not have to create a new session whenever switch happens > User will be able to get status of all async queries seamlessly. > > > Build Log : > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Lens Checkstyle Rules . SUCCESS [2.740s] > [INFO] Lens .. SUCCESS [5.788s] > [INFO] Lens API .. SUCCESS [48.711s] > [INFO] Lens API for server and extensions SUCCESS [33.537s] > [INFO] Lens Cube . SUCCESS [7:43.672s] > [INFO] Lens DB storage ... SUCCESS [29.279s
Re: Review Request 69554: Lens HA changes
> On Jan. 7, 2019, 3:48 a.m., Amareshwari Sriramadasu wrote: > > No unit tests added for the change introduced. Can you add unit tests? added test cases for lensserver dao functions. - Ankit --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69554/#review211711 --- On Jan. 22, 2019, 11:08 a.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69554/ > --- > > (Updated Jan. 22, 2019, 11:08 a.m.) > > > Review request for lens, Amareshwari Sriramadasu and Rajitha R. > > > Bugs: LENS-1538 > https://issues.apache.org/jira/browse/LENS-1538 > > > Repository: lens > > > Description > --- > > Implementation > > Session > > We need to persist all client session in database. > This involve persisting client session’s id and configuration in mysql > server. persisting entire session object is not required. We need to do this > as first thing while creating user session. > For each request from client lens server should not validate session id > from its in memory map. It should rather validate it from persisted session > in mysql server. > > if > > session is present in mysql and not present in lens server’s in memory > map(session manager) then it should create a lens session with same session > id and add it to in memory map > > else If > > session is not present in mysql and present in server’s in memory map then it > signifies that user have initiated close session from other host and we will > need to remove this session from current host’s in memory map as well > > else if > > session is not present in mysql and in server’s in memory map then return > “invalid session”. > > Whenever session expired/closed then we need to remove this session from > lens server’s in memory map and mysql. > > > > Query > > For sync queries lens server will close connection as soon as it is > stopped or failed with appropriate failure message. Client will be retrying > the query. > For async-light queries and async-heavy queries lens server takes care of > rescheduling all queries which were in running or queued state at the time of > restart. we can use this to make query ids available with both servers. > For async queries we need to persist all non finished async user queries > in mysql server in new table current_query. > If user executes query on host1 and it goes down and nginx points to > host2 then user should be able to poll on query status. Lens server will > first check query id in its inmemory maps if it is not present then it will > check in “finished_query” table in mysql else in “current_query” table in > mysql. In this case we will continue showing old status of query since it is > scheduled by host1. As soon as host1 comes up it will reschedule these > queries and will change the status. Optionally we can have host2 to move > these queries in “allqueries” map of its query service which will take care > of recovering, in this case we don't need to wait for host1 to come up and > reschedule. > While moving query from ”finished_quey” map to “finished_query” table in > mysql we will need to remove it from “current_query” tables as well. > This flow will remain same for request ids/query ids generated for query > plan/estimate. > > > > Issue addressed > > Lens downtime will be zero in case of deployment and if primary fail. > User will not have to create a new session whenever switch happens > User will be able to get status of all async queries seamlessly. > > > Build Log : > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Lens Checkstyle Rules . SUCCESS [2.740s] > [INFO] Lens .. SUCCESS [5.788s] > [INFO] Lens API .. SUCCESS [48.711s] > [INFO] Lens API for server and extensions SUCCESS [33.537s] > [INFO] Lens Cube . SUCCESS [7:43.672s] > [INFO] Lens DB storage ... SUCCESS [29.279s] > [INFO] Lens Query Library SUCCESS [22.507s] > [INFO] Lens Hive Driver .. SUCCESS [1:27.344s] > [
Re: Review Request 69554: Lens HA changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69554/ --- (Updated Dec. 18, 2018, 12:10 p.m.) Review request for lens, Amareshwari Sriramadasu and Rajitha R. Bugs: LENS-1538 https://issues.apache.org/jira/browse/LENS-1538 Repository: lens Description --- Implementation Session We need to persist all client session in database. This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session. For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server. if session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map else If session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well else if session is not present in mysql and in server’s in memory map then return “invalid session”. Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql. Query For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query. For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers. For async queries we need to persist all non finished async user queries in mysql server in new table current_query. If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule. While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well. This flow will remain same for request ids/query ids generated for query plan/estimate. Issue addressed Lens downtime will be zero in case of deployment and if primary fail. User will not have to create a new session whenever switch happens User will be able to get status of all async queries seamlessly. Diffs (updated) - lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191 lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658 lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java cb29c9ea lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4 lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f lens-server/src/main/java/org/apache/lens/server/scheduler/ScheduleResource.java 1d5959b2 lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70 Diff: https://reviews.apache.org/r/69554/diff/4/ Changes: https://reviews.apache.org/r/69554/diff/3-4/ Testing --- Unit testing Thanks, Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69554/ --- (Updated Dec. 18, 2018, 12:20 p.m.) Review request for lens, Amareshwari Sriramadasu and Rajitha R. Bugs: LENS-1538 https://issues.apache.org/jira/browse/LENS-1538 Repository: lens Description --- Implementation Session We need to persist all client session in database. This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session. For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server. if session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map else If session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well else if session is not present in mysql and in server’s in memory map then return “invalid session”. Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql. Query For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query. For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers. For async queries we need to persist all non finished async user queries in mysql server in new table current_query. If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule. While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well. This flow will remain same for request ids/query ids generated for query plan/estimate. Issue addressed Lens downtime will be zero in case of deployment and if primary fail. User will not have to create a new session whenever switch happens User will be able to get status of all async queries seamlessly. Diffs (updated) - lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191 lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658 lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java cb29c9ea lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4 lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f lens-server/src/main/java/org/apache/lens/server/scheduler/ScheduleResource.java 1d5959b2 lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70 Diff: https://reviews.apache.org/r/69554/diff/5/ Changes: https://reviews.apache.org/r/69554/diff/4-5/ Testing --- Unit testing Thanks, Ankit Kailaswar
Review Request 69554: Lens HA changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69554/ --- Review request for lens, Amareshwari Sriramadasu and Rajitha R. Bugs: LENS-1538 https://issues.apache.org/jira/browse/LENS-1538 Repository: lens Description --- Implementation Session We need to persist all client session in database. This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session. For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server. if session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map else If session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well else if session is not present in mysql and in server’s in memory map then return “invalid session”. Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql. Query For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query. For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers. For async queries we need to persist all non finished async user queries in mysql server in new table current_query. If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule. While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well. This flow will remain same for request ids/query ids generated for query plan/estimate. Issue addressed Lens downtime will be zero in case of deployment and if primary fail. User will not have to create a new session whenever switch happens User will be able to get status of all async queries seamlessly. Diffs - lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658 lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java a0d8e0ba lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098 lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java ccd2a3b7 lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4 lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70 lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 7d81375c Diff: https://reviews.apache.org/r/69554/diff/1/ Testing --- Unit testing Thanks, Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
> > Lines 2258 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113076#file2113076line2258>
> >
> > makes more sense to move this to all executequery apis than in
> > createcontext api to be in sync with the api definition
make more sense, moved.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
> > Lines 2367 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113076#file2113076line2367>
> >
> > How is this supposed to work?
> > Given the scenario where Lens 2 is given a query Q1 from Lens 1, the db
> > may still contain an older status which it could have updated before going
> > down
> > Who exactly is updating the latest status of Q1 in the db?
only lens server 1 is updating status. Lens 2 will just return its status from
db. lens1 will keep on updating status till it goes down. Lens 2 will always
return last updated status in db.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
> > Lines 3680 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113076#file2113076line3680>
> >
> > Why can't we put this code in existing validatesession in
> > Baselensservice than defining a new api altogether?
moved.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
> > Lines 431 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113078#file2113078line431>
> >
> > can't see any calls to this api anywhere
> > Also from our offline discussion, from my understanding , both
> > instances share a separate persistent state(on hdfs) and they read session
> > state from db only when it can't find a given session id in its memory
> > Why does it require changes in existing restoresession in the first
> > place?
removed, This is needed for phase2.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
> > Lines 590 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113078#file2113078line590>
> >
> > Instead of making a db call for every session , we need to have some
> > batching mechanism for better performance
This has to be instantaneous. If we batch then we might end up in situation
where session is closed on one server but it is still present on another.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java
> > Line 615 (original), 616 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113079#file2113079line616>
> >
> > public interface Externalizable extends Serializable {
removed.
- Ankit
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211269
---
On Dec. 12, 2018, 8:21 a.m., Ankit Kailaswar wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> ---
>
> (Updated Dec. 12, 2018, 8:21 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> ---
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql
> server. persisting entire session object is not required. We need to do this
> as first thing while creating user session.
> For each request from client lens server should not validate session id
> from its in memory map. It should rather validate it from persisted session
> in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory
> map(session manager) then it should create a lens session with same session
> id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it
> signifies
Re: Review Request 69554: Lens HA changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69554/ --- (Updated Dec. 14, 2018, 11:42 a.m.) Review request for lens, Amareshwari Sriramadasu and Rajitha R. Bugs: LENS-1538 https://issues.apache.org/jira/browse/LENS-1538 Repository: lens Description --- Implementation Session We need to persist all client session in database. This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session. For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server. if session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map else If session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well else if session is not present in mysql and in server’s in memory map then return “invalid session”. Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql. Query For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query. For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers. For async queries we need to persist all non finished async user queries in mysql server in new table current_query. If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule. While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well. This flow will remain same for request ids/query ids generated for query plan/estimate. Issue addressed Lens downtime will be zero in case of deployment and if primary fail. User will not have to create a new session whenever switch happens User will be able to get status of all async queries seamlessly. Diffs (updated) - lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658 lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java a0d8e0ba lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098 lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4 lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70 lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 7d81375c src/site/apt/admin/config.apt 4cee5ae2 Diff: https://reviews.apache.org/r/69554/diff/2/ Changes: https://reviews.apache.org/r/69554/diff/1-2/ Testing --- Unit testing Thanks, Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69554/ --- (Updated Dec. 24, 2018, 10:50 a.m.) Review request for lens, Amareshwari Sriramadasu and Rajitha R. Bugs: LENS-1538 https://issues.apache.org/jira/browse/LENS-1538 Repository: lens Description --- Implementation Session We need to persist all client session in database. This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session. For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server. if session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map else If session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well else if session is not present in mysql and in server’s in memory map then return “invalid session”. Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql. Query For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query. For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers. For async queries we need to persist all non finished async user queries in mysql server in new table current_query. If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule. While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well. This flow will remain same for request ids/query ids generated for query plan/estimate. Issue addressed Lens downtime will be zero in case of deployment and if primary fail. User will not have to create a new session whenever switch happens User will be able to get status of all async queries seamlessly. Diffs (updated) - lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191 lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658 lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4 lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70 Diff: https://reviews.apache.org/r/69554/diff/6/ Changes: https://reviews.apache.org/r/69554/diff/5-6/ Testing --- Unit testing Thanks, Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java > > Lines 623 (patched) > > <https://reviews.apache.org/r/69554/diff/1/?file=2113075#file2113075line623> > > > > Can this be shifted to final block? Also DBUtils.close(stmt) and > > DBUtils.close(rs) makes more sense. This applies here and everywhere else > > Ankit Kailaswar wrote: > DBUtils.close(stmt) and DBUtils.close(rs) throws SQLException, not a good > idea to have it in final block. > > Rajitha R wrote: > You could use closeQuietly in that case replaced. - Ankit --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69554/#review211269 ------- On Dec. 24, 2018, 10:50 a.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69554/ > --- > > (Updated Dec. 24, 2018, 10:50 a.m.) > > > Review request for lens, Amareshwari Sriramadasu and Rajitha R. > > > Bugs: LENS-1538 > https://issues.apache.org/jira/browse/LENS-1538 > > > Repository: lens > > > Description > --- > > Implementation > > Session > > We need to persist all client session in database. > This involve persisting client session’s id and configuration in mysql > server. persisting entire session object is not required. We need to do this > as first thing while creating user session. > For each request from client lens server should not validate session id > from its in memory map. It should rather validate it from persisted session > in mysql server. > > if > > session is present in mysql and not present in lens server’s in memory > map(session manager) then it should create a lens session with same session > id and add it to in memory map > > else If > > session is not present in mysql and present in server’s in memory map then it > signifies that user have initiated close session from other host and we will > need to remove this session from current host’s in memory map as well > > else if > > session is not present in mysql and in server’s in memory map then return > “invalid session”. > > Whenever session expired/closed then we need to remove this session from > lens server’s in memory map and mysql. > > > > Query > > For sync queries lens server will close connection as soon as it is > stopped or failed with appropriate failure message. Client will be retrying > the query. > For async-light queries and async-heavy queries lens server takes care of > rescheduling all queries which were in running or queued state at the time of > restart. we can use this to make query ids available with both servers. > For async queries we need to persist all non finished async user queries > in mysql server in new table current_query. > If user executes query on host1 and it goes down and nginx points to > host2 then user should be able to poll on query status. Lens server will > first check query id in its inmemory maps if it is not present then it will > check in “finished_query” table in mysql else in “current_query” table in > mysql. In this case we will continue showing old status of query since it is > scheduled by host1. As soon as host1 comes up it will reschedule these > queries and will change the status. Optionally we can have host2 to move > these queries in “allqueries” map of its query service which will take care > of recovering, in this case we don't need to wait for host1 to come up and > reschedule. > While moving query from ”finished_quey” map to “finished_query” table in > mysql we will need to remove it from “current_query” tables as well. > This flow will remain same for request ids/query ids generated for query > plan/estimate. > > > > Issue addressed > > Lens downtime will be zero in case of deployment and if primary fail. > User will not have to create a new session whenever switch happens > User will be able to get status of all async queries seamlessly. > > > Diffs > - > > > lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java > 65403191 > > lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java > 94734658 > lens-server/src/main/java/org/apache/lens/server/BaseLensService.java > 9
Re: Review Request 69554: Lens HA changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69554/ --- (Updated Dec. 14, 2018, 12:06 p.m.) Review request for lens, Amareshwari Sriramadasu and Rajitha R. Bugs: LENS-1538 https://issues.apache.org/jira/browse/LENS-1538 Repository: lens Description --- Implementation Session We need to persist all client session in database. This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session. For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server. if session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map else If session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well else if session is not present in mysql and in server’s in memory map then return “invalid session”. Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql. Query For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query. For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers. For async queries we need to persist all non finished async user queries in mysql server in new table current_query. If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule. While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well. This flow will remain same for request ids/query ids generated for query plan/estimate. Issue addressed Lens downtime will be zero in case of deployment and if primary fail. User will not have to create a new session whenever switch happens User will be able to get status of all async queries seamlessly. Diffs (updated) - lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658 lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098 lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4 lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70 src/site/apt/admin/config.apt 4cee5ae2 Diff: https://reviews.apache.org/r/69554/diff/3/ Changes: https://reviews.apache.org/r/69554/diff/2-3/ Testing --- Unit testing Thanks, Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69554/ --- (Updated Jan. 8, 2019, 10:52 a.m.) Review request for lens, Amareshwari Sriramadasu and Rajitha R. Bugs: LENS-1538 https://issues.apache.org/jira/browse/LENS-1538 Repository: lens Description --- Implementation Session We need to persist all client session in database. This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session. For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server. if session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map else If session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well else if session is not present in mysql and in server’s in memory map then return “invalid session”. Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql. Query For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query. For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers. For async queries we need to persist all non finished async user queries in mysql server in new table current_query. If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule. While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well. This flow will remain same for request ids/query ids generated for query plan/estimate. Issue addressed Lens downtime will be zero in case of deployment and if primary fail. User will not have to create a new session whenever switch happens User will be able to get status of all async queries seamlessly. Diffs (updated) - lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191 lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658 lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4 lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70 lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java PRE-CREATION lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java 066525b7 Diff: https://reviews.apache.org/r/69554/diff/7/ Changes: https://reviews.apache.org/r/69554/diff/6-7/ Testing --- Unit testing Thanks, Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69554/#review211719 --- lens-server/src/main/java/org/apache/lens/server/BaseLensService.java Lines 133 (patched) <https://reviews.apache.org/r/69554/#comment297271> made it static final. lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java Lines 1176 (patched) <https://reviews.apache.org/r/69554/#comment297274> We have a counters for insert and delete. update will be intermidiate operation between these 2 calls though of not having one more extra counter. Added though. lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java Lines 91 (patched) <https://reviews.apache.org/r/69554/#comment297272> this is not required here, can use base class instance. removed as member variable. lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java Lines 397 (patched) <https://reviews.apache.org/r/69554/#comment297273> removed - Ankit Kailaswar On Jan. 8, 2019, 10:52 a.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69554/ > --- > > (Updated Jan. 8, 2019, 10:52 a.m.) > > > Review request for lens, Amareshwari Sriramadasu and Rajitha R. > > > Bugs: LENS-1538 > https://issues.apache.org/jira/browse/LENS-1538 > > > Repository: lens > > > Description > --- > > Implementation > > Session > > We need to persist all client session in database. > This involve persisting client session’s id and configuration in mysql > server. persisting entire session object is not required. We need to do this > as first thing while creating user session. > For each request from client lens server should not validate session id > from its in memory map. It should rather validate it from persisted session > in mysql server. > > if > > session is present in mysql and not present in lens server’s in memory > map(session manager) then it should create a lens session with same session > id and add it to in memory map > > else If > > session is not present in mysql and present in server’s in memory map then it > signifies that user have initiated close session from other host and we will > need to remove this session from current host’s in memory map as well > > else if > > session is not present in mysql and in server’s in memory map then return > “invalid session”. > > Whenever session expired/closed then we need to remove this session from > lens server’s in memory map and mysql. > > > > Query > > For sync queries lens server will close connection as soon as it is > stopped or failed with appropriate failure message. Client will be retrying > the query. > For async-light queries and async-heavy queries lens server takes care of > rescheduling all queries which were in running or queued state at the time of > restart. we can use this to make query ids available with both servers. > For async queries we need to persist all non finished async user queries > in mysql server in new table current_query. > If user executes query on host1 and it goes down and nginx points to > host2 then user should be able to poll on query status. Lens server will > first check query id in its inmemory maps if it is not present then it will > check in “finished_query” table in mysql else in “current_query” table in > mysql. In this case we will continue showing old status of query since it is > scheduled by host1. As soon as host1 comes up it will reschedule these > queries and will change the status. Optionally we can have host2 to move > these queries in “allqueries” map of its query service which will take care > of recovering, in this case we don't need to wait for host1 to come up and > reschedule. > While moving query from ”finished_quey” map to “finished_query” table in > mysql we will need to remove it from “current_query” tables as well. > This flow will remain same for request ids/query ids generated for query > plan/estimate. > > > > Issue addressed > > Lens downtime will be zero in case of deployment and if primary fail. > User will not have to create a new session whenever switch happens > User will be able to get status of all async queries seamlessly. > > > Diffs > - > > > lens-server-api/src/main/java/org
Re: Review Request 69554: Lens HA changes
> On Dec. 21, 2018, 6:10 a.m., Rajitha R wrote: > > lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java > > Lines 31 (patched) > > <https://reviews.apache.org/r/69554/diff/5/?file=2114506#file2114506line31> > > > > Can we not retain the same name validateSession? changed name > On Dec. 21, 2018, 6:10 a.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/BaseLensService.java > > Line 392 (original), 401 (patched) > > <https://reviews.apache.org/r/69554/diff/5/?file=2114508#file2114508line401> > > > > validatesession not being called here getsession is internal call. We have to validate user session in mysql only for user requests and not for internal session calls. There wont be a scenario where getsession is called and session is not present in memory because it will be created itself on user's first request only. Also on other hand if we try to call validatesession here then it will be called for lot of time in query prep/execute/submission phases. Not an good idea. That could be the reason it was not in place for older implementation as well. > On Dec. 21, 2018, 6:10 a.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/BaseLensService.java > > Lines 607 (patched) > > <https://reviews.apache.org/r/69554/diff/5/?file=2114508#file2114508line608> > > > > Any particular reason we are throwing e here and not daoE? May be throw > > e should be outside the if condition. need to throw exception on condition here, corrected. > On Dec. 21, 2018, 6:10 a.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/BaseLensService.java > > Lines 611 (patched) > > <https://reviews.apache.org/r/69554/diff/5/?file=2114508#file2114508line612> > > > > can validateSession and validateSessionId api's be swapped here such > > that no changes will be required where validateSession api is called? changed name > On Dec. 21, 2018, 6:10 a.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java > > Lines 1175 (patched) > > <https://reviews.apache.org/r/69554/diff/5/?file=2114512#file2114512line1175> > > > > I am afraid if these continuous db updates will slow down the system This update happens 3-4 times in query lifecycle, once for each state change (SUBMITTED, QUEUED, RUNNING, FAILED). Considering query execution time this should not slow down the system. > On Dec. 21, 2018, 6:10 a.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java > > Line 1541 (original), 1566 (patched) > > <https://reviews.apache.org/r/69554/diff/5/?file=2114512#file2114512line1566> > > > > why is db update statement NOT called outside the switch statement and > > only in EXECUTED case ? it has to be for all, moved. > On Dec. 21, 2018, 6:10 a.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java > > Lines 538 (patched) > > <https://reviews.apache.org/r/69554/diff/5/?file=2114515#file2114515line540> > > > > Why is this required? Can we not implement these counters in the > > existing way by calling notifyevent? I cant see any event implementation for metric service in code also we are using metric service directly to handle metric counters at all places in code. Correct me if I am wrong. - Ankit --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69554/#review211463 --- On Jan. 8, 2019, 10:52 a.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69554/ > --- > > (Updated Jan. 8, 2019, 10:52 a.m.) > > > Review request for lens, Amareshwari Sriramadasu and Rajitha R. > > > Bugs: LENS-1538 > https://issues.apache.org/jira/browse/LENS-1538 > > > Repository: lens > > > Description > --- > > Implementation > > Session > > We need to persist all client session in database. > This involve persisting client session’s id and configuration in mysql > server. persisting entire session object is not required. We need to do this > as first thing while creating user session. > For each request from client lens server should not validate session id > from its in memory map. It should rather valida
Re: Review Request 69554: Lens HA changes
> On Dec. 12, 2018, 3:38 p.m., Rajitha R wrote: > > lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java > > Line 54 (original), 55 (patched) > > <https://reviews.apache.org/r/69554/diff/1/?file=2113070#file2113070line55> > > > > AbstractQueryContext already implements Serialzable missed that, reverted. > On Dec. 12, 2018, 3:38 p.m., Rajitha R wrote: > > lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java > > Lines 69 (patched) > > <https://reviews.apache.org/r/69554/diff/1/?file=2113072#file2113072line69> > > > > A new implementation for existing restoresession would make more sense > > than adding new api yes, make sense, reverted. > On Dec. 12, 2018, 3:38 p.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/BaseLensService.java > > Line 24 (original) > > <https://reviews.apache.org/r/69554/diff/1/?file=2113073#file2113073line24> > > > > can this be reverted ? reverted > On Dec. 12, 2018, 3:38 p.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/BaseLensService.java > > Line 583 (original), 580 (patched) > > <https://reviews.apache.org/r/69554/diff/1/?file=2113073#file2113073line584> > > > > "should not validate session id from its in memory map. It should > > rather validate it from persisted session in mysql server" > > The above statement says to validate session from mysql, but here its > > checking inmemory and throwing the exception we are validating session from memory map as well as from mysql. (please check getSession() definition). Current flow is, 1. check if it is present in memory first. 2. If it is present in memory then validate if its mark for expired or not. 3. If it is not present in memory then validate if its presnet in mysql, if it it presnet in mysql then validate if its mark for closed or not. 4. If it is not present in mysql too then throw an exception for sesion not present. - Ankit --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69554/#review211230 --- On Dec. 18, 2018, 12:20 p.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69554/ > --- > > (Updated Dec. 18, 2018, 12:20 p.m.) > > > Review request for lens, Amareshwari Sriramadasu and Rajitha R. > > > Bugs: LENS-1538 > https://issues.apache.org/jira/browse/LENS-1538 > > > Repository: lens > > > Description > --- > > Implementation > > Session > > We need to persist all client session in database. > This involve persisting client session’s id and configuration in mysql > server. persisting entire session object is not required. We need to do this > as first thing while creating user session. > For each request from client lens server should not validate session id > from its in memory map. It should rather validate it from persisted session > in mysql server. > > if > > session is present in mysql and not present in lens server’s in memory > map(session manager) then it should create a lens session with same session > id and add it to in memory map > > else If > > session is not present in mysql and present in server’s in memory map then it > signifies that user have initiated close session from other host and we will > need to remove this session from current host’s in memory map as well > > else if > > session is not present in mysql and in server’s in memory map then return > “invalid session”. > > Whenever session expired/closed then we need to remove this session from > lens server’s in memory map and mysql. > > > > Query > > For sync queries lens server will close connection as soon as it is > stopped or failed with appropriate failure message. Client will be retrying > the query. > For async-light queries and async-heavy queries lens server takes care of > rescheduling all queries which were in running or queued state at the time of > restart. we can use this to make query ids available with both servers. > For async queries we need to persist all non finished async user queries > in mysql server in new table current_query. > If user executes query on host1 and it goes down and nginx points to > host2 then user should be able to poll on query status.
Re: Review Request 69554: Lens HA changes
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java > > Lines 617 (patched) > > <https://reviews.apache.org/r/69554/diff/1/?file=2113075#file2113075line617> > > > > can this be moved to the if condition? this would ensure there is > > atleast one row we should rely on this to throw exception. If resultset has no rows or for any other type of error or exception we are relying on finsihed_query table to fetch query details. > On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java > > Lines 623 (patched) > > <https://reviews.apache.org/r/69554/diff/1/?file=2113075#file2113075line623> > > > > Can this be shifted to final block? Also DBUtils.close(stmt) and > > DBUtils.close(rs) makes more sense. This applies here and everywhere else DBUtils.close(stmt) and DBUtils.close(rs) throws SQLException, not a good idea to have it in final block. > On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java > > Line 390 (original), 827 (patched) > > <https://reviews.apache.org/r/69554/diff/1/?file=2113075#file2113075line827> > > > > while all of existing code in lensserverdao is using commons db utils, > > any particular reason we are using vanilla java code in the new code? to > > maintain consistency in the code, it seems better to use the db utils > > library to define the new apis as well, IMO db utils api sunc as DBRunner that we are using currently perofrms one to one serialisaition between java class and mysql table. In case of query context class and LensSession.PersistInfo it will add unnecessary complexity of modifying these table definition as and when we introduce new filed in these classes. > On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java > > Lines 3680 (patched) > > <https://reviews.apache.org/r/69554/diff/1/?file=2113076#file2113076line3680> > > > > Why can't we put this code in existing validatesession in > > Baselensservice than defining a new api altogether? > > Ankit Kailaswar wrote: > moved. SessionValidator interface is better place for this api, moved it there. > On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote: > > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java > > Line 579 (original), 616 (patched) > > <https://reviews.apache.org/r/69554/diff/1/?file=2113078#file2113078line616> > > > > While all the write operations on sessions are handled in this code, I > > am not able to see any changes in list and get kind of apis in this class > > For instance, what happens if getAllSessionParameters api gets called > > with a session id from other lens instance, in which case I assume , > > restoresessionfrom db needs to be called in the getSession api of > > BaseLensService, which doesn't seem to be handled here > > restoresessionfromdb might have to be called in other apis involving > > write operations as well, which I am unable to see we are calling it for 1. requests in query resource 2. get/set params , list/add resources in session. 3. all requests for scheduler service 4. all requests for metastore service. - Ankit --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69554/#review211269 --- On Dec. 18, 2018, 12:20 p.m., Ankit Kailaswar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69554/ > --- > > (Updated Dec. 18, 2018, 12:20 p.m.) > > > Review request for lens, Amareshwari Sriramadasu and Rajitha R. > > > Bugs: LENS-1538 > https://issues.apache.org/jira/browse/LENS-1538 > > > Repository: lens > > > Description > --- > > Implementation > > Session > > We need to persist all client session in database. > This involve persisting client session’s id and configuration in mysql > server. persisting entire session object is not required. We need to do this > as first thing while creating user session. > For each request from client lens server should not validate session id > from its in memory map. It should rather validate it from persisted session > in mysql server. > >
Re: Review Request 69554: Lens HA changes
Contributed Clients .. SUCCESS [0.226s] [INFO] Lens Python Client SUCCESS [0.231s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 30:34.060s [INFO] Finished at: Thu Jan 17 10:47:01 UTC 2019 [INFO] Final Memory: 300M/3097M [INFO] [Platform_Common_Job] $ /bin/sh -xe /tmp/jenkins5693507200862759570.sh + ruby /d0/jenkins/scripts/licenseCheck.rb /d0/jenkins/workspace/Platform_Common_Job 0 Diffs (updated) - lens-client/src/test/java/org/apache/lens/server/MockQueryExecutionServiceImpl.java 9b55fb62 lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098 lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 74806afe lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4 lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70 lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java PRE-CREATION lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java 066525b7 Diff: https://reviews.apache.org/r/69554/diff/10/ Changes: https://reviews.apache.org/r/69554/diff/9-10/ Testing --- Unit testing Thanks, Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
Contributed Clients .. SUCCESS [0.226s] [INFO] Lens Python Client SUCCESS [0.231s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 30:34.060s [INFO] Finished at: Thu Jan 17 10:47:01 UTC 2019 [INFO] Final Memory: 300M/3097M [INFO] [Platform_Common_Job] $ /bin/sh -xe /tmp/jenkins5693507200862759570.sh + ruby /d0/jenkins/scripts/licenseCheck.rb /d0/jenkins/workspace/Platform_Common_Job 0 Diffs (updated) - lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098 lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 74806afe lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4 lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70 lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java PRE-CREATION lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java 066525b7 Diff: https://reviews.apache.org/r/69554/diff/9/ Changes: https://reviews.apache.org/r/69554/diff/8-9/ Testing --- Unit testing Thanks, Ankit Kailaswar
[jira] [Created] (LENS-1461) Lens services state are not persisted correctly
Ankit Kailaswar created LENS-1461: - Summary: Lens services state are not persisted correctly Key: LENS-1461 URL: https://issues.apache.org/jira/browse/LENS-1461 Project: Apache Lens Issue Type: Bug Components: server Reporter: Ankit Kailaswar Assignee: Ankit Kailaswar Lens services state are not persisted correctly causing lens to fail at startup. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (LENS-1461) Lens services state are not persisted correctly
[
https://issues.apache.org/jira/browse/LENS-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131977#comment-16131977
]
Ankit Kailaswar edited comment on LENS-1461 at 8/18/17 9:49 AM:
There were few issues found while persisting service states and using it to
start lens server.
1. Session service is not getting persisted properly on few occasions. session
service try to serialize few sessions which are invalid using its session
handles throwing an exception uncaught at service level. Other services are
getting persisted successfully.
{panel:title=lens server log
|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1|bgColor=#CE}
14 Aug 2017 20:13:49 6daca8db-deb9-4ab0-ba2d-3f5e474ad545
Lens-server-snapshotter-Thread-1 WARN org.apache.lens.server.BaseLensService -
{color:red} *Session cbc183be-8f05-4356-bb56-6ba43217518b not found* {color}
org.apache.hive.service.cli.HiveSQLException: Invalid SessionHandle:
SessionHandle cbc183be-8f05-4356-bb56-6ba43217518b
at
org.apache.hive.service.cli.session.SessionManager.getSession(SessionManager.java:389)
~hive-service-2.1.3-inm.jar:2.1.3-inm
at
org.apache.lens.server.BaseLensService.getSession(BaseLensService.java:366)
~classes/:na
at
org.apache.lens.server.session.HiveSessionService.writeExternal(HiveSessionService.java:425)
classes/:na
at
org.apache.lens.server.LensServices.persistState(LensServices.java:369)
classes/:na
at
org.apache.lens.server.LensServices.persistLensServiceState(LensServices.java:354)
classes/:na
at org.apache.lens.server.LensServices.access$100(LensServices.java:63)
classes/:na
at org.apache.lens.server.LensServices$1.run(LensServices.java:305)
classes/:na
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
na:1.8.0_72
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
na:1.8.0_72
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
na:1.8.0_72
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
na:1.8.0_72
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
na:1.8.0_72
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
na:1.8.0_72
at java.lang.Thread.run(Thread.java:745) na:1.8.0_72
14 Aug 2017 20:13:49 ba8c2672-53a0-44ad-9d50-fc53e2286143
grizzly-http-server-12 INFO hive.metastore - Closed a connection to metastore,
current connections: 282
14 Aug 2017 20:13:49 ba8c2672-53a0-44ad-9d50-fc53e2286143
grizzly-http-server-12 INFO org.apache.lens.server.BaseLensService - Closed
session cbc183be-8f05-4356-bb56-6ba43217518b33a980a2-206c-4dbc-b3d9-ffa3a6b6de87
for yoda-admin user
14 Aug 2017 20:13:49 6daca8db-deb9-4ab0-ba2d-3f5e474ad545
Lens-server-snapshotter-Thread-1 {color:red} *ERROR
org.apache.lens.server.LensServices - Error while persisting state for service
session* {color}
javax.ws.rs.ClientErrorException: Session cbc183be-8f05-4356-bb56-6ba43217518b
is invalid cbc183be-8f05-4356-bb56-6ba43217518b33a980a2-206c-4dbc-b3d9-ffa3a6b6de87
at
org.apache.lens.server.BaseLensService.getSession(BaseLensService.java:370)
~classes/:na
at
org.apache.lens.server.session.HiveSessionService.writeExternal(HiveSessionService.java:425)
~classes/:na
at
org.apache.lens.server.LensServices.persistState(LensServices.java:369)
classes/:na
at
org.apache.lens.server.LensServices.persistLensServiceState(LensServices.java:354)
classes/:na
at org.apache.lens.server.LensServices.access$100(LensServices.java:63)
classes/:na
at org.apache.lens.server.LensServices$1.run(LensServices.java:305)
classes/:na
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
na:1.8.0_72
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
na:1.8.0_72
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
na:1.8.0_72
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
na:1.8.0_72
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
na:1.8.0_72
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
na:1.8.0_72
at java.lang.Thread.run(Thread.java:745) na:1.8.0_72
Caused by: org.apache.hive.service.cli.HiveSQLException: Invalid SessionHandle:
SessionHandle cbc183be-8f05-4356-bb56-6ba43217518b
at
org.apache.hive.service.cli.session.SessionManager.getSession(SessionManager.java:389)
~hive-service-2.1.3-inm.jar:2.1.3-inm
[jira] [Commented] (LENS-1461) Lens services state are not persisted correctly
[
https://issues.apache.org/jira/browse/LENS-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131977#comment-16131977
]
Ankit Kailaswar commented on LENS-1461:
---
There were few issues found while persisting service states and using it to
start lens server.
1. Session service is not getting persisted properly on few occasions. session
service try to serialize few sessions which are invalid using its session
handles throwing an exception uncaught at service level. Other services are
getting persisted successfully.
{panel:title=lens server log
|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1|bgColor=#CE}
14 Aug 2017 20:13:49 6daca8db-deb9-4ab0-ba2d-3f5e474ad545
Lens-server-snapshotter-Thread-1 WARN org.apache.lens.server.BaseLensService -
{color:red} *Session cbc183be-8f05-4356-bb56-6ba43217518b not found* {color}
org.apache.hive.service.cli.HiveSQLException: Invalid SessionHandle:
SessionHandle cbc183be-8f05-4356-bb56-6ba43217518b
at
org.apache.hive.service.cli.session.SessionManager.getSession(SessionManager.java:389)
~hive-service-2.1.3-inm.jar:2.1.3-inm
at
org.apache.lens.server.BaseLensService.getSession(BaseLensService.java:366)
~classes/:na
at
org.apache.lens.server.session.HiveSessionService.writeExternal(HiveSessionService.java:425)
classes/:na
at
org.apache.lens.server.LensServices.persistState(LensServices.java:369)
classes/:na
at
org.apache.lens.server.LensServices.persistLensServiceState(LensServices.java:354)
classes/:na
at org.apache.lens.server.LensServices.access$100(LensServices.java:63)
classes/:na
at org.apache.lens.server.LensServices$1.run(LensServices.java:305)
classes/:na
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
na:1.8.0_72
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
na:1.8.0_72
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
na:1.8.0_72
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
na:1.8.0_72
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
na:1.8.0_72
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
na:1.8.0_72
at java.lang.Thread.run(Thread.java:745) na:1.8.0_72
14 Aug 2017 20:13:49 ba8c2672-53a0-44ad-9d50-fc53e2286143
grizzly-http-server-12 INFO hive.metastore - Closed a connection to metastore,
current connections: 282
14 Aug 2017 20:13:49 ba8c2672-53a0-44ad-9d50-fc53e2286143
grizzly-http-server-12 INFO org.apache.lens.server.BaseLensService - Closed
session cbc183be-8f05-4356-bb56-6ba43217518b33a980a2-206c-4dbc-b3d9-ffa3a6b6de87
for yoda-admin user
14 Aug 2017 20:13:49 6daca8db-deb9-4ab0-ba2d-3f5e474ad545
Lens-server-snapshotter-Thread-1 ERROR org.apache.lens.server.LensServices -
Error while persisting state for service session
javax.ws.rs.ClientErrorException: Session cbc183be-8f05-4356-bb56-6ba43217518b
is invalid cbc183be-8f05-4356-bb56-6ba43217518b33a980a2-206c-4dbc-b3d9-ffa3a6b6de87
at
org.apache.lens.server.BaseLensService.getSession(BaseLensService.java:370)
~classes/:na
at
org.apache.lens.server.session.HiveSessionService.writeExternal(HiveSessionService.java:425)
~classes/:na
at
org.apache.lens.server.LensServices.persistState(LensServices.java:369)
classes/:na
at
org.apache.lens.server.LensServices.persistLensServiceState(LensServices.java:354)
classes/:na
at org.apache.lens.server.LensServices.access$100(LensServices.java:63)
classes/:na
at org.apache.lens.server.LensServices$1.run(LensServices.java:305)
classes/:na
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
na:1.8.0_72
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
na:1.8.0_72
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
na:1.8.0_72
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
na:1.8.0_72
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
na:1.8.0_72
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
na:1.8.0_72
at java.lang.Thread.run(Thread.java:745) na:1.8.0_72
Caused by: org.apache.hive.service.cli.HiveSQLException: Invalid SessionHandle:
SessionHandle cbc183be-8f05-4356-bb56-6ba43217518b
at
org.apache.hive.service.cli.session.SessionManager.getSession(SessionManager.java:389)
~hive-service-2.1.3-inm.jar:2.1.3-inm
at
org.apache.lens.server.BaseLensService.getSession(BaseLensService.java:366)
~classes/:na
[jira] [Commented] (LENS-1461) Lens services state are not persisted correctly
[ https://issues.apache.org/jira/browse/LENS-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139749#comment-16139749 ] Ankit Kailaswar commented on LENS-1461: --- review request : https://reviews.apache.org/r/61881/ > Lens services state are not persisted correctly > --- > > Key: LENS-1461 > URL: https://issues.apache.org/jira/browse/LENS-1461 > Project: Apache Lens > Issue Type: Bug > Components: server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar > > Lens services state are not persisted correctly causing lens to fail at > startup. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (LENS-1511) enable SSL/TLS for lens server and client communication
Ankit Kailaswar created LENS-1511: - Summary: enable SSL/TLS for lens server and client communication Key: LENS-1511 URL: https://issues.apache.org/jira/browse/LENS-1511 Project: Apache Lens Issue Type: Bug Components: client, server Reporter: Ankit Kailaswar Assignee: Ankit Kailaswar As a part of GDPR compliance for apache lens we will be providing functionality to enable https over lens server-client communication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LENS-1516) SSL changes to ignore cert for python module
Ankit Kailaswar created LENS-1516: - Summary: SSL changes to ignore cert for python module Key: LENS-1516 URL: https://issues.apache.org/jira/browse/LENS-1516 Project: Apache Lens Issue Type: Bug Components: client Reporter: Ankit Kailaswar Assignee: Ankit Kailaswar SSL changes for python client for lens when we want to ignore cert. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1516) SSL changes to ignore cert for python module
[ https://issues.apache.org/jira/browse/LENS-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1516: -- Attachment: Lens-1516.1.patch > SSL changes to ignore cert for python module > - > > Key: LENS-1516 > URL: https://issues.apache.org/jira/browse/LENS-1516 > Project: Apache Lens > Issue Type: Bug > Components: client > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1516.1.patch > > > SSL changes for python client for lens when we want to ignore cert. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1516) SSL changes to ignore cert for python module
[ https://issues.apache.org/jira/browse/LENS-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1516: -- Attachment: (was: Lens-1516.patch) > SSL changes to ignore cert for python module > - > > Key: LENS-1516 > URL: https://issues.apache.org/jira/browse/LENS-1516 > Project: Apache Lens > Issue Type: Bug > Components: client > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1516.1.patch > > > SSL changes for python client for lens when we want to ignore cert. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1515) For SSL enabled lens client must always use trust manager for root X509 cert
[ https://issues.apache.org/jira/browse/LENS-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1515: -- Attachment: Lens-1515.patch > For SSL enabled lens client must always use trust manager for root X509 cert > > > Key: LENS-1515 > URL: https://issues.apache.org/jira/browse/LENS-1515 > Project: Apache Lens > Issue Type: Bug > Components: client > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1515.patch > > > In case of if cert verification is disabled then we are returning root cert. > Also server cert validation will be taken care by default trust manager. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1516) SSL changes to ignore cert for python module
[ https://issues.apache.org/jira/browse/LENS-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1516: -- Attachment: Lens-1516.patch > SSL changes to ignore cert for python module > - > > Key: LENS-1516 > URL: https://issues.apache.org/jira/browse/LENS-1516 > Project: Apache Lens > Issue Type: Bug > Components: client > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1516.patch > > > SSL changes for python client for lens when we want to ignore cert. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1516) SSL changes to ignore cert for python module
[ https://issues.apache.org/jira/browse/LENS-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1516: -- Status: Patch Available (was: In Progress) > SSL changes to ignore cert for python module > - > > Key: LENS-1516 > URL: https://issues.apache.org/jira/browse/LENS-1516 > Project: Apache Lens > Issue Type: Bug > Components: client > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1516.patch > > > SSL changes for python client for lens when we want to ignore cert. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (LENS-1516) SSL changes to ignore cert for python module
[ https://issues.apache.org/jira/browse/LENS-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on LENS-1516 started by Ankit Kailaswar. - > SSL changes to ignore cert for python module > - > > Key: LENS-1516 > URL: https://issues.apache.org/jira/browse/LENS-1516 > Project: Apache Lens > Issue Type: Bug > Components: client > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1516.patch > > > SSL changes for python client for lens when we want to ignore cert. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1511) enable SSL/TLS for lens server and client communication
[ https://issues.apache.org/jira/browse/LENS-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1511: -- Attachment: (was: Lens-1511.5.patch) > enable SSL/TLS for lens server and client communication > --- > > Key: LENS-1511 > URL: https://issues.apache.org/jira/browse/LENS-1511 > Project: Apache Lens > Issue Type: Bug > Components: client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1511.9.patch > > > As a part of GDPR compliance for apache lens we will be providing > functionality to enable https over lens server-client communication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1511) enable SSL/TLS for lens server and client communication
[ https://issues.apache.org/jira/browse/LENS-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1511: -- Attachment: (was: Lens-1511.3.patch) > enable SSL/TLS for lens server and client communication > --- > > Key: LENS-1511 > URL: https://issues.apache.org/jira/browse/LENS-1511 > Project: Apache Lens > Issue Type: Bug > Components: client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > > As a part of GDPR compliance for apache lens we will be providing > functionality to enable https over lens server-client communication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1511) enable SSL/TLS for lens server and client communication
[ https://issues.apache.org/jira/browse/LENS-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1511: -- Attachment: Lens-1511.3.patch > enable SSL/TLS for lens server and client communication > --- > > Key: LENS-1511 > URL: https://issues.apache.org/jira/browse/LENS-1511 > Project: Apache Lens > Issue Type: Bug > Components: client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1511.3.patch > > > As a part of GDPR compliance for apache lens we will be providing > functionality to enable https over lens server-client communication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1511) enable SSL/TLS for lens server and client communication
[ https://issues.apache.org/jira/browse/LENS-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1511: -- Attachment: Lens-1511.9.patch > enable SSL/TLS for lens server and client communication > --- > > Key: LENS-1511 > URL: https://issues.apache.org/jira/browse/LENS-1511 > Project: Apache Lens > Issue Type: Bug > Components: client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1511.9.patch > > > As a part of GDPR compliance for apache lens we will be providing > functionality to enable https over lens server-client communication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1511) enable SSL/TLS for lens server and client communication
[ https://issues.apache.org/jira/browse/LENS-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1511: -- Attachment: Lens-1511.5.patch > enable SSL/TLS for lens server and client communication > --- > > Key: LENS-1511 > URL: https://issues.apache.org/jira/browse/LENS-1511 > Project: Apache Lens > Issue Type: Bug > Components: client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1511.5.patch > > > As a part of GDPR compliance for apache lens we will be providing > functionality to enable https over lens server-client communication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1518) Missing self for ignore cert variable in query.py file
[ https://issues.apache.org/jira/browse/LENS-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1518: -- Status: Patch Available (was: Open) > Missing self for ignore cert variable in query.py file > -- > > Key: LENS-1518 > URL: https://issues.apache.org/jira/browse/LENS-1518 > Project: Apache Lens > Issue Type: Bug > Components: client > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1518.patch > > > Self is missing for variable corresponding to ignore cert. This is causing > python query builder to fail while creating request object. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (LENS-1517) Python3 compatibility issues with python client
[
https://issues.apache.org/jira/browse/LENS-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Kailaswar resolved LENS-1517.
---
Resolution: Fixed
> Python3 compatibility issues with python client
> ---
>
> Key: LENS-1517
> URL: https://issues.apache.org/jira/browse/LENS-1517
> Project: Apache Lens
> Issue Type: Bug
> Components: python-client
>Affects Versions: 3.0, 2.8
> Environment: Python 3.5.2
> Commit: c6423ae01a4776383f0edcd8591124ac643b9e3e
>Reporter: Michael McCarthy
>Assignee: Michael McCarthy
>Priority: Major
>
> Using the python client in Python3 fails on import:
> {noformat}
> from lens.client import LensClient
> ---
> ImportError Traceback (most recent call last)
> in ()
> > 1 from lens.client import LensClient
> /data/d1/users/ds-exp/python-virtualenvs/python3.5/src/lenspythonclient/contrib/clients/python/lens/client/__init__.py
> in ()
> 15 # limitations under the License.
> 16 #
> ---> 17 from .main import LensClient
> 18
> 19 __all__ = ['LensClient']
> /data/d1/users/ds-exp/python-virtualenvs/python3.5/src/lenspythonclient/contrib/clients/python/lens/client/main.py
> in ()
> 18
> 19 from six import string_types
> ---> 20 from .log import LensLogClient
> 21 from .session import LensSessionClient
> 22 from .query import LensQueryClient
> /data/d1/users/ds-exp/python-virtualenvs/python3.5/src/lenspythonclient/contrib/clients/python/lens/client/log.py
> in ()
> 15 # limitations under the License.
> 16 #
> ---> 17 from .auth import SpnegoAuth
> 18 import requests
> 19
> /data/d1/users/ds-exp/python-virtualenvs/python3.5/src/lenspythonclient/contrib/clients/python/lens/client/auth.py
> in ()
> 19 import subprocess
> 20 import threading
> ---> 21 from urlparse import urlparse
> 22
> 23
> ImportError: No module named 'urlparse'
> {noformat}
> This appears to be due to the fact that in Python3, {{urlparse}} is
> {{urllib.parse}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (LENS-1518) Missing self for ignore cert variable in query.py file
Ankit Kailaswar created LENS-1518: - Summary: Missing self for ignore cert variable in query.py file Key: LENS-1518 URL: https://issues.apache.org/jira/browse/LENS-1518 Project: Apache Lens Issue Type: Bug Components: client Reporter: Ankit Kailaswar Assignee: Ankit Kailaswar Self is missing for variable corresponding to ignore cert. This is causing python query builder to fail while creating request object. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1518) Missing self for ignore cert variable in query.py file
[ https://issues.apache.org/jira/browse/LENS-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1518: -- Attachment: Lens-1518.patch > Missing self for ignore cert variable in query.py file > -- > > Key: LENS-1518 > URL: https://issues.apache.org/jira/browse/LENS-1518 > Project: Apache Lens > Issue Type: Bug > Components: client > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1518.patch > > > Self is missing for variable corresponding to ignore cert. This is causing > python query builder to fail while creating request object. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1526) Lens api to display number of sessions created per user
[ https://issues.apache.org/jira/browse/LENS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1526: -- Status: Patch Available (was: Open) > Lens api to display number of sessions created per user > --- > > Key: LENS-1526 > URL: https://issues.apache.org/jira/browse/LENS-1526 > Project: Apache Lens > Issue Type: Improvement > Components: api > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1526.patch > > > Exposing lens api to display number of sessions created per user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1526) Lens api to display number of sessions created per user
[ https://issues.apache.org/jira/browse/LENS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1526: -- Attachment: Lens-1526.patch > Lens api to display number of sessions created per user > --- > > Key: LENS-1526 > URL: https://issues.apache.org/jira/browse/LENS-1526 > Project: Apache Lens > Issue Type: Improvement > Components: api > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1526.patch > > > Exposing lens api to display number of sessions created per user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LENS-1526) Lens api to display number of sessions created per user
Ankit Kailaswar created LENS-1526: - Summary: Lens api to display number of sessions created per user Key: LENS-1526 URL: https://issues.apache.org/jira/browse/LENS-1526 Project: Apache Lens Issue Type: Improvement Components: api Reporter: Ankit Kailaswar Assignee: Ankit Kailaswar Exposing lens api to display number of sessions created per user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1506) Kerberos authentication in lens
[ https://issues.apache.org/jira/browse/LENS-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1506: -- Attachment: design3.png > Kerberos authentication in lens > --- > > Key: LENS-1506 > URL: https://issues.apache.org/jira/browse/LENS-1506 > Project: Apache Lens > Issue Type: Improvement > Components: client, driver-hive, python-client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: design3.png > > > Current Lens implementation is broken when we try to enable kerberos > authentication in lens as mentioned at > [https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2] in > following ways, > 1. openSession REST API fails to create new session for user. Currently it > supports only passwd types of authentication. > 2. If the underlying hive driver is running with kerberos authentication then > driver initialization flow to obtain hive transport for hive driver in lens > errors out. Hive server accepts only sasl messages but lens continues using > PLAINSASL. > 3. If hadoop cluster has kerberos authentication enabled then all hdfs calls > (persisting services, all hdfs path in conf etc) fail. > 4. Lens as if now doesnt supports refreshing KDC token before it expires. > Changes required in lens to fully support kerberose authentication are as > follows, > # lens's hive driver must use SASL for all communication in to kerberozied > hive. Current thrift client for hive doesn't support this functionality. > # Lens must refresh KDC ticket before it expires. > # All clients must be authenticated with kerberose authentication before > session creation. > # In kerberos mode all hive driver query should be executed with single > cluster user as "lens". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (LENS-1506) Kerberos authentication in lens
[ https://issues.apache.org/jira/browse/LENS-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar reassigned LENS-1506: - Assignee: Ankit Kailaswar > Kerberos authentication in lens > --- > > Key: LENS-1506 > URL: https://issues.apache.org/jira/browse/LENS-1506 > Project: Apache Lens > Issue Type: Improvement > Components: client, driver-hive, python-client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > > Current Lens implementation is broken when we try to enable kerberos > authentication in lens as mentioned at > [https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2] in > following ways, > 1. openSession REST API fails to create new session for user. Currently it > supports only passwd types of authentication. > 2. If the underlying hive driver is running with kerberos authentication then > driver initialization flow to obtain hive transport for hive driver in lens > errors out. Hive server accepts only sasl messages but lens continues using > PLAINSASL. > 3. If hadoop cluster has kerberos authentication enabled then all hdfs calls > (persisting services, all hdfs path in conf etc) fail. > 4. Lens as if now doesnt supports refreshing KDC token before it expires. > Changes required in lens to fully support kerberose authentication are as > follows, > # lens's hive driver must use SASL for all communication in to kerberozied > hive. Current thrift client for hive doesn't support this functionality. > # Lens must refresh KDC ticket before it expires. > # All clients must be authenticated with kerberose authentication before > session creation. > # In kerberos mode all hive driver query should be executed with single > cluster user as "lens". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1506) Kerberos authentication in lens
[ https://issues.apache.org/jira/browse/LENS-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1506: -- Attachment: (was: design3.png) > Kerberos authentication in lens > --- > > Key: LENS-1506 > URL: https://issues.apache.org/jira/browse/LENS-1506 > Project: Apache Lens > Issue Type: Improvement > Components: client, driver-hive, python-client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > > Current Lens implementation is broken when we try to enable kerberos > authentication in lens as mentioned at > [https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2] in > following ways, > 1. openSession REST API fails to create new session for user. Currently it > supports only passwd types of authentication. > 2. If the underlying hive driver is running with kerberos authentication then > driver initialization flow to obtain hive transport for hive driver in lens > errors out. Hive server accepts only sasl messages but lens continues using > PLAINSASL. > 3. If hadoop cluster has kerberos authentication enabled then all hdfs calls > (persisting services, all hdfs path in conf etc) fail. > 4. Lens as if now doesnt supports refreshing KDC token before it expires. > Changes required in lens to fully support kerberose authentication are as > follows, > # lens's hive driver must use SASL for all communication in to kerberozied > hive. Current thrift client for hive doesn't support this functionality. > # Lens must refresh KDC ticket before it expires. > # All clients must be authenticated with kerberose authentication before > session creation. > # In kerberos mode all hive driver query should be executed with single > cluster user as "lens". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (LENS-540) Skip deploying source archive and lens-server.war
[ https://issues.apache.org/jira/browse/LENS-540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar reassigned LENS-540: Assignee: Ankit Kailaswar (was: Jaideep Dhok) > Skip deploying source archive and lens-server.war > --- > > Key: LENS-540 > URL: https://issues.apache.org/jira/browse/LENS-540 > Project: Apache Lens > Issue Type: Improvement > Components: build >Reporter: Amareshwari Sriramadasu > Assignee: Ankit Kailaswar >Priority: Critical > > During 2.1 release deploy, i noticed we are deploying source-release.zip and > lens-server war to maven repository. > https://repository.apache.org/content/repositories/releases/org/apache/lens/apache-lens/2.1.0-beta-incubating/ > https://repository.apache.org/content/repositories/releases/org/apache/lens/lens-server/2.1.0-beta-incubating/ > Those should be skipped as their size is huge , and we are providing them > separate downloadable release artifacts through mirrors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LENS-1506) Kerberos authentication in lens
Ankit Kailaswar created LENS-1506: - Summary: Kerberos authentication in lens Key: LENS-1506 URL: https://issues.apache.org/jira/browse/LENS-1506 Project: Apache Lens Issue Type: Improvement Components: client, driver-hive, python-client, server Reporter: Ankit Kailaswar Current Lens implementation is broken when we try to enable kerberos authentication in lens as mentioned at [https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2] in following ways, 1. openSession REST API fails to create new session for user. Currently it supports only passwd types of authentication. 2. If the underlying hive driver is running with kerberos authentication then driver initialization flow to obtain hive transport for hive driver in lens errors out. Hive server accepts only sasl messages but lens continues using PLAINSASL. 3. If hadoop cluster has kerberos authentication enabled then all hdfs calls (persisting services, all hdfs path in conf etc) fail. 4. Lens as if now doesnt supports refreshing KDC token before it expires. Changes required in lens to fully support kerberose authentication are as follows, # lens's hive driver must use SASL for all communication in to kerberozied hive. Current thrift client for hive doesn't support this functionality. # Lens must refresh KDC ticket before it expires. # All clients must be authenticated with kerberose authentication before session creation. # In kerberos mode all hive driver query should be executed with single cluster user as "lens". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (LENS-540) Skip deploying source archive and lens-server.war
[ https://issues.apache.org/jira/browse/LENS-540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar reassigned LENS-540: Assignee: Jaideep Dhok (was: Ankit Kailaswar) > Skip deploying source archive and lens-server.war > --- > > Key: LENS-540 > URL: https://issues.apache.org/jira/browse/LENS-540 > Project: Apache Lens > Issue Type: Improvement > Components: build >Reporter: Amareshwari Sriramadasu >Assignee: Jaideep Dhok >Priority: Critical > > During 2.1 release deploy, i noticed we are deploying source-release.zip and > lens-server war to maven repository. > https://repository.apache.org/content/repositories/releases/org/apache/lens/apache-lens/2.1.0-beta-incubating/ > https://repository.apache.org/content/repositories/releases/org/apache/lens/lens-server/2.1.0-beta-incubating/ > Those should be skipped as their size is huge , and we are providing them > separate downloadable release artifacts through mirrors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1506) Kerberos authentication in lens
[ https://issues.apache.org/jira/browse/LENS-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1506: -- Attachment: design3.png > Kerberos authentication in lens > --- > > Key: LENS-1506 > URL: https://issues.apache.org/jira/browse/LENS-1506 > Project: Apache Lens > Issue Type: Improvement > Components: client, driver-hive, python-client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: design3.png > > > Current Lens implementation is broken when we try to enable kerberos > authentication in lens as mentioned at > [https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2] in > following ways, > 1. openSession REST API fails to create new session for user. Currently it > supports only passwd types of authentication. > 2. If the underlying hive driver is running with kerberos authentication then > driver initialization flow to obtain hive transport for hive driver in lens > errors out. Hive server accepts only sasl messages but lens continues using > PLAINSASL. > 3. If hadoop cluster has kerberos authentication enabled then all hdfs calls > (persisting services, all hdfs path in conf etc) fail. > 4. Lens as if now doesnt supports refreshing KDC token before it expires. > Changes required in lens to fully support kerberose authentication are as > follows, > # lens's hive driver must use SASL for all communication in to kerberozied > hive. Current thrift client for hive doesn't support this functionality. > # Lens must refresh KDC ticket before it expires. > # All clients must be authenticated with kerberose authentication before > session creation. > # In kerberos mode all hive driver query should be executed with single > cluster user as "lens". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LENS-1506) Kerberos authentication in lens
[ https://issues.apache.org/jira/browse/LENS-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400034#comment-16400034 ] Ankit Kailaswar commented on LENS-1506: --- review request : https://reviews.apache.org/r/66081/ > Kerberos authentication in lens > --- > > Key: LENS-1506 > URL: https://issues.apache.org/jira/browse/LENS-1506 > Project: Apache Lens > Issue Type: Improvement > Components: client, driver-hive, python-client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: design3.png > > > Current Lens implementation is broken when we try to enable kerberos > authentication in lens as mentioned at > [https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2] in > following ways, > 1. openSession REST API fails to create new session for user. Currently it > supports only passwd types of authentication. > 2. If the underlying hive driver is running with kerberos authentication then > driver initialization flow to obtain hive transport for hive driver in lens > errors out. Hive server accepts only sasl messages but lens continues using > PLAINSASL. > 3. If hadoop cluster has kerberos authentication enabled then all hdfs calls > (persisting services, all hdfs path in conf etc) fail. > 4. Lens as if now doesnt supports refreshing KDC token before it expires. > Changes required in lens to fully support kerberose authentication are as > follows, > # lens's hive driver must use SASL for all communication in to kerberozied > hive. Current thrift client for hive doesn't support this functionality. > # Lens must refresh KDC ticket before it expires. > # All clients must be authenticated with kerberose authentication before > session creation. > # In kerberos mode all hive driver query should be executed with single > cluster user as "lens". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1506) Kerberos authentication in lens
[ https://issues.apache.org/jira/browse/LENS-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1506: -- Attachment: Lens-1506_patch > Kerberos authentication in lens > --- > > Key: LENS-1506 > URL: https://issues.apache.org/jira/browse/LENS-1506 > Project: Apache Lens > Issue Type: Improvement > Components: client, driver-hive, python-client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1506_patch, design3.png > > > Current Lens implementation is broken when we try to enable kerberos > authentication in lens as mentioned at > [https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2] in > following ways, > 1. openSession REST API fails to create new session for user. Currently it > supports only passwd types of authentication. > 2. If the underlying hive driver is running with kerberos authentication then > driver initialization flow to obtain hive transport for hive driver in lens > errors out. Hive server accepts only sasl messages but lens continues using > PLAINSASL. > 3. If hadoop cluster has kerberos authentication enabled then all hdfs calls > (persisting services, all hdfs path in conf etc) fail. > 4. Lens as if now doesnt supports refreshing KDC token before it expires. > Changes required in lens to fully support kerberose authentication are as > follows, > # lens's hive driver must use SASL for all communication in to kerberozied > hive. Current thrift client for hive doesn't support this functionality. > # Lens must refresh KDC ticket before it expires. > # All clients must be authenticated with kerberose authentication before > session creation. > # In kerberos mode all hive driver query should be executed with single > cluster user as "lens". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LENS-1506) Kerberos authentication in lens
[ https://issues.apache.org/jira/browse/LENS-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423925#comment-16423925 ] Ankit Kailaswar commented on LENS-1506: --- Taken patch from review board and attaching. > Kerberos authentication in lens > --- > > Key: LENS-1506 > URL: https://issues.apache.org/jira/browse/LENS-1506 > Project: Apache Lens > Issue Type: Improvement > Components: client, driver-hive, python-client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: design3.png > > > Current Lens implementation is broken when we try to enable kerberos > authentication in lens as mentioned at > [https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2] in > following ways, > 1. openSession REST API fails to create new session for user. Currently it > supports only passwd types of authentication. > 2. If the underlying hive driver is running with kerberos authentication then > driver initialization flow to obtain hive transport for hive driver in lens > errors out. Hive server accepts only sasl messages but lens continues using > PLAINSASL. > 3. If hadoop cluster has kerberos authentication enabled then all hdfs calls > (persisting services, all hdfs path in conf etc) fail. > 4. Lens as if now doesnt supports refreshing KDC token before it expires. > Changes required in lens to fully support kerberose authentication are as > follows, > # lens's hive driver must use SASL for all communication in to kerberozied > hive. Current thrift client for hive doesn't support this functionality. > # Lens must refresh KDC ticket before it expires. > # All clients must be authenticated with kerberose authentication before > session creation. > # In kerberos mode all hive driver query should be executed with single > cluster user as "lens". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1506) Kerberos authentication in lens
[ https://issues.apache.org/jira/browse/LENS-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1506: -- Attachment: Lens-1506.diff > Kerberos authentication in lens > --- > > Key: LENS-1506 > URL: https://issues.apache.org/jira/browse/LENS-1506 > Project: Apache Lens > Issue Type: Improvement > Components: client, driver-hive, python-client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1506.diff, Lens-1506_patch, design3.png > > > Current Lens implementation is broken when we try to enable kerberos > authentication in lens as mentioned at > [https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2] in > following ways, > 1. openSession REST API fails to create new session for user. Currently it > supports only passwd types of authentication. > 2. If the underlying hive driver is running with kerberos authentication then > driver initialization flow to obtain hive transport for hive driver in lens > errors out. Hive server accepts only sasl messages but lens continues using > PLAINSASL. > 3. If hadoop cluster has kerberos authentication enabled then all hdfs calls > (persisting services, all hdfs path in conf etc) fail. > 4. Lens as if now doesnt supports refreshing KDC token before it expires. > Changes required in lens to fully support kerberose authentication are as > follows, > # lens's hive driver must use SASL for all communication in to kerberozied > hive. Current thrift client for hive doesn't support this functionality. > # Lens must refresh KDC ticket before it expires. > # All clients must be authenticated with kerberose authentication before > session creation. > # In kerberos mode all hive driver query should be executed with single > cluster user as "lens". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1506) Kerberos authentication in lens
[ https://issues.apache.org/jira/browse/LENS-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1506: -- Attachment: Lens-1506.4.patch > Kerberos authentication in lens > --- > > Key: LENS-1506 > URL: https://issues.apache.org/jira/browse/LENS-1506 > Project: Apache Lens > Issue Type: Improvement > Components: client, driver-hive, python-client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1506.1.patch, Lens-1506.2.patch, Lens-1506.3.patch, > Lens-1506.4.patch, Lens-1506_patch, design3.png > > > Current Lens implementation is broken when we try to enable kerberos > authentication in lens as mentioned at > [https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2] in > following ways, > 1. openSession REST API fails to create new session for user. Currently it > supports only passwd types of authentication. > 2. If the underlying hive driver is running with kerberos authentication then > driver initialization flow to obtain hive transport for hive driver in lens > errors out. Hive server accepts only sasl messages but lens continues using > PLAINSASL. > 3. If hadoop cluster has kerberos authentication enabled then all hdfs calls > (persisting services, all hdfs path in conf etc) fail. > 4. Lens as if now doesnt supports refreshing KDC token before it expires. > Changes required in lens to fully support kerberose authentication are as > follows, > # lens's hive driver must use SASL for all communication in to kerberozied > hive. Current thrift client for hive doesn't support this functionality. > # Lens must refresh KDC ticket before it expires. > # All clients must be authenticated with kerberose authentication before > session creation. > # In kerberos mode all hive driver query should be executed with single > cluster user as "lens". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1506) Kerberos authentication in lens
[ https://issues.apache.org/jira/browse/LENS-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1506: -- Attachment: (was: Lens-1506.diff) > Kerberos authentication in lens > --- > > Key: LENS-1506 > URL: https://issues.apache.org/jira/browse/LENS-1506 > Project: Apache Lens > Issue Type: Improvement > Components: client, driver-hive, python-client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1506_patch, design3.png > > > Current Lens implementation is broken when we try to enable kerberos > authentication in lens as mentioned at > [https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2] in > following ways, > 1. openSession REST API fails to create new session for user. Currently it > supports only passwd types of authentication. > 2. If the underlying hive driver is running with kerberos authentication then > driver initialization flow to obtain hive transport for hive driver in lens > errors out. Hive server accepts only sasl messages but lens continues using > PLAINSASL. > 3. If hadoop cluster has kerberos authentication enabled then all hdfs calls > (persisting services, all hdfs path in conf etc) fail. > 4. Lens as if now doesnt supports refreshing KDC token before it expires. > Changes required in lens to fully support kerberose authentication are as > follows, > # lens's hive driver must use SASL for all communication in to kerberozied > hive. Current thrift client for hive doesn't support this functionality. > # Lens must refresh KDC ticket before it expires. > # All clients must be authenticated with kerberose authentication before > session creation. > # In kerberos mode all hive driver query should be executed with single > cluster user as "lens". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1506) Kerberos authentication in lens
[ https://issues.apache.org/jira/browse/LENS-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1506: -- Status: Patch Available (was: In Progress) attached patch > Kerberos authentication in lens > --- > > Key: LENS-1506 > URL: https://issues.apache.org/jira/browse/LENS-1506 > Project: Apache Lens > Issue Type: Improvement > Components: client, driver-hive, python-client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1506.1.patch, Lens-1506_patch, design3.png > > > Current Lens implementation is broken when we try to enable kerberos > authentication in lens as mentioned at > [https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2] in > following ways, > 1. openSession REST API fails to create new session for user. Currently it > supports only passwd types of authentication. > 2. If the underlying hive driver is running with kerberos authentication then > driver initialization flow to obtain hive transport for hive driver in lens > errors out. Hive server accepts only sasl messages but lens continues using > PLAINSASL. > 3. If hadoop cluster has kerberos authentication enabled then all hdfs calls > (persisting services, all hdfs path in conf etc) fail. > 4. Lens as if now doesnt supports refreshing KDC token before it expires. > Changes required in lens to fully support kerberose authentication are as > follows, > # lens's hive driver must use SASL for all communication in to kerberozied > hive. Current thrift client for hive doesn't support this functionality. > # Lens must refresh KDC ticket before it expires. > # All clients must be authenticated with kerberose authentication before > session creation. > # In kerberos mode all hive driver query should be executed with single > cluster user as "lens". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1506) Kerberos authentication in lens
[ https://issues.apache.org/jira/browse/LENS-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1506: -- Attachment: Lens-1506.1.patch > Kerberos authentication in lens > --- > > Key: LENS-1506 > URL: https://issues.apache.org/jira/browse/LENS-1506 > Project: Apache Lens > Issue Type: Improvement > Components: client, driver-hive, python-client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1506.1.patch, Lens-1506_patch, design3.png > > > Current Lens implementation is broken when we try to enable kerberos > authentication in lens as mentioned at > [https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2] in > following ways, > 1. openSession REST API fails to create new session for user. Currently it > supports only passwd types of authentication. > 2. If the underlying hive driver is running with kerberos authentication then > driver initialization flow to obtain hive transport for hive driver in lens > errors out. Hive server accepts only sasl messages but lens continues using > PLAINSASL. > 3. If hadoop cluster has kerberos authentication enabled then all hdfs calls > (persisting services, all hdfs path in conf etc) fail. > 4. Lens as if now doesnt supports refreshing KDC token before it expires. > Changes required in lens to fully support kerberose authentication are as > follows, > # lens's hive driver must use SASL for all communication in to kerberozied > hive. Current thrift client for hive doesn't support this functionality. > # Lens must refresh KDC ticket before it expires. > # All clients must be authenticated with kerberose authentication before > session creation. > # In kerberos mode all hive driver query should be executed with single > cluster user as "lens". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1506) Kerberos authentication in lens
[ https://issues.apache.org/jira/browse/LENS-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1506: -- Attachment: Lens-1506.2.patch > Kerberos authentication in lens > --- > > Key: LENS-1506 > URL: https://issues.apache.org/jira/browse/LENS-1506 > Project: Apache Lens > Issue Type: Improvement > Components: client, driver-hive, python-client, server > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1506.1.patch, Lens-1506.2.patch, Lens-1506_patch, > design3.png > > > Current Lens implementation is broken when we try to enable kerberos > authentication in lens as mentioned at > [https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2] in > following ways, > 1. openSession REST API fails to create new session for user. Currently it > supports only passwd types of authentication. > 2. If the underlying hive driver is running with kerberos authentication then > driver initialization flow to obtain hive transport for hive driver in lens > errors out. Hive server accepts only sasl messages but lens continues using > PLAINSASL. > 3. If hadoop cluster has kerberos authentication enabled then all hdfs calls > (persisting services, all hdfs path in conf etc) fail. > 4. Lens as if now doesnt supports refreshing KDC token before it expires. > Changes required in lens to fully support kerberose authentication are as > follows, > # lens's hive driver must use SASL for all communication in to kerberozied > hive. Current thrift client for hive doesn't support this functionality. > # Lens must refresh KDC ticket before it expires. > # All clients must be authenticated with kerberose authentication before > session creation. > # In kerberos mode all hive driver query should be executed with single > cluster user as "lens". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (LENS-1510) hive's server princiapl in lens's hive driver
Ankit Kailaswar created LENS-1510: - Summary: hive's server princiapl in lens's hive driver Key: LENS-1510 URL: https://issues.apache.org/jira/browse/LENS-1510 Project: Apache Lens Issue Type: Bug Components: driver-hive Reporter: Ankit Kailaswar since the property name for lens and hive's principal name are conflicting we need explicitly need to mention hive's principal while creating hive driver for lens. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1510) hive's server princiapl in lens's hive driver
[ https://issues.apache.org/jira/browse/LENS-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Kailaswar updated LENS-1510: -- Status: Patch Available (was: In Progress) https://reviews.apache.org/r/66730/ > hive's server princiapl in lens's hive driver > - > > Key: LENS-1510 > URL: https://issues.apache.org/jira/browse/LENS-1510 > Project: Apache Lens > Issue Type: Bug > Components: driver-hive > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1510_1.diff > > > since the property name for lens and hive's principal name are conflicting we > need explicitly need to mention hive's principal while creating hive driver > for lens. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (LENS-1510) hive's server princiapl in lens's hive driver
[ https://issues.apache.org/jira/browse/LENS-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on LENS-1510 started by Ankit Kailaswar. - > hive's server princiapl in lens's hive driver > - > > Key: LENS-1510 > URL: https://issues.apache.org/jira/browse/LENS-1510 > Project: Apache Lens > Issue Type: Bug > Components: driver-hive > Reporter: Ankit Kailaswar > Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1510_1.diff > > > since the property name for lens and hive's principal name are conflicting we > need explicitly need to mention hive's principal while creating hive driver > for lens. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
