[GitHub] zeppelin pull request #2886: ZEPPELIN-3356: Zeppelin FileSystemStorage relog...
Github user gss2002 closed the pull request at: https://github.com/apache/zeppelin/pull/2886 ---
[GitHub] zeppelin issue #2886: ZEPPELIN-3356: Zeppelin FileSystemStorage reloginFromK...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/2886 @prabhjyotsingh @zjffdu I made changes to check if security is enabled and if it was logged in via a keytab and than i relogin with checktgt method vs relogining in every time causing excess load on the kdc ---
[GitHub] zeppelin issue #2886: ZEPPELIN-3356: Zeppelin FileSystemStorage reloginFromK...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/2886 @zjffdu I am going to cut the new improved fix based on original feedback. But yes you will have to adjust the KDC to test this as Java does not use ticket_lifetime or renew_lifetime from krb5.conf per this article not fixed until Java 9. https://stackoverflow.com/questions/38555244/how-do-you-set-the-kerberos-ticket-lifetime-from-java https://bugs.openjdk.java.net/browse/JDK-8044500 ---
[GitHub] zeppelin issue #2886: ZEPPELIN-3356: Zeppelin FileSystemStorage reloginFromK...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/2886 https://stackoverflow.com/questions/38555244/how-do-you-set-the-kerberos-ticket-lifetime-from-java https://bugs.openjdk.java.net/browse/JDK-8044500 ---
[GitHub] zeppelin issue #2886: ZEPPELIN-3356: Zeppelin FileSystemStorage reloginFromK...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/2886 @zjffdu you cannot just update the krb5.conf those are just recommendations on the client side. The KDC both with MIT Krb5 and Active Directory control the max_renewable_lifetime via /var/kerberos/krb5kdc/kdc.conf and settings in Windows registry. My co-worker and I tested this today and the ticket is still renewable because the KDC controls the max time and it looks as if Java takes info from the KDC... Using the CLI kinit/klist and hadoop fs the ticket is expired. But from the looks of it when logging in with a keytab via UGI which zeppelin does for the HDFS calls it takes the settings from the kdc... See below: JDK - KRB5 DEBUG OUTPUT from Zeppelin JVM: Native config name: /etc/krb5.conf Loaded from native config >>> KdcAccessibility: reset >>> KdcAccessibility: reset >>> KeyTabInputStream, readName(): UNIT.HDP.EXAMPLE.COM >>> KeyTabInputStream, readName(): zeppelin-unit >>> KeyTab: load() entry length: 88; type: 18 >>> KeyTabInputStream, readName(): UNIT.HDP.EXAMPLE.COM >>> KeyTabInputStream, readName(): zeppelin-unit >>> KeyTab: load() entry length: 72; type: 17 >>> KeyTabInputStream, readName(): UNIT.HDP.EXAMPLE.COM >>> KeyTabInputStream, readName(): zeppelin-unit >>> KeyTab: load() entry length: 72; type: 23 Looking for keys for: zeppelin-u...@unit.hdp.example.com Added key: 23version: 2 Added key: 17version: 2 Added key: 18version: 2 Looking for keys for: zeppelin-u...@unit.hdp.example.com Added key: 23version: 2 Added key: 17version: 2 Added key: 18version: 2 Using builtin default etypes for default_tkt_enctypes default etypes for default_tkt_enctypes: 18 17 16 23. >>> KrbAsReq creating message >>> KrbKdcReq send: kdc=ha21d51kd.unit.hdp.example.com TCP:88, timeout=3, number of retries =3, #bytes=174 >>> KDCCommunication: kdc=ha21d51kd.unit.hdp.example.com TCP:88, timeout=3,Attempt =1, #bytes=174 >>>DEBUG: TCPClient reading 769 bytes >>> KrbKdcReq send: #bytes read=769 >>> KdcAccessibility: remove ha21d51kd.unit.hdp.example.com Looking for keys for: zeppelin-u...@unit.hdp.example.com Added key: 23version: 2 Added key: 17version: 2 Added key: 18version: 2 >>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType >>> KrbAsRep cons in KrbAsReq.getReply zeppelin-unit Found ticket for zeppelin-u...@unit.hdp.example.com to go to krbtgt/unit.hdp.example@unit.hdp.example.com expiring on Wed Mar 28 23:28:46 EDT 2018 Entered Krb5Context.initSecContext with state=STATE_NEW Found ticket for zeppelin-u...@unit.hdp.example.com to go to krbtgt/unit.hdp.example@unit.hdp.example.com expiring on Wed Mar 28 23:28:46 EDT 2018 Service ticket not found in the subject >>> Credentials acquireServiceCreds: same realm Using builtin default etypes for default_tgs_enctypes default etypes for default_tgs_enctypes: 18 17 16 23. Showing Zeppelin was started after modifying /etc/krb5.conf 2m/5m ticket_lifetime/renew_lifetime [root@ha21d55en zeppelin]# ps guaxww | grep -i zeppelin zeppelin 89982 2.4 3.6 6872888 601888 ? Sl 13:28 0:30 /usr/jdk64/jdk1.8.0_102/bin/java -Dsun.security.krb5.debug=true -Dhdp.version=2.5.3.18-5 -Dspark.executor.memory=512m -Dspark.yarn.queue=default -Dfile.encoding=UTF-8 -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -Dlog4j.configuration=file:///usr/local/zeppelin/current/conf/log4j.properties -Dzeppelin.log.file=/var/log/zeppelin/zeppelin-zeppelin-ha21d55en.unit.hdp.example.com.log -cp ::/usr/local/zeppelin/current/lib/interpreter/*:/usr/local/zeppelin/current/lib/*:/usr/local/zeppelin/current/*::/usr/local/zeppelin/current/conf:/etc/hadoop/conf org.apache.zeppelin.server.ZeppelinServer zeppelin 90439 0.0 0.0 113124 1524 ?S13:30 0:00 /bin/bash /usr/local/zeppelin/current/bin/interpreter.sh -d /usr/local/zeppelin/current/interpreter/livy -c 10.70.57.5 -p 41478 -r : -l /usr/local/zeppelin/current/local-repo/livy1 -g livy1 zeppelin 90454 0.0 0.0 113120 836 ?S13:30 0:00 /bin/bash /usr/local/zeppelin/current/bin/interpreter.sh -d /usr/local/zeppelin/current/interpreter/livy -c 10.70.57.5 -p 41478 -r : -l /usr/local/zeppelin/current/local-repo/livy1 -g livy1 zeppelin 90455 0.3 1.3 5198944 214228 ? Sl 13:30 0:04 /usr/jdk64/jdk1.8.0_102/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/local/zeppelin/current/conf/log4j.properties -Dzeppelin.log.file=/var/log/zeppelin/zeppelin-interpreter-livy1-zeppelin-ha21d55en.unit.hdp.example.com.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -
[GitHub] zeppelin issue #2886: ZEPPELIN-3356: Zeppelin FileSystemStorage reloginFromK...
Github user gss2002 commented on the issue:
https://github.com/apache/zeppelin/pull/2886
@prabhjyotsingh I just read that same stackoverflow part of me says use
checktgtandreloginfronkeytab to be lighter on kdc thoughts? I will dig a bit
deeper in am but auto renewal thread that exists in ugi cannot go beyond max
renewal
@felixcheung I think you are right if I do
usergroupinformation.getCurrentUser().checkTGtAndReloginFromKeytab() would work
too
private void reloginFromKeytab(boolean checkTGT) throws IOException {
if (!shouldRelogin() || !isFromKeytab()) {
return;
}
HadoopLoginContext login = getLogin();
if (login == null) {
throw new KerberosAuthException(MUST_FIRST_LOGIN_FROM_KEYTAB);
}
if (checkTGT) {
KerberosTicket tgt = getTGT();
if (tgt != null && !shouldRenewImmediatelyForTests &&
Time.now() < getRefreshTime(tgt)) {
return;
}
}
relogin(login);
}
---
[GitHub] zeppelin issue #2886: ZEPPELIN-3356: Zeppelin FileSystemStorage reloginFromK...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/2886 @prabhjyotsingh @zjffdu can you help review if you feel this is a valid fix? Thanks again ---
[GitHub] zeppelin issue #2886: ZEPPELIN-3356: Zeppelin FileSystemStorage reloginFromK...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/2886 @zjffdu here is a patch that I think will fix this issue. I will know in 7 days if the issue comes back but has plagued our 4 different environments running Zeppelin over the last few days since it has reached max timeout. Let me know your thoughts on this patch. Also the CI failures look to be un-related. ---
[GitHub] zeppelin pull request #2886: ZEPPELIN-3356: Zeppelin FileSystemStorage relog...
GitHub user gss2002 opened a pull request: https://github.com/apache/zeppelin/pull/2886 ZEPPELIN-3356: Zeppelin FileSystemStorage reloginFromKeytab needed What is this PR for? During long runs of Apache Zeppelin using HDFS as the backing configuration and notebook storage. We noticed that when the Zeppelin Server ticket had reached 7 days our max renewal time the keytab is not re-logged in leaving the Zeppelin Server in an unusable state. The solution is to reLoginFromKeytab before any operations as it will check if the ticket needs to be relogged in. What type of PR is it? [Bug Fix] Todos What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3356 How should this be tested? Run Zeppelin Server for the max kerberos renewal time Screenshots (if appropriate) Questions: Does the licenses files need update? No Is there breaking changes for older versions? No Does this needs documentation? No Author: Greg Senia gse...@apache.org You can merge this pull request into a Git repository by running: $ git pull https://github.com/gss2002/zeppelin ZEPPELIN-3356 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2886.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2886 commit dc190e5979ffaca2ae36cdbc5a171624ce5868d5 Author: gss2002 <greg@...> Date: 2018-03-21T16:33:34Z ZEPPELIN-3356: Zeppelin FileSystem Storage reloginFromKeytab needed ---
[GitHub] zeppelin issue #2855: ZEPPELIN-3309. Import/Clone user not set in Paragraph ...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/2855 @zjffdu the only test not passing was the e2e test which I am re-runnin gnow.. Let me know if I need to do anything else. ---
[GitHub] zeppelin issue #2855: ZEPPELIN-3309. Import/Clone user not set in Paragraph ...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/2855 @zjffdu any reason why travis-ci is failing on disk issues? travis_time:end:20c877c0:start=1520617145805371744,finish=1520617145811589622,duration=6217878 [0Ktravis_fold:end:after_failure.11 [0Ktravis_fold:start:after_failure.12 [0Ktravis_time:start:09d64a64 [0K$ cat livy/target/tmp/livy-int-test/MiniYarnMain/target/org.apache.livy.test.framework.MiniYarnMain/*/*/*/stderr cat: livy/target/tmp/livy-int-test/MiniYarnMain/target/org.apache.livy.test.framework.MiniYarnMain/*/*/*/stderr: No such file or directory travis_time:end:09d64a64:start=1520617145818231645,finish=1520617145824458197,duration=6226552 [0Ktravis_fold:end:after_failure.12 [0K Done. Your build exited with 1. grep: write error: No space left on device ---
[GitHub] zeppelin issue #2855: ZEPPELIN-3309. Import/Clone user not set in Paragraph ...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/2855 @zjffdu I think this pull request should be good. There looks to be some issues with travis unrelated to the code changes. And I ran the tests a few times not sure whats up with it. ---
[GitHub] zeppelin pull request #2855: ZEPPELIN-3309. Import/Clone user not set in Par...
GitHub user gss2002 opened a pull request: https://github.com/apache/zeppelin/pull/2855 ZEPPELIN-3309. Import/Clone user not set in Paragraph causes NPE. What is this PR for? During Import/Clone Paragraph set "user" to eliminate NPEs thrown in Helium and other functions leaving unusable notebooks. What type of PR is it? [Bug Fix] Todos What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3309 How should this be tested? Manually tested using Import/Clone of Notebooks and attempt to adjust bound interpreters Screenshots (if appropriate) Questions: Does the licenses files need update? No Is there breaking changes for older versions? No Does this needs documentation? No Author: Greg Senia gse...@apache.org You can merge this pull request into a Git repository by running: $ git pull https://github.com/gss2002/zeppelin ZEPPELIN-3309 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2855.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2855 commit b8c35c964b07b9836ca09cbb5bb4a8509b2367cd Author: gss2002 <greg@...> Date: 2018-03-08T22:06:06Z ZEPPELIN-3309. Import/Clone user not set in Paragraph causes NPE. ---
[GitHub] zeppelin pull request #2851: ZEPPELIN-3309. Import/Clone user not set in Par...
Github user gss2002 closed the pull request at: https://github.com/apache/zeppelin/pull/2851 ---
[GitHub] zeppelin issue #2851: ZEPPELIN-3309. Import/Clone user not set in Paragraph ...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/2851 @zjffdu thanks for the insight. It's setup and running now. ---
[GitHub] zeppelin pull request #2851: ZEPPELIN-3309. Import/Clone doesn't set user in...
GitHub user gss2002 reopened a pull request: https://github.com/apache/zeppelin/pull/2851 ZEPPELIN-3309. Import/Clone doesn't set user in Paragraph causing NPE What is this PR for? During Import/Clone Paragraph set "user" to eliminate NPEs thrown in Helium and other functions leaving unusable notebooks. What type of PR is it? [Bug Fix] Todos - Task What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3309 How should this be tested? Manually tested using Import/Clone of Notebooks and attempt to adjust bound interpreters and tested using new unit test to clone notebooks to verify username is added during clone. Screenshots (if appropriate) Questions: Does the licenses files need update? No Is there breaking changes for older versions? No Does this needs documentation? No Author: Greg Senia gse...@apache.org You can merge this pull request into a Git repository by running: $ git pull https://github.com/gss2002/zeppelin ZEPPELIN-3309 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2851.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2851 commit 091130073ee545a4f637d53708ee1f7165bddebc Author: gss2002 <greg@...> Date: 2018-03-08T22:06:06Z ZEPPELIN-3309. Import/Clone doesn't set user in Paragraph causing NPE ---
[GitHub] zeppelin issue #2851: ZEPPELIN-3309. Import/Clone doesn't set user in Paragr...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/2851 @zjffdu any chance you would possibly know why the commit wouldn't be found? [2018-03-08 22:14:11] Can't find build for commit 091130073ee545a4f637d53708ee1f7165bddebc from gss2002 ---
[GitHub] zeppelin pull request #2851: ZEPPELIN-3309. Import/Clone doesn't set user in...
GitHub user gss2002 opened a pull request: https://github.com/apache/zeppelin/pull/2851 ZEPPELIN-3309. Import/Clone doesn't set user in Paragraph causing NPE What is this PR for? During Import/Clone Paragraph set "user" to eliminate NPEs thrown in Helium and other functions leaving unusable notebooks. What type of PR is it? [Bug Fix] Todos - Task What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3309 How should this be tested? Manually tested using Import/Clone of Notebooks and attempt to adjust bound interpreters and tested using new unit test to clone notebooks to verify username is added during clone. Screenshots (if appropriate) Questions: Does the licenses files need update? No Is there breaking changes for older versions? No Does this needs documentation? No Author: Greg Senia gse...@apache.org You can merge this pull request into a Git repository by running: $ git pull https://github.com/gss2002/zeppelin ZEPPELIN-3309 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2851.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2851 commit 091130073ee545a4f637d53708ee1f7165bddebc Author: gss2002 <greg@...> Date: 2018-03-08T22:06:06Z ZEPPELIN-3309. Import/Clone doesn't set user in Paragraph causing NPE ---
[GitHub] zeppelin issue #2849: ZEPPELIN-3309. Import/Clone doesn't set user in Paragr...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/2849 Closing this pull request for now will re-open with a new one which includes formatting changes and corrections and test cases. ---
[GitHub] zeppelin pull request #2849: ZEPPELIN-3309. Import/Clone doesn't set user in...
Github user gss2002 closed the pull request at: https://github.com/apache/zeppelin/pull/2849 ---
[GitHub] zeppelin issue #2849: ZEPPELIN-3309. Import/Clone doesn't set user in Paragr...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/2849 @zjffdu no problem. yes I will work on a unit test later tonight/tomorrow am. ---
[GitHub] zeppelin pull request #2849: ZEPPELIN-3309. Import/Clone doesn't set user in...
GitHub user gss2002 opened a pull request: https://github.com/apache/zeppelin/pull/2849 ZEPPELIN-3309. Import/Clone doesn't set user in Paragraph causing NPE ### What is this PR for? During Import/Clone Paragraph set "user" to eliminate NPEs thrown in Helium and other functions leaving unusable notebooks. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3309 ### How should this be tested? * Manually tested using Import/Clone of Notebooks and attempt to adjust bound interpreters ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Greg Senia <gse...@apache.org> You can merge this pull request into a Git repository by running: $ git pull https://github.com/gss2002/zeppelin ZEPPELIN-3309 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2849.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2849 commit e4f01773e1f848d45d3706ad81a1f9801d256802 Author: gss2002 <greg@...> Date: 2018-03-08T01:39:19Z ZEPPELIN-3309. Import/Clone doesn't set user in Paragraph causing NPE ---
[GitHub] zeppelin pull request #1516: ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 E...
Github user gss2002 closed the pull request at: https://github.com/apache/zeppelin/pull/1516 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1503: NPE LivySparkSQLInterpreter thrown with %livy.s...
Github user gss2002 closed the pull request at: https://github.com/apache/zeppelin/pull/1503 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1503: NPE LivySparkSQLInterpreter thrown with %livy.sql inte...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1503 @zjffdu all set --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1614: ZEPPELIN-1472 - Create new LdapRealm based on A...
GitHub user gss2002 reopened a pull request:
https://github.com/apache/zeppelin/pull/1614
ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm:
Provides LdapRealm Functionality similar to Apache Knox
### What is this PR for?
ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm:
Provides LdapRealm Functionality similar to what Apache Knox provides. This is
critical as in large enterprise environments Active Directory Global Catalogs
are used for lookup with samAccountName and using a DN Template is not an
option as their are multiple OUs. Also searching on "userPrincipalName" is
risky in an AD environment since the explicit UPN vs Implicit UPN can be
different this is definitely the case with environments using Office 365. And
the LDAP userPrincipalName attribute is the explicit UPN which can be defined
by the directory administrator to any value and it can be duplicated.
SamAccountName is unique per domain and Microsoft states best practice is to
not allow duplicate samAccountName's across the forest.
In addition to the above changes I have adjusted and moved the
LdapGroupRealm and ActiveDirectoryGroupRealm into the org.apache.zeppelin.realm
package structure to make all Realm's consistent.
The LdapRealm class also works with role to group mapping for usage within
Zeppelin for notebook authorization.
I have adjusted SecurityUtils to use ClassName vs realmName in determining
what to use as you may have companies that decide to use their own custom
realmname in shiro.ini and may not realize you cannot so using className is
much safer.
Example - SecurityUtils
String name = realm.getClass().getName();
if (name.equals("org.apache.shiro.realm.text.IniRealm")) {
allRoles = ((IniRealm) realm).getIni().get("roles");
break;
} else if (name.equals("org.apache.zeppelin.realm.LdapRealm")) {
allRoles = ((LdapRealm) realm).getListRoles();
break;
}
Example - SecurityRestApi:
String name = realm.getClass().getName();
if (LOG.isDebugEnabled()) {
LOG.debug("RealmClass.getName: " + name);
}
if (name.equals("org.apache.shiro.realm.text.IniRealm")) {
usersList.addAll(getUserListObj.getUserList((IniRealm) realm));
rolesList.addAll(getUserListObj.getRolesList((IniRealm) realm));
} else if
(name.equals("org.apache.zeppelin.realm.LdapGroupRealm")) {
usersList.addAll(getUserListObj.getUserList((JndiLdapRealm)
realm, searchText));
} else if (name.equals("org.apache.zeppelin.realm.LdapRealm")) {
usersList.addAll(getUserListObj.getUserList((LdapRealm) realm,
searchText));
rolesList.addAll(getUserListObj.getRolesList((LdapRealm)
realm));
} else if
(name.equals("org.apache.zeppelin.realm.ActiveDirectoryGroupRealm")) {
usersList.addAll(getUserListObj.getUserList((ActiveDirectoryGroupRealm) realm,
searchText));
} else if (name.equals("org.apache.shiro.realm.jdbc.JdbcRealm")) {
usersList.addAll(getUserListObj.getUserList((JdbcRealm) realm));
}
Please see feedback from previous PRs related to this JIRA:
https://github.com/apache/zeppelin/pull/1513
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1472
### How should this be tested?
Update shiro.ini to use configuration similar to below:
# Sample LDAP configuration, for user Authentication, currently tested for
single Realm
[main]
ldapADGCRealm = org.apache.zeppelin.realm.LdapRealm
ldapADGCRealm.contextFactory.systemUsername =
CN=hdplookup,OU=hadoop,DC=hdpusr,DC=senia,DC=org
ldapADGCRealm.contextFactory.systemPassword = ldapBindPassword
ldapADGCRealm.searchBase = dc=hdpusr,dc=senia,dc=org
ldapADGCRealm.userSearchBase = dc=hdpusr,dc=senia,dc=org
ldapADGCRealm.groupSearchBase = dc=hdpusr,dc=senia,dc=org
ldapADGCRealm.authorizationEnabled = true
ldapADGCRealm.contextFactory.url = ldap://seniadc1.hdpusr.senia.org:3268
ldapADGCRealm.userSearchAttributeName = sAMAccountName
ldapADGCRealm.contextFactory.authenticationMechanism = simple
ldapADGCRealm.groupObjectClass = group
ldapADGCRealm.memberAttribute = member
ldapADGCRealm.rolesByGroup = hdpeng: admin, \
hadoopusers: user
securityManager.realms = $ldapADGCRealm
sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager
### If caching of user is required then uncomment below lines
#ca
[GitHub] zeppelin pull request #1614: ZEPPELIN-1472 - Create new LdapRealm based on A...
Github user gss2002 closed the pull request at: https://github.com/apache/zeppelin/pull/1614 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1614: ZEPPELIN-1472 - Create new LdapRealm based on A...
GitHub user gss2002 reopened a pull request:
https://github.com/apache/zeppelin/pull/1614
ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm:
Provides LdapRealm Functionality similar to Apache Knox
### What is this PR for?
ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm:
Provides LdapRealm Functionality similar to what Apache Knox provides. This is
critical as in large enterprise environments Active Directory Global Catalogs
are used for lookup with samAccountName and using a DN Template is not an
option as their are multiple OUs. Also searching on "userPrincipalName" is
risky in an AD environment since the explicit UPN vs Implicit UPN can be
different this is definitely the case with environments using Office 365. And
the LDAP userPrincipalName attribute is the explicit UPN which can be defined
by the directory administrator to any value and it can be duplicated.
SamAccountName is unique per domain and Microsoft states best practice is to
not allow duplicate samAccountName's across the forest.
In addition to the above changes I have adjusted and moved the
LdapGroupRealm and ActiveDirectoryGroupRealm into the org.apache.zeppelin.realm
package structure to make all Realm's consistent.
The LdapRealm class also works with role to group mapping for usage within
Zeppelin for notebook authorization.
I have adjusted SecurityUtils to use ClassName vs realmName in determining
what to use as you may have companies that decide to use their own custom
realmname in shiro.ini and may not realize you cannot so using className is
much safer.
Example - SecurityUtils
String name = realm.getClass().getName();
if (name.equals("org.apache.shiro.realm.text.IniRealm")) {
allRoles = ((IniRealm) realm).getIni().get("roles");
break;
} else if (name.equals("org.apache.zeppelin.realm.LdapRealm")) {
allRoles = ((LdapRealm) realm).getListRoles();
break;
}
Example - SecurityRestApi:
String name = realm.getClass().getName();
if (LOG.isDebugEnabled()) {
LOG.debug("RealmClass.getName: " + name);
}
if (name.equals("org.apache.shiro.realm.text.IniRealm")) {
usersList.addAll(getUserListObj.getUserList((IniRealm) realm));
rolesList.addAll(getUserListObj.getRolesList((IniRealm) realm));
} else if
(name.equals("org.apache.zeppelin.realm.LdapGroupRealm")) {
usersList.addAll(getUserListObj.getUserList((JndiLdapRealm)
realm, searchText));
} else if (name.equals("org.apache.zeppelin.realm.LdapRealm")) {
usersList.addAll(getUserListObj.getUserList((LdapRealm) realm,
searchText));
rolesList.addAll(getUserListObj.getRolesList((LdapRealm)
realm));
} else if
(name.equals("org.apache.zeppelin.realm.ActiveDirectoryGroupRealm")) {
usersList.addAll(getUserListObj.getUserList((ActiveDirectoryGroupRealm) realm,
searchText));
} else if (name.equals("org.apache.shiro.realm.jdbc.JdbcRealm")) {
usersList.addAll(getUserListObj.getUserList((JdbcRealm) realm));
}
Please see feedback from previous PRs related to this JIRA:
https://github.com/apache/zeppelin/pull/1513
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1472
### How should this be tested?
Update shiro.ini to use configuration similar to below:
# Sample LDAP configuration, for user Authentication, currently tested for
single Realm
[main]
ldapADGCRealm = org.apache.zeppelin.realm.LdapRealm
ldapADGCRealm.contextFactory.systemUsername =
CN=hdplookup,OU=hadoop,DC=hdpusr,DC=senia,DC=org
ldapADGCRealm.contextFactory.systemPassword = ldapBindPassword
ldapADGCRealm.searchBase = dc=hdpusr,dc=senia,dc=org
ldapADGCRealm.userSearchBase = dc=hdpusr,dc=senia,dc=org
ldapADGCRealm.groupSearchBase = dc=hdpusr,dc=senia,dc=org
ldapADGCRealm.authorizationEnabled = true
ldapADGCRealm.contextFactory.url = ldap://seniadc1.hdpusr.senia.org:3268
ldapADGCRealm.userSearchAttributeName = sAMAccountName
ldapADGCRealm.contextFactory.authenticationMechanism = simple
ldapADGCRealm.groupObjectClass = group
ldapADGCRealm.memberAttribute = member
ldapADGCRealm.rolesByGroup = hdpeng: admin, \
hadoopusers: user
securityManager.realms = $ldapADGCRealm
sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager
### If caching of user is required then uncomment below lines
#ca
[GitHub] zeppelin issue #1614: ZEPPELIN-1472 - Create new LdapRealm based on Apache K...
Github user gss2002 commented on the issue:
https://github.com/apache/zeppelin/pull/1614
Going to close and re-open to re kick the tests. These tests have been
flaky I dont think the error is related to the patch...
15:38:55,459 ERROR org.apache.zeppelin.AbstractZeppelinIT:136 - Exception
in ParagraphActionsIT while testEditOnDoubleClick
org.openqa.selenium.ElementNotVisibleException: Element is not currently
visible and so may not be interacted with
Command duration or timeout: 30.04 seconds
Build info: version: '2.48.2', revision:
'41bccdd10cf2c0560f637404c2d96164b67d9d67', time: '2015-10-09 13:08:06'
System info: host: 'testing-docker-60ee1fc8-0996-4929-93bf-f3f4ab1d7d4e',
ip: '172.17.0.8', os.name: 'Linux', os.arch: 'amd64', os.version:
'4.4.0-47-generic', java.version: '1.7.0_76'
Session ID: e568225a-5433-4a6e-b11a-85faf279113b
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities [{platform=LINUX, acceptSslCerts=true, javascriptEnabled=true,
cssSelectorsEnabled=true, databaseEnabled=true, browserName=firefox,
handlesAlerts=true, nativeEvents=false, webStorageEnabled=true,
rotatable=false, locationContextEnabled=true, applicationCacheEnabled=true,
takesScreenshot=true, version=31.0}]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:206)
at
org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:158)
at
org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:647)
at
org.openqa.selenium.remote.RemoteWebElement.execute(RemoteWebElement.java:326)
at
org.openqa.selenium.remote.RemoteWebElement.sendKeys(RemoteWebElement.java:121)
at
org.apache.zeppelin.integration.ParagraphActionsIT.testEditOnDoubleClick(ParagraphActionsIT.java:443)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1614: ZEPPELIN-1472 - Create new LdapRealm based on A...
Github user gss2002 closed the pull request at: https://github.com/apache/zeppelin/pull/1614 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1614: ZEPPELIN-1472 - Create new LdapRealm based on A...
Github user gss2002 closed the pull request at: https://github.com/apache/zeppelin/pull/1614 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1614: ZEPPELIN-1472 - Create new LdapRealm based on A...
GitHub user gss2002 reopened a pull request:
https://github.com/apache/zeppelin/pull/1614
ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm:
Provides LdapRealm Functionality similar to Apache Knox
### What is this PR for?
ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm:
Provides LdapRealm Functionality similar to what Apache Knox provides. This is
critical as in large enterprise environments Active Directory Global Catalogs
are used for lookup with samAccountName and using a DN Template is not an
option as their are multiple OUs. Also searching on "userPrincipalName" is
risky in an AD environment since the explicit UPN vs Implicit UPN can be
different this is definitely the case with environments using Office 365. And
the LDAP userPrincipalName attribute is the explicit UPN which can be defined
by the directory administrator to any value and it can be duplicated.
SamAccountName is unique per domain and Microsoft states best practice is to
not allow duplicate samAccountName's across the forest.
In addition to the above changes I have adjusted and moved the
LdapGroupRealm and ActiveDirectoryGroupRealm into the org.apache.zeppelin.realm
package structure to make all Realm's consistent.
The LdapRealm class also works with role to group mapping for usage within
Zeppelin for notebook authorization.
I have adjusted SecurityUtils to use ClassName vs realmName in determining
what to use as you may have companies that decide to use their own custom
realmname in shiro.ini and may not realize you cannot so using className is
much safer.
Example - SecurityUtils
String name = realm.getClass().getName();
if (name.equals("org.apache.shiro.realm.text.IniRealm")) {
allRoles = ((IniRealm) realm).getIni().get("roles");
break;
} else if (name.equals("org.apache.zeppelin.realm.LdapRealm")) {
allRoles = ((LdapRealm) realm).getListRoles();
break;
}
Example - SecurityRestApi:
String name = realm.getClass().getName();
if (LOG.isDebugEnabled()) {
LOG.debug("RealmClass.getName: " + name);
}
if (name.equals("org.apache.shiro.realm.text.IniRealm")) {
usersList.addAll(getUserListObj.getUserList((IniRealm) realm));
rolesList.addAll(getUserListObj.getRolesList((IniRealm) realm));
} else if
(name.equals("org.apache.zeppelin.realm.LdapGroupRealm")) {
usersList.addAll(getUserListObj.getUserList((JndiLdapRealm)
realm, searchText));
} else if (name.equals("org.apache.zeppelin.realm.LdapRealm")) {
usersList.addAll(getUserListObj.getUserList((LdapRealm) realm,
searchText));
rolesList.addAll(getUserListObj.getRolesList((LdapRealm)
realm));
} else if
(name.equals("org.apache.zeppelin.realm.ActiveDirectoryGroupRealm")) {
usersList.addAll(getUserListObj.getUserList((ActiveDirectoryGroupRealm) realm,
searchText));
} else if (name.equals("org.apache.shiro.realm.jdbc.JdbcRealm")) {
usersList.addAll(getUserListObj.getUserList((JdbcRealm) realm));
}
Please see feedback from previous PRs related to this JIRA:
https://github.com/apache/zeppelin/pull/1513
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1472
### How should this be tested?
Update shiro.ini to use configuration similar to below:
# Sample LDAP configuration, for user Authentication, currently tested for
single Realm
[main]
ldapADGCRealm = org.apache.zeppelin.realm.LdapRealm
ldapADGCRealm.contextFactory.systemUsername =
CN=hdplookup,OU=hadoop,DC=hdpusr,DC=senia,DC=org
ldapADGCRealm.contextFactory.systemPassword = ldapBindPassword
ldapADGCRealm.searchBase = dc=hdpusr,dc=senia,dc=org
ldapADGCRealm.userSearchBase = dc=hdpusr,dc=senia,dc=org
ldapADGCRealm.groupSearchBase = dc=hdpusr,dc=senia,dc=org
ldapADGCRealm.authorizationEnabled = true
ldapADGCRealm.contextFactory.url = ldap://seniadc1.hdpusr.senia.org:3268
ldapADGCRealm.userSearchAttributeName = sAMAccountName
ldapADGCRealm.contextFactory.authenticationMechanism = simple
ldapADGCRealm.groupObjectClass = group
ldapADGCRealm.memberAttribute = member
ldapADGCRealm.rolesByGroup = hdpeng: admin, \
hadoopusers: user
securityManager.realms = $ldapADGCRealm
sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager
### If caching of user is required then uncomment below lines
#ca
[GitHub] zeppelin issue #1614: ZEPPELIN-1472 - Create new LdapRealm based on Apache K...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1614 No problem Let me know if you see any others or if you need me to adjust this code at all --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1614: ZEPPELIN-1472 - Create new LdapRealm based on A...
GitHub user gss2002 reopened a pull request:
https://github.com/apache/zeppelin/pull/1614
ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm:
Provides LdapRealm Functionality similar to Apache Knox
### What is this PR for?
ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm:
Provides LdapRealm Functionality similar to what Apache Knox provides. This is
critical as in large enterprise environments Active Directory Global Catalogs
are used for lookup with samAccountName and using a DN Template is not an
option as their are multiple OUs. Also searching on "userPrincipalName" is
risky in an AD environment since the explicit UPN vs Implicit UPN can be
different this is definitely the case with environments using Office 365. And
the LDAP userPrincipalName attribute is the explicit UPN which can be defined
by the directory administrator to any value and it can be duplicated.
SamAccountName is unique per domain and Microsoft states best practice is to
not allow duplicate samAccountName's across the forest.
In addition to the above changes I have adjusted and moved the
LdapGroupRealm and ActiveDirectoryGroupRealm into the org.apache.zeppelin.realm
package structure to make all Realm's consistent.
The LdapRealm class also works with role to group mapping for usage within
Zeppelin for notebook authorization.
I have adjusted SecurityUtils to use ClassName vs realmName in determining
what to use as you may have companies that decide to use their own custom
realmname in shiro.ini and may not realize you cannot so using className is
much safer.
Example - SecurityUtils
String name = realm.getClass().getName();
if (name.equals("org.apache.shiro.realm.text.IniRealm")) {
allRoles = ((IniRealm) realm).getIni().get("roles");
break;
} else if (name.equals("org.apache.zeppelin.realm.LdapRealm")) {
allRoles = ((LdapRealm) realm).getListRoles();
break;
}
Example - SecurityRestApi:
String name = realm.getClass().getName();
if (LOG.isDebugEnabled()) {
LOG.debug("RealmClass.getName: " + name);
}
if (name.equals("org.apache.shiro.realm.text.IniRealm")) {
usersList.addAll(getUserListObj.getUserList((IniRealm) realm));
rolesList.addAll(getUserListObj.getRolesList((IniRealm) realm));
} else if
(name.equals("org.apache.zeppelin.realm.LdapGroupRealm")) {
usersList.addAll(getUserListObj.getUserList((JndiLdapRealm)
realm, searchText));
} else if (name.equals("org.apache.zeppelin.realm.LdapRealm")) {
usersList.addAll(getUserListObj.getUserList((LdapRealm) realm,
searchText));
rolesList.addAll(getUserListObj.getRolesList((LdapRealm)
realm));
} else if
(name.equals("org.apache.zeppelin.realm.ActiveDirectoryGroupRealm")) {
usersList.addAll(getUserListObj.getUserList((ActiveDirectoryGroupRealm) realm,
searchText));
} else if (name.equals("org.apache.shiro.realm.jdbc.JdbcRealm")) {
usersList.addAll(getUserListObj.getUserList((JdbcRealm) realm));
}
Please see feedback from previous PRs related to this JIRA:
https://github.com/apache/zeppelin/pull/1513
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1472
### How should this be tested?
Update shiro.ini to use configuration similar to below:
# Sample LDAP configuration, for user Authentication, currently tested for
single Realm
[main]
ldapADGCRealm = org.apache.zeppelin.realm.LdapRealm
ldapADGCRealm.contextFactory.systemUsername =
CN=hdplookup,OU=hadoop,DC=hdpusr,DC=senia,DC=org
ldapADGCRealm.contextFactory.systemPassword = ldapBindPassword
ldapADGCRealm.searchBase = dc=hdpusr,dc=senia,dc=org
ldapADGCRealm.userSearchBase = dc=hdpusr,dc=senia,dc=org
ldapADGCRealm.groupSearchBase = dc=hdpusr,dc=senia,dc=org
ldapADGCRealm.authorizationEnabled = true
ldapADGCRealm.contextFactory.url = ldap://seniadc1.hdpusr.senia.org:3268
ldapADGCRealm.userSearchAttributeName = sAMAccountName
ldapADGCRealm.contextFactory.authenticationMechanism = simple
ldapADGCRealm.groupObjectClass = group
ldapADGCRealm.memberAttribute = member
ldapADGCRealm.rolesByGroup = hdpeng: admin, \
hadoopusers: user
securityManager.realms = $ldapADGCRealm
sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager
### If caching of user is required then uncomment below lines
#ca
[GitHub] zeppelin issue #1614: ZEPPELIN-1472 - Create new LdapRealm based on Apache K...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1614 reopen for clean up --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1513: ZEPPELIN-1472 - Create new LdapRealm based on A...
Github user gss2002 closed the pull request at: https://github.com/apache/zeppelin/pull/1513 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1513: ZEPPELIN-1472 - Create new LdapRealm based on Apache K...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1513 @nazgul33 refactoring some code fix coming.. To utilize groups and users --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1516: ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 Exceptio...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1516 @zjffdu went and confirmed the previous error from the pull 19 days ago was in a set of tests that ran completely fine this time. So I think this patch is good to go --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1516: ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 Exceptio...
Github user gss2002 commented on the issue:
https://github.com/apache/zeppelin/pull/1516
@zjffdu I re-run the tests. This error doesn't look to have anything to do
with the code change here. Let me know your thoughts and how we want to proceed
Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 196.06 sec
<<< FAILURE! - in org.apache.zeppelin.integration.ParagraphActionsIT
testEditOnDoubleClick(org.apache.zeppelin.integration.ParagraphActionsIT)
Time elapsed: 37.614 sec <<< ERROR!
org.openqa.selenium.ElementNotVisibleException: Element is not currently
visible and so may not be interacted with
Command duration or timeout: 30.08 seconds
Build info: version: '2.48.2', revision:
'41bccdd10cf2c0560f637404c2d96164b67d9d67', time: '2015-10-09 13:08:06'
System info: host: 'testing-worker-linux-docker-cccaeb55-3455-linux-3', ip:
'172.17.1.168', os.name: 'Linux', os.arch: 'amd64', os.version:
'3.13.0-40-generic', java.version: '1.7.0_76'
Session ID: 461cd88c-98e9-467e-88bd-e44eab394a92
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities [{platform=LINUX, acceptSslCerts=true, javascriptEnabled=true,
cssSelectorsEnabled=true, databaseEnabled=true, browserName=firefox,
handlesAlerts=true, nativeEvents=false, webStorageEnabled=true,
rotatable=false, locationContextEnabled=true, applicationCacheEnabled=true,
takesScreenshot=true, version=31.0}]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:206)
at
org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:158)
at
org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:647)
at
org.openqa.selenium.remote.RemoteWebElement.execute(RemoteWebElement.java:326)
at
org.openqa.selenium.remote.RemoteWebElement.sendKeys(RemoteWebElement.java:121)
at
org.apache.zeppelin.integration.ParagraphActionsIT.testEditOnDoubleClick(ParagraphActionsIT.java:443)
Caused by: org.openqa.selenium.ElementNotVisibleException: Element is not
currently visible and so may not be interacted with
Build info: version: '2.48.2', revision:
'41bccdd10cf2c0560f637404c2d96164b67d9d67', time: '2015-10-09 13:08:06'
System info: host: 'testing-worker-linux-docker-cccaeb55-3455-linux-3', ip:
'172.17.1.168', os.name: 'Linux', os.arch: 'amd64', os.version:
'3.13.0-40-generic', java.version: '1.7.0_76'
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1516: ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 E...
GitHub user gss2002 reopened a pull request:
https://github.com/apache/zeppelin/pull/1516
ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 Exception not caught Kerberos
Enabled
### What is this PR for?
ZEPPELIN-1546: Zeppelin Livy Interpreter 404 Exception not caught with
Kerberos
Livy Interpreter gets a NestedRuntimeException when running with Kerberized
components.
### What type of PR is it?
[Bug Fix]
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1546
### How should this be tested?
Enable Kerberos for Zeppelin to talk to Livy
"2BXS1CND2": {
"id": "2BXS1CND2",
"name": "livy",
"group": "livy",
"properties": {
"livy.spark.driver.cores": "",
"zeppelin.livy.principal": "zeppelin-clustern...@example.com",
"zeppelin.livy.keytab":
"/etc/security/keytabs/zeppelin.server.kerberos.keytab",
### Screenshots (if appropriate)
### Questions:
- Does the licenses files need update? n
- Is there breaking changes for older versions? n
- Does this needs documentation? n
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gss2002/zeppelin ZEPPELIN-1546
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1516.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1516
commit 54d8fe43adff15097aa8075a7976d6b96cd68b3f
Author: gss2002 <g...@senia.org>
Date: 2016-10-14T03:24:35Z
ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 Exception not caught with
KerberosTemplate being enabled
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1516: ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 E...
Github user gss2002 closed the pull request at: https://github.com/apache/zeppelin/pull/1516 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1516: ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 Exceptio...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1516 Reopen to re-check code and error --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1514: ZEPPELIN-1516. NPE LivySparkSQLInterpreter thrown with...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1514 @prabhjyotsingh I have only seen this issue with livy.sql as it is the only one that shares the HashMap between livy.spark and livy.sql. livy.sparkr and livy.pyspark do not share session info. In the future yes? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Github user gss2002 commented on the issue:
https://github.com/apache/zeppelin/pull/1447
@spektom / @zjffdu @purechoc there is definitely an additional condition.
Not sure if it's because the ConcurrentHashMaps are not being used correctly.
But the exception doesn't get caught completely or correctly at times with the
fix proposed here..
ERROR [2016-10-19 14:19:57,638] ({pool-2-thread-11}
LivyHelper.java[executeHTTP]:378) - Error with 404 StatusCode: "Session '9' not
found."
ERROR [2016-10-19 14:19:57,638] ({pool-2-thread-11}
LivyHelper.java[interpretInput]:229) - error in interpretInput
org.apache.zeppelin.livy.LivyHelper$LivyNoSessionException: Session not
found, Livy server would have restarted, or lost session.
at
org.apache.zeppelin.livy.LivyHelper.executeCommand(LivyHelper.java:312)
at org.apache.zeppelin.livy.LivyHelper.interpret(LivyHelper.java:241)
at
org.apache.zeppelin.livy.LivyHelper.interpretInput(LivyHelper.java:189)
at
org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:106)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:390)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at
org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO [2016-10-19 14:19:57,639] ({pool-2-thread-11}
SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1476901197622
finished by scheduler org.apache.zeppelin.livy.LivySparkInterpreter37814848
INFO [2016-10-19 14:19:57,819] ({pool-2-thread-34}
SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1476901197819
started by scheduler org.apache.zeppelin.livy.LivySparkInterpreter37814848
ERROR [2016-10-19 14:19:57,835] ({pool-2-thread-34}
LivyHelper.java[executeHTTP]:378) - Error with 404 StatusCode: "Session '9' not
found."
ERROR [2016-10-19 14:19:57,835] ({pool-2-thread-34}
LivyHelper.java[interpretInput]:229) - error in interpretInput
org.apache.zeppelin.livy.LivyHelper$LivyNoSessionException: Session not
found, Livy server would have restarted, or lost session.
at
org.apache.zeppelin.livy.LivyHelper.executeCommand(LivyHelper.java:312)
at org.apache.zeppelin.livy.LivyHelper.interpret(LivyHelper.java:241)
at
org.apache.zeppelin.livy.LivyHelper.interpretInput(LivyHelper.java:189)
at
org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:106)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:390)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at
org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO [2016-10-19 14:19:57,836] ({pool-2-thread-34}
SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1476901197819
finished by scheduler org.apache.zeppelin.livy.LivySparkInterpreter37814848
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Github user gss2002 commented on the issue:
https://github.com/apache/zeppelin/pull/1447
@spektom just tested against my build by catching the exception and
rethrowing.. It definitely solves the issue.
} catch (LivyNoSessionException e) {
throw e;
} catch (Exception e) {
LOGGER.error("error in interpretInput", e);
return new InterpreterResult(Code.ERROR, e.getMessage());
}
}
public InterpreterResult interpret(String stringLines,
The code base I'm using is here with you patch and a few of @zjffdu patches
and one of mine for NestedRuntimeException for 404's with KerberosTemplate:
https://github.com/gss2002/zeppelin/blob/GSS_PROD_BUILD/livy/src/main/java/org/apache/zeppelin/livy/LivyHelper.java
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Github user gss2002 commented on the issue:
https://github.com/apache/zeppelin/pull/1447
@spektom I think what happens here is this code fires.. which has nothing
to do with the fix here..
in LivySparkInterpreter:
return livyHelper.interpretInput(line, interpreterContext,
userSessionMap, out,
sessionId2AppIdMap.get(sessionId),
sessionId2WebUIMap.get(sessionId), displayAppInfo);
That gets called before the NoSessionException occurs.. And then in
LivyHelper --> public InterpreterResult interpretInput grabs the exception
and handles it. I guess the question is can we do a rootcause on this and
rethrow?
} catch (Exception e) {
LOGGER.error("error in interpretInput", e);
return new InterpreterResult(Code.ERROR, e.getMessage());
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1513: ZEPPELIN-1472 - Create new LdapRealm based on Apache K...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1513 @khalidhuseynov and @zjffdu can we look at committing this since tests have passed. If not let me know what else is needed. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1516: ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 Exceptio...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1516 @zjffdu and @prabhjyotsingh is this safe to be committed at this point? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1447 @spektom this fix is good. Did some extensive load testing with it this AM and it solves the session expiration issues. Thanks for the contribution --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1458: [ZEPPELIN-1486] Allow configuring whether shown values...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1458 Also input validation is secure java coding best practice regardless.. http://www.oracle.com/technetwork/java/seccodeguide-139067.html#5 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1458: [ZEPPELIN-1486] Allow configuring whether shown values...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1458 Well we will just agree to disagree --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1513: ZEPPELIN-1472 - Create new LdapRealm based on Apache K...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1513 Rerun build as error is not related to this patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1513: ZEPPELIN-1472 - Create new LdapRealm based on A...
GitHub user gss2002 reopened a pull request: https://github.com/apache/zeppelin/pull/1513 ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm ### What is this PR for? ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm: Provides LdapRealm Functionality similar to what Apache Knox provides. This is critical as in large enterprise environments Active Directory Global Catalogs are used for lookup with samAccountName and using a DN Template is not an option as their are multiple OUs. Also searching on "userPrincipalName" is risky in an AD environment since the explicit UPN vs Implicit UPN can be different this is definitely the case with environments using Office 365. And the LDAP userPrincipalName attribute is the explicit UPN which can be defined by the directory administrator to any value and it can be duplicated. SamAccountName is unique per domain and Microsoft states best practice is to not allow duplicate samAccountName's across the forest. ### What type of PR is it? [Improvement] ### Todos None ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1472 ### How should this be tested? Setup shiro.ini to use the following configuration: ldapADGCRealm = org.apache.zeppelin.realm.LdapRealm ldapADGCRealm.contextFactory.systemUsername = CN=hdpbind,OU=Svc,DC=exadc,DC=w2k,DC=example,DC=com ldapADGCRealm.contextFactory.systemPassword = ldapPassword ldapADGCRealm.searchBase = dc=w2k,dc=example,dc=com ldapADGCRealm.userSearchBase = dc=w2k,dc=example,dc=com ldapADGCRealm.groupSearchBase = dc=w2k,dc=example,dc=com ldapADGCRealm.contextFactory.url = ldap://exampledc1.exadc.w2k.example.com:3268 ldapADGCRealm.userSearchAttributeName = sAMAccountName ldapADGCRealm.contextFactory.authenticationMechanism = simple ldapADGCRealm.userObjectClass = user ldapADGCRealm.groupObjectClass = group ldapADGCRealm.memberAttribute = member ### Questions: * Does the licenses files need update? n * Is there breaking changes for older versions? n * Does this needs documentation? n You can merge this pull request into a Git repository by running: $ git pull https://github.com/gss2002/zeppelin ZEPPELIN-1472 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/1513.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1513 commit 34938754ac7e220a03cc1817bf93f2cf2d189ee9 Author: gss2002 <g...@senia.org> Date: 2016-10-11T03:58:51Z ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm Class commit 8991d647b024d04eed7005173b4a8eec07b18c6c Author: gss2002 <g...@senia.org> Date: 2016-10-14T00:48:25Z Merge remote-tracking branch 'upstream/master' into ZEPPELIN-1472 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1513: ZEPPELIN-1472 - Create new LdapRealm based on A...
Github user gss2002 closed the pull request at: https://github.com/apache/zeppelin/pull/1513 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1516: ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 E...
Github user gss2002 closed the pull request at: https://github.com/apache/zeppelin/pull/1516 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1516: ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 E...
GitHub user gss2002 reopened a pull request: https://github.com/apache/zeppelin/pull/1516 ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 Exception not caught Kerberos Enabled ### What is this PR for? ZEPPELIN-1546: Zeppelin Livy Interpreter 404 Exception not caught with Kerberos Livy Interpreter gets a NestedRuntimeException when running with Kerberized components. ### What type of PR is it? [Bug Fix] ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1546 ### How should this be tested? Enable Kerberos ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? n * Is there breaking changes for older versions? n * Does this needs documentation? n KerberosTemplate being enabled You can merge this pull request into a Git repository by running: $ git pull https://github.com/gss2002/zeppelin ZEPPELIN-1546 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/1516.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1516 commit 54d8fe43adff15097aa8075a7976d6b96cd68b3f Author: gss2002 <g...@senia.org> Date: 2016-10-14T03:24:35Z ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 Exception not caught with KerberosTemplate being enabled --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1516: ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 Exceptio...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1516 @zjffdu and @prabhjyotsingh here is the 404 NestedRuntime Fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1516: ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 E...
GitHub user gss2002 opened a pull request: https://github.com/apache/zeppelin/pull/1516 ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 Exception not caught Kerberos Enabled ### What is this PR for? ZEPPELIN-1546: Zeppelin Livy Interpreter 404 Exception not caught with Kerberos Livy Interpreter gets a NestedRuntimeException when running with Kerberized components. ### What type of PR is it? [Bug Fix] ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1546 ### How should this be tested? Enable Kerberos ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? n * Is there breaking changes for older versions? n * Does this needs documentation? n KerberosTemplate being enabled You can merge this pull request into a Git repository by running: $ git pull https://github.com/gss2002/zeppelin ZEPPELIN-1546 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/1516.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1516 commit ed439589c13c2f57c8d376634c4574e1f3697675 Author: gss2002 <g...@senia.org> Date: 2016-10-14T03:24:35Z ZEPPELIN-1546 - Zeppelin Livy Interpreter 404 Exception not caught with KerberosTemplate being enabled --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1503: NPE LivySparkSQLInterpreter thrown with %livy.sql inte...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1503 @zjffdu this may be me misunderstanding the code. but if you have 10 users in Zeppelin all accessing PySpark or SparkR there is no possibility of multiple users stepping on the map? So you are saying only 1 thread to interpret is available? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1503: NPE LivySparkSQLInterpreter thrown with %livy.sql inte...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1503 @zjffdu Not a problem. Also I assume we will want to fix SparkR and PySpark also to use ConcurrentHashMaps since they could have multiuser's hitting it. I noticed some of the other interpreters use ConcurrentHashmaps. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1512: Zeppelin 1516 1546
Github user gss2002 closed the pull request at: https://github.com/apache/zeppelin/pull/1512 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1503: NPE LivySparkSQLInterpreter thrown with %livy.sql inte...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1503 @zjffdu Also those new maps added with ZEPPELIN-1430 those should be changed to ConcurrentHashMap. Basically any map that is static with gets/puts/removes etc should be ConcurrentHashMap if not trouble will ensue - https://dzone.com/articles/java-7-hashmap-vs https://issues.apache.org/jira/browse/ZEPPELIN-1293 - The code in this needs to be adjust to check for null also. If json is null create new session as other errors could occur and cause issues as I saw during debug the last few days. I will adjust the ZEPPELIN-1546 JIRA to just handle the NestedRuntimeException. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1426: ZEPPELIN-1430. Display appId and webui link in LivyInt...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1426 @zjffdu and @prabhjyotsingh the new static Maps can these be converted to ConcurrentHashMaps's to prevent possible future contention with multiple users/threads. https://dzone.com/articles/java-7-hashmap-vs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Github user gss2002 commented on the issue:
https://github.com/apache/zeppelin/pull/1447
@spektom and @zjffdu is it possible to add a null check also. I've been
doing some debugging over the past few days and certain situations can cause
nulls to be returned and in theory if a null is being returned the session is
dead.
**So instead of this:**
if (json.matches("^(\")?Session (\'[0-9]\' )?not found(.?\"?)$")) {
throw new LivyNoSessionException();
**This:**
boolean clearSession = false;
if (json != null) {
if (json.matches("^(\")?Session (\'[0-9]\' )?not found(.?\"?)$")) {
clearSession = true;
}
} else {
clearSession = true;
}
if (clearSession) {
throw new LivyNoSessionException();
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1503: NPE LivySparkSQLInterpreter thrown with %livy.sql inte...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1503 @zjffdu The change you made solved the issue. I see difference now. Your quick workaround is good to go. I am going to fix the LivyHelper to handle the 404 with NestedException if that works under ZEPPELIN-1546. cc @prabhjyotsingh --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1503: NPE LivySparkSQLInterpreter thrown with %livy.sql inte...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1503 @zjffdu testing the changes again with another build. I see a slight difference between your's and mine. Testing again ill advise shortly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1503: NPE LivySparkSQLInterpreter thrown with %livy.sql inte...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1503 @zjffdu I tried that same code multiple times and it did not work to be completely honest.. I have an email out to a JDK Developer/Debugger for one of the large JDK vendors to find out why this isn't specifically working. The reason I moved to the Singleton model for those classes is to prevent some bad stuff from happening as there is no guarantee when you have multiple requests coming in at the same time that you won't end up with multiple maps. Also I opened ZEPPELIN-1546 to try to solve the 404 problem which I was able to solve. If you want go ahead and merge. I will refactor the code after the fact but honestly I would try to move to the singleton model if multiple request and users will be hitting the Interpreter. I do understand trying to keep this code simple but unfortunately this is not a simple function. if you would like to discuss this further offline shoot me an email. I based my implementation off of this and some issues I hit supporting some large scale apps over the years. http://stackoverflow.com/questions/11165852/java-singleton-and-synchronization --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1513: ZEPPELIN-1472 - Create new LdapRealm based on Apache K...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1513 @khalidhuseynov made the requested changes and updated documentation. Please let me know what you think. Also I will be willing to create a jira and move the other Realms. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1513: ZEPPELIN-1472 - Create new LdapRealm based on A...
GitHub user gss2002 opened a pull request: https://github.com/apache/zeppelin/pull/1513 ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm ### What is this PR for? ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm: Provides LdapRealm Functionality similar to what Apache Knox provides. This is critical as in large enterprise environments Active Directory Global Catalogs are used for lookup with samAccountName and using a DN Template is not an option as their are multiple OUs. Also searching on "userPrincipalName" is risky in an AD environment since the explicit UPN vs Implicit UPN can be different this is definitely the case with environments using Office 365. And the LDAP userPrincipalName attribute is the explicit UPN which can be defined by the directory administrator to any value and it can be duplicated. SamAccountName is unique per domain and Microsoft states best practice is to not allow duplicate samAccountName's across the forest. ### What type of PR is it? [Improvement] ### Todos None ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1472 ### How should this be tested? Setup shiro.ini to use the following configuration: ldapADGCRealm = org.apache.zeppelin.realm.LdapRealm ldapADGCRealm.contextFactory.systemUsername = CN=hdpbind,OU=Svc,DC=exadc,DC=w2k,DC=example,DC=com ldapADGCRealm.contextFactory.systemPassword = ldapPassword ldapADGCRealm.searchBase = dc=w2k,dc=example,dc=com ldapADGCRealm.userSearchBase = dc=w2k,dc=example,dc=com ldapADGCRealm.groupSearchBase = dc=w2k,dc=example,dc=com ldapADGCRealm.contextFactory.url = ldap://exampledc1.exadc.w2k.example.com:3268 ldapADGCRealm.userSearchAttributeName = sAMAccountName ldapADGCRealm.contextFactory.authenticationMechanism = simple ldapADGCRealm.userObjectClass = user ldapADGCRealm.groupObjectClass = group ldapADGCRealm.memberAttribute = member ### Questions: * Does the licenses files need update? n * Is there breaking changes for older versions? n * Does this needs documentation? n You can merge this pull request into a Git repository by running: $ git pull https://github.com/gss2002/zeppelin ZEPPELIN-1472 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/1513.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1513 commit 34938754ac7e220a03cc1817bf93f2cf2d189ee9 Author: gss2002 <g...@senia.org> Date: 2016-10-11T03:58:51Z ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm Class commit 8991d647b024d04eed7005173b4a8eec07b18c6c Author: gss2002 <g...@senia.org> Date: 2016-10-14T00:48:25Z Merge remote-tracking branch 'upstream/master' into ZEPPELIN-1472 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1493: ZEPPELIN-1472 - Create new LdapRealm based on Apache K...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1493 @khalidhuseynov I'm going to make the changes and move this under realms. I will also close this pull request and open a new one --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1493: ZEPPELIN-1472 - Create new LdapRealm based on A...
Github user gss2002 closed the pull request at: https://github.com/apache/zeppelin/pull/1493 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1503: NPE LivySparkSQLInterpreter thrown with %livy.sql inte...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1503 Also Static HashMaps are not threadsafe with multiple puts in the case of multiple users. Hence why I built out Singletons to keep track of these objects to guarantee only one instance gets created. this also allowed for the corrupt/lost session fix to be built out. Basically I propose to merge ZEPPELIN-1516 and ZEPPELIN-1546 and to solve the last remaining issues. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1512: Zeppelin 1516 1546
GitHub user gss2002 opened a pull request:
https://github.com/apache/zeppelin/pull/1512
Zeppelin 1516 1546
### What is this PR for?
Livy Interpreter gets a NestedRuntimException when running with Kerberized
components.
ERROR [2016-10-11 22:44:47,769] (
{pool-2-thread-11} LivyHelper.java[interpretInput]:192) - Interpreter
exception
org.springframework.web.client.RestClientException: Error running rest
call; nested exception is
org.springframework.web.client.HttpClientErrorException: 404 Not Found
at
org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:196)
at
org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580)
at
org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498)
at org.apache.zeppelin.livy.LivyHelper.executeHTTP(LivyHelper.java:377)
at org.apache.zeppelin.livy.LivyHelper.executeCommand(LivyHelper.java:301)
at org.apache.zeppelin.livy.LivyHelper.interpret(LivyHelper.java:239)
at org.apache.zeppelin.livy.LivyHelper.interpretInput(LivyHelper.java:190)
at
org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:79)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:390)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
### What type of PR is it?
[Improvement | Feature | Refactoring]
### Todos
Doc and Merge latest changes from LivyHelper into code base
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1546
### How should this be tested?
Kerberize Hadoop Cluster, Kerberize Livy Server and Enable Security around
Zeppelin with multi-user and stop/start running LivyServer
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? n
* Is there breaking changes for older versions? n
* Does this needs documentation? n
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gss2002/zeppelin ZEPPELIN-1516_1546
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1512.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1512
commit 11bcd2d1db02ff2dbb2a8e560e3f09fe37057796
Author: gss2002 <g...@senia.org>
Date: 2016-10-11T04:09:23Z
ZEPPELIN-1516 Static/Singleton for userSessionMaps
commit 93c2b1488f6f3610c28a10537b18e3d394c8f9e2
Author: gss2002 <g...@senia.org>
Date: 2016-10-13T16:28:42Z
ZEPPELIN-1546 Zeppelin Livy Interpreter Session Management Recovery
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1503: NPE LivySparkSQLInterpreter thrown with %livy.sql inte...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1503 @zjffdu I tried to do that initially problem is instantiation of the shared object. I can probably consolidate down to one SessionMap but I think if there is sharing needed between sparkSQL and Spark it's best to do it this way minus other solutions. I'll ping @vinayshukla I was discussing this issue with him at Hadoop World a few weeks ago. Maybe we can setup a call to discuss? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1503: NPE LivySparkSQLInterpreter thrown with %livy.sql inte...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1503 Attempted to use the work around (protected static Map<String, Integer> userSessionMap = new HashMap<>(); ) provided by @zjffdu it did not work. So I went with the more involved re-write and it has solved our problems in our large scale environment and provides the correct sharing between session contexts. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1503: NPE LivySparkSQLInterpreter thrown with %livy.s...
GitHub user gss2002 opened a pull request: https://github.com/apache/zeppelin/pull/1503 NPE LivySparkSQLInterpreter thrown with %livy.sql interpreter function ### What is this PR for? The LivySparkSQLInterpreter class does not correctly process the userSessionMap throwing back an NPE when using %livy.sql or %sql when livy is default binding. This prevents correct sharing between Spark Sessions and SparkSQL Sessions. I have attached a fix that implements seperate static single instance classes that manage userSessionMaps. ### What type of PR is it? Bug Fix/Improvement ### Todos Documentation/Unit Tests ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1516 ### How should this be tested? Run %sql or %livy.sql against a sql enabled dataset such as hive metastore. ### Questions: * Does the licenses files need update? n * Is there breaking changes for older versions? n * Does this needs documentation? m You can merge this pull request into a Git repository by running: $ git pull https://github.com/gss2002/zeppelin ZEPPELIN-1516 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/1503.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1503 commit 11bcd2d1db02ff2dbb2a8e560e3f09fe37057796 Author: gss2002 <g...@senia.org> Date: 2016-10-11T04:09:23Z ZEPPELIN-1516 Static/Singleton for userSessionMaps --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1493: ZEPPELIN-1472 - Create new LdapRealm based on Apache K...
Github user gss2002 commented on the issue: https://github.com/apache/zeppelin/pull/1493 @vinayshukla here is the pull request for the LdapRealm enhancement we discussed last week at HadoopWorld/Strata --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1493: ZEPPELIN-1472 - Create new LdapRealm based on A...
GitHub user gss2002 opened a pull request: https://github.com/apache/zeppelin/pull/1493 ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm ### What is this PR for? Provides LdapRealm Functionality similar to what Apache Knox provides. This is critical as in large enterprise environments Active Directory Global Catalogs are used for lookup with samAccountName and using a DN Template is not an option as their are multiple OUs. Also searching on "userPrincipalName" is risky in an AD environment since the explicit UPN vs Implicit UPN can be different this is definitely the case with environments using Office 365. And the LDAP userPrincipalName attribute is the explicit UPN which can be defined by the directory administrator to any value and it can be duplicated. SamAccountName is unique per domain and Microsoft states best practice is to not allow duplicate samAccountName's across the forest. Information about samAccountName and userPrincipalName with ActiveDirectory http://windowsitpro.com/active-directory/q-does-samaccountname-object-have-be-unique-active-directory-domain-or-entire-fores https://jorgequestforknowledge.wordpress.com/2010/10/12/user-principal-names-in-ad-part-1/ ### What type of PR is it? Improvement ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1472 ### How should this be tested? shiro.ini [main] ldapRealm = org.apache.zeppelin.server.LdapRealm ldapRealm.contextFactory.systemUsername = CN=hdpbind,OU=Svc,DC=exadc,DC=w2k,DC=example,DC=com ldapRealm.contextFactory.systemPassword = ldapPassword ldapRealm.searchBase = dc=w2k,dc=example,dc=com ldapRealm.userSearchBase = dc=w2k,dc=example,dc=com ldapRealm.groupSearchBase = dc=w2k,dc=example,dc=com ldapRealm.contextFactory.url = ldap://exampledc1.exadc.w2k.example.com:3268 ldapRealm.userSearchAttributeName = sAMAccountName ldapRealm.contextFactory.authenticationMechanism = simple ldapRealm.userObjectClass = user ldapRealm.groupObjectClass = group ldapRealm.memberAttribute = member securityManager.realms = $ldapRealm ### Questions: * Does the licenses files need update? n * Is there breaking changes for older versions? n * Does this needs documentation? y You can merge this pull request into a Git repository by running: $ git pull https://github.com/gss2002/zeppelin master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/1493.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1493 commit 4b5963a2019f1fded13e6ce9942033101ef2acf1 Author: Initial Commit <gse...@apache.org> Date: 2016-10-07T00:55:42Z ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm Class In our environment we attempted to use the ActiveDirectoryGroupRealm and the LdapGroupRealm but unfortunately those implementations against Shiro do not support ADLDAP Global Catalog. Also searching on "userPrincipalName" is risky in an AD environment since the explicit UPN vs Implicit UPN can be different. And the LDAP userPrincipalName attribute is the explicit UPN which can be defined by the directory administrator to any value and it can be duplicated.. SamAccountName is unique per domain and Microsoft states best practice is to not allow duplicate samAccountName's per the forest. I have attached a semi-working modified KnoxLdapRealm which works against samAccountName and global catalog for auth. http://windowsitpro.com/active-directory/q-does-samaccountname-object-have-be-unique-active-directory-domain-or-entire-fores https://jorgequestforknowledge.wordpress.com/2010/10/12/user-principal-names-in-ad-part-1/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
