[GitHub] zeppelin issue #1840: [ZEPPELIN-1730, 1587] add spark impersonation through ...
Github user jongyoul commented on the issue: https://github.com/apache/zeppelin/pull/1840 Merged it into master and branch-0.7 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1840: [ZEPPELIN-1730, 1587] add spark impersonation through ...
Github user khalidhuseynov commented on the issue: https://github.com/apache/zeppelin/pull/1840 I just pushed changes to keep compatibility using `ZEPPELIN_IMPERSONATE_SPARK_PROXY_USER` env. variable that will disable usage of `--proxy-user` option. after [SPARK-19143](https://issues.apache.org/jira/browse/SPARK-19143) resolved, maybe can come back to it again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1840: [ZEPPELIN-1730, 1587] add spark impersonation through ...
Github user Tagar commented on the issue: https://github.com/apache/zeppelin/pull/1840 As far as credentials refresh are concerned, please see new comments in [SPARK-19143](https://issues.apache.org/jira/browse/SPARK-19143). Hope this helps. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1840: [ZEPPELIN-1730, 1587] add spark impersonation through ...
Github user khalidhuseynov commented on the issue: https://github.com/apache/zeppelin/pull/1840 Also @Leemoonsoo review on this one would be helpful --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1840: [ZEPPELIN-1730, 1587] add spark impersonation through ...
Github user khalidhuseynov commented on the issue: https://github.com/apache/zeppelin/pull/1840 @zjffdu I agree about bringing security related features together in longer term, possibly `Credentials` menu could be used for that. Also regarding previously discussed running of `--proxy-user` with yarn cluster mode, I believe it's currently not supported in Zeppelin. As far as I know only standalone and yarn-client modes are supported by pure Spark interpreter. @Tagar right, if used in that way, kerberos tickets wouldn't be renewed automatically. However as i said, I think Spark interpreter doesn't support yarn cluster mode, so using `ZEPPELIN_IMPERSONATE_CMD` with `kinit` wouldn't be required in that case. also anyone having yarn cluster mode setup with kerberos is more than welcome to test it :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1840: [ZEPPELIN-1730, 1587] add spark impersonation through ...
Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/1840 ``` user configures export ZEPPELIN_IMPERSONATE_CMD in here with kinit @ -k -t and then it's run before spark-submit ``` One concern is that this requires all the interpreters of one user share the same keytab/principal. e.g. spark interpreter may affect shell interpreter if they use different keytab/principal for the same user. For the long term, we may need to put security related settings in one central place rather than in each interpreter setting. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1840: [ZEPPELIN-1730, 1587] add spark impersonation through ...
Github user Tagar commented on the issue: https://github.com/apache/zeppelin/pull/1840 Thank you @khalidhuseynov . On > user configures export ZEPPELIN_IMPERSONATE_CMD in here with kinit @ -k -t and then it's run before spark-submit The only problem I see with this option is that Kerberos tickets will not be renewed automatically, and will expire at some point. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1840: [ZEPPELIN-1730, 1587] add spark impersonation through ...
Github user khalidhuseynov commented on the issue: https://github.com/apache/zeppelin/pull/1840 @zjffdu i didn't try secured cluster mode yet, but as i checked spark documentation, they indeed don't allow using `--principal` & `--keytab` for spark-submit alongside with `--proxy-user` because of security issue on exposing keytab. Then possible solutions could be: 1. user configures `export ZEPPELIN_IMPERSONATE_CMD` in [here](https://github.com/apache/zeppelin/blob/d1fc86b7b2d2012c0323345166c98cc02886e9f1/conf/zeppelin-env.sh.template#L83) with `kinit @ -k -t ` and then it's run before `spark-submit` 2. don't use `--proxy-user` in cluster mode 3. other suggestions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1840: [ZEPPELIN-1730, 1587] add spark impersonation through ...
Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/1840 @khalidhuseynov Have you try it in secured cluster ? IIRC, `--proxy-user` can not work with `--principal` & `--keytab` together, that means in secured cluster, user have to run `kinit` instead of using `--principal` & `--keytab'. This might not be user expect. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1840: [ZEPPELIN-1730, 1587] add spark impersonation through ...
Github user khalidhuseynov commented on the issue: https://github.com/apache/zeppelin/pull/1840 this is ready for review. @prabhjyotsingh plz help review as original author, also @zjffdu @astroshim @Leemoonsoo as followup from #1566. CI failure in first profile is irrelevant and due to rat problem under [ZEPPELIN-1850](https://issues.apache.org/jira/browse/ZEPPELIN-1850) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
