[GitHub] spark pull request: [branch-1.5][SPARK-11821] Propagate Kerberos k...
Github user woj-i closed the pull request at: https://github.com/apache/spark/pull/9837 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159915514 Thank you @vanzin for your help. I commited the nits in the documentation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159499824 @vanzin thanks for your support. I've made a commit with documentation update and clean the code as you asked. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159367661 I see my problem. Defining the environment variable HADOOP_CONF_DIR is not enough and doesn't work (not read config files there). Copy of configs to the classpath works. Thanks @vanzin . You are right about the hive-site. All in all- I don't need to set additional config, just have keytab properties in YARN mode as well as LOCAL mode. I've made a commit. I plan to read data from tables by `sql` method on a `HiveContext` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159351434 I had little time today to see this issue, but even if I provide a core-site.xml in my classpath the authentication method in the UGI config is updated to "simple" a one moment (haven't spotted when so far). I need to check it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159214470 Hi! I checked the code, you are right, by default xmls are loaded. However, in local mode it's cumbersome to attach hadoop config from the cluster. What do you think to set this parameter in local mode only? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159064971 I'll check it tomorrow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159063343 No. I need to load the property, because it's not loaded automatically by `UserGroupInformation.loginUserFromKeytab(args.principal, args.keytab)`, so it's required to load a keytab properly for objects created in another context than mine (e.g. classloader context for ClientWrapper). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159060652 Ok, I can prepare a commit with: 1. Shrinking the scope of this feature to YARN or LOCAL modes 2. I'll try to load the "authentication method" property from Hadoop config file specified under a HADOOP_CONF_DIR environment variable or the classpath (file core-site.xml). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159057200 Let's see the situation when I wrap the context of SparkContext with my UGI and even wrap the HiveContext with my UGI then most of Spark functions work with my provided credentials. However, the ClientWrapper object is created in another (the classloader) context, so the UGI provided in application doesn't work anymore. That's why the property has to be set. Is it clear or should I describe it in details? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159052976 No It's not enough. Please look at the code of the method UserGroupInformation.loginUserFromKeytab(args.principal, args.keytab). There is a check of authentication type. If it's simple (what is default in the classloader context) then parameters are not used. So that I think the recent patch need this fix. I faced the situation I described. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159010452 Hi! Can you test it on all environments? If not, then I can shrink the scope to the YARN and the local mode. In case of HADOOP_SECURITY_AUTHENTICATION there isn't this parameter in SparkConf, neither in HiveContext. What can you suggest? I think if the keytab is defined then authentication method should be Kerberos, shouldn't it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-158503831 This is the one parameter needed to update kerberos credentials based on a keytab. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [branch-1.5][SPARK-11821] Propagate Kerberos k...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9837#issuecomment-158494911 Title changed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on a diff in the pull request:
https://github.com/apache/spark/pull/9837#discussion_r45495036
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -166,7 +168,11 @@ private[hive] class ClientWrapper(
" specified in spark.yarn.keytab does not exist")
} else {
logInfo("Attempting to login to Kerberos" +
- s" using principal: ${principalName} and keytab:
${keytabFileName}")
+s" using principal: ${principalName} and keytab:
${keytabFileName}")
+val hadoopConfiguration = new Configuration()
+
hadoopConfiguration.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
--- End diff --
Yes, I did.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
GitHub user woj-i opened a pull request: https://github.com/apache/spark/pull/9859 [SPARK-11821] Propagate Kerberos keytab for all environments @andrewor14 the same PR as in branch 1.5 @harishreedharan You can merge this pull request into a Git repository by running: $ git pull https://github.com/woj-i/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9859.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9859 commit 0b55df5c433cd1052793e03eea4022b3f3b5f94b Author: woj-i Date: 2015-11-18T18:14:31Z [SPARK-11821] location of Kerberos keytab in a property for all environments, bugfix of renewing credentials in the Classloader context commit b3b0d59bfb92b86434d673280e43ce1ba3c6c080 Author: woj-i Date: 2015-11-20T09:51:05Z [SPARK-11821] bugfix of renewing Kerberos credentials in context of the classloader --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on a diff in the pull request:
https://github.com/apache/spark/pull/9837#discussion_r45450002
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -166,7 +168,11 @@ private[hive] class ClientWrapper(
" specified in spark.yarn.keytab does not exist")
} else {
logInfo("Attempting to login to Kerberos" +
- s" using principal: ${principalName} and keytab:
${keytabFileName}")
+s" using principal: ${principalName} and keytab:
${keytabFileName}")
+val hadoopConfiguration = new Configuration()
+
hadoopConfiguration.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
--- End diff --
Hi!
I think it is required because of another context of execution in this part
of code. Here is a context of the classloader, which cause an empty
environment, with default value of authentication, which is "simple". In case
of "simple" a keytab is not used and credentials cannot be granted.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9837#issuecomment-158020139 I also see, that a change of authentication method from a simple to a kerberos was required while renewing credentials. I've made a commit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
GitHub user woj-i opened a pull request: https://github.com/apache/spark/pull/9837 [SPARK-11821] Propagate Kerberos keytab for all environments I prepared a patch for recent bugfix. The scope of the previous bugfix is too narrow- it works only on YARN. I need it on local mode and I think the other modes also need the information (because reflection works the same for each environment having JVM). You can merge this pull request into a Git repository by running: $ git pull https://github.com/woj-i/spark branch-1.5 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9837.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9837 commit 7803fd30be4c14b23611a61216a4560da5604194 Author: woj-i Date: 2015-11-19T09:03:55Z [SPARK-11821] Propagate Kerberos keytab for all environments (not only YARN) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
