[jira] [Commented] (HIVE-10594) Remote Spark client doesn't use Kerberos keytab to authenticate [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14650577#comment-14650577
]
Xuefu Zhang commented on HIVE-10594:
Merged to master and cherry-picked to branch-1.
Remote Spark client doesn't use Kerberos keytab to authenticate [Spark Branch]
--
Key: HIVE-10594
URL: https://issues.apache.org/jira/browse/HIVE-10594
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: 1.1.0
Reporter: Chao Sun
Assignee: Xuefu Zhang
Fix For: spark-branch, 1.3.0, 2.0.0
Attachments: HIVE-10594.1-spark.patch
Reporting problem found by one of the HoS users:
Currently, if user is running Beeline on a different host than HS2, and
he/she didn't do kinit on the HS2 host, then he/she may get the following
error:
{code}
2015-04-29 15:49:34,614 INFO org.apache.hive.spark.client.SparkClientImpl:
15/04/29 15:49:34 WARN UserGroupInformation: PriviledgedActionException
as:hive (auth:KERBEROS) cause:java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)]
2015-04-29 15:49:34,652 INFO org.apache.hive.spark.client.SparkClientImpl:
Exception in thread main java.io.IOException: Failed on local exception:
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]; Host Details : local host is:
secure-hos-1.ent.cloudera.com/10.20.77.79; destination host is:
secure-hos-1.ent.cloudera.com:8032;
2015-04-29 15:49:34,653 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
2015-04-29 15:49:34,653 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
2015-04-29 15:49:34,654 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
2015-04-29 15:49:34,654 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
2015-04-29 15:49:34,654 INFO org.apache.hive.spark.client.SparkClientImpl:
at com.sun.proxy.$Proxy11.getClusterMetrics(Unknown Source)
2015-04-29 15:49:34,655 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:202)
2015-04-29 15:49:34,655 INFO org.apache.hive.spark.client.SparkClientImpl:
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2015-04-29 15:49:34,655 INFO org.apache.hive.spark.client.SparkClientImpl:
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
2015-04-29 15:49:34,656 INFO org.apache.hive.spark.client.SparkClientImpl:
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2015-04-29 15:49:34,656 INFO org.apache.hive.spark.client.SparkClientImpl:
at java.lang.reflect.Method.invoke(Method.java:606)
2015-04-29 15:49:34,656 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at com.sun.proxy.$Proxy12.getClusterMetrics(Unknown Source)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:461)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:91)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:91)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.spark.Logging$class.logInfo(Logging.scala:59)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.spark.deploy.yarn.Client.logInfo(Client.scala:49)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:90)
[jira] [Commented] (HIVE-10594) Remote Spark client doesn't use Kerberos keytab to authenticate [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596295#comment-14596295
]
Chao Sun commented on HIVE-10594:
-
+1
Remote Spark client doesn't use Kerberos keytab to authenticate [Spark Branch]
--
Key: HIVE-10594
URL: https://issues.apache.org/jira/browse/HIVE-10594
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: 1.1.0
Reporter: Chao Sun
Assignee: Xuefu Zhang
Attachments: HIVE-10594.1-spark.patch
Reporting problem found by one of the HoS users:
Currently, if user is running Beeline on a different host than HS2, and
he/she didn't do kinit on the HS2 host, then he/she may get the following
error:
{code}
2015-04-29 15:49:34,614 INFO org.apache.hive.spark.client.SparkClientImpl:
15/04/29 15:49:34 WARN UserGroupInformation: PriviledgedActionException
as:hive (auth:KERBEROS) cause:java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)]
2015-04-29 15:49:34,652 INFO org.apache.hive.spark.client.SparkClientImpl:
Exception in thread main java.io.IOException: Failed on local exception:
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]; Host Details : local host is:
secure-hos-1.ent.cloudera.com/10.20.77.79; destination host is:
secure-hos-1.ent.cloudera.com:8032;
2015-04-29 15:49:34,653 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
2015-04-29 15:49:34,653 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
2015-04-29 15:49:34,654 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
2015-04-29 15:49:34,654 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
2015-04-29 15:49:34,654 INFO org.apache.hive.spark.client.SparkClientImpl:
at com.sun.proxy.$Proxy11.getClusterMetrics(Unknown Source)
2015-04-29 15:49:34,655 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:202)
2015-04-29 15:49:34,655 INFO org.apache.hive.spark.client.SparkClientImpl:
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2015-04-29 15:49:34,655 INFO org.apache.hive.spark.client.SparkClientImpl:
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
2015-04-29 15:49:34,656 INFO org.apache.hive.spark.client.SparkClientImpl:
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2015-04-29 15:49:34,656 INFO org.apache.hive.spark.client.SparkClientImpl:
at java.lang.reflect.Method.invoke(Method.java:606)
2015-04-29 15:49:34,656 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at com.sun.proxy.$Proxy12.getClusterMetrics(Unknown Source)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:461)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:91)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:91)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.spark.Logging$class.logInfo(Logging.scala:59)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.spark.deploy.yarn.Client.logInfo(Client.scala:49)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:90)
2015-04-29 15:49:34,658 INFO org.apache.hive.spark.client.SparkClientImpl:
at
[jira] [Commented] (HIVE-10594) Remote Spark client doesn't use Kerberos keytab to authenticate [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593772#comment-14593772
]
Hive QA commented on HIVE-10594:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12740689/HIVE-10594.1-spark.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7987 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.initializationError
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_index_bitmap_auto
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/897/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/897/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-897/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12740689 - PreCommit-HIVE-SPARK-Build
Remote Spark client doesn't use Kerberos keytab to authenticate [Spark Branch]
--
Key: HIVE-10594
URL: https://issues.apache.org/jira/browse/HIVE-10594
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: 1.1.0
Reporter: Chao Sun
Assignee: Xuefu Zhang
Attachments: HIVE-10594.1-spark.patch
Reporting problem found by one of the HoS users:
Currently, if user is running Beeline on a different host than HS2, and
he/she didn't do kinit on the HS2 host, then he/she may get the following
error:
{code}
2015-04-29 15:49:34,614 INFO org.apache.hive.spark.client.SparkClientImpl:
15/04/29 15:49:34 WARN UserGroupInformation: PriviledgedActionException
as:hive (auth:KERBEROS) cause:java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)]
2015-04-29 15:49:34,652 INFO org.apache.hive.spark.client.SparkClientImpl:
Exception in thread main java.io.IOException: Failed on local exception:
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]; Host Details : local host is:
secure-hos-1.ent.cloudera.com/10.20.77.79; destination host is:
secure-hos-1.ent.cloudera.com:8032;
2015-04-29 15:49:34,653 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
2015-04-29 15:49:34,653 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
2015-04-29 15:49:34,654 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
2015-04-29 15:49:34,654 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
2015-04-29 15:49:34,654 INFO org.apache.hive.spark.client.SparkClientImpl:
at com.sun.proxy.$Proxy11.getClusterMetrics(Unknown Source)
2015-04-29 15:49:34,655 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:202)
2015-04-29 15:49:34,655 INFO org.apache.hive.spark.client.SparkClientImpl:
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2015-04-29 15:49:34,655 INFO org.apache.hive.spark.client.SparkClientImpl:
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
2015-04-29 15:49:34,656 INFO org.apache.hive.spark.client.SparkClientImpl:
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2015-04-29 15:49:34,656 INFO org.apache.hive.spark.client.SparkClientImpl:
at java.lang.reflect.Method.invoke(Method.java:606)
2015-04-29 15:49:34,656 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at
[jira] [Commented] (HIVE-10594) Remote Spark client doesn't use Kerberos keytab to authenticate [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14528640#comment-14528640
]
Bruce Nelson commented on HIVE-10594:
-
I have confirmed the issue with a few more specifics :
1. Confirmed using CDH 5.4.0 with Kerberos, OpenLDAP/SSSD and Sentry (no
impersonation)
2. Problem is seem even if beeline is run on the HS2 server,
3. Unless the hive/hs2 host princ@DOMAIN runs kinit, setting
hive-execution.engine=spark will result in a failed SQL execution. Once the
hive principal runs kinit, then the hive on spark query succeeds.
4. The problem is specific to HS2 - it must be able to find the TGT cache for
the hive principal in the default or KRB5CCNAME location or hive on spark will
fail.
Remote Spark client doesn't use Kerberos keytab to authenticate [Spark Branch]
--
Key: HIVE-10594
URL: https://issues.apache.org/jira/browse/HIVE-10594
Project: Hive
Issue Type: Bug
Components: Spark
Affects Versions: 1.1.0
Reporter: Chao Sun
Reporting problem found by one of the HoS users:
Currently, if user is running Beeline on a different host than HS2, and
he/she didn't do kinit on the HS2 host, then he/she may get the following
error:
{code}
2015-04-29 15:49:34,614 INFO org.apache.hive.spark.client.SparkClientImpl:
15/04/29 15:49:34 WARN UserGroupInformation: PriviledgedActionException
as:hive (auth:KERBEROS) cause:java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)]
2015-04-29 15:49:34,652 INFO org.apache.hive.spark.client.SparkClientImpl:
Exception in thread main java.io.IOException: Failed on local exception:
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]; Host Details : local host is:
secure-hos-1.ent.cloudera.com/10.20.77.79; destination host is:
secure-hos-1.ent.cloudera.com:8032;
2015-04-29 15:49:34,653 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
2015-04-29 15:49:34,653 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
2015-04-29 15:49:34,654 INFO org.apache.hive.spark.client.SparkClientImpl:
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
2015-04-29 15:49:34,654 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
2015-04-29 15:49:34,654 INFO org.apache.hive.spark.client.SparkClientImpl:
at com.sun.proxy.$Proxy11.getClusterMetrics(Unknown Source)
2015-04-29 15:49:34,655 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:202)
2015-04-29 15:49:34,655 INFO org.apache.hive.spark.client.SparkClientImpl:
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2015-04-29 15:49:34,655 INFO org.apache.hive.spark.client.SparkClientImpl:
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
2015-04-29 15:49:34,656 INFO org.apache.hive.spark.client.SparkClientImpl:
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2015-04-29 15:49:34,656 INFO org.apache.hive.spark.client.SparkClientImpl:
at java.lang.reflect.Method.invoke(Method.java:606)
2015-04-29 15:49:34,656 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at com.sun.proxy.$Proxy12.getClusterMetrics(Unknown Source)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:461)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:91)
2015-04-29 15:49:34,657 INFO org.apache.hive.spark.client.SparkClientImpl:
at
org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:91)
2015-04-29 15:49:34,657 INFO
