[
https://issues.apache.org/jira/browse/SPARK-31909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126248#comment-17126248
]
Apache Spark commented on SPARK-31909:
--------------------------------------
User 'jacobwu123' has created a pull request for this issue:
https://github.com/apache/spark/pull/28731
> Spark Beeline is not able to pick up krb5.conf address specified in
> spark_env.sh
> --------------------------------------------------------------------------------
>
> Key: SPARK-31909
> URL: https://issues.apache.org/jira/browse/SPARK-31909
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.4
> Reporter: Chenhao Wu
> Priority: Major
>
> The krb5.conf address is defined in file spark_env.sh using SPARK_SUBMIT_OPTS.
> When using Spark beeline to connect to either Hive Thrift Server Or Spark
> Thrift Server in Kerberos environment, it will not able to pick up those
> variables. And thus it generates errors like:
> {code:java}
> java.lang.IllegalStateException: Unable to get current login user:
> java.io.IOException: failure to loginjava.lang.IllegalStateException: Unable
> to get current login user: java.io.IOException: failure to login at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge.createClientWithConf(HadoopThriftAuthBridge.java:87)
> at
> org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:55)
> at
> org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:445)
> at
> org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:201) at
> org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176) at
> org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) at
> java.sql.DriverManager.getConnection(DriverManager.java:664) at
> java.sql.DriverManager.getConnection(DriverManager.java:208) at
> org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:142)
> at
> org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:207)
> at org.apache.hive.beeline.Commands.connect(Commands.java:1149) at
> org.apache.hive.beeline.Commands.connect(Commands.java:1070) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:970) at
> org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:707) at
> org.apache.hive.beeline.BeeLine.begin(BeeLine.java:757) at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484) at
> org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)Caused by:
> java.io.IOException: failure to login at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:824)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge.createClientWithConf(HadoopThriftAuthBridge.java:85)
> ... 21 moreCaused by: javax.security.auth.login.LoginException:
> java.lang.IllegalArgumentException: Illegal principal name [email protected]:
> org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule:
> No rules applied to [email protected] at
> org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:200)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at
> javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at
> javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at
> javax.security.auth.login.LoginContext.login(LoginContext.java:588) at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:799)
> ... 23 more
> {code}
> In spark-class script, the final exec command only generates this
> {code:java}
> exec /usr/lib/jvm/zulu-8//bin/java -cp
> '/usr/lib/spark/conf/:/usr/lib/spark/jars/*:/etc/hadoop/conf/' -Xmx1g
> org.apache.hive.beeline.BeeLine -u
> 'jdbc:hive2://some_string:12000/;principal=some_principal;auth=kerberos;'
> {code}
> which doesn't have the SPARK_SUBMIT_OPT variable contents.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
