[
https://issues.apache.org/jira/browse/SPARK-31909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chenhao Wu updated SPARK-31909:
-------------------------------
Description:
The krb5.conf address is defined in file spark_env.sh using SPARK_SUBMIT_OPTS.
When using Spark beeline to connect to either Hive Thrift Server Or Spark
Thrift Server in Kerberos environment, it will not able to pick up those
variables. And thus it generates errors like:
{code:java}
java.lang.IllegalStateException: Unable to get current login user:
java.io.IOException: failure to loginjava.lang.IllegalStateException: Unable to
get current login user: java.io.IOException: failure to login at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge.createClientWithConf(HadoopThriftAuthBridge.java:87)
at
org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:55)
at
org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:445)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:201)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176) at
org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) at
java.sql.DriverManager.getConnection(DriverManager.java:664) at
java.sql.DriverManager.getConnection(DriverManager.java:208) at
org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:142)
at
org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:207)
at org.apache.hive.beeline.Commands.connect(Commands.java:1149) at
org.apache.hive.beeline.Commands.connect(Commands.java:1070) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:970) at
org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:707) at
org.apache.hive.beeline.BeeLine.begin(BeeLine.java:757) at
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484) at
org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)Caused by:
java.io.IOException: failure to login at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:824)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge.createClientWithConf(HadoopThriftAuthBridge.java:85)
... 21 moreCaused by: javax.security.auth.login.LoginException:
java.lang.IllegalArgumentException: Illegal principal name n...@example.com:
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No
rules applied to n...@example.com at
org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:200)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at
javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at
javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at
javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at
javax.security.auth.login.LoginContext.login(LoginContext.java:588) at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:799)
... 23 more
{code}
In spark-class script, the final exec command only generates this
{code:java}
exec /usr/lib/jvm/zulu-8//bin/java -cp
'/usr/lib/spark/conf/:/usr/lib/spark/jars/*:/etc/hadoop/conf/' -Xmx1g
org.apache.hive.beeline.BeeLine -u
'jdbc:hive2://some_string:12000/;principal=some_principal;auth=kerberos;'
{code}
which doesn't have the SPARK_SUBMIT_OPT variable contents.
was:
The krb5.conf address is defined in file spark_env.sh using SPARK_SUBMIT_OPTS.
When using Spark beeline to connect to either Hive Thrift Server Or Spark
Thrift Server in Kerberos environment, it will not able to pick up those
variables. And thus it generates errors like:
{code:java}
java.lang.IllegalStateException: Unable to get current login user:
java.io.IOException: failure to loginjava.lang.IllegalStateException: Unable to
get current login user: java.io.IOException: failure to login at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge.createClientWithConf(HadoopThriftAuthBridge.java:87)
at
org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:55)
at
org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:445)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:201)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176) at
org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) at
java.sql.DriverManager.getConnection(DriverManager.java:664) at
java.sql.DriverManager.getConnection(DriverManager.java:208) at
org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:142)
at
org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:207)
at org.apache.hive.beeline.Commands.connect(Commands.java:1149) at
org.apache.hive.beeline.Commands.connect(Commands.java:1070) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:970) at
org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:707) at
org.apache.hive.beeline.BeeLine.begin(BeeLine.java:757) at
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484) at
org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)Caused by:
java.io.IOException: failure to login at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:824)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge.createClientWithConf(HadoopThriftAuthBridge.java:85)
... 21 moreCaused by: javax.security.auth.login.LoginException:
java.lang.IllegalArgumentException: Illegal principal name n...@example.com:
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No
rules applied to n...@example.com at
org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:200)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at
javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at
javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at
javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at
javax.security.auth.login.LoginContext.login(LoginContext.java:588) at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:799)
... 23 more
{code}
In spark-class script, the final exec command only generates this
{code:java}
exec /usr/lib/jvm/zulu-8//bin/java -cp
'/usr/lib/spark/conf/:/usr/lib/spark/jars/*:/etc/hadoop/conf/' -Xmx1g
org.apache.hive.beeline.BeeLine -u
'jdbc:hive2://some_string:12000/;principal=some_principal;auth=kerberos;'
{code}
> Spark Beeline is not able to pick up krb5.conf address specified in
> spark_env.sh
> --------------------------------------------------------------------------------
>
> Key: SPARK-31909
> URL: https://issues.apache.org/jira/browse/SPARK-31909
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.4
> Reporter: Chenhao Wu
> Priority: Major
>
> The krb5.conf address is defined in file spark_env.sh using SPARK_SUBMIT_OPTS.
> When using Spark beeline to connect to either Hive Thrift Server Or Spark
> Thrift Server in Kerberos environment, it will not able to pick up those
> variables. And thus it generates errors like:
> {code:java}
> java.lang.IllegalStateException: Unable to get current login user:
> java.io.IOException: failure to loginjava.lang.IllegalStateException: Unable
> to get current login user: java.io.IOException: failure to login at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge.createClientWithConf(HadoopThriftAuthBridge.java:87)
> at
> org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:55)
> at
> org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:445)
> at
> org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:201) at
> org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176) at
> org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) at
> java.sql.DriverManager.getConnection(DriverManager.java:664) at
> java.sql.DriverManager.getConnection(DriverManager.java:208) at
> org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:142)
> at
> org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:207)
> at org.apache.hive.beeline.Commands.connect(Commands.java:1149) at
> org.apache.hive.beeline.Commands.connect(Commands.java:1070) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:970) at
> org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:707) at
> org.apache.hive.beeline.BeeLine.begin(BeeLine.java:757) at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484) at
> org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)Caused by:
> java.io.IOException: failure to login at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:824)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge.createClientWithConf(HadoopThriftAuthBridge.java:85)
> ... 21 moreCaused by: javax.security.auth.login.LoginException:
> java.lang.IllegalArgumentException: Illegal principal name n...@example.com:
> org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule:
> No rules applied to n...@example.com at
> org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:200)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at
> javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at
> javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at
> javax.security.auth.login.LoginContext.login(LoginContext.java:588) at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:799)
> ... 23 more
> {code}
> In spark-class script, the final exec command only generates this
> {code:java}
> exec /usr/lib/jvm/zulu-8//bin/java -cp
> '/usr/lib/spark/conf/:/usr/lib/spark/jars/*:/etc/hadoop/conf/' -Xmx1g
> org.apache.hive.beeline.BeeLine -u
> 'jdbc:hive2://some_string:12000/;principal=some_principal;auth=kerberos;'
> {code}
> which doesn't have the SPARK_SUBMIT_OPT variable contents.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
