Chenhao Wu created SPARK-31909: ---------------------------------- Summary: Spark Beeline is not able to pick up krb5.conf address specified in spark_env.sh Key: SPARK-31909 URL: https://issues.apache.org/jira/browse/SPARK-31909 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.4.4 Reporter: Chenhao Wu
The krb5.conf address is defined in file spark_env.sh using SPARK_SUBMIT_OPTS. When using Spark beeline to connect to either Hive Thrift Server Or Spark Thrift Server in Kerberos environment, it will not able to pick up those variables. And thus it generates errors like: {code:java} java.lang.IllegalStateException: Unable to get current login user: java.io.IOException: failure to loginjava.lang.IllegalStateException: Unable to get current login user: java.io.IOException: failure to login at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge.createClientWithConf(HadoopThriftAuthBridge.java:87) at org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:55) at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:445) at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:201) at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176) at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:208) at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:142) at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:207) at org.apache.hive.beeline.Commands.connect(Commands.java:1149) at org.apache.hive.beeline.Commands.connect(Commands.java:1070) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:970) at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:707) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:757) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484) at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)Caused by: java.io.IOException: failure to login at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:824) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge.createClientWithConf(HadoopThriftAuthBridge.java:85) ... 21 moreCaused by: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name n...@example.com: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to n...@example.com at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:200) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at javax.security.auth.login.LoginContext.login(LoginContext.java:588) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:799) ... 23 more {code} In spark-class script, the final exec command only generates this {code:java} exec /usr/lib/jvm/zulu-8//bin/java -cp '/usr/lib/spark/conf/:/usr/lib/spark/jars/*:/etc/hadoop/conf/' -Xmx1g org.apache.hive.beeline.BeeLine -u 'jdbc:hive2://some_string:12000/;principal=some_principal;auth=kerberos;' {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org