[ https://issues.apache.org/jira/browse/SPARK-8672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Lian updated SPARK-8672: ------------------------------ Shepherd: Cheng Lian Environment: Hive 0.13.1 > throws NPE when running spark sql thrift server with session state > authenticator > -------------------------------------------------------------------------------- > > Key: SPARK-8672 > URL: https://issues.apache.org/jira/browse/SPARK-8672 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.4.0 > Environment: Hive 0.13.1 > Reporter: Min Zhou > > Here is the configuration > {noformat} > <property> > <name>hive.security.authorization.enabled</name> > <value>false</value> > </property> > <property> > <name>hive.security.authorization.manager</name> > > <value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value> > </property> > <property> > <name>hive.security.metastore.authorization.manager</name> > > <value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value> > </property> > <property> > <name>hive.security.authenticator.manager</name> > > <value>org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator</value> > </property> > <property> > <name>hive.server2.enable.doAs</name> > <value>true</value> > </property> > {noformat} > I started spark thrift server with -hiveconf > hive.security.authorization.enabled=true > when logining to beeline, connect with jdbc and create a table after that, a > NPE throws. The stack is like below > {noformat} > FAILED: NullPointerException null > org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:333) > org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:310) > org.apache.spark.sql.hive.client.ClientWrapper.withHiveState(ClientWrapper.scala:139) > org.apache.spark.sql.hive.client.ClientWrapper.runHive(ClientWrapper.scala:310) > org.apache.spark.sql.hive.client.ClientWrapper.runSqlHive(ClientWrapper.scala:300) > org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:472) > org.apache.spark.sql.hive.execution.HiveNativeCommand.run(HiveNativeCommand.scala:33) > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57) > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57) > org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:68) > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88) > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88) > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:148) > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:87) > org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:939) > org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:939) > org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:144) > org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:128) > org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51) > org.apache.spark.sql.SQLContext.sql(SQLContext.scala:744) > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.run(Shim13.scala:178) > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:231) > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:218) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:606) > org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79) > org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37) > org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64) > java.security.AccessController.doPrivileged(Native Method) > javax.security.auth.Subject.doAs(Subject.java:415) > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:493) > org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60) > com.sun.proxy.$Proxy39.executeStatementAsync(Unknown Source) > org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:233) > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:344) > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55) > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > {noformat} > After some deep tracing, i notice the root cause is a null username inside a > SessionState object, which is passed to a SessionStateUserAuthenticator > instance. The stack above shows it doesn't get the right SessionState. > Instead get a SessionState from thread local. > On the other hand, every time a user login. a new SessionState with a correct > username will be created > {noformat} > org/apache/hadoop/hive/ql/session/SessionState.<init>(Lorg/apache/hadoop/hive/conf/HiveConf;Ljava/lang/String;)V > call by thread [pool-20-thread-3] > > org.apache.hive.service.cli.session.HiveSessionImpl.<init>(HiveSessionImpl.java:104) > > org.apache.hive.service.cli.session.HiveSessionImplwithUGI.<init>(HiveSessionImplwithUGI.java:49) > > org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:122) > > org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(Shim13.scala:241) > > org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:161) > > org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:265) > > org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:191) > > org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253) > > org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238) > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > > org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55) > > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > {noformat} > However, the CREATE TABLE statement never retrieve this sessionstate, running > queries upon this right instance. > So , NPE happens -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org