[
https://issues.apache.org/jira/browse/HIVE-20016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526257#comment-16526257
]
Yongzhi Chen edited comment on HIVE-20016 at 6/28/18 2:37 PM:
--------------------------------------------------------------
The test failed for the second time. Revert the test from master for now.
The thread (2128)who is working on the failed stack worked on Session
e14a5024-5241-4ae6-84f5-b5e1cbb6fa15. Around that time,
HiveServer2-Background-Pool: Thread-2154 works on that session two (because of
Async compilation):
{noformat}
2018-06-27T22:24:46,717 INFO [HiveServer2-Background-Pool: Thread-2154]
sqlstd.SQLStdHiveAccessController: Created SQLStdHiveAccessController for
session context : HiveAuthzSessionContext
[sessionString=e14a5024-5241-4ae6-84f5-b5e1cbb6fa15, clientType=HIVESERVER2]
....
2018-06-27T22:24:46,719 DEBUG [HiveServer2-Background-Pool: Thread-2154]
session.SessionState: Session is using authorization class class
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerImpl
{noformat}
GetSchemasOperation.runInternal can not blocked by Compile lock for the
session, so it can be called when HiveServer2-Background-Pool: Thread-2154 is
compiling. And the getAuthorizationMode() and setupAuth() are never
synchronized, so there is race condition.
{noformat}
2018-06-27T22:24:46,719 INFO [e14a5024-5241-4ae6-84f5-b5e1cbb6fa15
HiveServer2-2018-06-27T22:24:46,717 INFO [e14a5024-5241-4ae6-84f5-b5e1cbb6fa15
HiveServer2-Handler-Pool: Thread-2128] operation.GetSchemasOperation: Fetching
schema metadata
.....
2018-06-27T22:24:46,720 WARN [HiveServer2-Handler-Pool: Thread-2128]
thrift.ThriftCLIService: Error getting schemas:
java.lang.RuntimeException: java.lang.AssertionError: Authorization plugins not
initialized!
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_102]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_102]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
~[hadoop-common-3.1.0.jar:?]
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at com.sun.proxy.$Proxy41.getSchemas(Unknown Source) ~[?:?]
at
org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:349)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:610)
[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1617)
[hive-service-rpc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1602)
[hive-service-rpc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
[libthrift-0.9.3.jar:0.9.3]
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
[libthrift-0.9.3.jar:0.9.3]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
Caused by: java.lang.AssertionError: Authorization plugins not initialized!
at
org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:1846)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:1850)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.operation.MetadataOperation.isAuthV2Enabled(MetadataOperation.java:126)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.operation.GetSchemasOperation.runInternal(GetSchemasOperation.java:68)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:251)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:625)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_102]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_102]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_102]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
... 18 more
{noformat}
was (Author: ychena):
The test failed for the second time. Revert the test from master for now.
The thread (2128)who is working on the failed stack worked on Session
e14a5024-5241-4ae6-84f5-b5e1cbb6fa15. Around that time,
HiveServer2-Background-Pool: Thread-2154 works on that session two (because of
Async compilation):
{noformat}
2018-06-27T22:24:46,717 INFO [HiveServer2-Background-Pool: Thread-2154]
sqlstd.SQLStdHiveAccessController: Created SQLStdHiveAccessController for
session context : HiveAuthzSessionContext
[sessionString=e14a5024-5241-4ae6-84f5-b5e1cbb6fa15, clientType=HIVESERVER2]
....
2018-06-27T22:24:46,719 DEBUG [HiveServer2-Background-Pool: Thread-2154]
session.SessionState: Session is using authorization class class
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerImpl
{noformat}
GetSchemasOperation.runInternal can not blocked by Compile lock for the
session, so it can be called when HiveServer2-Background-Pool: Thread-2154 is
compiling. And the getAuthorizationMode() and setupAuth() are never
synchronized, so there is race condition.
{noformat}
2018-06-27T22:24:46,720 WARN [HiveServer2-Handler-Pool: Thread-2128]
thrift.ThriftCLIService: Error getting schemas:
java.lang.RuntimeException: java.lang.AssertionError: Authorization plugins not
initialized!
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_102]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_102]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
~[hadoop-common-3.1.0.jar:?]
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at com.sun.proxy.$Proxy41.getSchemas(Unknown Source) ~[?:?]
at
org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:349)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:610)
[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1617)
[hive-service-rpc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1602)
[hive-service-rpc-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
[libthrift-0.9.3.jar:0.9.3]
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
[libthrift-0.9.3.jar:0.9.3]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
Caused by: java.lang.AssertionError: Authorization plugins not initialized!
at
org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:1846)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:1850)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.operation.MetadataOperation.isAuthV2Enabled(MetadataOperation.java:126)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.operation.GetSchemasOperation.runInternal(GetSchemasOperation.java:68)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:251)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:625)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_102]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_102]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_102]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
... 18 more
{noformat}
> Investigate random test failure
> --------------------------------
>
> Key: HIVE-20016
> URL: https://issues.apache.org/jira/browse/HIVE-20016
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 4.0.0
> Reporter: Yongzhi Chen
> Assignee: Yongzhi Chen
> Priority: Major
>
> org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3 failed with:
> java.lang.AssertionError: Concurrent Statement failed:
> org.apache.hive.service.cli.HiveSQLException: java.lang.AssertionError:
> Authorization plugins not initialized!
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hive.jdbc.TestJdbcWithMiniHS2.finishTasks(TestJdbcWithMiniHS2.java:374)
> at
> org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3(TestJdbcWithMiniHS2.java:304)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
