bharath v created IMPALA-8008:
---------------------------------
Summary: InvocationTargetException when making HMS RPCs
Key: IMPALA-8008
URL: https://issues.apache.org/jira/browse/IMPALA-8008
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 3.2.0
Reporter: bharath v
Assignee: Fredy Wijaya
A bunch of tests failed with the following HMS exception in logs. Looks like a
flaky HMS issue but raising this jira to rule out any obvious bugs.
*Failed tests*
{noformat}
authorization.test_grant_revoke.TestGrantRevoke.test_same_name_for_role_and_user_invalidate_metadata
authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_with_grant[protocol:
beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
0} | table_format: text/none]
authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_disabled[protocol:
beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
0} | table_format: text/none]
authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_without_grant[protocol:
beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
0} | table_format: text/none]
authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_different_cases[protocol:
beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
0} | table_format: text/none]
authorization.test_show_grant.TestShowGrant.test_show_grant_user[protocol:
beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
0} | table_format: text/none]
authorization.test_show_grant.TestShowGrant.test_show_grant_in_hs2[protocol:
beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
0} | table_format: text/none]
query_test.test_insert.TestInsertQueries.test_insert_large_string[protocol:
beeswax | exec_option: {'sync_ddl': 0, 'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
0} | table_format: text/none]
{noformat}
*Full stack trace*
{noformat}
2018-12-19T14:29:05,730 WARN [pool-8-thread-3] metastore.ObjectStore: Failed
to get database test_owner_privileges_without_grant_fd071aba, returning
NoSuchObjectException
2018-12-19T14:29:05,779 INFO [pool-8-thread-3] common.FileUtils: Creating
directory if it doesn't exist:
s3a://impala-test-uswest2-2/test-warehouse/test_owner_privileges_without_grant_fd071aba.db
2018-12-19T14:29:05,950 ERROR [pool-8-thread-3]
transport.RetryClientInvocationHandler: failed to execute notifyHmsEvent
java.lang.reflect.InvocationTargetException: null
at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
at
org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:95)
[sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
at
org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41)
[sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
at com.sun.proxy.$Proxy38.notifyHmsEvent(Unknown Source) [?:?]
at
org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.notifyHmsEvent(SentrySyncHMSNotificationsPostEventListener.java:252)
[sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
at
org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.onCreateDatabase(SentrySyncHMSNotificationsPostEventListener.java:195)
[sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$15.notify(MetaStoreListenerNotifier.java:64)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:243)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:896)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:928)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at com.sun.proxy.$Proxy26.create_database(Unknown Source) [?:?]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10589)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10573)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_144]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
[hadoop-common-3.0.0-cdh6.x-20181219.140835-1.jar:?]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_144]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: org.apache.sentry.core.common.exception.SentryUserException: Thrift
exception occurred
at
org.apache.sentry.api.service.thrift.SentryPolicyServiceClientDefaultImpl.notifyHmsEvent(SentryPolicyServiceClientDefaultImpl.java:1188)
~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
... 33 more
Caused by: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.sentry.api.service.thrift.SentryPolicyService$Client.recv_sentry_notify_hms_event(SentryPolicyService.java:663)
~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
at
org.apache.sentry.api.service.thrift.SentryPolicyService$Client.sentry_notify_hms_event(SentryPolicyService.java:650)
~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
at
org.apache.sentry.api.service.thrift.SentryPolicyServiceClientDefaultImpl.notifyHmsEvent(SentryPolicyServiceClientDefaultImpl.java:1184)
~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
... 33 more
2018-12-19T14:29:05,956 ERROR [pool-8-thread-3]
transport.RetryClientInvocationHandler: Thrift call failed
org.apache.thrift.transport.TTransportException:
org.apache.thrift.transport.TTransportException
at
org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:110)
[sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
at
org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41)
[sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
at com.sun.proxy.$Proxy38.notifyHmsEvent(Unknown Source) [?:?]
at
org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.notifyHmsEvent(SentrySyncHMSNotificationsPostEventListener.java:252)
[sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
at
org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.onCreateDatabase(SentrySyncHMSNotificationsPostEventListener.java:195)
[sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$15.notify(MetaStoreListenerNotifier.java:64)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:243)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:896)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:928)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at com.sun.proxy.$Proxy26.create_database(Unknown Source) [?:?]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10589)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10573)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_144]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
[hadoop-common-3.0.0-cdh6.x-20181219.140835-1.jar:?]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_144]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
at
org.apache.sentry.api.service.thrift.SentryPolicyService$Client.recv_sentry_notify_hms_event(SentryPolicyService.java:663)
~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
at
org.apache.sentry.api.service.thrift.SentryPolicyService$Client.sentry_notify_hms_event(SentryPolicyService.java:650)
~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
at
org.apache.sentry.api.service.thrift.SentryPolicyServiceClientDefaultImpl.notifyHmsEvent(SentryPolicyServiceClientDefaultImpl.java:1184)
~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
at
org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:95)
~[sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
... 29 more
2018-12-19T14:29:06,258 INFO [pool-8-thread-3] metastore.HiveMetaStore: 3:
source:127.0.0.1 get_database: test_owner_privileges_without_grant_fd071aba
2018-12-19T14:29:06,258 INFO [pool-8-thread-3] HiveMetaStore.audit:
ugi=jenkins ip=127.0.0.1 cmd=source:127.0.0.1 get_database:
test_owner_privileges_without_grant_fd071aba
2018-12-19T14:29:06,271 INFO [pool-8-thread-173] metastore.HiveMetaStore: 54:
source:127.0.0.1 get_database: owner_priv_db
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
