[
https://issues.apache.org/jira/browse/IMPALA-8008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fredy Wijaya resolved IMPALA-8008.
----------------------------------
Resolution: Fixed
Fix Version/s: Impala 3.2.0
> InvocationTargetException when making HMS RPCs
> ----------------------------------------------
>
> Key: IMPALA-8008
> URL: https://issues.apache.org/jira/browse/IMPALA-8008
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 3.2.0
> Reporter: bharath v
> Assignee: Fredy Wijaya
> Priority: Major
> Labels: broken-build, flaky
> Fix For: Impala 3.2.0
>
>
> A bunch of tests failed with the following HMS exception in logs. Looks like
> a flaky HMS issue but raising this jira to rule out any obvious bugs.
> *Failed tests*
> {noformat}
> authorization.test_grant_revoke.TestGrantRevoke.test_same_name_for_role_and_user_invalidate_metadata
> authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_with_grant[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: text/none]
> authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_disabled[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: text/none]
> authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_without_grant[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: text/none]
> authorization.test_owner_privileges.TestOwnerPrivileges.test_owner_privileges_different_cases[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: text/none]
> authorization.test_show_grant.TestShowGrant.test_show_grant_user[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: text/none]
> authorization.test_show_grant.TestShowGrant.test_show_grant_in_hs2[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: text/none]
> query_test.test_insert.TestInsertQueries.test_insert_large_string[protocol:
> beeswax | exec_option: {'sync_ddl': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: text/none]
> {noformat}
> *Full stack trace*
> {noformat}
> 2018-12-19T14:29:05,730 WARN [pool-8-thread-3] metastore.ObjectStore: Failed
> to get database test_owner_privileges_without_grant_fd071aba, returning
> NoSuchObjectException
> 2018-12-19T14:29:05,779 INFO [pool-8-thread-3] common.FileUtils: Creating
> directory if it doesn't exist:
> s3a://impala-test-uswest2-2/test-warehouse/test_owner_privileges_without_grant_fd071aba.db
> 2018-12-19T14:29:05,950 ERROR [pool-8-thread-3]
> transport.RetryClientInvocationHandler: failed to execute notifyHmsEvent
> java.lang.reflect.InvocationTargetException: null
> at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
> at
> org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:95)
> [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at
> org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41)
> [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at com.sun.proxy.$Proxy38.notifyHmsEvent(Unknown Source) [?:?]
> at
> org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.notifyHmsEvent(SentrySyncHMSNotificationsPostEventListener.java:252)
> [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at
> org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.onCreateDatabase(SentrySyncHMSNotificationsPostEventListener.java:195)
> [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$15.notify(MetaStoreListenerNotifier.java:64)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:243)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:896)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:928)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at com.sun.proxy.$Proxy26.create_database(Unknown Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10589)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10573)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_144]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
> [hadoop-common-3.0.0-cdh6.x-20181219.140835-1.jar:?]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_144]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_144]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
> Caused by: org.apache.sentry.core.common.exception.SentryUserException:
> Thrift exception occurred
> at
> org.apache.sentry.api.service.thrift.SentryPolicyServiceClientDefaultImpl.notifyHmsEvent(SentryPolicyServiceClientDefaultImpl.java:1188)
> ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> ... 33 more
> Caused by: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.sentry.api.service.thrift.SentryPolicyService$Client.recv_sentry_notify_hms_event(SentryPolicyService.java:663)
> ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at
> org.apache.sentry.api.service.thrift.SentryPolicyService$Client.sentry_notify_hms_event(SentryPolicyService.java:650)
> ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at
> org.apache.sentry.api.service.thrift.SentryPolicyServiceClientDefaultImpl.notifyHmsEvent(SentryPolicyServiceClientDefaultImpl.java:1184)
> ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> ... 33 more
> 2018-12-19T14:29:05,956 ERROR [pool-8-thread-3]
> transport.RetryClientInvocationHandler: Thrift call failed
> org.apache.thrift.transport.TTransportException:
> org.apache.thrift.transport.TTransportException
> at
> org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:110)
> [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at
> org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41)
> [sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at com.sun.proxy.$Proxy38.notifyHmsEvent(Unknown Source) [?:?]
> at
> org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.notifyHmsEvent(SentrySyncHMSNotificationsPostEventListener.java:252)
> [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at
> org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener.onCreateDatabase(SentrySyncHMSNotificationsPostEventListener.java:195)
> [sentry-binding-hive-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$15.notify(MetaStoreListenerNotifier.java:64)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:243)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:896)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:928)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at com.sun.proxy.$Proxy26.create_database(Unknown Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10589)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:10573)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_144]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
> [hadoop-common-3.0.0-cdh6.x-20181219.140835-1.jar:?]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_144]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_144]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
> Caused by: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.sentry.api.service.thrift.SentryPolicyService$Client.recv_sentry_notify_hms_event(SentryPolicyService.java:663)
> ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at
> org.apache.sentry.api.service.thrift.SentryPolicyService$Client.sentry_notify_hms_event(SentryPolicyService.java:650)
> ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at
> org.apache.sentry.api.service.thrift.SentryPolicyServiceClientDefaultImpl.notifyHmsEvent(SentryPolicyServiceClientDefaultImpl.java:1184)
> ~[sentry-service-api-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
> at
> org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:95)
> ~[sentry-core-common-2.1.0-cdh6.x-SNAPSHOT.jar:?]
> ... 29 more
> 2018-12-19T14:29:06,258 INFO [pool-8-thread-3] metastore.HiveMetaStore: 3:
> source:127.0.0.1 get_database: test_owner_privileges_without_grant_fd071aba
> 2018-12-19T14:29:06,258 INFO [pool-8-thread-3] HiveMetaStore.audit:
> ugi=jenkins ip=127.0.0.1 cmd=source:127.0.0.1 get_database:
> test_owner_privileges_without_grant_fd071aba
> 2018-12-19T14:29:06,271 INFO [pool-8-thread-173] metastore.HiveMetaStore:
> 54: source:127.0.0.1 get_database: owner_priv_db
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
