[ https://issues.apache.org/jira/browse/KUDU-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854238#comment-16854238 ]
Alexey Serbin commented on KUDU-2779: ------------------------------------- Seems to be just another manifestation of the issue. Full log attached, the relevant snippet is below: [^master-stress-test.1.txt.xz] {noformat} master-stress-test.cc:328] Check failed: _s.ok() Bad status: Illegal state: failed to alter Hive MetaStore table: TException - service has thrown: InvalidOperationException(message=default.table_d913a97b355d45879ed3c199ae7c3cc8 table not found) @ 0x7f8166309c37 gsignal at ??:0 @ 0x7f816630d028 abort at ??:0 @ 0x45cdc2 kudu::MasterStressTest::AlterTableThread() at /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/master-stress-test.cc:328 (discriminator 25) @ 0x4706c7 std::_Mem_fn<>::_M_call<>() at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/functional:526 (discriminator 4) @ 0x4700d0 std::_Mem_fn<>::operator()<>() at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/functional:579 @ 0x46faf5 _ZNSt12_Bind_simpleIFSt7_Mem_fnIMN4kudu16MasterStressTestEFvvEEPNS1_26MasterStressTest_Test_TestEEE9_M_invokeIILm0EEEEvSt12_Index_tupleIIXspT_EEE at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/functional:1701 @ 0x46f24d std::_Bind_simple<>::operator()() at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/functional:1688 @ 0x46e56e std::thread::_Impl<>::_M_run() at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/thread:115 @ 0x7f816cea0184 start_thread at ??:0 @ 0x7f81663d0ffd clone at ??:0 {noformat} > MasterStressTest is flaky when HMS is enabled > --------------------------------------------- > > Key: KUDU-2779 > URL: https://issues.apache.org/jira/browse/KUDU-2779 > Project: Kudu > Issue Type: Test > Reporter: Hao Hao > Priority: Major > Attachments: master-stress-test.1.txt.xz > > > Encountered failure in master-stress-test.cc when HMS integration is enabled: > {noformat} > 22:30:11.487 [HMS - ERROR - pool-8-thread-2] (HiveAlterHandler.java:341) > Failed to alter table default.table_1529084adeeb48719dd0a1d18572b357 > 22:30:11.494 [HMS - ERROR - pool-8-thread-3] (HiveAlterHandler.java:341) > Failed to alter table default.table_4657eb1f8bbe4b60b03db2cbf07803a3 > 22:30:11.506 [HMS - ERROR - pool-8-thread-2] (RetryingHMSHandler.java:200) > MetaException(message:java.lang.IllegalStateException: Event not set up > correctly) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6189) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:4063) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:4020) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107) > at com.sun.proxy.$Proxy24.alter_table_with_environment_context(Unknown > Source) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:11631) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:11615) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at > org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:103) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.IllegalStateException: Event not set up correctly > at > org.apache.hadoop.hive.metastore.messaging.AlterTableMessage.checkValid(AlterTableMessage.java:49) > at > org.apache.hadoop.hive.metastore.messaging.json.JSONAlterTableMessage.<init>(JSONAlterTableMessage.java:57) > at > org.apache.hadoop.hive.metastore.messaging.json.JSONMessageFactory.buildAlterTableMessage(JSONMessageFactory.java:115) > at > org.apache.hive.hcatalog.listener.DbNotificationListener.onAlterTable(DbNotificationListener.java:187) > at > org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$8.notify(MetaStoreListenerNotifier.java:107) > at > org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175) > at > org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205) > at > org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:317) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:4049) > ... 16 more > Caused by: org.apache.thrift.protocol.TProtocolException: Unexpected > character:{ > at > org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:337) > at > org.apache.thrift.protocol.TJSONProtocol$JSONPairContext.read(TJSONProtocol.java:246) > at > org.apache.thrift.protocol.TJSONProtocol.readJSONObjectStart(TJSONProtocol.java:793) > at > org.apache.thrift.protocol.TJSONProtocol.readStructBegin(TJSONProtocol.java:840) > at > org.apache.hadoop.hive.metastore.api.Table$TableStandardScheme.read(Table.java:1577) > at > org.apache.hadoop.hive.metastore.api.Table$TableStandardScheme.read(Table.java:1573) > at org.apache.hadoop.hive.metastore.api.Table.read(Table.java:1407) > at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:81) > at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:67) > at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:98) > at > org.apache.hadoop.hive.metastore.messaging.json.JSONMessageFactory.getTObj(JSONMessageFactory.java:270) > at > org.apache.hadoop.hive.metastore.messaging.json.JSONAlterTableMessage.getTableObjAfter(JSONAlterTableMessage.java:97) > at > org.apache.hadoop.hive.metastore.messaging.AlterTableMessage.checkValid(AlterTableMessage.java:41) > ... 24 more > 22:30:11.506 [HMS - ERROR - pool-8-thread-3] (RetryingHMSHandler.java:200) > MetaException(message:java.lang.IllegalStateException: Event not set up > correctly) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6189) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:4063) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:4020) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107) > at com.sun.proxy.$Proxy24.alter_table_with_environment_context(Unknown > Source) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:11631) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:11615) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at > org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:103) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.IllegalStateException: Event not set up correctly > at > org.apache.hadoop.hive.metastore.messaging.AlterTableMessage.checkValid(AlterTableMessage.java:49) > at > org.apache.hadoop.hive.metastore.messaging.json.JSONAlterTableMessage.<init>(JSONAlterTableMessage.java:57) > at > org.apache.hadoop.hive.metastore.messaging.json.JSONMessageFactory.buildAlterTableMessage(JSONMessageFactory.java:115) > at > org.apache.hive.hcatalog.listener.DbNotificationListener.onAlterTable(DbNotificationListener.java:187) > at > org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$8.notify(MetaStoreListenerNotifier.java:107) > at > org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175) > at > org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205) > at > org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:317) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:4049) > ... 16 more > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:68) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) > at > org.apache.thrift.protocol.TJSONProtocol$LookaheadReader.read(TJSONProtocol.java:270) > at > org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:335) > at > org.apache.thrift.protocol.TJSONProtocol.readJSONString(TJSONProtocol.java:646) > at > org.apache.thrift.protocol.TJSONProtocol.readFieldBegin(TJSONProtocol.java:860) > at > org.apache.hadoop.hive.metastore.api.Table$TableStandardScheme.read(Table.java:1580) > at > org.apache.hadoop.hive.metastore.api.Table$TableStandardScheme.read(Table.java:1573) > at org.apache.hadoop.hive.metastore.api.Table.read(Table.java:1407) > at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:81) > at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:67) > at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:98) > at > org.apache.hadoop.hive.metastore.messaging.json.JSONMessageFactory.getTObj(JSONMessageFactory.java:270) > at > org.apache.hadoop.hive.metastore.messaging.json.JSONAlterTableMessage.getTableObjAfter(JSONAlterTableMessage.java:97) > at > org.apache.hadoop.hive.metastore.messaging.AlterTableMessage.checkValid(AlterTableMessage.java:41) > ... 24 more > {noformat} > This seems due to HIVE-19874. We may probably want to bump up Hive version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)