[
https://issues.apache.org/jira/browse/IMPALA-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16535485#comment-16535485
]
Vuk Ercegovac commented on IMPALA-7154:
---------------------------------------
Looking at the logs from the failed test (original post) and not the data-load
failure, there are a number of issues with that run (from catalogd log):
{noformat}
java.lang.IllegalStateException at
com.google.common.base.Preconditions.checkState(Preconditions.java:129) at
org.apache.impala.common.FileSystemUtil.getDistributedFileSystem(FileSystemUtil.java:376)
at
org.apache.impala.catalog.CatalogServiceCatalog$CachePoolReader.run(CatalogServiceCatalog.java:317)
...
W0608 02:49:26.329107 727 RetryingMetaStoreClient.java:148] MetaStoreClient
lost connection. Attempting to reconnect. Java exception follows:
org.apache.thrift.transport.TTransportException at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1294)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1280)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1282)
...
E0608 02:51:10.309448 424 MetaStoreUtils.java:1350] Got exception:
org.apache.thrift.transport.TTransportException null Java exception follows:
org.apache.thrift.transport.TTransportException at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_databases(ThriftHiveMetastore.java:7\
87) at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_databases(ThriftHiveMetastore.java:775)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:1105)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101)
at com.sun.proxy.$Proxy5.getAllDatabases(Unknown Source)
...
{noformat}
A lot of fs and metastore accesses were failing. From the hive log, looks like
a few 'db not found' errors, but otherwise, looks like the db was created and
accessed successfully. Here is what the hms log looks like around the time of
the exception on the Impala side:
{noformat}
...
2018-06-08 05:49:26,101 INFO log.PerfLogger (PerfLogger.java:PerfLogBegin(127))
- <PERFLOG method=get_database
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
2018-06-08 05:49:26,101 INFO metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(836)) - 25: source:127.0.0.1
get_database:test_resolution_by_name_56b45511
2018-06-08 05:49:26,101 INFO HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(405)) - ugi=jenkins ip=127.0.0.1
cmd=source:127.0.0.1 get_database: test_resolution_by_name_56b45511
2018-06-08 05:49:26,101 INFO metastore.HiveMetaStore
(HiveMetaStore.java:newRawStore(679)) - 25: Opening raw store with
implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2018-06-08 05:49:26,103 INFO metastore.ObjectStore
(ObjectStore.java:initialize(340)) - ObjectStore, initialize called
2018-06-08 05:49:26,109 INFO metastore.MetaStoreDirectSql
(MetaStoreDirectSql.java:<init>(136)) - Using direct SQL, underlying DB is OTHER
2018-06-08 05:49:26,109 INFO metastore.ObjectStore
(ObjectStore.java:setConf(323)) - Initialized ObjectStore
2018-06-08 05:49:26,110 WARN metastore.ObjectStore
(ObjectStore.java:getDatabase(628)) - Failed to get database
test_resolution_by_name_56b45511, returning NoSuchObjectException
...{noformat}
Will look into the unique_database fixture to see if further retries make sense
there.
> Error making 'dropDatabase' RPC to Hive Metastore
> -------------------------------------------------
>
> Key: IMPALA-7154
> URL: https://issues.apache.org/jira/browse/IMPALA-7154
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 2.13.0
> Reporter: Tim Armstrong
> Assignee: Vuk Ercegovac
> Priority: Critical
> Labels: broken-build, flaky
> Attachments: TEST-impala-parallel.log.gz,
> TEST-impala-parallel.xml.gz,
> catalogd.ec2-m2-4xlarge-centos-6-4-0f46.vpc.cloudera.com.jenkins.log.INFO.20180608-024815.32143.gz,
> hive.log.gz
>
>
> {noformat}
> conftest.py:293: in cleanup
> {'sync_ddl': sync_ddl})
> common/impala_test_suite.py:528: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:535: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options)
> common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:339: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:335: in execute_query_async
> return self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:460: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
> E MESSAGE: ImpalaRuntimeException: Error making 'dropDatabase' RPC to Hive
> Metastore:
> E CAUSED BY: NoSuchObjectException: test_resolution_by_name_56b45511
> {noformat}
> The backtrace in the catalogd log is:
> {noformat}
> I0608 05:49:26.111824 24195 jni-util.cc:230]
> org.apache.impala.common.ImpalaRuntimeException: Error making 'dropDatabase'
> RPC to Hive Metastore:
> at
> org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1309)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:300)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
> Caused by: NoSuchObjectException(message:test_resolution_by_name_56b45511)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_resultStandardScheme.read(ThriftHiveMetastore.java:16387)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_resultStandardScheme.read(ThriftHiveMetastore.java:16364)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result.read(ThriftHiveMetastore.java:16295)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:702)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:689)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1232)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropDatabase(HiveMetaStoreClient.java:791)
> at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101)
> at com.sun.proxy.$Proxy5.dropDatabase(Unknown Source)
> at
> org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1305)
> ... 2 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
