[
https://issues.apache.org/jira/browse/IMPALA-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16514246#comment-16514246
]
Tianyi Wang commented on IMPALA-7154:
-------------------------------------
Saw it again. I took a brief look. The problem was that the connection to HMS
was lost after the drop database commend was sent. HMS client however didn't
know the command took effect and retried so it failed.
{noformat}
I0614 21:46:06.395942 14644 ClientCnxn.java:975] Opening socket connection to
server localhost/0:0:0:
0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
W0614 21:46:06.396255 14644 ClientCnxn.java:1102] Session 0x0 for server null,
unexpected error, clos
ing socket connection and attempting reconnect
Java exception follows:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
W0614 21:46:06.548285 12859 RetryingMetaStoreClient.java:148] MetaStoreClient
lost connection. Attemp
ting to reconnect.
Java exception follows:
org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_drop_database(ThriftHiveMet
astore.java:733)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.drop_database(ThriftHiveMetastor
e.java:718)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropDatabase(HiveMetaStoreClient.java:810)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101
)
at com.sun.proxy.$Proxy5.dropDatabase(Unknown Source)
at
org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1305)
at
org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:300)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 16 more
I0614 21:46:06.550910 12859 HiveMetaStoreClient.java:547] Closed a connection
to metastore, current c
onnections: 9
I0614 21:46:06.550966 12859 HiveMetaStoreClient.java:391] Trying to connect to
metastore with URI thr
ift://localhost:9083
I0614 21:46:06.551259 12859 HiveMetaStoreClient.java:465] Opened a connection
to metastore, current c
onnections: 10
I0614 21:46:06.551961 12859 HiveMetaStoreClient.java:518] Connected to
metastore.
I0614 21:46:06.555757 12859 jni-util.cc:230]
org.apache.impala.common.ImpalaRuntimeException: Error m
aking 'dropDatabase' RPC to Hive Metastore:
at
org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1309)
at
org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:300)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
Caused by: NoSuchObjectException(message:test_fuzz_nested_types_67367717)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_result
StandardScheme.read(ThriftHiveMetastore.java:16387)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_result
StandardScheme.read(ThriftHiveMetastore.java:16364)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result.read(ThriftHiveMeta
store.java:16295)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMeta
store.java:702)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore
.java:689)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1232)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropDatabase(HiveMetaStoreClient.java:791)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101
)
at com.sun.proxy.$Proxy5.dropDatabase(Unknown Source)
at
org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1305)
... 2 more
{noformat}
> Error making 'dropDatabase' RPC to Hive Metastore
> -------------------------------------------------
>
> Key: IMPALA-7154
> URL: https://issues.apache.org/jira/browse/IMPALA-7154
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 2.13.0
> Reporter: Tim Armstrong
> Assignee: Vuk Ercegovac
> Priority: Critical
> Labels: broken-build, flaky
> Attachments: TEST-impala-parallel.log.gz,
> TEST-impala-parallel.xml.gz,
> catalogd.ec2-m2-4xlarge-centos-6-4-0f46.vpc.cloudera.com.jenkins.log.INFO.20180608-024815.32143.gz,
> hive.log.gz
>
>
> {noformat}
> conftest.py:293: in cleanup
> {'sync_ddl': sync_ddl})
> common/impala_test_suite.py:528: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:535: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options)
> common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:339: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:335: in execute_query_async
> return self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:460: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
> E MESSAGE: ImpalaRuntimeException: Error making 'dropDatabase' RPC to Hive
> Metastore:
> E CAUSED BY: NoSuchObjectException: test_resolution_by_name_56b45511
> {noformat}
> The backtrace in the catalogd log is:
> {noformat}
> I0608 05:49:26.111824 24195 jni-util.cc:230]
> org.apache.impala.common.ImpalaRuntimeException: Error making 'dropDatabase'
> RPC to Hive Metastore:
> at
> org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1309)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:300)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
> Caused by: NoSuchObjectException(message:test_resolution_by_name_56b45511)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_resultStandardScheme.read(ThriftHiveMetastore.java:16387)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result$get_database_resultStandardScheme.read(ThriftHiveMetastore.java:16364)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_database_result.read(ThriftHiveMetastore.java:16295)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:702)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:689)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1232)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropDatabase(HiveMetaStoreClient.java:791)
> at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101)
> at com.sun.proxy.$Proxy5.dropDatabase(Unknown Source)
> at
> org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:1305)
> ... 2 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
