Tim Armstrong created IMPALA-9345:
-------------------------------------
Summary: Hang in TestOrc::test_type_conversions when running
create table in hive
Key: IMPALA-9345
URL: https://issues.apache.org/jira/browse/IMPALA-9345
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Reporter: Tim Armstrong
{noformat}
query_test.test_scanners.TestOrc.test_type_conversions[protocol: beeswax |
exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
orc/def/block] (from pytest)
Failing for the past 1 build (Since Failed#93 )
Took 2 hr 0 min.
add description
Error Message
query_test/test_scanners.py:1337: in test_type_conversions
self.run_test_case('DataErrorsTest/orc-type-checks', vector, unique_database)
common/impala_test_suite.py:659: in run_test_case result = exec_fn(query,
user=test_section.get('USER', '').strip() or None)
common/impala_test_suite.py:610: in __exec_in_hive result =
h.execute(query, user=user) common/impala_connection.py:334: in execute r =
self.__fetch_results(handle, profile_format=profile_format)
common/impala_connection.py:441: in __fetch_results
cursor._wait_to_finish()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:405:
in _wait_to_finish resp = self._last_operation._rpc('GetOperationStatus',
req)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:992:
in _rpc response = self._execute(func_name, request)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:1009:
in _execute return func(request)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:517:
in GetOperationStatus return self.recv_GetOperationStatus()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:529:
in recv_GetOperationStatus (fname, mtype, rseqid) =
iprot.readMessageBegin()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:126:
in readMessageBegin sz = self.readI32()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:206:
in readI32 buff = self.trans.readAll(4)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58:
in readAll chunk = self.read(sz - have)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:159:
in read self._read_frame()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:163:
in _read_frame header = read_all_compat(self._trans, 4)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/six.py:31:
in <lambda> read_all_compat = lambda trans, sz: trans.readAll(sz)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58:
in readAll chunk = self.read(sz - have)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TSocket.py:105:
in read buff = self.handle.recv(sz) E Failed: Timeout >7200s
Stacktrace
query_test/test_scanners.py:1337: in test_type_conversions
self.run_test_case('DataErrorsTest/orc-type-checks', vector,
unique_database)
common/impala_test_suite.py:659: in run_test_case
result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
common/impala_test_suite.py:610: in __exec_in_hive
result = h.execute(query, user=user)
common/impala_connection.py:334: in execute
r = self.__fetch_results(handle, profile_format=profile_format)
common/impala_connection.py:441: in __fetch_results
cursor._wait_to_finish()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:405:
in _wait_to_finish
resp = self._last_operation._rpc('GetOperationStatus', req)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:992:
in _rpc
response = self._execute(func_name, request)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:1009:
in _execute
return func(request)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:517:
in GetOperationStatus
return self.recv_GetOperationStatus()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:529:
in recv_GetOperationStatus
(fname, mtype, rseqid) = iprot.readMessageBegin()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:126:
in readMessageBegin
sz = self.readI32()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:206:
in readI32
buff = self.trans.readAll(4)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58:
in readAll
chunk = self.read(sz - have)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:159:
in read
self._read_frame()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:163:
in _read_frame
header = read_all_compat(self._trans, 4)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/six.py:31:
in <lambda>
read_all_compat = lambda trans, sz: trans.readAll(sz)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58:
in readAll
chunk = self.read(sz - have)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TSocket.py:105:
in read
buff = self.handle.recv(sz)
E Failed: Timeout >7200s
{noformat}
This has happened on multiple builds, all running against S3. It seems to be
happening with high frequency on S3, but isn't deterministic - I saw this
commit both succeeed and fail:
{noformat}
IMPALA-9334: Log node blacklisting removals in ClusterMembershipMgr
{noformat}
Sample hive log:
{noformat}
tarmstrong@tarmstrong-box:~/tmplogs/impala-asf-master-core-s3-data-cache-93/logs/cluster$
grep a6c81474e8c hive*
grep: hive: Is a directory
hive-server2.log:2020-01-28T11:10:49,558 INFO
[e00c5c82-1a05-4fc7-bed2-1c62751c2223 HiveServer2-Handler-Pool: Thread-41]
ql.Driver: Compiling
command(queryId=jenkins_20200128111049_91b0e648-7740-431f-88bb-a6c81474e8c9):
create external table test_type_conversions_3e81315e.union_complextypes(
hive-server2.log:2020-01-28T11:10:50,798 INFO
[e00c5c82-1a05-4fc7-bed2-1c62751c2223 HiveServer2-Handler-Pool: Thread-41]
ql.Driver: Completed compiling
command(queryId=jenkins_20200128111049_91b0e648-7740-431f-88bb-a6c81474e8c9);
Time taken: 1.274 seconds
hive-server2.log:2020-01-28T13:10:16,567 INFO
[e00c5c82-1a05-4fc7-bed2-1c62751c2223 HiveServer2-Handler-Pool: Thread-41]
operation.Operation: The running operation has been successfully interrupted:
jenkins_20200128111049_91b0e648-7740-431f-88bb-a6c81474e8c9
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
