Tianyi Wang created IMPALA-6257:
-----------------------------------
Summary:
TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan hangs in S3 build
Key: IMPALA-6257
URL: https://issues.apache.org/jira/browse/IMPALA-6257
Project: IMPALA
Issue Type: Bug
Affects Versions: Impala 2.11.0
Reporter: Tianyi Wang
Priority: Blocker
In a recent S3 build,
TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan failed because a
fragment never finishes. The error message is:
{noformat}
07:49:57
TestKuduMemLimits.test_low_mem_limit_low_selectivity_scan[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1,
'exec_single_node_rows_threshold': 0} | table_format: text/none-1]
07:49:57 query_test/test_kudu.py:1062: in
test_low_mem_limit_low_selectivity_scan
07:49:57 v.wait_for_metric("impala-server.num-fragments-in-flight", 0,
timeout=30)
07:49:57 verifiers/metric_verifier.py:62: in wait_for_metric
07:49:58 self.impalad_service.wait_for_metric_value(metric_name,
expected_value, timeout)
07:49:58 common/impala_service.py:89: in wait_for_metric_value
07:49:58 (metric_name, expected_value, timeout)
07:49:58 E AssertionError: Metric value impala-server.num-fragments-in-flight
did not reach value 0 in 30s
{noformat}
Looking at the coordinator log, the query hangs for 30 seconds before cancelled
by the client:
{noformat}
I1127 07:30:37.677126 5939 coordinator.cc:789] Coordinator waiting for
backends to finish, 1 remaining
I1127 07:30:37.677865 5939 coordinator.cc:794] All backends finished
successfully.
I1127 07:30:37.677877 5939 coordinator.cc:1090] Release admssion control
resources for query 4044d47830c1cc98:8a63d27b00000000
I1127 07:30:37.677942 6043 data-stream-mgr.cc:238] DeregisterRecvr():
fragment_instance_id=4044d47830c1cc98:8a63d27b00000000, node=1
I1127 07:30:37.677963 6043 data-stream-recvr.cc:235] cancelled stream:
fragment_instance_id_=4044d47830c1cc98:8a63d27b00000000 node_id=1
I1127 07:30:37.678045 6043 query-state.cc:385] Instance completed.
instance_id=4044d47830c1cc98:8a63d27b00000000 #in-flight=1 status=OK
I1127 07:30:37.678073 6043 query-exec-mgr.cc:149] ReleaseQueryState():
query_id=4044d47830c1cc98:8a63d27b00000000 refcnt=2
I1127 07:31:08.084748 5939 impala-hs2-server.cc:699] CloseOperation():
query_id=4044d47830c1cc98:8a63d27b00000000
I1127 07:31:08.084779 5939 impala-server.cc:992] UnregisterQuery():
query_id=4044d47830c1cc98:8a63d27b00000000
I1127 07:31:08.084790 5939 impala-server.cc:1075] Cancel():
query_id=4044d47830c1cc98:8a63d27b00000000
I1127 07:31:08.084808 5939 coordinator.cc:895] Cancel()
query_id=4044d47830c1cc98:8a63d27b00000000
I1127 07:31:08.084827 5939 coordinator.cc:905] CancelBackends()
query_id=4044d47830c1cc98:8a63d27b00000000, tried to cancel 0 backends
I1127 07:31:08.090203 5939 query-exec-mgr.cc:149] ReleaseQueryState():
query_id=4044d47830c1cc98:8a63d27b00000000 refcnt=1
I1127 07:31:08.101332 5939 impala-hs2-server.cc:381] CloseSession():
request=TCloseSessionReq {
{noformat}
Compared to the same but successful query, the #in-flight differs:
{noformat}
I1128 07:35:45.863986 7571 coordinator.cc:789] Coordinator waiting for
backends to finish, 1 remaining
I1128 07:35:45.864734 7571 coordinator.cc:794] All backends finished
successfully.
I1128 07:35:45.864749 7571 coordinator.cc:1090] Release admssion control
resources for query 6f43945cef3a95be:744fd2a300000000
I1128 07:35:45.864823 7671 data-stream-mgr.cc:238] DeregisterRecvr():
fragment_instance_id=6f43945cef3a95be:744fd2a300000000, node=1
I1128 07:35:45.864845 7671 data-stream-recvr.cc:235] cancelled stream:
fragment_instance_id_=6f43945cef3a95be:744fd2a300000000 node_id=1
I1128 07:35:45.864953 7671 query-state.cc:385] Instance completed.
instance_id=6f43945cef3a95be:744fd2a300000000 #in-flight=0 status=OK
I1128 07:35:45.864982 7671 query-exec-mgr.cc:149] ReleaseQueryState():
query_id=6f43945cef3a95be:744fd2a300000000 refcnt=2
I1128 07:35:45.944633 7571 impala-hs2-server.cc:699] CloseOperation():
query_id=6f43945cef3a95be:744fd2a300000000
I1128 07:35:45.944655 7571 impala-server.cc:992] UnregisterQuery():
query_id=6f43945cef3a95be:744fd2a300000000
I1128 07:35:45.944666 7571 impala-server.cc:1075] Cancel():
query_id=6f43945cef3a95be:744fd2a300000000
I1128 07:35:45.944691 7571 coordinator.cc:895] Cancel()
query_id=6f43945cef3a95be:744fd2a300000000
I1128 07:35:45.944710 7571 coordinator.cc:905] CancelBackends()
query_id=6f43945cef3a95be:744fd2a300000000, tried to cancel 0 backends
I1128 07:35:45.949873 7571 query-exec-mgr.cc:149] ReleaseQueryState():
query_id=6f43945cef3a95be:744fd2a300000000 refcnt=1
I1128 07:35:45.959841 7571 impala-hs2-server.cc:381] CloseSession():
request=TCloseSessionReq {
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
