[
https://issues.apache.org/jira/browse/IMPALA-9073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16964428#comment-16964428
]
Tim Armstrong commented on IMPALA-9073:
---------------------------------------
I think this is because of IMPALA-8803 - releasing resources per-backend. It is
possible to have > 3 concurrent queries even with only 3 slots per backend if
the previous queries had released resources on those backends. I think the test
should actually be checking the number of queries on each backend.
{noformat}
I1024 22:24:48.146070 106355 admission-controller.cc:1303] Stats:
agg_num_running=3, agg_num_queued=2, agg_mem_reserved=815.26 KB,
local_host(local_mem_admitted=300.05 MB, num_admitted_running=3, num_queued=2,
backend_mem_reserved=75.87 KB)
I1024 22:24:48.146086 106355 admission-controller.cc:1509] Could not dequeue
query id=3e4bce299fd907b1:f92ff5cf00000000 reason: Not enough admission control
slots available on host ip-172-31-20-105:22000. Needed 1 slots but 3/3 are
already in use.
I1024 22:24:48.179689 107036 coordinator.cc:508] ExecState: query
id=f9492c4fae791967:35b096ec00000000 execution completed
I1024 22:24:48.179709 107036 coordinator.cc:644] Coordinator waiting for
backends to finish, 1 remaining. query_id=f9492c4fae791967:35b096ec00000000
I1024 22:24:48.179702 107084 krpc-data-stream-mgr.cc:298]
f9492c4fae791967:35b096ec00000000] DeregisterRecvr():
fragment_instance_id=f9492c4fae791967:35b096ec00000000, node=1
I1024 22:24:48.179896 107084 query-state.cc:652]
f9492c4fae791967:35b096ec00000000] Instance completed.
instance_id=f9492c4fae791967:35b096ec00000000 #in-flight=2 status=OK
I1024 22:24:48.179916 107081 query-state.cc:287]
f9492c4fae791967:35b096ec00000000] UpdateBackendExecState(): last report for
f9492c4fae791967:35b096ec00000000
I1024 22:24:48.179930 107088 krpc-data-stream-mgr.cc:298]
e147d5691862e504:f48934e700000000] DeregisterRecvr():
fragment_instance_id=e147d5691862e504:f48934e700000000, node=1
I1024 22:24:48.179937 107038 coordinator.cc:508] ExecState: query
id=e147d5691862e504:f48934e700000000 execution completed
I1024 22:24:48.179980 107038 coordinator.cc:644] Coordinator waiting for
backends to finish, 1 remaining. query_id=e147d5691862e504:f48934e700000000
I1024 22:24:48.180224 107088 query-state.cc:652]
e147d5691862e504:f48934e700000000] Instance completed.
instance_id=e147d5691862e504:f48934e700000000 #in-flight=1 status=OK
I1024 22:24:48.180279 107083 query-state.cc:287]
e147d5691862e504:f48934e700000000] UpdateBackendExecState(): last report for
e147d5691862e504:f48934e700000000
I1024 22:24:48.180637 107035 coordinator.cc:508] ExecState: query
id=1c48aa10421e20d8:677b55c400000000 execution completed
I1024 22:24:48.180656 107089 krpc-data-stream-mgr.cc:298]
1c48aa10421e20d8:677b55c400000000] DeregisterRecvr():
fragment_instance_id=1c48aa10421e20d8:677b55c400000000, node=1
I1024 22:24:48.180670 107035 coordinator.cc:644] Coordinator waiting for
backends to finish, 1 remaining. query_id=1c48aa10421e20d8:677b55c400000000
I1024 22:24:48.180932 107089 query-state.cc:652]
1c48aa10421e20d8:677b55c400000000] Instance completed.
instance_id=1c48aa10421e20d8:677b55c400000000 #in-flight=0 status=OK
I1024 22:24:48.180950 107082 query-state.cc:287]
1c48aa10421e20d8:677b55c400000000] UpdateBackendExecState(): last report for
1c48aa10421e20d8:677b55c400000000
I1024 22:24:48.181324 106187 coordinator.cc:768] Backend completed:
host=ip-172-31-20-105:22000 remaining=1
query_id=f9492c4fae791967:35b096ec00000000
I1024 22:24:48.181390 107036 coordinator.cc:960] Release admission control
resources for query_id=f9492c4fae791967:35b096ec00000000
I1024 22:24:48.181421 106355 admission-controller.cc:1291] Trying to admit
id=3e4bce299fd907b1:f92ff5cf00000000 in pool_name=default-pool
executor_group_name=default-pool-group1 per_host_mem_estimate=176.02 MB
dedicated_coord_mem_estimate=100.02 MB max_requests=-1 (configured statically)
max_queued=200 (configured statically) max_mem=-1.00 B (configured statically)
I1024 22:24:48.181447 106355 admission-controller.cc:1303] Stats:
agg_num_running=3, agg_num_queued=2, agg_mem_reserved=815.26 KB,
local_host(local_mem_admitted=200.03 MB, num_admitted_running=3, num_queued=2,
backend_mem_reserved=75.87 KB)
I1024 22:24:48.181470 106355 admission-controller.cc:1443] Admitting from
queue: query=3e4bce299fd907b1:f92ff5cf00000000
I1024 22:24:48.181493 107081 query-state.cc:448]
f9492c4fae791967:35b096ec00000000] Cancelling fragment instances as directed by
the coordinator. Returned status: Cancelled
I1024 22:24:48.181511 107081 query-state.cc:669]
f9492c4fae791967:35b096ec00000000] Cancel:
query_id=f9492c4fae791967:35b096ec00000000
I1024 22:24:48.181520 107081 krpc-data-stream-mgr.cc:329]
f9492c4fae791967:35b096ec00000000] cancelling active streams for
fragment_instance_id=f9492c4fae791967:35b096ec00000000
I1024 22:24:48.181553 106355 admission-controller.cc:1291] Trying to admit
id=704b58a9d42f96a2:3d711cc000000000 in pool_name=default-pool
executor_group_name=default-pool-group1 per_host_mem_estimate=176.02 MB
dedicated_coord_mem_estimate=100.02 MB max_requests=-1 (configured statically)
max_queued=200 (configured statically) max_mem=-1.00 B (configured statically)
I1024 22:24:48.181578 106355 admission-controller.cc:1303] Stats:
agg_num_running=4, agg_num_queued=1, agg_mem_reserved=815.26 KB,
local_host(local_mem_admitted=652.08 MB, num_admitted_running=4, num_queued=1,
backend_mem_reserved=75.87 KB)
I1024 22:24:48.181602 106355 admission-controller.cc:1509] Could not dequeue
query id=704b58a9d42f96a2:3d711cc000000000 reason: Not enough admission control
slots available on host ip-172-31-20-105:22000. Needed 1 slots but 3/3 are
already in use.
{noformat}
> Failed test during pre-commit:
> custom_cluster.test_executor_groups.TestExecutorGroups.test_executor_concurrency
> ---------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-9073
> URL: https://issues.apache.org/jira/browse/IMPALA-9073
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Anurag Mantripragada
> Assignee: Tim Armstrong
> Priority: Critical
> Labels: build-failure, flaky
> Attachments: TEST-impala-custom-cluster.log,
> TEST-impala-custom-cluster.xml,
> impalad.ip-172-31-3-83.ubuntu.log.INFO.20191020-182539.109469
>
>
> Observed this test failure in pre-commit test of an unrelated change. Looks
> like the expected number of concurrent queries reached 4 while it is expected
> to be 3 or less.
> {code:java}
> custom_cluster/test_executor_groups.py:293: in test_executor_concurrency
> assert max(num_running) == 3, \
> E AssertionError: Unexpected number of running queries: [3, 3, 3, 3, 3, 3,
> 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3]
> E assert 4 == 3
> E + where 4 = max([3, 3, 3, 3, 3, 3, ...]) {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
