[
https://issues.apache.org/jira/browse/IMPALA-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217253#comment-17217253
]
Tim Armstrong commented on IMPALA-9884:
---------------------------------------
{noformat}
I1017 01:11:24.339452 25165 admission-controller.cc:1638]
3144178b629c699c:dde994b700000000] Stats: agg_num_running=5, agg_num_queued=0,
agg_mem_reserved=0, local_host(local_mem_admitted=12.00 GB,
num_admitted_running=5, num_queued=0, backend_mem_reserved=0, topN_query_stats:
queries=[], total_mem_consumed=0; pool_level_stats: num_running=0, min=0,
max=0, pool_total_mem=0)
...
I1017 01:11:24.339519 25165 admission-controller.cc:1195]
3144178b629c699c:dde994b700000000] Queuing, query
id=3144178b629c699c:dde994b700000000 reason: Not enough aggregate memory
available in pool default-pool with max mem resources 12.00 GB. Needed 2.40 GB
but only 18.00 B was available.
...
I1017 01:11:29.640173 24428 admission-controller.cc:1630] Trying to admit
id=3144178b629c699c:dde994b700000000 in pool_name=default-pool
executor_group_name=default per_host_mem_estimate=81.29 MB
dedicated_coord_mem_estimate=101.29 MB max_requests=150 max_queued=10
max_mem=12.00 GB
I1017 01:11:29.640328 24428 admission-controller.cc:1652] Cannot admit query
3144178b629c699c:dde994b700000000 to group default: Not enough aggregate memory
available in pool default-pool with max mem resources 12.00 GB. Needed 2.40 GB
but only 18.00 B was available. Details:
I1017 01:11:29.640334 24428 admission-controller.cc:1851] Could not dequeue
query id=3144178b629c699c:dde994b700000000 reason: Not enough aggregate memory
available in pool default-pool with max mem resources 12.00 GB. Needed 2.40 GB
but only 18.00 B was available.
I1017 01:11:29.677559 24428 admission-controller.cc:1630] Trying to admit
id=3144178b629c699c:dde994b700000000 in pool_name=default-pool
executor_group_name=default per_host_mem_estimate=81.29 MB
dedicated_coord_mem_estimate=101.29 MB max_requests=150 max_queued=10
max_mem=12.00 GB
I1017 01:11:29.677701 24428 admission-controller.cc:1786] Admitting from queue:
query=3144178b629c699c:dde994b700000000
I1017 01:11:29.677712 24428 admission-controller.cc:1878] For Query
3144178b629c699c:dde994b700000000 per_backend_mem_limit set to: 819.20 MB
per_backend_mem_to_admit set to: 819.20 MB coord_backend_mem_limit set to:
819.20 MB coord_backend_mem_to_admit set to: 819.20 MB
I1017 01:11:29.677990 25165 admission-controller.cc:1273]
3144178b629c699c:dde994b700000000] Admitted queued query
id=3144178b629c699c:dde994b700000000
I1017 01:11:29.678004 25165 admission-controller.cc:1274]
3144178b629c699c:dde994b700000000] Final: agg_num_running=6, agg_num_queued=9,
agg_mem_reserved=9.60 GB, local_host(local_mem_admitted=12.00 GB,
num_admitted_running=6, num_queued=9, backend_mem_reserved=4.00 GB,
topN_query_stats: queries=[8f462fa2ce60d289:e063147100000000,
d5466702e1e5c14e:43f31d3000000000, 1b4e1ee5d51fc461:1221932500000000,
cf498fd1ece032b6:b4f673d100000000, 4a4d18e5caa85310:022e022900000000],
total_mem_consumed=59.95 MB, fraction_of_pool_total_mem=1; pool_level_stats:
num_running=5, min=5.03 MB, max=13.76 MB, pool_total_mem=59.95 MB,
average_per_query=11.99 MB)
{noformat}
It looks like this was able to be dequeued because a query finished running on
a backend:
{noformat}
I1017 01:11:29.639609 24226 coordinator.cc:959] Backend completed:
host=impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com:27001
remaining=3 query_id= [^impalad-executors.tar.gz] 00
I1017 01:11:29.639629 24226 coordinator-backend-state.cc:362]
query_id=1b4e1ee5d51fc461:1221932500000000: first in-progress backend:
impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com:27000
I1017 01:11:29.639644 24226 admission-controller.cc:759] Update admitted mem
reserved for
host=impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com:27001
prev=2.40 GB new=1.60 GB
I1017 01:11:29.639657 24226 admission-controller.cc:764] Update admitted
queries for
host=impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com:27001 prev=3
new=2
I1017 01:11:29.639659 24226 admission-controller.cc:769] Update slots in use
for host=impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com:27001
prev=3 new=2
I1017 01:11:29.639701 24226 admission-controller.cc:1337] Released query
backend(s) impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com:27001
for query id=1b4e1ee5d51fc461:1221932500000000 agg_num_running=5,
agg_num_queued=10, agg_mem_reserved=12.00 GB,
local_host(local_mem_admitted=9.60 GB, num_admitted_running=5, num_queued=10,
backend_mem_reserved=4.00 GB, topN_query_stats:
queries=[cf498fd1ece032b6:b4f673d100000000, d5466702e1e5c14e:43f31d3000000000,
1b4e1ee5d51fc461:1221932500000000, 8f462fa2ce60d289:e063147100000000,
4a4d18e5caa85310:022e022900000000], total_mem_consumed=37.12 MB,
fraction_of_pool_total_mem=1; pool_level_stats: num_running=5, min=666.05 KB,
max=11.81 MB, pool_total_mem=37.12 MB, average_per_query=7.42 MB)
{noformat}
Attaching the executor logs too - [^impalad-executors.tar.gz]
> TestAdmissionControllerStress.test_mem_limit failing occasionally
> -----------------------------------------------------------------
>
> Key: IMPALA-9884
> URL: https://issues.apache.org/jira/browse/IMPALA-9884
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.0
> Reporter: Vihang Karajgaonkar
> Assignee: Tim Armstrong
> Priority: Blocker
> Labels: broken-build, flaky
> Attachments: impalad-executors.tar.gz,
> impalad.impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com.jenkins.log.INFO.20201017-011116.23933.gz
>
>
> Recently, I saw this test failing with the exception trace below.
> {noformat}
> custom_cluster/test_admission_controller.py:1782: in test_mem_limit
> {'request_pool': self.pool_name, 'mem_limit': query_mem_limit})
> custom_cluster/test_admission_controller.py:1638: in run_admission_test
> assert metric_deltas['dequeued'] == 0,\
> E AssertionError: Queued queries should not run until others are made to
> finish
> E assert 1 == 0
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
