[
https://issues.apache.org/jira/browse/IMPALA-7064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikramjeet Vig updated IMPALA-7064:
-----------------------------------
Description:
Due to a delay in updating (host_mem_reserved_[coord_id]) memory reserved
locally on a host in impala, multiple queries submitted in succession after
every previous succeeds can result in queries being queued momentarily due to
not enough memory being available. This happens because unlike pool stats that
immediately update local metrics, the host level memory reserved metric is only
updated on the next statestore update. This delay can cause the admission
controller to believe that memory reserved for a previous query has not been
released.
Repro steps:
start an impala cluster using:
start-impala-cluster.py --impalad_args="--default_pool_max_requests=1
--queue_wait_timeout_ms=1000000 --mem_limit=5G --default_pool_mem_limit=10G
--queue_wait_timeout_ms=1"
run queries in succession like this:
./bin/impala-shell.sh -q "set mem_limit=5G;set num_nodes=1;select * from
functional.alltypesagg, (select 1) B limit 1;select * from
functional.alltypesagg, (select 1) B limit 1;"
Due to queue wait timeout set to 1ms, you will notice that the second query
fails.
was:
Due to a delay in updating memory admitted locally on a host in impala,
multiple queries submitted in succession after every previous succeeds can
result in queries being queued momentarily due to not enough memory being
available. This happens because unlike pool stats that immediately update local
metrics, the host level memory admitted metric is only updated on the next
statestore update. This delay can cause the admission controller to believe
that memory admitted for a previous query has not been released.
Repro steps:
start an impala cluster using:
start-impala-cluster.py --impalad_args="--default_pool_max_requests=1
--queue_wait_timeout_ms=1000000 --mem_limit=5G --default_pool_mem_limit=10G
--queue_wait_timeout_ms=1"
run queries in succession like this:
./bin/impala-shell.sh -q "set mem_limit=5G;set num_nodes=1;select * from
functional.alltypesagg, (select 1) B limit 1;select * from
functional.alltypesagg, (select 1) B limit 1;"
Due to queue wait timeout set to 1ms, you will notice that the second query
fails.
> Local memory admitted metric does not get updated immediately
> --------------------------------------------------------------
>
> Key: IMPALA-7064
> URL: https://issues.apache.org/jira/browse/IMPALA-7064
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.0
> Reporter: Bikramjeet Vig
> Assignee: Bikramjeet Vig
> Priority: Major
> Labels: admission-control
>
> Due to a delay in updating (host_mem_reserved_[coord_id]) memory reserved
> locally on a host in impala, multiple queries submitted in succession after
> every previous succeeds can result in queries being queued momentarily due to
> not enough memory being available. This happens because unlike pool stats
> that immediately update local metrics, the host level memory reserved metric
> is only updated on the next statestore update. This delay can cause the
> admission controller to believe that memory reserved for a previous query has
> not been released.
> Repro steps:
> start an impala cluster using:
> start-impala-cluster.py --impalad_args="--default_pool_max_requests=1
> --queue_wait_timeout_ms=1000000 --mem_limit=5G --default_pool_mem_limit=10G
> --queue_wait_timeout_ms=1"
> run queries in succession like this:
> ./bin/impala-shell.sh -q "set mem_limit=5G;set num_nodes=1;select * from
> functional.alltypesagg, (select 1) B limit 1;select * from
> functional.alltypesagg, (select 1) B limit 1;"
> Due to queue wait timeout set to 1ms, you will notice that the second query
> fails.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
