Zoltán Borók-Nagy created IMPALA-15678:
------------------------------------------
Summary: Query fails with "Failed to get minimum memory
reservation" right after the previous query, while that query's memory is still
being released
Key: IMPALA-15678
URL: https://issues.apache.org/jira/browse/IMPALA-15678
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Zoltán Borók-Nagy
A serial runner that submits each query as soon as the previous one has
returned all rows and been closed sometimes gets an admitted query that fails
on an executor at startup:
{noformat}
Failed to get minimum memory reservation of 6.19 GB on daemon <executor>:27000
for query <id>
due to following error: Memory limit exceeded: Could not allocate memory while
trying to
increase reservation.
{noformat}
TPC-DS at SF3000 on 8 executors (mem_limit=40g, MT_DOP=8, PLANNER=CALCITE,
queries run one at a time):
* q80's first execution failed this way right after q79 in 4 full-workload runs.
* q78's second execution failed this way right after its first (3.51 GB minimum
reservation).
* q80's second execution, right after the failed one, succeeded each time.
The claim fails against the executor's process memory limit. The most likely
cause is that the previous query's QueryState on that executor still holds its
memory, while admission control already counts it as released.
*Possible fixes:*
* When the initial reservation claim fails while another query on the executor
is being torn down, wait for it, with a bound, and retry.
* Make admission control count a query's memory on an executor until that
executor reports the QueryState released.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
