Zoltán Borók-Nagy created IMPALA-15679:
------------------------------------------
Summary: Queries whose minimum reservation nearly fills the
reservation limit fail instead of running with lower parallelism (TPC-DS q14_1
at SF3000)
Key: IMPALA-15679
URL: https://issues.apache.org/jira/browse/IMPALA-15679
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Zoltán Borók-Nagy
TPC-DS q14_1 at SF3000 on 8 executors with mem_limit=40g, MT_DOP=8 and
PLANNER=CALCITE, without CTE materialization, needs a minimum reservation of
29.28 GB per host. The query reservation limit is 32 GB.
* First run: the initial reservation claim failed:
{noformat}
Failed to get minimum memory reservation of 29.28 GB on daemon <executor>:27000
...
Memory limit exceeded: Could not allocate memory while trying to increase
reservation.
Query(<id>) could not allocate 29.28 GB without exceeding limit.
{noformat}
* Second run: it ran for 133 s, then failed allocating an 8 KB row batch:
{noformat}
Memory limit exceeded: Failed to allocate row batch EXCHANGE_NODE (id=707)
could not allocate
8.00 KB without exceeding limit.
Query(<id>): Reservation=31.99 GB ReservationLimit=32.00 GB OtherMemory=2.16 GB
Total=34.16 GB
Unclaimed reservations: Reservation=9.01 GB OtherMemory=0 Total=9.01 GB
{noformat}
Lower parallelism makes it fit. Per-host minimum reservation from EXPLAIN:
29.72 GB at MT_DOP=8, 15.92 GB at MT_DOP=4, 8.18 GB at MT_DOP=2. With MT_DOP=4
the same plan completes in 279 s with the correct result; with MT_DOP=8 and the
original planner the query completes in 300 s. (With CTE materialization it
fails differently, see IMPALA-15313.)
The minimum reservation scales with the instance count, so a plan whose minimum
reservation is close to the limit leaves no room for the reservation growth of
spilling operators and for non-reserved memory.
*Proposal:* when the plan's per-host minimum reservation exceeds a fraction of
the per-query reservation limit (e.g. 50%), lower the parallelism of the query,
or of the fragments that need the most reservation, at planning or admission
time. Also worth checking: why a non-reserved 8 KB allocation fails while the
query still holds 9 GB of unclaimed initial reservation.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
