Yida Wu created IMPALA-15758:
--------------------------------
Summary: Dynamic memory reallocation between grouping sets in
streaming preaggregation
Key: IMPALA-15758
URL: https://issues.apache.org/jira/browse/IMPALA-15758
Project: IMPALA
Issue Type: Improvement
Components: Backend
Affects Versions: Impala 5.0.0
Reporter: Yida Wu
Assignee: Yida Wu
When executing a streaming preaggregation with multiple grouping sets (such as
GROUPING SETS, ROLLUP, or multiple DISTINCT aggregates), Impala creates one
Aggregator per set. These aggregators all share the same memory reservation on
the node.
Currently, the sets take memory from the shared reservation on a first-come,
first-served basis, and memory is never moved between them once taken. If a
grouping set runs out of hash table space, it is forced to pass its rows
straight through to the downstream merge aggregation. This happens even if that
set has a high reduction rate and would benefit greatly from more memory.
Meanwhile, another set on the same node may hold memory while reducing very few
rows. Because memory cannot be shifted between the sets, overall preaggregation
efficiency drops, forcing the exchange and merge phases to take on extra,
avoidable work.
One approach is to introduce a flush mechanism, when a highly reducing grouping
set becomes starved for memory, we identify another grouping set that is
reducing poorly (a "victim"). We can flush the victim's hash tables downstream,
thereby freeing up its memory reservation and handing it over to the starved
grouping set. A prototype of this approach can cut the query time by about 25%
on TPC-DS SF10 q67 with mem_limit=350m.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
