Hello Aman Sinha, Thomas Tauber-Marshall, Sahil Takiar, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15463
to look at the new patch set (#3).
Change subject: IMPALA-9530: query option to limit preagg memory
......................................................................
IMPALA-9530: query option to limit preagg memory
This adds an advanced PREAGG_BYTES_LIMIT query option that
allows limiting the memory consumption of streaming
preaggregation operators in a query.
It works by setting a maximum reservation on each grouping
aggregator in a preaggregation node. The aggregators switch
to passthrough mode automatically when hitting this limit,
the same as if they were hitting the query memory limit.
This does not override the minimum reservation computed for
the aggregation - if the limit is less than the minimum
reservation, the minimum reservation is used as the limit
instead.
The default behaviour is unchanged.
Testing:
Add a planner test with estimates higher and lower than limit
to ensure that resource estimates correctly reflect the option.
Add an end-to-end test that verifies that the option forces
passthrough when the memory limit is hit.
Change-Id: I87f7a5c68da93d068e304ef01afbcbb0d56807d9
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A
testdata/workloads/functional-planner/queries/PlannerTest/preagg-bytes-limit.test
M testdata/workloads/tpch/queries/tpch-passthrough-aggregations.test
9 files changed, 203 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/15463/3
--
To view, visit http://gerrit.cloudera.org:8080/15463
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87f7a5c68da93d068e304ef01afbcbb0d56807d9
Gerrit-Change-Number: 15463
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Sahil Takiar <[email protected]>
Gerrit-Reviewer: Thomas Tauber-Marshall <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
