Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15463 )
Change subject: IMPALA-9530: query option to limit preagg memory ...................................................................... IMPALA-9530: query option to limit preagg memory This adds an advanced PREAGG_BYTES_LIMIT query option that allows limiting the memory consumption of streaming preaggregation operators in a query. It works by setting a maximum reservation on each grouping aggregator in a preaggregation node. The aggregators switch to passthrough mode automatically when hitting this limit, the same as if they were hitting the query memory limit. This does not override the minimum reservation computed for the aggregation - if the limit is less than the minimum reservation, the minimum reservation is used as the limit instead. The default behaviour is unchanged. Testing: Add a planner test with estimates higher and lower than limit to ensure that resource estimates correctly reflect the option. Add an end-to-end test that verifies that the option forces passthrough when the memory limit is hit. Change-Id: I87f7a5c68da93d068e304ef01afbcbb0d56807d9 Reviewed-on: http://gerrit.cloudera.org:8080/15463 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/preagg-bytes-limit.test M testdata/workloads/tpch/queries/tpch-passthrough-aggregations.test 9 files changed, 203 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15463 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I87f7a5c68da93d068e304ef01afbcbb0d56807d9 Gerrit-Change-Number: 15463 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Sahil Takiar <[email protected]> Gerrit-Reviewer: Thomas Tauber-Marshall <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
