Tim Armstrong has posted comments on this change. Change subject: IMPALA-5713: always reserve memory for preaggs ......................................................................
Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/7739/3/fe/src/main/java/org/apache/impala/planner/AggregationNode.java File fe/src/main/java/org/apache/impala/planner/AggregationNode.java: PS3, Line 327: 16b > I vote for omitting that last calculation. Done Line 330: perInstanceMinReservation = (bufferSize + 64 * 1024) * PARTITION_FANOUT; > Tim once suggested that the pre-agg should really be a separate exec node. > Why even have 16 partitions etc. I tried changing that once and it didn't go well :). It's actually much slower to have a single partition for some aggregations. There's a strong pre-partitioning effect that improves memory locality in the merge aggregation. But yeah it's ripe for a lot of cleanup - we could separate out the ExecNode implementations and figure out a better way to exploit the pre-partitioning (since the effect is now accidental and undocumented). -- To view, visit http://gerrit.cloudera.org:8080/7739 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I2b544f9ec1a906719e2bbb074743926b95a3a573 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Mostafa Mokhtar <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-HasComments: Yes
