David Rorke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15963 )
Change subject: IMPALA-6692: Trigger sort node run before hitting memory limit. ...................................................................... Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/15963/9//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15963/9//COMMIT_MSG@23 PS9, Line 23: This patch speedup the decision to start the sort without waiting it : to hit memory limit first by capping the intermediary quicksort run to : lower memory limit, > Great idea! I will try to implement it that way. I'm not necessarily opposed to this approach (enforcing limit only after spilling starts) but if we had confidence in the memory estimate it seems like we could enforce the limit from the start if the estimate is > the memory limit (we're very likely to spill). Unfortunately in some of the queries I'm looking at our estimates are lower than the actual peak consumed (with no limit) by an order of magnitude even though the queries end up spilling heavily. So we'll have to look into why those estimates are so bad, but for now maybe we should file a follow up JIRA to go back and make the application of the limit consider the estimate once we've improved the estimates (and consider a TODO in the code). -- To view, visit http://gerrit.cloudera.org:8080/15963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a0ba7c4bae4f1d300d4d9d7f594f63ced06a240 Gerrit-Change-Number: 15963 Gerrit-PatchSet: 9 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: David Rorke <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Comment-Date: Thu, 04 Jun 2020 22:25:36 +0000 Gerrit-HasComments: Yes
