David Rorke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15963 )
Change subject: IMPALA-6692: Trigger sort node run before hitting memory limit. ...................................................................... Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/15963/9//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15963/9//COMMIT_MSG@23 PS9, Line 23: This patch speedup the decision to start the sort without waiting it : to hit memory limit first by capping the intermediary quicksort run to : lower memory limit, > > Since the patch can have immediate benefit in some well known cases like The following estimates vs actual from a partitioned parquet insert of catalog sales. Multiple runs with differing sort limits. All with 10 TB scale factor on 20 nodes with an explicit 32 GB memory limit set: Sort Limit. Rows. Est Rows. Peak Mem. Est Peak Mem None 14.40B. 15.59B. 25.60 GB 2.43 GB 8GB 14.40B 15.59B 8.00 GB 2.43 GB 2GB 14.40B 15.59B 2.00 GB 2.43 GB 512MB. 14.40B 15.59B 678.65 MB 2.43 GB 256MB 14.40B 15.59B 1.51 GB 2.43 GB So cardinality estimate is pretty good in this case, estimated peak mem is very low for the reason you gave (sort estimate doesn't assume the full input will be kept in memory). -- To view, visit http://gerrit.cloudera.org:8080/15963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a0ba7c4bae4f1d300d4d9d7f594f63ced06a240 Gerrit-Change-Number: 15963 Gerrit-PatchSet: 9 Gerrit-Owner: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: David Rorke <dro...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Comment-Date: Fri, 05 Jun 2020 19:45:56 +0000 Gerrit-HasComments: Yes
