Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/9943 )
Change subject: IMPALA-5706: Spilling sort optimisations ...................................................................... Patch Set 12: (2 comments) http://gerrit.cloudera.org:8080/#/c/9943/8/be/src/runtime/sorter.cc File be/src/runtime/sorter.cc: http://gerrit.cloudera.org:8080/#/c/9943/8/be/src/runtime/sorter.cc@1676 PS8, Line 1676: --num_runs_in_one_merge; > It's possible to have var-len slots but no var-len pages if all strings are Thanks for the heads-up! I count for 2 buffers for an intermediate result then. http://gerrit.cloudera.org:8080/#/c/9943/12/tests/query_test/test_sort.py File tests/query_test/test_sort.py: http://gerrit.cloudera.org:8080/#/c/9943/12/tests/query_test/test_sort.py@135 PS12, Line 135: doesn't consume all the memory from the second Sort. This query takes approx. 25s to I gave this a second look and apparently I mis-measured something the last time. This in fact runs for at leas 25s according to my measurements. I spent a few hours to reduce this but according to my observations in order to see this behavior manifest in the # of merges we need this scale of data and runtime. I'm open for suggestions about the next step here. I thought this shouldn't run in core tests for sure, so I restricted it for the exhaustive runs. However, that would run this test in a number of permutations of test configs and multiplying that with this 25s per run I find it too much. Currently I'd prefer just to drop this test. What do you think? -- To view, visit http://gerrit.cloudera.org:8080/9943 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74857c1694802e81f1cfc765d2b4e8bc644387f9 Gerrit-Change-Number: 9943 Gerrit-PatchSet: 12 Gerrit-Owner: Gabor Kaszab <gaborkas...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Gabor Kaszab <gaborkas...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Comment-Date: Thu, 17 May 2018 15:21:11 +0000 Gerrit-HasComments: Yes
