Hello Tim Armstrong, Csaba Ringhofer,

I'd like you to reexamine a change. Please visit


to look at the new patch set (#11).

Change subject: IMPALA-5706: Spilling sort optimisations

IMPALA-5706: Spilling sort optimisations

This patch covers multiple changes with the purpose of optimizing
spilling sort mechanism:
  - Remove the hard-coded maximum limit of buffers that can be used
    for merging the sorted runs. Instead this number is calculated
    based on the available memory through buffer pool.
  - The already sorted runs are distributed more optimally between
    the last intermediate merge and the final merge to avoid that a
    heavy intermediate merge is followed by a light final merge.
  - Right before starting the merging phase Sorter tries to allocate
    additional memory through the buffer pool.
  - An output run is not allocated anymore for the final merge.

Note, double-buffering the runs during a merge was also planned with
this patch. However, performance testing showed that except some
exotic queries with unreasonably small amount of buffer pool memory
available double-buffering doesn't add to the overall performance.
It's basically because the half of the available buffers have to be
sacrificed to do double-buffering and as a result the merge tree can
get deeper. In addition the amount of I/O wait time is not reaching
the level where double-buffering could countervail the reduced number
of runs during a particular merge.

Performance measurements were made during manual testing to verify
that this is in fact an optimization:
  - In case doing a sort on top of a join when working with a
    restricted amount of memory then the Sort node successfully
    allocates additional memory right before the merging phase. This
    is feasible because once Join finishes sending new input data and
    calls InputDone() then it releases memory that can be picked up
    by the Sorter. This results in shallower merging trees (more runs
    grabbed for a merge).
  - On a multi-node cluster I verified that in cases when at least one
    merging step is done then this change reduces the execution time
    for sorts.
  - The more merging steps are done the bigger the performance gain is
    compared to the baseline.

Change-Id: I74857c1694802e81f1cfc765d2b4e8bc644387f9
M be/src/runtime/sorter.cc
M be/src/runtime/sorter.h
M tests/query_test/test_sort.py
3 files changed, 188 insertions(+), 114 deletions(-)

  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/9943/11
To view, visit http://gerrit.cloudera.org:8080/9943
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I74857c1694802e81f1cfc765d2b4e8bc644387f9
Gerrit-Change-Number: 9943
Gerrit-PatchSet: 11
Gerrit-Owner: Gabor Kaszab <gaborkas...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <gaborkas...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>

Reply via email to