Paul Rogers created DRILL-5019:
----------------------------------
Summary: ExternalSortBatch spills all batches to disk even if even
one spills
Key: DRILL-5019
URL: https://issues.apache.org/jira/browse/DRILL-5019
Project: Apache Drill
Issue Type: Improvement
Affects Versions: 1.8.0
Reporter: Paul Rogers
Priority: Minor
The ExternalSortBatch (ESB) operator sorts batches while spilling to disk to
stay within a defined memory budget.
Assume the memory budget is 10 GB. Assume that the actual volume of data to be
sorted is 10.1 GB. The ESB spills the extra 0.1 GB to disk. (Actually spills
more than that, say 5 GB.)
At the completion of the run, ESB has read all incoming batches. It must now
merge those batches. It does so by spilling **all** batches to disk, then doing
a disk-based merge.
This means that exceeding the memory limit by even a small amount is the same
as having a very low memory limit: all batches must spill.
This solution is simple, it works, and has some amount of logic.
But, it would be better to have a slightly more advanced solution that spills
only the smallest possible set of batches to disk, then does a hybrid
in-memory, on-disk merge, saving the unnecessary write/read cycle.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
