[
https://issues.apache.org/jira/browse/DRILL-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886810#comment-15886810
]
ASF GitHub Bot commented on DRILL-5284:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/761#discussion_r103332364
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/managed/ExternalSortBatch.java
---
@@ -392,22 +448,31 @@ private void configure(DrillConfig config) {
// Set too large and the ratio between memory and input data sizes
becomes
// small. Set too small and disk seek times dominate performance.
- spillBatchSize =
config.getBytes(ExecConstants.EXTERNAL_SORT_SPILL_BATCH_SIZE);
- spillBatchSize = Math.max(spillBatchSize, MIN_SPILL_BATCH_SIZE);
+ preferredSpillBatchSize =
config.getBytes(ExecConstants.EXTERNAL_SORT_SPILL_BATCH_SIZE);
+
+ // In low memory, use no more than 1/4 of memory for each spill batch.
Ensures we
+ // can merge.
+
+ preferredSpillBatchSize = Math.min(preferredSpillBatchSize,
memoryLimit / 4);
--- End diff --
In low memory conditions, restrict the spill batch size to 1/4 of memory.
Why?
* We need to accumulate at least 2 such batches to do a merge. (Now at 1/2
of memory.)
* We need to create an output batch from the two inputs (3/4 of memory).
* Need overhead for other direct memory uses. (Remaining 1/4 of memory.)
Sadly, memory management in Drill is not very precise: batch sizes can't be
predicted with any accuracy. Trying to use, say, 1/3 of memory for the spill
batch would seem more logical. (Two batches into the merge, one out), but the
allocator issues a fatal error if we guess wrong by even one byte. So, we are
forced to be conservative.
If we had better control, or a more forgiving allocator, we could make
different choices.
Also, why try to sort GBs of data in 20 MB? Yet, this is the test case that
had to be solved and that this particular fix enables.
I'm open to suggestions for better solutions; this is a very tricky area...
> Roll-up of final fixes for managed sort
> ---------------------------------------
>
> Key: DRILL-5284
> URL: https://issues.apache.org/jira/browse/DRILL-5284
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.10.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Fix For: 1.10.0
>
>
> The managed external sort was introduced in DRILL-5080. Since that time,
> extensive testing has identified a number of minor fixes and improvements.
> Given the long PR cycles, it is not practical to spend a week or two to do a
> PR for each fix individually. This ticket represents a roll-up of a
> combination of a number of fixes. Small fixes are listed here, larger items
> appear as sub-tasks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
