[
https://issues.apache.org/jira/browse/DRILL-6030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297218#comment-16297218
]
ASF GitHub Bot commented on DRILL-6030:
---------------------------------------
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1075
One additional thought. This bug was found when sorting 18 GB of data in 8
GB of memory. That is, a case in which the sort must spill.
What happens in the case in which the 18 GB of data is sorted in, say, 20
GB of memory (an in-memory sort)? We don't want the merge limit to force a
spill in this case; kind of defeats the purpose of an in-memory sort.
So:
1. Does the limit affect in memory sort? If so, we need to revise the
solution.
2. Does the in-memory sort suffer from a similar performance issue? If so,
we need to revise the in memory sort.
One possible solution is to:
1. Defer sorting of individual batches until necessary.
2. Sort batches just before spilling.
3. If all batches fit in memory, do a single, combined sort (using an SV4).
> Managed sort should minimize number of batches in a k-way merge
> ---------------------------------------------------------------
>
> Key: DRILL-6030
> URL: https://issues.apache.org/jira/browse/DRILL-6030
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Vlad Rozov
> Assignee: Vlad Rozov
>
> The time complexity of the algorithm is O(n*k*log(k)) where k is a number of
> batches to merge and n is a number of records in each batch (assuming equal
> size batches). As n*k is the total number of record to merge and it can be
> quite large, minimizing k should give better results.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
