[
https://issues.apache.org/jira/browse/DRILL-6030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297169#comment-16297169
]
ASF GitHub Bot commented on DRILL-6030:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1075#discussion_r157830252
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/managed/SortConfig.java
---
@@ -84,7 +85,7 @@ public SortConfig(DrillConfig config) {
if (limit > 0) {
mergeLimit = Math.max(limit, MIN_MERGE_LIMIT);
} else {
- mergeLimit = Integer.MAX_VALUE;
+ mergeLimit = DEFAULT_MERGE_LIMIT;
--- End diff --
The merge limit is already a config option. (I'd forgotten about that.) The
comment on the config option says "Limit on the number of spilled batches that
can be merged in a single pass." So, let's just set that default (in
`drill-override-conf`) to your new value of 128 and leave the code here
unchanged.
> Managed sort should minimize number of batches in a k-way merge
> ---------------------------------------------------------------
>
> Key: DRILL-6030
> URL: https://issues.apache.org/jira/browse/DRILL-6030
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Vlad Rozov
> Assignee: Vlad Rozov
>
> The time complexity of the algorithm is O(n*k*log(k)) where k is a number of
> batches to merge and n is a number of records in each batch (assuming equal
> size batches). As n*k is the total number of record to merge and it can be
> quite large, minimizing k should give better results.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
