[
https://issues.apache.org/jira/browse/DRILL-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15849327#comment-15849327
]
ASF GitHub Bot commented on DRILL-5080:
---------------------------------------
Github user Ben-Zvi commented on a diff in the pull request:
https://github.com/apache/drill/pull/717#discussion_r99045403
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -177,13 +177,47 @@ drill.exec: {
sort: {
purge.threshold : 1000,
external: {
- batch.size : 4000,
+ // Drill uses the managed External Sort Batch by default.
+ // Set this to true to use the legacy, unmanaged version.
+ // Disabled in the intial commit, to be enabled after
+ // tests are committed.
+ disable_managed: true
+ // Limit on the number of batches buffered in memory.
+ // Primarily for testing.
+ // 0 = unlimited
+ batch_limit: 0
+ // Limit on the amount of memory used for xsort. Overrides the
+ // value provided by Foreman. Primarily for testing.
+ // 0 = unlimited, Supports HOCON memory suffixes.
+ mem_limit: 0
+ // Limit on the number of spilled batches that can be merged in
+ // a single pass. Limits the number of open file handles.
+ // 0 = unlimited
+ merge_limit: 0
spill: {
- batch.size : 4000,
- group.size : 40000,
- threshold : 40000,
- directories : [ "/tmp/drill/spill" ],
- fs : "file:///"
+ // Deprecated for managed xsort; used only by legacy xsort
+ group.size: 40000,
+ // Deprecated for managed xsort; used only by legacy xsort
+ threshold: 40000,
+ // Minimum number of in-memory batches to spill per spill file
+ // Affects only spilling from memory to disk.
+ // Primarily for testing.
+ min_batches: 2,
+ // Maximum number of in-memory batches to spill per spill file
+ // Affects only spilling from memory to disk.
+ // Primarily for testing.
+ // 0 = unlimited
+ max_batches: 0,
+ // File system to use. Local file system by default.
+ fs: "file:///"
+ // List of directories to use. Directories are created
--- End diff --
List of (comma separated) directories....
> Create a memory-managed version of the External Sort operator
> -------------------------------------------------------------
>
> Key: DRILL-5080
> URL: https://issues.apache.org/jira/browse/DRILL-5080
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.8.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Fix For: 1.10.0
>
> Attachments: ManagedExternalSortDesign.pdf
>
>
> We propose to create a "managed" version of the external sort operator that
> works to a clearly-defined memory limit. Attached is a design specification
> for the work.
> The project will include fixing a number of bugs related to the external
> sort, include as sub-tasks of this umbrella task.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
