[
https://issues.apache.org/jira/browse/DRILL-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070479#comment-16070479
]
ASF GitHub Bot commented on DRILL-5601:
---------------------------------------
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/860
# Additional Revisions
The code contains an optimization for short queries. If the query has just
one batch then the sort need not copy that batch to produce the output.
Instead, the sort simply passes along the input batch along with an SV2. This
commit fixes some issues with transferring that single, buffered, input batch
to the sort’s output container.
While reproducing problems found by QA, it turned out to be useful to be
able to parse SQL statements from a file in the context of the test framework.
Added test methods for this purpose.
The “AllocationHelper” class turns out to throw an exception if asked to
allocate a vector with zero elements. Since a zero-size can occasionally come
from the record batch sizer (via the “smart allocation helper”, special code
was added to handle this case.
Backed out some debugging code that accidentally appeared in the original
PR.
With this set of improvements, the revised, managed sort has become more
stable and reliable than the original version. As a result, this commit enables
the managed sort by default.
> Rollup of External Sort memory management fixes
> -----------------------------------------------
>
> Key: DRILL-5601
> URL: https://issues.apache.org/jira/browse/DRILL-5601
> Project: Apache Drill
> Issue Type: Task
> Affects Versions: 1.11.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> Rollup of a set of specific JIRA entries that all relate to the very
> difficult problem of managing memory within Drill in order for the external
> sort to stay within a memory budget. In general, the fixes relate to better
> estimating memory used by the three ways that Drill allocates vector memory
> (see DRILL-5522) and to predicting the size of vectors that the sort will
> create, to avoid repeated realloc-copy cycles (see DRILL-5594).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
