[
https://issues.apache.org/jira/browse/DRILL-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-5008:
-------------------------------
Description:
ExternalSortBatch provides a spillable sort operator for Drill. The code works
fine, but can be a hard to follow and understand. Make the following changes to
improve ease-of-use for developers:
1. Refactor the large methods into bite-sized chunks to aid understanding.
2. Provide additional explanation of the theory and operation of this operator.
The memory limit cases for the spill-needed test seem inconsistent:
For the test for in-memory sort:
{code}
long currentlyAvailable = popConfig.getMaxAllocation() -
oAllocator.getAllocatedMemory();
{code}
For reaching the memory limit:
{code}
oAllocator.getAllocatedMemory() > .95 * oAllocator.getLimit()
{code}
That is, one uses `oAllocator.getLimit` ("the current maximum limit this
allocator imposes"), the other uses `popConfig.getMaxAllocation` ("The maximum
memory this operator can allocate".)
was:
ExternalSortBatch provides a spillable sort operator for Drill. The code works
fine, but can be a hard to follow and understand. Make the following changes to
improve ease-of-use for developers:
1. Refactor the large methods into bite-sized chunks to aid understanding.
2. Provide additional explanation of the theory and operation of this operator.
The memory limit cases for the spill-needed test seem inconsistent:
For the test for in-memory sort:
{code}
long currentlyAvailable = popConfig.getMaxAllocation() -
oAllocator.getAllocatedMemory();
{code}
For reaching the memory limit:
{code}
oAllocator.getAllocatedMemory() > .95 * oAllocator.getLimit()
{code}
That is, one uses `oAllocator.getLimit` ("the current maximum limit this
allocator imposes"), the other uses `popConfig.getMaxAllocation` (not
documented, but the memory limit for the operator as defined by the Foreman.)
> Refactor, document and simplify ExternalSortBatch
> -------------------------------------------------
>
> Key: DRILL-5008
> URL: https://issues.apache.org/jira/browse/DRILL-5008
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.8.0
> Reporter: Paul Rogers
> Priority: Minor
>
> ExternalSortBatch provides a spillable sort operator for Drill. The code
> works fine, but can be a hard to follow and understand. Make the following
> changes to improve ease-of-use for developers:
> 1. Refactor the large methods into bite-sized chunks to aid understanding.
> 2. Provide additional explanation of the theory and operation of this
> operator.
> The memory limit cases for the spill-needed test seem inconsistent:
> For the test for in-memory sort:
> {code}
> long currentlyAvailable = popConfig.getMaxAllocation() -
> oAllocator.getAllocatedMemory();
> {code}
> For reaching the memory limit:
> {code}
> oAllocator.getAllocatedMemory() > .95 * oAllocator.getLimit()
> {code}
> That is, one uses `oAllocator.getLimit` ("the current maximum limit this
> allocator imposes"), the other uses `popConfig.getMaxAllocation` ("The
> maximum memory this operator can allocate".)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
