[
https://issues.apache.org/jira/browse/DRILL-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959735#comment-15959735
]
ASF GitHub Bot commented on DRILL-5325:
---------------------------------------
GitHub user paul-rogers opened a pull request:
https://github.com/apache/drill/pull/808
DRILL-5325: Unit tests for the managed sort
Uses the sub-operator test framework (DRILL-5318), including the test
row set abstraction (DRILL-5323) to enable unit testing of the
“managed” external sort. This PR allows early review of the code, but
cannot be pulled until the dependencies (mentioned above) are pulled.
Refactors the external sort code into small chunks that can be unit
tested, then “wraps” that code in tests for all interesting data types,
record batch sizes, and so on.
Refactors some of the operator definitions to more easily allow
programmatic setup in the unit tests.
Fixes a number of bugs discovered by the unit tests. The biggest
changes were in the new code: the code that computes spilling and
merging based on memory levels.
Otherwise, although GitHub will show many files change, most of the
changes are simply moving blocks of code around to create smaller units
that can be tested independently.
Includes a refactoring of the code that does spilling, along with a
complete set of low-level unit tests.
Since this PR is dependent on others, it cannot (yet) be built on its
own. The PR will be rebased once the dependencies are in master so that
it will build prior to commit to master.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/paul-rogers/drill DRILL-5325
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/808.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #808
----
commit ccd0f15889b8cc7ec1011a7a7f8126b16c6bbf40
Author: Paul Rogers <[email protected]>
Date: 2017-04-06T20:57:19Z
DRILL-5325: Unit tests for the managed sort
Uses the sub-operator test framework (DRILL-5318), including the test
row set abstraction (DRILL-5323) to enable unit testing of the
“managed” external sort. This PR allows early review of the code, but
cannot be pulled until the dependencies (mentioned above) are pulled.
Refactors the external sort code into small chunks that can be unit
tested, then “wraps” that code in tests for all interesting data types,
record batch sizes, and so on.
Refactors some of the operator definitions to more easily allow
programmatic setup in the unit tests.
Fixes a number of bugs discovered by the unit tests. The biggest
changes were in the new code: the code that computes spilling and
merging based on memory levels.
Otherwise, although GitHub will show many files change, most of the
changes are simply moving blocks of code around to create smaller units
that can be tested independently.
Includes a refactoring of the code that does spilling, along with a
complete set of low-level unit tests.
Since this PR is dependent on others, it cannot (yet) be built on its
own. The PR will be rebased once the dependencies are in master so that
it will build prior to commit to master.
----
> Implement sub-operator unit tests for managed external sort
> -----------------------------------------------------------
>
> Key: DRILL-5325
> URL: https://issues.apache.org/jira/browse/DRILL-5325
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Tools, Build & Test
> Affects Versions: 1.11.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> Validate the proposed sub-operator test framework, by creating low-level unit
> tests for the managed version of the external sort.
> The external sort has a small number of existing tests, but those tests are
> quite superficial; the "managed sort" project found many bugs. The managed
> sort itself was tested with ad-hoc system-level tests created using the new
> "cluster fixture" framework. But, again, such tests could not reach deep
> inside the sort code to exercise very specific conditions.
> As a result, we spent far too much time using QA functional tests to identify
> specific code issues.
> Using the sub-opeator unit test framework, we can instead test each bit of
> functionality at the unit test level.
> If doing so works, and is practical, it can serve as a model for other
> operator testing projects.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
