ASF GitHub Bot commented on DRILL-6148:

Github user paul-rogers commented on a diff in the pull request:

    --- Diff: 
    @@ -59,6 +59,7 @@ public static void setup() throws Exception {
         ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher)
             .configProperty(ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 1) // 
             .configProperty(ExecConstants.EXTERNAL_SORT_SPILL_GROUP_SIZE, 1) 
// Unmanaged
    +        .configProperty(ExecConstants.EXTERNAL_SORT_MAX_MEMORY, 10 * 1024 
* 1024) //use less memory for sorting.
    --- End diff --
    There is something very wrong here. This means that the memory calcs are 
not working and users will try to sort in unusually small amounts of memory. 
The query probably has one sort. It should have just a few threads. (We might 
want to control this if we don't now.) Take the 60 MB of memory, divide by the 
number of sorts (1) and you get 60 MB for the sort in this case. If we are 
getting 40, then something is wrong. Let's find out where things went off the 
    The minimum buffer op memory is supposed to set a floor for those cases in 
which a query has, say, 5 sorts on a machine with 30 cores and we try to divide 
the 2 GB by 150 and end up trying to sort in, say, 13 MB of memory. 

> TestSortSpillWithException is sometimes failing.
> ------------------------------------------------
>                 Key: DRILL-6148
>                 URL: https://issues.apache.org/jira/browse/DRILL-6148
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Tools, Build & Test
>    Affects Versions: 1.12.0
>            Reporter: Hanumath Rao Maduri
>            Assignee: Hanumath Rao Maduri
>            Priority: Minor
>              Labels: ready-to-commit
>             Fix For: 1.13.0
> TestSortSpillWithException#testSpillLeakManaged is sometimes failing. However 
> for some reason this is being observed only in one of my branch. 
> TestSpillLeakManaged tests for leak when an exception is thrown during the 
> spilling of the rows in ExternalSort. In the test failure case it happens 
> that ExternalSort is able to sort the data with the given memory and not 
> spill at all. Hence the injection interruption path is not hit at all and 
> hence no exception is thrown.
> The test case should use drill.exec.sort.external.mem_limit to force it to 
> use as less memory as possible so as to test the case.

This message was sent by Atlassian JIRA

Reply via email to