Github user paul-rogers commented on a diff in the pull request:
    --- Diff: 
    @@ -59,6 +59,7 @@ public static void setup() throws Exception {
         ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher)
             .configProperty(ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 1) // 
             .configProperty(ExecConstants.EXTERNAL_SORT_SPILL_GROUP_SIZE, 1) 
// Unmanaged
    +        .configProperty(ExecConstants.EXTERNAL_SORT_MAX_MEMORY, 10 * 1024 
* 1024) //use less memory for sorting.
    --- End diff --
    There is something very wrong here. This means that the memory calcs are 
not working and users will try to sort in unusually small amounts of memory. 
The query probably has one sort. It should have just a few threads. (We might 
want to control this if we don't now.) Take the 60 MB of memory, divide by the 
number of sorts (1) and you get 60 MB for the sort in this case. If we are 
getting 40, then something is wrong. Let's find out where things went off the 
    The minimum buffer op memory is supposed to set a floor for those cases in 
which a query has, say, 5 sorts on a machine with 30 cores and we try to divide 
the 2 GB by 150 and end up trying to sort in, say, 13 MB of memory. 


Reply via email to