[
https://issues.apache.org/jira/browse/DRILL-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481446#comment-16481446
]
Paul Rogers commented on DRILL-6429:
------------------------------------
[~timothyfarkas], good plan. BTW, forgot to mention, leniency is only turned on
in production, it is off by default in testing (so we can catch errors.)
(Assertion state is used as the trigger.) Just go ahead and add a config option
or something to force leniency off in your tests.
One suggestion: memory planning must be done up front as reactive planning will
fail when dealing with queries with a sort: sorts start with little memory use,
but grow to maximum use just before spilling, then drop their use. Better to
budget for the full amount allocated to each operator.
Drill makes a very strong assumption: all nodes are identical. If we use
up-front planning, we already know the amount of memory available on each node.
Choosing a memory-per fragment, and allocating memory per fragment (uniformly
across nodes) avoids the need to track per-node memory fluctuations.
Then, we know that, statistically, not all fragments will use their full memory
allotment at all times. This provides some wiggle-room.
To achieve your goal, just do testing with leniency turned off. Since tests do
pass, we know that, at least for our test cases, the memory management in sort
does work. It will be much better once the work is done to control batch sizes
elsewhere. You can decide later if you still need the margin of safety provided
by leniency. Would be great if not.
Looking forward to the result of all this great work!
> SortImpl Should Not Use BufferAllocator.setLenient()
> ----------------------------------------------------
>
> Key: DRILL-6429
> URL: https://issues.apache.org/jira/browse/DRILL-6429
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Timothy Farkas
> Assignee: Timothy Farkas
> Priority: Major
>
> The Sort operator should always obey its memory limit. This is required for
> resource management.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
