[
https://issues.apache.org/jira/browse/DRILL-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481353#comment-16481353
]
Timothy Farkas commented on DRILL-6429:
---------------------------------------
[~paul-rogers] We want to start implementing true resource management for
memory. This would entail a resource monitor that keeps track of all available
memory on each node, and planning queries to fit within available memory. In
order to do this, we need to be sure that when we tell an operator to run
within X amount of memory it will do so. If we tell an operator to run within X
and it decides to use more memory, then our resource monitor would be
inaccurate and we can get queries that unexpectedly OOM.
It is currently possibly to do this reliable with heuristics. I have done it
for HashJoin, which withstood a battery of tests without using setLenient().
See
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinMemoryCalculatorImpl.java
. Furthermore as the batch sizing project you and Padma have been working on
progresses, we will eventually be able to move away from heuristics and deal
with certainties.
The goal here is to look at SortImpl and see if it can be augmented to not
require setLenient, so our resource management system can make the assumption
that all memory limits are obeyed and are absolute. If we can make sure that
assumption is true, the difficulty of implementing a reliable resource
management system drastically reduces. I suspect it is possible to do since it
has been done for HashJoin, and I suspect it would not involve any major
changes to Sort.
> SortImpl Should Not Use BufferAllocator.setLenient()
> ----------------------------------------------------
>
> Key: DRILL-6429
> URL: https://issues.apache.org/jira/browse/DRILL-6429
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Timothy Farkas
> Assignee: Timothy Farkas
> Priority: Major
>
> The Sort operator should always obey its memory limit. This is required for
> resource management.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
