[
https://issues.apache.org/jira/browse/DRILL-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959941#comment-15959941
]
Paul Rogers commented on DRILL-4181:
------------------------------------
The sort has been modified to work in low-memory, and to make better use of
memory in all conditions.
The next step is to better assign memory to each sort, which must start with
assigning memory to queries.
A key challenge for memory limits is to know how much memory to give to each
query. This, in turn, depends on the number of queries to run. 1? 10? 100? If
we run 20 queries, say, then, ceteris paribus, each query should get 1/20th of
memory.
We need a way to learn the number of active queries. In particular, we need a
way to LIMIT the number since we can’t change the memory allocation once a
planning decision is made, if it turns out that 100 more queries arrive.
The answer is to use the existing Zookeeper queues to set a limit on the number
of active queries. We already know the total direct memory available. A bit of
math tells us how much memory to assign per admitted query.
The next step is to adapt the existing memory planning code to use the memory
limit as the amount of memory to give each query.
At the implementation level, the code for queues and memory assignment is
factored out into a new resource manager. The code provides three versions:
null (no query, old-style memory assignment), existing Zookeeper, and a new
single-node, in-process queue for testing. An RM framework lets advanced users
create their own queueing solution.
All of this entailed moving existing code around to set things right for the
above work.
> Improve memory limit assignment for Sort operators
> --------------------------------------------------
>
> Key: DRILL-4181
> URL: https://issues.apache.org/jira/browse/DRILL-4181
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.4.0
> Reporter: Aman Sinha
> Assignee: Paul Rogers
>
> Currently, the max_query_memory_per_node is divided among all the Sort
> operators in the query plan, regardless of whether the Sort operators are
> executing at the same time. Since Sort is a blocking operator, the number of
> concurrent Sorts running is limited. Thus, the Sort on both sides of a
> MergeJoin could be concurrently executing but a Sort that occurs above the
> MergeJoin cannot start until the MergeJoin produces a row which is only when
> both the child Sorts have produced a row.
> Due to the conservative estimate, we have seen queries such as TPC-H Q8 with
> forced MergeJoin run out-of-memory even with high max_query_memory_per_node.
> This query plan has 15 Sort operators and with max_width = 23,
> max_query_memory_per_node = 20GB, each Sort gets only 63MB (20GB/(23*15)).
> We should improve the algorithm for computing the memory limit.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
