[ https://issues.apache.org/jira/browse/DRILL-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kunal Khatua closed DRILL-5716. ------------------------------- Verified in Drill 1.12.0 > Queue-based memory assignment for buffering operators > ----------------------------------------------------- > > Key: DRILL-5716 > URL: https://issues.apache.org/jira/browse/DRILL-5716 > Project: Apache Drill > Issue Type: Improvement > Affects Versions: 1.11.0 > Reporter: Paul Rogers > Assignee: Paul Rogers > Priority: Major > Labels: doc-impacting > Fix For: 1.12.0 > > > Apache Drill already has a queueing feature based on ZK semaphores. We did a > bit of testing to show that the feature does, in fact work. We propose to > enhance the feature with some light revisions to make work with the "managed" > external sort and the newly-added spilling feature for the hash agg operator. > The key requirement is to build on what we have for now; we may want to > tackle a larger project to create a more complete solution later. > Existing functionality: > * Two ZK-based queues called the “small” and “large” query queues. > * A threshold, call it T, given as a query cost, to determine the queue into > which a query will go. > * Admit levels for the two queues: call them Qs and Ql. > Basically, when a query comes in: > * Plan the query as usual. > * Obtain the final query cost from the planner, call this C. > * If C<T, the query goes into the small queue, else it goes into the large > queue. > * Suppose the small queue. Ask ZK if the query can run. > * ZK checks if Qs queries are already running. If so, the query waits, else > the query runs. > The proposed changes include: > * Refactor the code to provide a queueing API that supports a variety of > queuing mechanisms. > * Provide three: the null queue (default), an in-process queue (for testing) > and the ZK queues. > * Modify the query profile web UI to show two new bits of information about > queues: > - The queue to which the query was sent. > - The total planning cost. > * Modify the query profile web UI to show two memory assignment numbers: > - Total memory allocated to the query > - Memory per sort or hash-add operator > Then, add to the queue mechanism the ability to do memory assignment: > * Provide a weight, W: every small query gets 1 unit, every large query gets > W units. > * Use the queue admit levels to determine total units: U = Qs + W * Ql. > * Obtain total direct memory from the system. M. > * Subtract a reserve percent R for overhead. > * Do the math to get the memory per query for each query: > * For the small queue: (M - R) / U > * For the large queue: (M - R) / U * W > * Use this memory amount as the “memory per query” number in the existing > sort/hash-agg memory assignment (instead of the fixed 2 GB.) > The result will be a nice incremental addition to what we already have, and > should make it a bit easier people to actually use the feature (because they > can see the planning numbers and see the queues used, allowing them to > effectively tune the system.) > The API used for the above features also allow third parties to add on a more > robust admission control feature as needed, perhaps tying into an existing > queueing mechanism of their choice. -- This message was sent by Atlassian JIRA (v7.6.3#76005)