[
https://issues.apache.org/jira/browse/DRILL-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151320#comment-16151320
]
ASF GitHub Bot commented on DRILL-5716:
---------------------------------------
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/928
Responses to the general comments:
> I think we need to limit the number of queries waiting for queues.
Please see the detailed notes. The ZK-based mechanism does not (AFAIK)
provide the means to limit queue size. This is why this feature is preliminary,
and may be why it has been disabled, though it has been available in the code
for several years.
> For short queries, this feature might add more overhead/latency (going to
zookeeper, extra synchronization mechanisms added etc.). This will be a problem
with high concurrency. Did we quantify that ?
This is not a feature for situations of very high concurrency (= very small
queries). It is a feature for situations in which a cluster runs enough
distributed queries that the cluster becomes overloaded. For the case with
1000s of small, single-node queries, a app could use the framework provided to
implement an in-process queue. In fact, one exists, but it is intended for unit
tests.
> Not being to cancel once enqueued is also a problem. Can we do that by
having cancel release the semaphore ?
Yes, cancelation is an issue. Since the ZK-based queues were an existing
feature, I didn't dive too deeply into how they handle query cancellation. This
would be a good area for QA to exercise.
> Also, does semaphore count give an indication how many are waiting in the
queue ?
There is no ready means to determine this, though I suspect that something
could be put together that queries the internal znode and counts the ephemeral
nodes that represent waiting processes. Doing so would be handy to display
queue size in the Drill web UI. This would be a good enhancement once we prove
that the basic mechanism works.
> Queue-based memory assignment for buffering operators
> -----------------------------------------------------
>
> Key: DRILL-5716
> URL: https://issues.apache.org/jira/browse/DRILL-5716
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.12.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
>
> Apache Drill already has a queueing feature based on ZK semaphores. We did a
> bit of testing to show that the feature does, in fact work. We propose to
> enhance the feature with some light revisions to make work with the "managed"
> external sort and the newly-added spilling feature for the hash agg operator.
> The key requirement is to build on what we have for now; we may want to
> tackle a larger project to create a more complete solution later.
> Existing functionality:
> * Two ZK-based queues called the “small” and “large” query queues.
> * A threshold, call it T, given as a query cost, to determine the queue into
> which a query will go.
> * Admit levels for the two queues: call them Qs and Ql.
> Basically, when a query comes in:
> * Plan the query as usual.
> * Obtain the final query cost from the planner, call this C.
> * If C<T, the query goes into the small queue, else it goes into the large
> queue.
> * Suppose the small queue. Ask ZK if the query can run.
> * ZK checks if Qs queries are already running. If so, the query waits, else
> the query runs.
> The proposed changes include:
> * Refactor the code to provide a queueing API that supports a variety of
> queuing mechanisms.
> * Provide three: the null queue (default), an in-process queue (for testing)
> and the ZK queues.
> * Modify the query profile web UI to show two new bits of information about
> queues:
> - The queue to which the query was sent.
> - The total planning cost.
> * Modify the query profile web UI to show two memory assignment numbers:
> - Total memory allocated to the query
> - Memory per sort or hash-add operator
> Then, add to the queue mechanism the ability to do memory assignment:
> * Provide a weight, W: every small query gets 1 unit, every large query gets
> W units.
> * Use the queue admit levels to determine total units: U = Qs + W * Ql.
> * Obtain total direct memory from the system. M.
> * Subtract a reserve percent R for overhead.
> * Do the math to get the memory per query for each query:
> * For the small queue: (M - R) / U
> * For the large queue: (M - R) / U * W
> * Use this memory amount as the “memory per query” number in the existing
> sort/hash-agg memory assignment (instead of the fixed 2 GB.)
> The result will be a nice incremental addition to what we already have, and
> should make it a bit easier people to actually use the feature (because they
> can see the planning numbers and see the queues used, allowing them to
> effectively tune the system.)
> The API used for the above features also allow third parties to add on a more
> robust admission control feature as needed, perhaps tying into an existing
> queueing mechanism of their choice.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
