[
https://issues.apache.org/jira/browse/DRILL-5808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16173659#comment-16173659
]
Boaz Ben-Zvi commented on DRILL-5808:
-------------------------------------
Implementing a dynamic system of memory "quotas" would work better and
address the above memory limitations as well. One simple implementation of this
system: Divide the memory available in 2, and keep one half as "reserve". The
other half can be equally divided among all the buffered operators as their
"quotas". Then enhance the allocator that in case it hits the limit, it should
request more memory from the "reserve" (if not available then OOM). Also when
each operator needs no more memory (e.g., Hash Join finished the build phase) ,
then this operator can return the leftover quota to the "reserve".
> Reduce memory allocator strictness for "managed" operators
> ----------------------------------------------------------
>
> Key: DRILL-5808
> URL: https://issues.apache.org/jira/browse/DRILL-5808
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.11.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Fix For: 1.12.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Drill 1.11 and 1.12 introduce new "managed" versions of the sort and hash agg
> that enforce memory limits, spilling to disk when necessary.
> Drill's internal memory system is very "lumpy" and unpredictable. The
> operators have no control over the incoming batch size; an overly large batch
> can cause the operator to exceed its memory limit before it has a chance to
> do any work.
> Vector allocations grow in power-of-two sizes. Adding a single record can
> double the memory allocated to a vector.
> Drill has no metadata, so operators cannot predict the size of VarChar
> columns nor the cardinality of arrays. The "Record Batch Sizer" tries to
> extract this information on each batch, but it works with averages, and
> specific column patterns can still throw off the memory calculations. (For
> example, having a series of very wide columns for A-M and very narrow columns
> for N-Z will cause a moderate average. But, once sorted, the A-M rows, and
> batches, will be much larger than expected, causing out-of-memory errors.)
> At present, if an operator is wrong in its memory usage by a single byte, the
> entire query is killed. That is, the user pays the death penalty (of queries)
> for poor design decisions within Drill. This leads to a less-than-optimal
> user experience.
> The proposal here is to make the memory allocator less strict for "managed"
> operators.
> First, we recognize that the managed operators do attempt to control memory
> and, if designed well, will, on average hit their targets.
> Second, we recognize that, due to the lumpiness issues above, any single
> operator may exceed, or be under, the configured maximum memory.
> Given this, the proposal here is:
> 1. An operator identifies itself as managed to the memory allocator.
> 2. In managed mode, the allocator has soft limits. It emits a warning to the
> log when the limit is exceeded.
> 3. For safety, in managed mode, the allocator enforces a hard limit larger
> than the configured limit.
> The enforcement limit might be:
> * For memory sizes < 100MB, up to 2x the configured limit.
> * For larger memory sizes, no more than 100MB over the configured limit.
> The exact numbers can be made configurable.
> Now, during testing, scripts should look for over-memory warnings. Each
> should be fixed as we fix OOM issues today. But, during production, user
> queries are far less likely to fail due to any remaining corner cases that
> throw off the memory calculations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
