[
https://issues.apache.org/jira/browse/DRILL-5815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206336#comment-16206336
]
ASF GitHub Bot commented on DRILL-5815:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/960#discussion_r144926648
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/util/MemoryAllocationUtilities.java
---
@@ -56,30 +59,90 @@ public static void
setupBufferedOpsMemoryAllocations(final PhysicalPlan plan, fi
}
// if there are any sorts, compute the maximum allocation, and set it
on them
- if (bufferedOpList.size() > 0) {
- final OptionManager optionManager = queryContext.getOptions();
- double cpu_load_average =
optionManager.getOption(ExecConstants.CPU_LOAD_AVERAGE);
- final long maxWidth =
optionManager.getOption(ExecConstants.MAX_WIDTH_PER_NODE);
- final long maxWidthPerNode =
ExecConstants.MAX_WIDTH_PER_NODE.computeMaxWidth(cpu_load_average,maxWidth);
- long maxAllocPerNode = Math.min(DrillConfig.getMaxDirectMemory(),
-
queryContext.getConfig().getLong(RootAllocatorFactory.TOP_LEVEL_MAX_ALLOC));
- maxAllocPerNode = Math.min(maxAllocPerNode,
-
optionManager.getOption(ExecConstants.MAX_QUERY_MEMORY_PER_NODE_KEY).num_val);
- final long maxOperatorAlloc = maxAllocPerNode /
(bufferedOpList.size() * maxWidthPerNode);
- logger.debug("Max buffered operator alloc: {}", maxOperatorAlloc);
-
- // User configurable option to allow forcing minimum memory.
- // Ensure that the buffered ops receive the minimum memory needed to
make progress.
- // Without this, the math might work out to allocate too little
memory.
- final long opMinMem =
queryContext.getOptions().getOption(ExecConstants.MIN_MEMORY_PER_BUFFERED_OP_KEY).num_val;
-
- for(final PhysicalOperator op : bufferedOpList) {
-
- long alloc = Math.max(maxOperatorAlloc, op.getInitialAllocation());
- alloc = Math.max(alloc, opMinMem);
- op.setMaxAllocation(alloc);
- }
- }
plan.getProperties().hasResourcePlan = true;
+ if (bufferedOpList.isEmpty()) {
+ return;
+ }
+
+ // Setup options, etc.
+
+ final OptionManager optionManager = queryContext.getOptions();
+ final long directMemory = DrillConfig.getMaxDirectMemory();
+
+ // Compute per-node, per-query memory.
+
+ final long maxAllocPerNode =
computeQueryMemory(queryContext.getConfig(), optionManager, directMemory);
+ logger.debug("Memory per query per node: {}", maxAllocPerNode);
+
+ // Now divide up the memory by slices and operators.
+
+ final long opMinMem = computeOperatorMemory(optionManager,
maxAllocPerNode, bufferedOpList.size());
--- End diff --
MEP? Do you mean Drill 1.13?
The dynamic memory idea needs thought. In fact, this is exactly what we do
today: each operator grabs the memory it needs; and sends messages downstream
if it is memory-constrained. The reason we are doing so much work is that the
existing technique is very, very hard to get right.
We are moving toward fixed budgets. Once that works, we can set a fixed
budget per fragment and "slosh" memory between operators in the same fragment
based on an empirical study of memory use.
It will be far, far harder to slosh memory between fragments as a fragment
lifecycle, with buffering operators, is caret-shaped: ramp up to a peak, then
ramp down. Coordination is needed to know, if a fragment is using only 100 MB
of 1 GB, that this is because the fragment just started, and is ramping up (to
full 1 GB use), vs. the operator is ramping down and won't be using additional
memory.
Since all of this is a big project, the present change tries to do the best
we can with the situation we currently find ourselves in.
> Provide option to set query memory as percent of total
> ------------------------------------------------------
>
> Key: DRILL-5815
> URL: https://issues.apache.org/jira/browse/DRILL-5815
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.11.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.12.0
>
>
> Drill provides a parameter to set the memory per query as a static number
> which defaults to 2 GB. This number is a wonderful setting for the default
> Drillbit configuration of 8 GB heap; it allows 2-3 concurrent queries. But,
> as Drillbit memory increases, the default becomes a bit constraining. While
> users can change the setting, they seldom do.
> In addition, provide an option that sets memory as a percent of total memory.
> If the allocation is 10%, say, and total memory is 128 GB, then each query
> gets ~13GB, which is a big improvement.
> The existing option acts as a floor: the query must receive at least that
> much memory.
> *DOCUMENTATION*
> New option should be documented - planner.memory.percent_per_query
> Default - 0.05 (which is equivalent to 5 %)
> To disable feature set to 0.
> More information can be found in pull request description.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
