[
https://issues.apache.org/jira/browse/DRILL-5815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190800#comment-16190800
]
ASF GitHub Bot commented on DRILL-5815:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/960#discussion_r142579717
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -351,7 +351,35 @@
MAX_QUERY_MEMORY_PER_NODE_KEY, 1024 * 1024, Long.MAX_VALUE);
/**
- * Minimum memory alocated to each buffered operator instance.
+ * Alternative way to compute per-query-per-node memory as a percent
+ * of the total available system memory.
+ * <p>
+ * Suggestion for computation.
+ * <ul>
+ * <li>Assume an allowance for non-managed operators. Default assumption:
+ * 50%</li>
+ * <li>Assume a desired number of concurrent queries. Default assumption:
+ * 10.</li>
+ * <li>The value of this parameter is<br>
+ * (1 - non-managed allowance) / concurrency</li>
--- End diff --
I thought about that. Then I remembered that we use ratios elsewhere such
as `planner.cpu_load_average`, `planner.affinity_factor`, and
`planner.memory.hash_agg_table_factor`. I didn't see any place we use the
percent-as-integer form.
> Provide option to set query memory as percent of total
> ------------------------------------------------------
>
> Key: DRILL-5815
> URL: https://issues.apache.org/jira/browse/DRILL-5815
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.11.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Fix For: 1.12.0
>
>
> Drill provides a parameter to set the memory per query as a static number
> which defaults to 2 GB. This number is a wonderful setting for the default
> Drillbit configuration of 8 GB heap; it allows 2-3 concurrent queries. But,
> as Drillbit memory increases, the default becomes a bit constraining. While
> users can change the setting, they seldom do.
> In addition, provide an option that sets memory as a percent of total memory.
> If the allocation is 10%, say, and total memory is 128 GB, then each query
> gets ~13GB, which is a big improvement.
> The existing option acts as a floor: the query must receive at least that
> much memory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
