[
https://issues.apache.org/jira/browse/DRILL-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142605#comment-16142605
]
ASF GitHub Bot commented on DRILL-5741:
---------------------------------------
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/922
-1
This seems like a good idea. But, I don't think this is the way we should
go for reasons identified in the JIRA.
For one, we count on the user to provide a parameter to describe their
system memory to work around a problem where the user provides another
parameter that is not consistent with their system memory. Why would the user
set one correctly but the other wrong?
This change overly complicates the shell scripts. This change conflicts
with the way that YARN, Warden and other tools allocate memory.
A better solution would be:
1. When the Drillbit starts, obtain (from the system) the available memory.
(Or, if running under YARN, obtain the YARN-allocated memory from an env var.
Same for Warden or Mesos or...)
2. Do the math on the memory allocations.
3. If the memory is too large, write a warning to the log file.
> During startup Drill should not exceed the available memory
> -----------------------------------------------------------
>
> Key: DRILL-5741
> URL: https://issues.apache.org/jira/browse/DRILL-5741
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
> Affects Versions: 1.11.0
> Reporter: Kunal Khatua
> Assignee: Kunal Khatua
> Fix For: 1.12.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Currently, during startup, a Drillbit can be assigned large values for the
> following:
> * Xmx (Heap)
> * XX:MaxDirectMemorySize
> * XX:ReservedCodeCacheSize
> * XX:MaxPermSize
> All of this, potentially, can exceed the available memory on a system when a
> Drillbit is under heavy load. It would be good to have the Drillbit ensure
> during startup itself that the cumulative value of these parameters does not
> exceed a pre-defined upper limit for the Drill process.
> The proposal is to have the
> [runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit]
> script look for an additional environment variable:
> {{DRILLBIT_MAX_PROC_MEM}}
> The parameter can specify the maximum in GB/MB (similar in syntax to how the
> Java's MaxHeap is defined), or in terms of percentage of available memory
> (not to exceed 95%).
> The
> [runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit]
> script will perform the calculation of the sum of memory required by the
> memory spaces (heap, direct, etc) and ensure that it is within the limit
> defined by the {{DRILLBIT_MAX_PROC_MEM}} env variable.
> In the absence of this parameter, there will be no restriction. A node admin
> can then define this variable in the default terminal's environment (e.g.
> {{/root/.bashrc}} ) files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
