[
https://issues.apache.org/jira/browse/DRILL-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144197#comment-16144197
]
ASF GitHub Bot commented on DRILL-5741:
---------------------------------------
Github user kkhatua commented on the issue:
https://github.com/apache/drill/pull/922
@paul-rogers
The reason this would be needed is because the system admin might not be
the one managing the specifics of the Drill memory allocations. This allows a
node admin to define the limits within which Drill will work, while the
specifications can be delegated to the drill-env.sh script which is managed by
a power user.
There isn't a conflict with the way a service like Warden would allocate
memory, because it does not need to manage the internal allocations, only the
overall. The Drill service will continue to own its configuration's specifics.
Your solution is pretty much in line with what the PR proposes. The
exception being that, in the presence of the env variable defined, the
specifications **cannot** oversubscribe the available (or max permissible)
memory. In case of oversubscribing, Drill will not startup.
If there are uber resource managers (for e.g. YARN), they have the option
of either
- providing this limit and relying on files like ```drill-env.sh``` to
specify the limits...
or
- provide the explicit parameters for Heap, DirectMemory, etc to the JVM
and **not** have to specify ```DRILLBIT_MAX_PROC_MEM``` .
> During startup Drill should not exceed the available memory
> -----------------------------------------------------------
>
> Key: DRILL-5741
> URL: https://issues.apache.org/jira/browse/DRILL-5741
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
> Affects Versions: 1.11.0
> Reporter: Kunal Khatua
> Assignee: Kunal Khatua
> Fix For: 1.12.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Currently, during startup, a Drillbit can be assigned large values for the
> following:
> * Xmx (Heap)
> * XX:MaxDirectMemorySize
> * XX:ReservedCodeCacheSize
> * XX:MaxPermSize
> All of this, potentially, can exceed the available memory on a system when a
> Drillbit is under heavy load. It would be good to have the Drillbit ensure
> during startup itself that the cumulative value of these parameters does not
> exceed a pre-defined upper limit for the Drill process.
> The proposal is to have the
> [runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit]
> script look for an additional environment variable:
> {{DRILLBIT_MAX_PROC_MEM}}
> The parameter can specify the maximum in GB/MB (similar in syntax to how the
> Java's MaxHeap is defined), or in terms of percentage of available memory
> (not to exceed 95%).
> The
> [runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit]
> script will perform the calculation of the sum of memory required by the
> memory spaces (heap, direct, etc) and ensure that it is within the limit
> defined by the {{DRILLBIT_MAX_PROC_MEM}} env variable.
> In the absence of this parameter, there will be no restriction. A node admin
> can then define this variable in the default terminal's environment (e.g.
> {{/root/.bashrc}} ) files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
