[
https://issues.apache.org/jira/browse/DRILL-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142446#comment-16142446
]
ASF GitHub Bot commented on DRILL-5741:
---------------------------------------
GitHub user kkhatua opened a pull request:
https://github.com/apache/drill/pull/922
DRILL-5741: During startup Drill should not exceed the available memory
Providing an environment variable - DRILLBIT_MAX_PROC_MEM to ensure that a
Drillbit's max memory parameters, cumulatively, don't exceed the value
specified.
The variable can be defined in KB, MB, or GB; similar in syntax to how the
JVM MaxHeap is specified.
e.g.
```
DRILLBIT_MAX_PROC_MEM=13G
DRILLBIT_MAX_PROC_MEM=8192m
DRILLBIT_MAX_PROC_MEM=4194304K
```
In addition, you can specify it as a percent of the available free memory
prior to the Drillbit starting up:
`DRILLBIT_MAX_PROC_MEM=40%
`
For a system with with 28GB free memory, when set to 40% the Drillbit (with
settings defined in drill-env.sh) fails startup with the following message:
```
2017-08-25 14:58:57 [ERROR] Unable to start Drillbit due to memory
constraint violations
Max Memory Permitted : 40% (11 of 28 GB free memory)
Total Memory Requested : 26 GB
Check the following settings to possibly modify (or increase the Max
Memory Permitted):
-Xmx8g
-XX:MaxDirectMemorySize=16g
-XX:ReservedCodeCacheSize=1G
-XX:MaxPermSize=512M
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kkhatua/drill DRILL-5741
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/922.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #922
----
commit 04b04d8e4b0c24fef053bbbd6ecebfaebad99d9c
Author: Kunal Khatua <[email protected]>
Date: 2017-08-25T23:35:37Z
DRILL-5741: During startup Drill should not exceed the available memory
Providing an environment variable - DRILLBIT_MAX_PROC_MEM to ensure that a
Drillbit's max memory parameters, cumulatively, don't exceed the value
specified.
The variable can be defined in KB, MB, or GB; similar in syntax to how the
JVM MaxHeap is specified.
e.g.
```
DRILLBIT_MAX_PROC_MEM=13G
DRILLBIT_MAX_PROC_MEM=8192m
DRILLBIT_MAX_PROC_MEM=4194304K
```
In addition, you can specify it as a percent of the available free memory
prior to the Drillbit starting up:
`DRILLBIT_MAX_PROC_MEM=40%
`
For a system with with 28GB free memory, when set to 40% the Drillbit (with
settings defined in drill-env.sh) fails startup with the following message:
```
2017-08-25 14:58:57 [ERROR] Unable to start Drillbit due to memory
constraint violations
Max Memory Permitted : 40% (11 of 28 GB free memory)
Total Memory Requested : 26 GB
Check the following settings to possibly modify (or increase the Max
Memory Permitted):
-Xmx8g
-XX:MaxDirectMemorySize=16g
-XX:ReservedCodeCacheSize=1G
-XX:MaxPermSize=512M
```
----
> During startup Drill should not exceed the available memory
> -----------------------------------------------------------
>
> Key: DRILL-5741
> URL: https://issues.apache.org/jira/browse/DRILL-5741
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
> Affects Versions: 1.11.0
> Reporter: Kunal Khatua
> Assignee: Kunal Khatua
> Fix For: 1.12.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Currently, during startup, a Drillbit can be assigned large values for the
> following:
> * Xmx (Heap)
> * XX:MaxDirectMemorySize
> * XX:ReservedCodeCacheSize
> * XX:MaxPermSize
> All of this, potentially, can exceed the available memory on a system when a
> Drillbit is under heavy load. It would be good to have the Drillbit ensure
> during startup itself that the cumulative value of these parameters does not
> exceed a pre-defined upper limit for the Drill process.
> The proposal is to have the
> [runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit]
> script look for an additional environment variable:
> {{DRILLBIT_MAX_PROC_MEM}}
> The parameter can specify the maximum in GB/MB (similar in syntax to how the
> Java's MaxHeap is defined), or in terms of percentage of available memory
> (not to exceed 95%).
> The
> [runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit]
> script will perform the calculation of the sum of memory required by the
> memory spaces (heap, direct, etc) and ensure that it is within the limit
> defined by the {{DRILLBIT_MAX_PROC_MEM}} env variable.
> In the absence of this parameter, there will be no restriction. A node admin
> can then define this variable in the default terminal's environment (e.g.
> {{/root/.bashrc}} ) files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
