Kunal Khatua created DRILL-5741:
-----------------------------------
Summary: Drillbit during startup should not exceed the available
memory on a node
Key: DRILL-5741
URL: https://issues.apache.org/jira/browse/DRILL-5741
Project: Apache Drill
Issue Type: Improvement
Components: Server
Affects Versions: 1.11.0
Reporter: Kunal Khatua
Fix For: 1.12.0
Currently, during startup, a Drillbit can be assigned large values for the
following:
* Xmx (Heap)
* XX:MaxDirectMemorySize
* XX:ReservedCodeCacheSize
* XX:MaxPermSize
All of this, potentially, can exceed the available memory on a system when a
Drillbit is under heavy load. It would be good to have the Drillbit ensure
during startup itself that the cumulative value of these parameters does not
exceed a pre-defined upper limit for the Drill process.
The proposal is to have the
[runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit]
script look for an additional environment variable:
{{DRILLBIT_MAX_PROC_MEM}}
The parameter can specify the maximum in GB/MB (similar in syntax to how the
Java's MaxHeap is defined), or in terms of percentage of available memory (not
to exceed 95%).
The
[runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit]
script will perform the calculation of the sum of memory required by the
memory spaces (heap, direct, etc) and ensure that it is within the limit
defined by the {{DRILLBIT_MAX_PROC_MEM}} env variable.
In the absence of this parameter, there will be no restriction. A node admin
can then define this variable in the default terminal's environment (e.g.
{{/root/.bashrc}} ) files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
