[
https://issues.apache.org/jira/browse/DRILL-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310408#comment-16310408
]
ASF GitHub Bot commented on DRILL-5741:
---------------------------------------
GitHub user kkhatua opened a pull request:
https://github.com/apache/drill/pull/1082
DRILL-5741: Automatically manage memory allocations during startup
Providing an environment variable - DRILLBIT_MAX_PROC_MEM to ensure that a
Drillbit's max memory parameters, cumulatively, don't exceed the value
specified.
The variable can be defined in KB, MB, or GB; similar in syntax to how the
JVM MaxHeap is specified. For e.g.
```
DRILLBIT_MAX_PROC_MEM=13G
DRILLBIT_MAX_PROC_MEM=8192m
DRILLBIT_MAX_PROC_MEM=4194304K
```
In addition, you can specify it as a percent of the total sytem memory
prior to the Drillbit starting up:
`DRILLBIT_MAX_PROC_MEM=40%`
For a system with with 48GB free memory, when set to (say) 25% (with
settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are
defined; the Drillbit fails startup with the following message:
```
2018-01-03 14:27:57 [WARN] 25% of System Memory (47 GB) translates to 12 GB
2018-01-03 14:27:57 [ERROR] Unable to start Drillbit due to memory
constraint violations
Total Memory Requested : 19 GB
Check the following settings to possibly modify (or increase the Max
Memory Permitted):
DRILLBIT_MAX_PROC_MEM=25%
DRILL_HEAP=8G
DRILL_MAX_DIRECT_MEMORY=10G
DRILLBIT_CODE_CACHE_SIZE=1024m
*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as
this will be auto-computed based on the HeapSize and would not exceed 1GB
```
For all other combinations, the undefined parameters are adjusted to ensure
that the total memory allocated is within the value specified by
`DRILLBIT_MAX_PROC_MEM`,
For a system with with 48GB free memory, when set to (say) 50% (with
settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are
defined; the Drillbit startup with the following warning:
```
2018-01-03 14:31:06 [WARN] 50% of System Memory (47 GB) translates to 24 GB
2018-01-03 14:31:06 [WARN] You have an allocation of 4 GB that is
currently unused from a total of 24 GB. You can increase your existing memory
configuration to use this extra memory
DRILLBIT_MAX_PROC_MEM=50%
DRILL_HEAP=8G
DRILL_MAX_DIRECT_MEMORY=10G
DRILLBIT_CODE_CACHE_SIZE=1024m
*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as
this will be auto-computed based on the HeapSize and would not exceed 1GB
```
In addition, if the available free memory is less than the allocation, an
additional warning is provided under the assumption that the OS will reclaim
more free memory when required:
```
2018-01-03 14:31:06 [WARN] Total Memory Allocation for Drillbit (19GB)
exceeds available free memory (11GB)
2018-01-03 14:31:06 [WARN] Drillbit will start up, but can potentially
crash due to oversubscribing of system memory.
```
For more details, refer the attachments in
https://issues.apache.org/jira/browse/DRILL-5741
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kkhatua/drill DRILL-5741
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/1082.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1082
----
commit 2fea048835f729175f77b0a0dea731f741bb70e9
Author: Kunal Khatua <kkhatua@...>
Date: 2018-01-03T21:22:52Z
DRILL-6068: Support user/distrib-specific config checks during startup
1. Allows for distrib/user specific checks to be done
2. Place-holder files for distribution and user specific checks
3. Moved JVM Version Check to head of script
commit 266529f2338c607bd7845d408cddb721a41ae4ac
Author: Kunal Khatua <kkhatua@...>
Date: 2018-01-03T22:15:03Z
DRILL-5741: Automatically manage memory allocations during startup
Providing an environment variable - DRILLBIT_MAX_PROC_MEM to ensure that a
Drillbit's max memory parameters, cumulatively, don't exceed the value
specified.
The variable can be defined in KB, MB, or GB; similar in syntax to how the
JVM MaxHeap is specified.
e.g.
```
DRILLBIT_MAX_PROC_MEM=13G
DRILLBIT_MAX_PROC_MEM=8192m
DRILLBIT_MAX_PROC_MEM=4194304K
```
In addition, you can specify it as a percent of the total sytem memory
prior to the Drillbit starting up:
`DRILLBIT_MAX_PROC_MEM=40%`
For a system with with 48GB free memory, when set to (say) 25% (with
settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are
defined; the Drillbit fails startup with the following message:
```
2018-01-03 14:27:57 [WARN] 25% of System Memory (47 GB) translates to 12 GB
2018-01-03 14:27:57 [ERROR] Unable to start Drillbit due to memory
constraint violations
Total Memory Requested : 19 GB
Check the following settings to possibly modify (or increase the Max
Memory Permitted):
DRILLBIT_MAX_PROC_MEM=25%
DRILL_HEAP=8G
DRILL_MAX_DIRECT_MEMORY=10G
DRILLBIT_CODE_CACHE_SIZE=1024m
*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as
this will be auto-computed based on the HeapSize and would not exceed 1GB
```
For all other combinations, the undefined parameters are adjusted to ensure
that the total memory allocated is within the value specified by
`DRILLBIT_MAX_PROC_MEM`,
For a system with with 48GB free memory, when set to (say) 50% (with
settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are
defined; the Drillbit startup with the following warning:
```
2018-01-03 14:31:06 [WARN] 50% of System Memory (47 GB) translates to 24 GB
2018-01-03 14:31:06 [WARN] You have an allocation of 4 GB that is
currently unused from a total of 24 GB. You can increase your existing memory
configuration to use this extra memory
DRILLBIT_MAX_PROC_MEM=50%
DRILL_HEAP=8G
DRILL_MAX_DIRECT_MEMORY=10G
DRILLBIT_CODE_CACHE_SIZE=1024m
*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as
this will be auto-computed based on the HeapSize and would not exceed 1GB
```
In addition, if the available free memory is less than the allocation, an
additional warning is provided under the assumption that the OS will reclaim
more free memory when required:
```
2018-01-03 14:31:06 [WARN] Total Memory Allocation for Drillbit (19GB)
exceeds available free memory (11GB)
2018-01-03 14:31:06 [WARN] Drillbit will start up, but can potentially
crash due to oversubscribing of system memory.
```
For more details, refer the attachments in
https://issues.apache.org/jira/browse/DRILL-5741
----
> Automatically manage memory allocations during startup
> ------------------------------------------------------
>
> Key: DRILL-5741
> URL: https://issues.apache.org/jira/browse/DRILL-5741
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
> Affects Versions: 1.11.0
> Reporter: Kunal Khatua
> Assignee: Kunal Khatua
> Fix For: 1.13.0
>
> Attachments: Auto Mem Allocation Proposal - Computation Logic.pdf,
> Auto Mem Allocation Proposal - Scenarios.pdf
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Currently, during startup, a Drillbit can be assigned large values for the
> following:
> * Xmx (Heap)
> * XX:MaxDirectMemorySize
> * XX:ReservedCodeCacheSize
> * XX:MaxPermSize
> All of this, potentially, can exceed the available memory on a system when a
> Drillbit is under heavy load. It would be good to have the Drillbit ensure
> during startup itself that the cumulative value of these parameters does not
> exceed a pre-defined upper limit for the Drill process.
> This JIRA is a *proposal* to allow for automatic configuration (based on
> configuration patterns observed in production Drill clusters). It leverages
> the capability of providing distribution (and user-specific) checks during
> Drill Startup from DRILL-6068.
> The idea is to remove the need for a user to worry about managing the tuning
> parameters, by providing the optimal values. In addition, it also allows for
> the memory allocation to be implicitly managed by simply providing the Drill
> process with a single dimensional of total process memory (either in absolute
> values, or as a percentage of the total system memory), while
> {{distrib-auto.sh}} provides the individual allocations.
> This allocation is then partitioned into allocations for Heap and Direct
> Memory, with a small portion allocated for the Generated Java CodeCache as
> well. If any of the individual allocations are also specified (via
> {{distrib-env.sh}} or {{drill-env.sh}}), the remaining unspecified
> allocations are adjusted to stay +within the limits+ of the total memory
> allocation.
> The *details* of the proposal are here:
> https://docs.google.com/spreadsheets/d/1N6VYlQFiPoTV4iD46XbkIrvEQesiGFUU9-GWXYsAPXs/edit#gid=0
> For those unable to access the Google Document, PDFs are attached:
> * [^Auto Mem Allocation Proposal - Computation Logic.pdf] - Provides the
> equation used for computing the heap, direct and code cache allocations for a
> given input
> * [^Auto Mem Allocation Proposal - Scenarios.pdf] - Describes the various
> inputs, and their expected allocations
> The variables that are (_optionally_) defined (in memory, {{distrib-env.sh}}
> or {{drill-env.sh}} ) are:
> * {{DRILLBIT_MAX_PROC_MEM}} : Total Process Memory
> * {{DRILL_HEAP}} : JVM Max Heap Size
> * {{DRILL_MAX_DIRECT_MEMORY}} : JVM Max Direct Memory Size
> * {{DRILLBIT_CODE_CACHE_SIZE}} : JVM Code Cache Size
> Note: _With JDK8, MaxPermSize is no longer supported, so we do not account
> for this any more, and will unset the variable if JDK8 or higher is detected._
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
