GitHub user kkhatua opened a pull request: https://github.com/apache/drill/pull/1082
DRILL-5741: Automatically manage memory allocations during startup Providing an environment variable - DRILLBIT_MAX_PROC_MEM to ensure that a Drillbit's max memory parameters, cumulatively, don't exceed the value specified. The variable can be defined in KB, MB, or GB; similar in syntax to how the JVM MaxHeap is specified. For e.g. ``` DRILLBIT_MAX_PROC_MEM=13G DRILLBIT_MAX_PROC_MEM=8192m DRILLBIT_MAX_PROC_MEM=4194304K ``` In addition, you can specify it as a percent of the total sytem memory prior to the Drillbit starting up: `DRILLBIT_MAX_PROC_MEM=40%` For a system with with 48GB free memory, when set to (say) 25% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit fails startup with the following message: ``` 2018-01-03 14:27:57 [WARN] 25% of System Memory (47 GB) translates to 12 GB 2018-01-03 14:27:57 [ERROR] Unable to start Drillbit due to memory constraint violations Total Memory Requested : 19 GB Check the following settings to possibly modify (or increase the Max Memory Permitted): DRILLBIT_MAX_PROC_MEM=25% DRILL_HEAP=8G DRILL_MAX_DIRECT_MEMORY=10G DRILLBIT_CODE_CACHE_SIZE=1024m *NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB ``` For all other combinations, the undefined parameters are adjusted to ensure that the total memory allocated is within the value specified by `DRILLBIT_MAX_PROC_MEM`, For a system with with 48GB free memory, when set to (say) 50% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit startup with the following warning: ``` 2018-01-03 14:31:06 [WARN] 50% of System Memory (47 GB) translates to 24 GB 2018-01-03 14:31:06 [WARN] You have an allocation of 4 GB that is currently unused from a total of 24 GB. You can increase your existing memory configuration to use this extra memory DRILLBIT_MAX_PROC_MEM=50% DRILL_HEAP=8G DRILL_MAX_DIRECT_MEMORY=10G DRILLBIT_CODE_CACHE_SIZE=1024m *NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB ``` In addition, if the available free memory is less than the allocation, an additional warning is provided under the assumption that the OS will reclaim more free memory when required: ``` 2018-01-03 14:31:06 [WARN] Total Memory Allocation for Drillbit (19GB) exceeds available free memory (11GB) 2018-01-03 14:31:06 [WARN] Drillbit will start up, but can potentially crash due to oversubscribing of system memory. ``` For more details, refer the attachments in https://issues.apache.org/jira/browse/DRILL-5741 You can merge this pull request into a Git repository by running: $ git pull https://github.com/kkhatua/drill DRILL-5741 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/1082.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1082 ---- commit 2fea048835f729175f77b0a0dea731f741bb70e9 Author: Kunal Khatua <kkhatua@...> Date: 2018-01-03T21:22:52Z DRILL-6068: Support user/distrib-specific config checks during startup 1. Allows for distrib/user specific checks to be done 2. Place-holder files for distribution and user specific checks 3. Moved JVM Version Check to head of script commit 266529f2338c607bd7845d408cddb721a41ae4ac Author: Kunal Khatua <kkhatua@...> Date: 2018-01-03T22:15:03Z DRILL-5741: Automatically manage memory allocations during startup Providing an environment variable - DRILLBIT_MAX_PROC_MEM to ensure that a Drillbit's max memory parameters, cumulatively, don't exceed the value specified. The variable can be defined in KB, MB, or GB; similar in syntax to how the JVM MaxHeap is specified. e.g. ``` DRILLBIT_MAX_PROC_MEM=13G DRILLBIT_MAX_PROC_MEM=8192m DRILLBIT_MAX_PROC_MEM=4194304K ``` In addition, you can specify it as a percent of the total sytem memory prior to the Drillbit starting up: `DRILLBIT_MAX_PROC_MEM=40%` For a system with with 48GB free memory, when set to (say) 25% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit fails startup with the following message: ``` 2018-01-03 14:27:57 [WARN] 25% of System Memory (47 GB) translates to 12 GB 2018-01-03 14:27:57 [ERROR] Unable to start Drillbit due to memory constraint violations Total Memory Requested : 19 GB Check the following settings to possibly modify (or increase the Max Memory Permitted): DRILLBIT_MAX_PROC_MEM=25% DRILL_HEAP=8G DRILL_MAX_DIRECT_MEMORY=10G DRILLBIT_CODE_CACHE_SIZE=1024m *NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB ``` For all other combinations, the undefined parameters are adjusted to ensure that the total memory allocated is within the value specified by `DRILLBIT_MAX_PROC_MEM`, For a system with with 48GB free memory, when set to (say) 50% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit startup with the following warning: ``` 2018-01-03 14:31:06 [WARN] 50% of System Memory (47 GB) translates to 24 GB 2018-01-03 14:31:06 [WARN] You have an allocation of 4 GB that is currently unused from a total of 24 GB. You can increase your existing memory configuration to use this extra memory DRILLBIT_MAX_PROC_MEM=50% DRILL_HEAP=8G DRILL_MAX_DIRECT_MEMORY=10G DRILLBIT_CODE_CACHE_SIZE=1024m *NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB ``` In addition, if the available free memory is less than the allocation, an additional warning is provided under the assumption that the OS will reclaim more free memory when required: ``` 2018-01-03 14:31:06 [WARN] Total Memory Allocation for Drillbit (19GB) exceeds available free memory (11GB) 2018-01-03 14:31:06 [WARN] Drillbit will start up, but can potentially crash due to oversubscribing of system memory. ``` For more details, refer the attachments in https://issues.apache.org/jira/browse/DRILL-5741 ---- ---