DRILL-5741: Automatically manage memory allocations during startup
**Note:** This commit is rebased on (hence, requires) DRILL-6068 commit
This commit provides a way for distributions and users to automatically define
the Drillbit JVM's Heap, Direct and CodeCache allocations. To do this, it
leverages the DRILL-6068 commit, which provides for configuration setup.
The only new variable introduced is an environment variable -
`DRILLBIT_MAX_PROC_MEM` that defines a Drillbit's max memory parameters. The
logic defined by `auto-setup.sh` will check that if any (or all) of the JVM
memory parameters have been declared; they, cumulatively, don't exceed the
value specified by `DRILLBIT_MAX_PROC_MEM`.
```
NOTE: To disable this, simply unset (or don't define) the environment variable
DRILLBIT_MAX_PROC_MEM
```
The variable can be defined in KB, MB, or GB; similar in syntax to how the JVM
MaxHeap is specified. For e.g.
```
DRILLBIT_MAX_PROC_MEM=13G
DRILLBIT_MAX_PROC_MEM=8192m
DRILLBIT_MAX_PROC_MEM=4194304K
```
In addition, you can specify it as a percent of the total system memory prior
to the Drillbit starting up:
`DRILLBIT_MAX_PROC_MEM=40%`
For a system with with 48GB free memory, when set to (say) 25% (with settings
defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the
Drillbit fails startup with the following message:
```
2018-01-03 14:27:57 [WARN] 25% of System Memory (47 GB) translates to 12 GB
2018-01-03 14:27:57 [ERROR] Unable to start Drillbit due to memory
constraint violations
Total Memory Requested : 19 GB
Check the following settings to possibly modify (or increase the Max Memory
Permitted):
DRILLBIT_MAX_PROC_MEM=25%
DRILL_HEAP=8G
DRILL_MAX_DIRECT_MEMORY=10G
DRILLBIT_CODE_CACHE_SIZE=1024m
*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as
this will be auto-computed based on the HeapSize and would not exceed 1GB
```
For all other combinations, the undefined parameters are adjusted to ensure
that the total memory allocated is within the value specified by
`DRILLBIT_MAX_PROC_MEM`,
For a system with with 48GB free memory, when set to (say) 50% (with settings
defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the
Drillbit startup with the following warning:
```
2018-01-03 14:31:06 [WARN] 50% of System Memory (47 GB) translates to 24 GB
2018-01-03 14:31:06 [WARN] You have an allocation of 4 GB that is currently
unused from a total of 24 GB. You can increase your existing memory
configuration to use this extra memory
DRILLBIT_MAX_PROC_MEM=50%
DRILL_HEAP=8G
DRILL_MAX_DIRECT_MEMORY=10G
DRILLBIT_CODE_CACHE_SIZE=1024m
*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as
this will be auto-computed based on the HeapSize and would not exceed 1GB
```
In addition, if the available free memory is less than the allocation, an
additional warning is provided under the assumption that the OS will reclaim
more free memory when required:
```
2018-01-03 14:31:06 [WARN] Total Memory Allocation for Drillbit (19GB) exceeds
available free memory (11GB)
2018-01-03 14:31:06 [WARN] Drillbit will start up, but can potentially crash
due to oversubscribing of system memory.
```
For more details, refer the attachments in
https://issues.apache.org/jira/browse/DRILL-5741
Changes to auto configure messaging
Publishing final values prior to startup
Minor update for printing to console's err stream
close apache/drill#1082
Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/50efb806
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/50efb806
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/50efb806
Branch: refs/heads/master
Commit: 50efb806bb03494e1da4d6b48f90fbf58d699c18
Parents: 266250c
Author: Kunal Khatua <[email protected]>
Authored: Thu Jan 11 17:35:13 2018 -0800
Committer: Aman Sinha <[email protected]>
Committed: Fri Feb 23 17:55:36 2018 -0800
----------------------------------------------------------------------
distribution/src/resources/auto-setup.sh | 202 ++++++++++++++++++++++-
distribution/src/resources/distrib-setup.sh | 2 +-
distribution/src/resources/drill-env.sh | 10 ++
3 files changed, 212 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/drill/blob/50efb806/distribution/src/resources/auto-setup.sh
----------------------------------------------------------------------
diff --git a/distribution/src/resources/auto-setup.sh
b/distribution/src/resources/auto-setup.sh
index 75bdda0..141648c 100644
--- a/distribution/src/resources/auto-setup.sh
+++ b/distribution/src/resources/auto-setup.sh
@@ -25,5 +25,205 @@
# if [ $status == "FAILED" ]; return 1; fi
###==========================================================================
-# FEATURES (Added here to describe supported additions)
+# FEATURES
+# 1. Provides checks and auto-configuration for memory settings
###==========================================================================
+
+# Convert Java memory value to MB
+function valueInMB() {
+ if [ -z "$1" ]; then echo ""; return; fi
+ local inputTxt=`echo $1| tr '[A-Z]' '[a-z]'`
+ local inputValue=`echo ${inputTxt:0:${#inputTxt}-1}`;
+ # Extracting Numeric Value
+ if [[ "$inputTxt" == *g ]]; then
+ let valueInMB=$inputValue*1024
+ elif [[ "$DbitMaxProcMem" == *k ]]; then
+ let valueInMB=$inputValue/1024
+ elif [[ "$inputTxt" == *m ]]; then
+ let valueInMB=$inputValue
+ elif [[ "$inputTxt" == *% ]]; then
+ #TotalRAM_inMB*percentage [Works on Linux]
+ let valueInMB=$inputValue*$totalRAM_inMB/100;
+ else
+ echo error;
+ return 1;
+ fi
+ echo "$valueInMB"
+ return
+}
+
+# Convert Java memory value to GB
+function valueInGB() {
+ if [ -z "$1" ]; then echo ""; return; fi
+ local inputTxt=`echo $1| tr '[A-Z]' '[a-z]'`
+ local inputValue=`echo ${inputTxt:0:${#inputTxt}-1}`;
+ # Extracting Numeric Value
+ if [[ "$inputTxt" == *g ]]; then
+ let valueInGB=$inputValue
+ elif [[ "$DbitMaxProcMem" == *k ]]; then
+ let valueInGB=$inputValue/1024/1024
+ elif [[ "$inputTxt" == *m ]]; then
+ let valueInGB=$inputValue/1024
+ elif [[ "$inputTxt" == *% ]]; then
+ #TotalRAM_inMB*percentage [Works on Linux]
+ let valueInGB=$inputValue*`cat /proc/meminfo | grep MemTotal | tr ' '
'\n'| grep '[0-9]'`/1024/1024/100;
+ else
+ echo error;
+ return 1;
+ fi
+ echo "$valueInGB"
+ return
+}
+
+# Estimates code cache based on total heap and direct
+function estCodeCacheInMB() {
+ local totalHeapAndDirect=$1
+ if [ $totalHeapAndDirect -le 4096 ]; then echo 512;
+ elif [ $totalHeapAndDirect -le 10240 ]; then echo 768;
+ else echo 1024;
+ fi
+}
+
+#Print Current Allocation
+function printCurrAllocation()
+{
+ if [ -n "$DRILLBIT_MAX_PROC_MEM" ]; then echo -e "
DRILLBIT_MAX_PROC_MEM=$DRILLBIT_MAX_PROC_MEM" 1>&2; fi
+ if [ -n "$DRILL_HEAP" ]; then echo -e " DRILL_HEAP=$DRILL_HEAP" 1>&2; fi
+ if [ -n "$DRILL_MAX_DIRECT_MEMORY" ]; then echo -e "
DRILL_MAX_DIRECT_MEMORY=$DRILL_MAX_DIRECT_MEMORY" 1>&2; fi
+ if [ -n "$DRILLBIT_CODE_CACHE_SIZE" ]; then
+ echo -e " DRILLBIT_CODE_CACHE_SIZE=$DRILLBIT_CODE_CACHE_SIZE " 1>&2
+ echo -e " *NOTE: It is recommended not to specify
DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize
and would not exceed 1GB" 1>&2
+ fi
+}
+
+#============================================================================
+# Check and auto-configuration for memory settings
+#----------------------------------------------------------------------------
+#Default (Track status of this check: "" => Continue checking ; "PASSED" => no
more check required)
+AutoMemConfigStatus=""
+
+#Computing existing system information
+# Tested on Linux (CentOS/RHEL/Ubuntu); Cygwin (Win10Pro-64bit)
+if [[ "$OSTYPE" == *linux* ]] || [[ "$OSTYPE" == cygwin* ]]; then
+ let totalRAM_inMB=`cat /proc/meminfo | grep MemTotal | tr ' ' '\n'| grep
'[0-9]'`/1024
+ let freeRAM_inMB=`cat /proc/meminfo | grep MemFree | tr ' ' '\n'| grep
'[0-9]'`/1024
+elif [[ "$OSTYPE" == darwin* ]]; then
+ # Mac OSX
+ #Refer for math: https://apple.stackexchange.com/a/196925
+ #Page Size
+ let macOSPageSize=`vm_stat | grep 'page size' | grep -o -E '[0-9]+'`
+ #MemoryUsage on MacOS
+ let freePg=`vm_stat | grep free | awk '{ print $NF }' | sed 's/\.//'`
+ let activePg=`vm_stat | grep -w 'active:' | awk '{ print $NF }' | sed
's/\.//'`
+ let speculativePg=`vm_stat | grep speculative | awk '{ print $NF }' | sed
's/\.//'`
+ let fileCachePg=`vm_stat | grep File-backed | awk '{ print $NF }' | sed
's/\.//'`
+ let wiredMemPg=`vm_stat | grep 'wired down' | awk '{ print $NF }' | sed
's/\.//'`
+ let compressedPg=`vm_stat | grep 'occupied by compressor' | awk '{ print $NF
}' | sed 's/\.//'`
+ #Total
+ let
totalRAM_inPages=$freePg+$activePg+$speculativePg+$fileCachePg+$wiredMemPg+$compressedPg
+ let totalRAM_inMB=$totalRAM_inPages*$macOSPageSize/1048576
+ let freeRAM_inMB=$freePg*$macOSPageSize/1048576
+elif [[ "$OSTYPE" == "msys" ]]; then
+ # Msys env on MinGW (TODO: Pending verification)
+ let totalRAM_inMB=`cat /proc/meminfo | grep MemTotal | tr ' ' '\n'| grep
'[0-9]'`/1024
+ let freeRAM_inMB=`cat /proc/meminfo | grep MemFree | tr ' ' '\n'| grep
'[0-9]'`/1024
+else
+ # Unknown OS
+ echo `date +%Y-%m-%d" "%H:%M:%S`" [WARN] Unknown OS ("$OSTYPE"). Will not
attempt to auto-configure memory" 1>&2
+ AutoMemConfigStatus="PASSED"
+fi
+
+#Read current values
+DbitMaxProcMem=$(valueInMB $DRILLBIT_MAX_PROC_MEM)
+DbitMaxDirectMem=$(valueInMB $DRILL_MAX_DIRECT_MEMORY)
+DbitMaxHeapMem=$(valueInMB $DRILL_HEAP)
+DbitMaxCodeCacheMem=$(valueInMB $DRILLBIT_CODE_CACHE_SIZE)
+
+# Alert for %age usage
+if [[ "$DRILLBIT_MAX_PROC_MEM" == *% ]] && [ -z "$AutoMemConfigStatus" ]; then
+ echo `date +%Y-%m-%d" "%H:%M:%S`" [WARN] "$DRILLBIT_MAX_PROC_MEM" of System
Memory ("$(valueInGB $totalRAM_inMB'm')" GB) translates to "$(valueInGB
$DbitMaxProcMem'm')" GB" 1>&2
+fi
+
+### Performing Auto-Configuration
+if [ -z "$DbitMaxProcMem" ] && [ -z "$AutoMemConfigStatus" ]; then
+ if [ -n "$DbitMaxDirectMem" ] && [ -n "$DbitMaxHeapMem" ]; then
+ ## [SCENARIO 1]: TotalCap is NOT Defined, but Heap&Direct ARE Defined
(i.e. no limit)
+ let currTotal=$DbitMaxDirectMem+$DbitMaxHeapMem
+ #Estimating CodeCache size of current total
+ if [ -z "$DbitMaxCodeCacheMem" ]; then export
DRILLBIT_CODE_CACHE_SIZE=$(estCodeCacheInMB $currTotal)'m'; fi
+ fi
+ # Default values will be loaded for unspecified memory parameters
+ AutoMemConfigStatus="PASSED"
+elif [ -z "$AutoMemConfigStatus" ]; then
+ ## Scenario: Total IS Defined
+ if [ -z "$DbitMaxCodeCacheMem" ]; then
+ let DbitMaxCodeCacheMem=$(estCodeCacheInMB $DbitMaxProcMem)
+ export DRILLBIT_CODE_CACHE_SIZE=$DbitMaxCodeCacheMem'm'
+ fi
+ if [ -n "$DbitMaxHeapMem" ] && [ -n "$DbitMaxDirectMem" ]; then
+ ## [SCENARIO 2]: Heap & Direct ARE Defined
+ let calcTotalInMB=$DbitMaxDirectMem+$DbitMaxHeapMem+$DbitMaxCodeCacheMem
+ # Fail if exceeding process limit
+ if [ $calcTotalInMB -gt $DbitMaxProcMem ]; then
+ echo "[ERROR] Unable to start Drillbit due to memory constraint
violations" 1>&2
+ echo " Total Memory Requested : "$(valueInGB $calcTotalInMB'm')" GB"
1>&2
+ echo " Check the following settings to possibly modify (or increase the
Max Memory Permitted):" 1>&2
+ printCurrAllocation
+ exit 127
+ else
+ #All numbers align
+ let deltaInGB=($DbitMaxProcMem-$calcTotalInMB)/1024
+ if [ $deltaInGB -gt 1 ]; then
+ echo "[WARN] You have an allocation of "$deltaInGB" GB that is
currently unused from a total of "$(valueInGB $DbitMaxProcMem'm')" GB. You can
increase your existing memory configuration to use this extra memory" 1>&2
+ printCurrAllocation
+ fi
+ fi
+ elif [ -n "$DbitMaxHeapMem" ] && [ -z "$DbitMaxDirectMem" ]; then
+ ## [SCENARIO 3]: Total and only Heap is defined
+ echo "[WARN] Only DRILL_HEAP is defined. Auto-configuring for Direct
memory" 1>&2
+ let DbitMaxDirectMem=$DbitMaxProcMem-$DbitMaxHeapMem-$DbitMaxCodeCacheMem
+ elif [ -z "$DbitMaxHeapMem" ] && [ -n "$DbitMaxDirectMem" ]; then
+ ## [SCENARIO 4]: Total and only Direct is defined
+ echo "[WARN] Only DRILL_MAX_DIRECT_MEMORY is defined. Auto-configuring for
Heap" 1>&2
+ let DbitMaxHeapMem=$DbitMaxProcMem-$DbitMaxDirectMem-$DbitMaxCodeCacheMem
+ elif [ -z "$DbitMaxDirectMem" ] && [ -z "$DbitMaxHeapMem" ]; then
+ ## [SCENARIO 5]: Only Total is defined
+ echo "[WARN] Only DRILLBIT_MAX_PROC_MEM is defined. Auto-configuring for
Heap & Direct memory" 1>&2
+ ## Compute Direct & Heap
+ let DbitMaxProcMemInGB=$(valueInGB $DbitMaxProcMem'm')
+ let DbitMaxHeapMemInGB=`echo $DbitMaxProcMemInGB | awk
'{heap=-13.2+6.12*log($1); if (heap<1) {heap=1}; printf "%0.0f\n", heap }'`
+ let DbitMaxHeapMem=$(valueInMB $DbitMaxHeapMemInGB'g')
+ let DbitMaxDirectMem=$DbitMaxProcMem-$DbitMaxHeapMem-$DbitMaxCodeCacheMem
+ fi
+ ## Export computed values
+ export DRILL_HEAP=$(valueInGB $DbitMaxHeapMem'm')"G"
+ export DRILL_MAX_DIRECT_MEMORY=$(valueInGB $DbitMaxDirectMem'm')"G"
+ export DRILLBIT_CODE_CACHE_SIZE=$DbitMaxCodeCacheMem'm'
+fi
+
+### Broad check for System Level capacity
+if [ -z "$AutoMemConfigStatus" ]; then
+ # Rereading for recently exported env var
+ DbitMaxDirectMem=$(valueInMB $DRILL_MAX_DIRECT_MEMORY)
+ DbitMaxHeapMem=$(valueInMB $DRILL_HEAP)
+ DbitMaxCodeCacheMem=$(valueInMB $DRILLBIT_CODE_CACHE_SIZE)
+ echo "[INFO] Attempting to start up Drill with the following settings" 1>&2
+ echo " DRILL_HEAP="$DRILL_HEAP 1>&2
+ echo " DRILL_MAX_DIRECT_MEMORY="$DRILL_MAX_DIRECT_MEMORY 1>&2
+ echo " DRILLBIT_CODE_CACHE_SIZE="$DRILLBIT_CODE_CACHE_SIZE 1>&2
+ let totalDBitMem_inMB=$DbitMaxDirectMem+$DbitMaxHeapMem+$DbitMaxCodeCacheMem
1>&2
+ if [ $totalDBitMem_inMB -gt $totalRAM_inMB ]; then
+ echo "[ERROR] Total Memory Allocation for Drillbit ("$(valueInGB
$totalDBitMem_inMB'm')"GB) exceeds total system memory ("$(valueInGB
$totalRAM_inMB'm')"GB)" 1>&2
+ echo "[ERROR] Drillbit not will start up. Please check your allocations"
1>&2
+ exit 127
+ elif [ $totalDBitMem_inMB -gt $freeRAM_inMB ]; then
+ echo "[WARN] Total Memory Allocation for Drillbit ("$(valueInGB
$totalDBitMem_inMB'm')"GB) exceeds available free memory ("$(valueInGB
$freeRAM_inMB'm')"GB)" 1>&2
+ echo "[WARN] Drillbit will start up, but can potentially crash due to
oversubscribing of system memory." 1>&2
+ fi
+fi
+
+#Implicit that checks have passed
+AutoMemConfigStatus="PASSED"
+#----------------------------------------------------------------------------
+# AT THIS POINT: Check and auto-configuration for memory settings [PASSED]
+#============================================================================
http://git-wip-us.apache.org/repos/asf/drill/blob/50efb806/distribution/src/resources/distrib-setup.sh
----------------------------------------------------------------------
diff --git a/distribution/src/resources/distrib-setup.sh
b/distribution/src/resources/distrib-setup.sh
index dbd2b90..e641b76 100644
--- a/distribution/src/resources/distrib-setup.sh
+++ b/distribution/src/resources/distrib-setup.sh
@@ -25,5 +25,5 @@
# if [ $status == "FAILED" ]; return 1; fi
###==========================================================================
-# FEATURES (Added here to describe supported additions)
+# FEATURES
###==========================================================================
http://git-wip-us.apache.org/repos/asf/drill/blob/50efb806/distribution/src/resources/drill-env.sh
----------------------------------------------------------------------
diff --git a/distribution/src/resources/drill-env.sh
b/distribution/src/resources/drill-env.sh
index 8211c3c..02f1b2d 100644
--- a/distribution/src/resources/drill-env.sh
+++ b/distribution/src/resources/drill-env.sh
@@ -37,6 +37,16 @@
#
# or a more specialized form.
+# Amount of total memory for the Drillbit process. This value is defined as
the limit
+# that the startup script will try to enforce on the Drill JVM. The values can
be
+# defined in terms of percentage of the available system memory, or in terms
of actual
+# values, similar to how we define the actual JVM memory parameters like Heap
Size.
+# There is no default and depends on how much can be allotted on a machine.
+# This enables Drill's memory auto-configuration logic to kick in, and should
be unset
+# if the intent is to not use the auto-configuration.
+
+#export DRILLBIT_MAX_PROC_MEM=${DRILLBIT_MAX_PROC_MEM:-"13G"}
+
# Amount of heap memory for the Drillbit process. Values are those supported by
# the Java -Xms option. The default is 4G.
