[
https://issues.apache.org/jira/browse/FLINK-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745098#comment-14745098
]
ASF GitHub Bot commented on FLINK-2641:
---------------------------------------
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1129#discussion_r39488999
--- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh ---
@@ -51,13 +51,41 @@ if [[ $STARTSTOP == "start" ]]; then
fi
fi
- if [[ ! ${FLINK_TM_HEAP} =~ ${IS_NUMBER} ]]; then
- echo "[ERROR] Configured TaskManager JVM heap size is not a
number. Please set '$KEY_TASKM_HEAP_MB' in $FLINK_CONF_FILE."
+ if [[ ! ${FLINK_TM_HEAP} =~ ${IS_NUMBER} ]] || [[ "${FLINK_TM_HEAP}"
-lt "0" ]]; then
+ echo "[ERROR] Configured TaskManager JVM heap size is not a
number. Please set '${KEY_TASKM_MEM_SIZE}' in ${FLINK_CONF_FILE}."
exit 1
fi
- if [ "$FLINK_TM_HEAP" -gt 0 ]; then
- export JVM_ARGS="$JVM_ARGS -Xms"$FLINK_TM_HEAP"m
-Xmx"$FLINK_TM_HEAP"m"
+ if [ "${FLINK_TM_HEAP}" -gt "0" ]; then
+
+ TM_HEAP_SIZE=${FLINK_TM_HEAP}
+ TM_OFFHEAP_SIZE=0
+ # some space for Netty initilization
+ NETTY_BUFFERS=$((1024 * 1024))
+
+ if useOffHeapMemory; then
+ if [[ "${FLINK_TM_MEM_MANAGED_SIZE}" -gt "0" ]]; then
+ # We split up the total memory in heap and off-heap memory
+ if [[ "${FLINK_TM_HEAP}" -le
"${FLINK_TM_MEM_MANAGED_SIZE}" ]]; then
+ echo "[ERROR] Configured TaskManager memory size
('${KEY_TASKM_MEM_SIZE}') must be larger than the managed memory size
('${KEY_TASKM_MEM_MANAGED_SIZE}')."
+ exit 1
+ fi
+ TM_OFFHEAP_SIZE=${FLINK_TM_MEM_MANAGED_SIZE}
+ TM_HEAP_SIZE=$((FLINK_TM_HEAP - FLINK_TM_MEM_MANAGED_SIZE))
+ else
+ # We calculate the memory using a fraction of the total
memory
+ if [[ `bc -l <<< "${FLINK_TM_MEM_MANAGED_FRACTION} >=
1.0"` != "0" ]]; then
+ echo "[ERROR] Configured TaskManager managed memory
fraction is not a valid value. Please set '${KEY_TASKM_MEM_MANAGED_FRACTION}'
in ${FLINK_CONF_FILE}"
+ exit 1
+ fi
+ # recalculate the JVM heap memory by taking the off-heap
ratio into account
+ TM_OFFHEAP_SIZE=`printf '%.0f\n' $(bc -l <<<
"${FLINK_TM_HEAP} * ${FLINK_TM_MEM_MANAGED_FRACTION}")`
+ TM_HEAP_SIZE=$((FLINK_TM_HEAP - TM_OFFHEAP_SIZE))
+ fi
+ fi
+
+ export JVM_ARGS="${JVM_ARGS} -Xms${TM_HEAP_SIZE}M
-Xmx${TM_HEAP_SIZE}M -XX:MaxDirectMemorySize=$((TM_OFFHEAP_SIZE * 1024 * 1024 +
FLINK_TM_MEM_NETWORK_SIZE + NETTY_BUFFERS))"
--- End diff --
Append "M" to -XX:MaxDirectMemorySize parameter?
> Integrate the off-heap memory configuration with the TaskManager start script
> -----------------------------------------------------------------------------
>
> Key: FLINK-2641
> URL: https://issues.apache.org/jira/browse/FLINK-2641
> Project: Flink
> Issue Type: New Feature
> Components: Start-Stop Scripts
> Affects Versions: 0.10
> Reporter: Stephan Ewen
> Assignee: Maximilian Michels
> Fix For: 0.10
>
>
> The TaskManager start script needs to adjust the {{-Xmx}}, {{-Xms}}, and
> {{-XX:MaxDirectMemorySize}} parameters according to the off-heap memory
> settings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
