[
https://issues.apache.org/jira/browse/FLINK-8826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-8826:
----------------------------------
Description:
When I tried running some job on the cluster, despite setting
taskmanager.heap.mb = 3072
taskmanager.network.memory.fraction: 0.4
and reported in the console
{code:java}
Cluster specification: ClusterSpecification{masterMemoryMB=768,
taskManagerMemoryMB=3072, numberTaskManagers=92, slotsPerTaskManager=1}{code}
The actual settings were:
{noformat}
2018-03-01 14:53:18,918 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
-
--------------------------------------------------------------------------------
2018-03-01 14:53:18,921 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Starting YARN TaskExecutor runner (Version: 1.5-SNAPSHOT,
Rev:e92eb39, Date:28.02.2018 @ 17:43:39 UTC)
2018-03-01 14:53:18,921 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- OS current user: yarn
2018-03-01 14:53:19,780 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Current Hadoop/Kerberos user: hadoop
2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- JVM: OpenJDK 64-Bit Server VM - Oracle Corporation -
1.8/25.161-b14
2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Maximum heap size: 245 MiBytes
2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- JAVA_HOME: /usr/lib/jvm/java-openjdk
2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Hadoop version: 2.4.1
2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- JVM Options:
2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Xms255m
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Xmx255m
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -XX:MaxDirectMemorySize=769m
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
-
-Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1150/container_1516373731080_1150_01_000105/taskmanager.log
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Dlogback.configurationFile=file:./logback.xml
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Dlog4j.configuration=file:./log4j.properties
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Program Arguments:
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- --configDir{noformat}
Heap was set to 255, while with default cuts of it should be 1383. 255MB seems
like coming from default taskmanager.heap.mb value of 1024.
When starting in non flip6 everything works as expected:
{noformat}
2018-03-01 14:04:49,650 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
--------------------------------------------------------------------------------
2018-03-01 14:04:49,700 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Starting YARN
TaskManager (Version: 1.5-SNAPSHOT, Rev:e92eb39, Date:28.02.2018 @ 17:43:39 UTC)
2018-03-01 14:04:49,700 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - OS current
user: yarn
2018-03-01 14:04:53,277 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Current
Hadoop/Kerberos user: hadoop
2018-03-01 14:04:53,278 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JVM: OpenJDK
64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
2018-03-01 14:04:53,279 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Maximum heap
size: 1326 MiBytes
2018-03-01 14:04:53,279 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JAVA_HOME:
/usr/lib/jvm/java-openjdk
2018-03-01 14:04:53,282 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Hadoop
version: 2.4.1
2018-03-01 14:04:53,284 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JVM Options:
2018-03-01 14:04:53,284 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Xms1383m
2018-03-01 14:04:53,284 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Xmx1383m
2018-03-01 14:04:53,284 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
-XX:MaxDirectMemorySize=1689m
2018-03-01 14:04:53,284 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
-Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1138/container_1516373731080_1138_01_000063/taskmanager.log
2018-03-01 14:04:53,285 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
-Dlogback.configurationFile=file:./logback.xml
2018-03-01 14:04:53,286 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
-Dlog4j.configuration=file:./log4j.properties
2018-03-01 14:04:53,287 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Program
Arguments:
2018-03-01 14:04:53,287 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - --configDir
2018-03-01 14:04:53,287 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - .{noformat}
CC [~till.rohrmann]
was:
When I tried running some job on the cluster, despite setting
taskmanager.heap.mb = 3072
taskmanager.network.memory.fraction: 0.4
and reported in the console
{{
Cluster specification: ClusterSpecification\{masterMemoryMB=768,
taskManagerMemoryMB=3072, numberTaskManagers=92, slotsPerTaskManager=1}
}}
The actual settings were:
{{
2018-03-01 14:53:18,918 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
-
--------------------------------------------------------------------------------
2018-03-01 14:53:18,921 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Starting YARN TaskExecutor runner (Version: 1.5-SNAPSHOT,
Rev:e92eb39, Date:28.02.2018 @ 17:43:39 UTC)
2018-03-01 14:53:18,921 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- OS current user: yarn
2018-03-01 14:53:19,780 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Current Hadoop/Kerberos user: hadoop
2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- JVM: OpenJDK 64-Bit Server VM - Oracle Corporation -
1.8/25.161-b14
2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Maximum heap size: 245 MiBytes
2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- JAVA_HOME: /usr/lib/jvm/java-openjdk
2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Hadoop version: 2.4.1
2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- JVM Options:
2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Xms255m
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Xmx255m
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -XX:MaxDirectMemorySize=769m
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
-
-Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1150/container_1516373731080_1150_01_000105/taskmanager.log
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Dlogback.configurationFile=file:./logback.xml
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Dlog4j.configuration=file:./log4j.properties
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Program Arguments:
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- --configDir
}}
Heap was set to 255, while with default cuts of it should be 1383. 255MB seems
like coming from default taskmanager.heap.mb value of 1024.
When starting in non flip6 everything works as expected:
{{
2018-03-01 14:04:49,650 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
--------------------------------------------------------------------------------
2018-03-01 14:04:49,700 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Starting YARN
TaskManager (Version: 1.5-SNAPSHOT, Rev:e92eb39, Date:28.02.2018 @ 17:43:39 UTC)
2018-03-01 14:04:49,700 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - OS current
user: yarn
2018-03-01 14:04:53,277 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Current
Hadoop/Kerberos user: hadoop
2018-03-01 14:04:53,278 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JVM: OpenJDK
64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
2018-03-01 14:04:53,279 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Maximum heap
size: 1326 MiBytes
2018-03-01 14:04:53,279 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JAVA_HOME:
/usr/lib/jvm/java-openjdk
2018-03-01 14:04:53,282 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Hadoop
version: 2.4.1
2018-03-01 14:04:53,284 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JVM Options:
2018-03-01 14:04:53,284 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Xms1383m
2018-03-01 14:04:53,284 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Xmx1383m
2018-03-01 14:04:53,284 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
-XX:MaxDirectMemorySize=1689m
2018-03-01 14:04:53,284 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
-Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1138/container_1516373731080_1138_01_000063/taskmanager.log
2018-03-01 14:04:53,285 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
-Dlogback.configurationFile=file:./logback.xml
2018-03-01 14:04:53,286 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
-Dlog4j.configuration=file:./log4j.properties
2018-03-01 14:04:53,287 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Program
Arguments:
2018-03-01 14:04:53,287 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - --configDir
2018-03-01 14:04:53,287 INFO
org.apache.flink.yarn.YarnTaskManagerRunnerFactory - .
}}
CC [~till.rohrmann]
> In Flip6 mode, when starting yarn cluster, configured taskmanager.heap.mb is
> ignored
> ------------------------------------------------------------------------------------
>
> Key: FLINK-8826
> URL: https://issues.apache.org/jira/browse/FLINK-8826
> Project: Flink
> Issue Type: Bug
> Components: ResourceManager, YARN
> Affects Versions: 1.5.0
> Reporter: Piotr Nowojski
> Priority: Blocker
>
> When I tried running some job on the cluster, despite setting
> taskmanager.heap.mb = 3072
> taskmanager.network.memory.fraction: 0.4
> and reported in the console
> {code:java}
> Cluster specification: ClusterSpecification{masterMemoryMB=768,
> taskManagerMemoryMB=3072, numberTaskManagers=92, slotsPerTaskManager=1}{code}
> The actual settings were:
> {noformat}
>
> 2018-03-01 14:53:18,918 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> -
> --------------------------------------------------------------------------------
> 2018-03-01 14:53:18,921 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - Starting YARN TaskExecutor runner (Version: 1.5-SNAPSHOT,
> Rev:e92eb39, Date:28.02.2018 @ 17:43:39 UTC)
> 2018-03-01 14:53:18,921 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - OS current user: yarn
> 2018-03-01 14:53:19,780 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - Current Hadoop/Kerberos user: hadoop
> 2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation -
> 1.8/25.161-b14
> 2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - Maximum heap size: 245 MiBytes
> 2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - JAVA_HOME: /usr/lib/jvm/java-openjdk
> 2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - Hadoop version: 2.4.1
> 2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - JVM Options:
> 2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - -Xms255m
> 2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - -Xmx255m
> 2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - -XX:MaxDirectMemorySize=769m
> 2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> -
> -Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1150/container_1516373731080_1150_01_000105/taskmanager.log
> 2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - -Dlogback.configurationFile=file:./logback.xml
> 2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - -Dlog4j.configuration=file:./log4j.properties
> 2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - Program Arguments:
> 2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - --configDir{noformat}
> Heap was set to 255, while with default cuts of it should be 1383. 255MB
> seems like coming from default taskmanager.heap.mb value of 1024.
> When starting in non flip6 everything works as expected:
> {noformat}
>
> 2018-03-01 14:04:49,650 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
> --------------------------------------------------------------------------------
> 2018-03-01 14:04:49,700 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Starting
> YARN TaskManager (Version: 1.5-SNAPSHOT, Rev:e92eb39, Date:28.02.2018 @
> 17:43:39 UTC)
> 2018-03-01 14:04:49,700 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory - OS current
> user: yarn
> 2018-03-01 14:04:53,277 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Current
> Hadoop/Kerberos user: hadoop
> 2018-03-01 14:04:53,278 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JVM: OpenJDK
> 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
> 2018-03-01 14:04:53,279 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Maximum heap
> size: 1326 MiBytes
> 2018-03-01 14:04:53,279 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JAVA_HOME:
> /usr/lib/jvm/java-openjdk
> 2018-03-01 14:04:53,282 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Hadoop
> version: 2.4.1
> 2018-03-01 14:04:53,284 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JVM Options:
> 2018-03-01 14:04:53,284 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Xms1383m
> 2018-03-01 14:04:53,284 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Xmx1383m
> 2018-03-01 14:04:53,284 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
> -XX:MaxDirectMemorySize=1689m
> 2018-03-01 14:04:53,284 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
> -Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1138/container_1516373731080_1138_01_000063/taskmanager.log
> 2018-03-01 14:04:53,285 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
> -Dlogback.configurationFile=file:./logback.xml
> 2018-03-01 14:04:53,286 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
> -Dlog4j.configuration=file:./log4j.properties
> 2018-03-01 14:04:53,287 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Program
> Arguments:
> 2018-03-01 14:04:53,287 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
> --configDir
> 2018-03-01 14:04:53,287 INFO
> org.apache.flink.yarn.YarnTaskManagerRunnerFactory -
> .{noformat}
>
> CC [~till.rohrmann]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
