Hi, Alexander I could not reproduce it in my local environment. Normally, Mesos RM will calculate all the mem config and add it to the launch command. Unfortunately, all the log I could found for this command is at the DEBUG level. Would you mind changing the log level to DEBUG or sharing anything about the taskmanager launch command you could found in the current log?
Best, Yangze Guo On Thu, Mar 12, 2020 at 1:38 PM Alexander Kasyanenko <as.kasyane...@gmail.com> wrote: > > Hi folks, > > I have a question related configuration for new memory introduced in flink > 1.10. Has anyone encountered similar problem? > I'm trying to make use of taskmanager.memory.process.size configuration key > in combination with mesos session cluster, but I get an error like this: > > 2020-03-11 11:44:09,771 [main] ERROR > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - Error while > starting the TaskManager > org.apache.flink.configuration.IllegalConfigurationException: Failed to > create TaskExecutorResourceSpec > at > org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.resourceSpecFromConfig(TaskExecutorResourceUtils.java:72) > at > org.apache.flink.runtime.taskexecutor.TaskManagerRunner.startTaskManager(TaskManagerRunner.java:356) > at > org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<init>(TaskManagerRunner.java:152) > at > org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:308) > at > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner.lambda$main$0(MesosTaskExecutorRunner.java:106) > at java.base/java.security.AccessController.doPrivileged(Native Method) > at java.base/javax.security.auth.Subject.doAs(Subject.java:423) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692) > at > org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner.main(MesosTaskExecutorRunner.java:105) > Caused by: org.apache.flink.configuration.IllegalConfigurationException: The > required configuration option Key: 'taskmanager.memory.task.heap.size' , > default: null (fallback keys: []) is not set > at > org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.checkConfigOptionIsSet(TaskExecutorResourceUtils.java:90) > at > org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.lambda$checkTaskExecutorResourceConfigSet$0(TaskExecutorResourceUtils.java:84) > at java.base/java.util.Arrays$ArrayList.forEach(Arrays.java:4390) > at > org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.checkTaskExecutorResourceConfigSet(TaskExecutorResourceUtils.java:84) > at > org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.resourceSpecFromConfig(TaskExecutorResourceUtils.java:70) > ... 9 more > > But when task manager is launched, it correctly parses process memory key: > > 2020-03-11 11:43:55,376 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - > -------------------------------------------------------------------------------- > 2020-03-11 11:43:55,377 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - Starting > MesosTaskExecutorRunner (Version: 1.10.0, Rev:aa4eb8f, Date:07.02.2020 @ > 19:18:19 CET) > 2020-03-11 11:43:55,377 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - OS current > user: root > 2020-03-11 11:43:57,347 [main] WARN org.apache.hadoop.util.NativeCodeLoader > - Unable to load native-hadoop library for your > platform... using builtin-java classes where applicable > 2020-03-11 11:43:57,535 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - JVM: OpenJDK > 64-Bit Server VM - AdoptOpenJDK - 11/11.0.2+9 > 2020-03-11 11:43:57,535 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - Maximum heap > size: 746 MiBytes > 2020-03-11 11:43:57,535 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - JAVA_HOME: > (not set) > 2020-03-11 11:43:57,539 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - Hadoop > version: 2.6.5 > 2020-03-11 11:43:57,539 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - JVM Options: > 2020-03-11 11:43:57,539 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - > -Xmx781818251 > 2020-03-11 11:43:57,539 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - > -Xms781818251 > 2020-03-11 11:43:57,540 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - > -XX:MaxDirectMemorySize=317424929 > 2020-03-11 11:43:57,540 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - > -XX:MaxMetaspaceSize=100663296 > 2020-03-11 11:43:57,540 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - > -Dlog.file=/var/log/flink-session-cluster/taskmanager.log > 2020-03-11 11:43:57,540 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - > -Dlog4j.configuration=file:/opt/flink/conf/log4j.properties > 2020-03-11 11:43:57,540 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - > -Dlogback.configurationFile=file:/opt/flink/conf/logback.xml > 2020-03-11 11:43:57,540 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - Program > Arguments: (none) > 2020-03-11 11:43:57,540 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - Classpath: > /opt/flink/lib/apache-log4j-extras-1.2.17.jar:/opt/flink/lib/flink-metrics-graphite-1.10.0.jar:/opt/flink/lib/flink-shaded-hadoop-2-uber-2.6.5-8.0.jar:/opt/flink/lib/flink-table-blink_2.12-1.10.0.jar:/opt/flink/lib/flink-table_2.12-1.10.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/flink/lib/flink-dist_2.12-1.10.0.jar: > 2020-03-11 11:43:57,541 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - > -------------------------------------------------------------------------------- > 2020-03-11 11:43:57,542 [main] INFO > org.apache.flink.mesos.entrypoint.MesosTaskExecutorRunner - Registered > UNIX signal handlers for [TERM, HUP, INT] > 2020-03-11 11:43:57,550 [main] INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: taskmanager.memory.process.size, 2g > 2020-03-11 11:43:57,550 [main] INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: taskmanager.cpu.cores, 2 > 2020-03-11 11:43:57,551 [main] INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: taskmanager.numberOfTaskSlots, 4 > 2020-03-11 11:43:57,551 [main] INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: parallelism.default, 1 > ... > > Judging by the docs specifying taskmanager.memory.process.size key should be > enough to launch the job, but it seems like this value is ignored. > I would appreciate any suggestion. > > Regards and thanks in advance, > Alex.
