Alex Smirnov created FLINK-9132:
-----------------------------------
Summary: Cluster runs out of task slots when a job falls into
restart loop
Key: FLINK-9132
URL: https://issues.apache.org/jira/browse/FLINK-9132
Project: Flink
Issue Type: Bug
Affects Versions: 1.4.2
Environment: env.java.opts in flink-conf.yaml file:
env.java.opts: -Xloggc:/home/user/flink/log/flinkServer-gc.log -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseG1GC
-XX:MaxGCPauseMillis=150 -XX:InitiatingHeapOccupancyPercent=55
-XX:+ParallelRefProcEnabled -XX:ParallelGCThreads=2 -XX:-ResizePLAB
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=100M
Reporter: Alex Smirnov
Attachments: FailedJob.java, jconsole-classes.png
If there's a job which is restarting in a loop, then Task Manager hosting it
goes down after some time. Job manager automatically assigns the job to another
Task Manager and the new Task Manager goes down as well. After some time, all
Task Managers are gone. Cluster becomes paralyzed.
I've attached to TaskManager's java process using jconsole and noticed that
number of loaded classes increases dramatically if a job is in restarting loop
and restores from checkpoint.
See attachment for the graph with G1GC enabled for the node. Standard GC
performs even worse - task manager shuts down within 20 minutes since the
restart loop start.
I've also attached minimal program to reproduce the problem
please let me know if additional information is required from me.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
