Hi. I found the exception attached below, for our simple job. It states that our task-manager was killed du to exceed memory limit on 2.7GB. But when I look at Flink metricts just 30 sec before it use 1.3 GB heap and 712 MB Non-Heap around 2 GB. So something else are also using memory inside the conatianer any idea how to figure out what? As a side note we use RockDBStateBackend with this configuration
env.getCheckpointConfig().setMinPauseBetweenCheckpoints((long)(config.checkPointInterval * 0.75)); env.enableCheckpointing(config.checkPointInterval, CheckpointingMode.AT_LEAST_ONCE); env.setStateBackend(new RocksDBStateBackend(config.checkpointDataUri)); Where checkpointDataUri point to S3 Lasse Nedergaard 2018-04-09 16:52:01,239 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Diagnostics for container container_1522921976871_0001_01_000079 in state COMPLETE : exitStatus=Pmem limit exceeded (-104) diagnostics=Container [pid=30118,containerID=container_1522921976871_0001_01_000079] is running beyond physical memory limits. Current usage: 2.7 GB of 2.7 GB physical memory used; 4.9 GB of 13.4 GB virtual memory used. Killing container. Dump of the process-tree for container_1522921976871_0001_01_000079 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 30136 30118 30118 30118 (java) 245173 68463 5193723904 703845 /usr/lib/jvm/java-openjdk/bin/java -Xms2063m -Xmx2063m -Dlog.file=/var/log/hadoop-yarn/containers/application_1522921976871_0001/container_1522921976871_0001_01_000079/taskmanager.log -Dlogback.configurationFile=file:./logback.xml -Dlog4j.configuration=file:./log4j.properties org.apache.flink.yarn.YarnTaskManager --configDir . |- 30118 30116 30118 30118 (bash) 0 0 115818496 674 /bin/bash -c /usr/lib/jvm/java-openjdk/bin/java -Xms2063m -Xmx2063m -Dlog.file=/var/log/hadoop-yarn/containers/application_1522921976871_0001/container_1522921976871_0001_01_000079/taskmanager.log -Dlogback.configurationFile=file:./logback.xml -Dlog4j.configuration=file:./log4j.properties org.apache.flink.yarn.YarnTaskManager --configDir . 1> /var/log/hadoop-yarn/containers/application_1522921976871_0001/container_1522921976871_0001_01_000079/taskmanager.out 2> /var/log/hadoop-yarn/containers/application_1522921976871_0001/container_1522921976871_0001_01_000079/taskmanager.err 2018-04-09 16:51:26,659 DEBUG org.trackunit.tm2.LogReporter - gauge.ip-10-1-1-181.taskmanager.container_1522921976871_0001_01_000079.Status.JVM.Memory.Heap.Used=1398739496 2018-04-09 16:51:26,659 DEBUG org.trackunit.tm2.LogReporter - gauge.ip-10-1-1-181.taskmanager.container_1522921976871_0001_01_000079.Status.JVM.Memory.NonHeap.Used=746869520 2018-04-09 23:52 GMT+02:00 Ken Krugler <kkrugler_li...@transpac.com>: > Hi Chesnay, > > Don’t know if this helps, but I’d run into this as well, though I haven’t > hooked up YourKit to analyze exactly what’s causing the memory problem. > > E.g. after about 3.5 hours running locally, it failed with memory issues. > > In the TaskManager logs, I start seeing exceptions in my code…. > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > And then eventually... > > 2018-04-07 21:55:25,686 WARN > org.apache.flink.runtime.accumulators.AccumulatorRegistry > - Failed to serialize accumulators for task. > java.lang.OutOfMemoryError: GC overhead limit exceeded > > Immediately after this, one of my custom functions gets a close() call, > and I see a log msg about it "switched from RUNNING to FAILED”. > > After this, I see messages that the job is being restarted, but the > TaskManager log output abruptly ends. > > In the Job Manager log, this is what is output following the time of the > last TaskManager logging output: > > 2018-04-07 21:57:33,702 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator > - Triggering checkpoint 129 @ 1523163453702 > 2018-04-07 21:58:43,916 WARN akka.remote.ReliableDeliverySupervisor > - Association with remote system [ > akka.tcp://fl...@kens-mbp.hsd1.ca.comcast.net:63780] has failed, address > is now gated for [5000] ms. Reason: [Disassociated] > 2018-04-07 21:58:51,084 WARN akka.remote.transport.netty.NettyTransport > - Remote connection to [null] failed with > java.net.ConnectException: Connection refused: kens- > mbp.hsd1.ca.comcast.net/192.168.3.177:63780 > 2018-04-07 21:58:51,086 WARN akka.remote.ReliableDeliverySupervisor > - Association with remote system [akka.tcp:// > fl...@kens-mbp.hsd1.ca.comcast.net:63780] has failed, address is now > gated for [5000] ms. Reason: [Association failed with [akka.tcp:// > fl...@kens-mbp.hsd1.ca.comcast.net:63780]] Caused by: [Connection > refused: kens-mbp.hsd1.ca.comcast.net/192.168.3.177:63780] > 2018-04-07 21:59:01,047 WARN akka.remote.transport.netty.NettyTransport > - Remote connection to [null] failed with > java.net.ConnectException: Connection refused: > kens-mbp.hsd1.ca.comcast.net/192.168.3.177:63780 > 2018-04-07 21:59:01,050 WARN akka.remote.ReliableDeliverySupervisor > - Association with remote system [akka.tcp:// > fl...@kens-mbp.hsd1.ca.comcast.net:63780] has failed, address is now > gated for [5000] ms. Reason: [Association failed with [akka.tcp:// > fl...@kens-mbp.hsd1.ca.comcast.net:63780]] Caused by: [Connection > refused: kens-mbp.hsd1.ca.comcast.net/192.168.3.177:63780] > 2018-04-07 21:59:11,057 WARN akka.remote.ReliableDeliverySupervisor > - Association with remote system [akka.tcp:// > fl...@kens-mbp.hsd1.ca.comcast.net:63780] has failed, address is now > gated for [5000] ms. Reason: [Association failed with [akka.tcp:// > fl...@kens-mbp.hsd1.ca.comcast.net:63780]] Caused by: [Connection > refused: kens-mbp.hsd1.ca.comcast.net/192.168.3.177:63780] > 2018-04-07 21:59:11,058 WARN akka.remote.transport.netty.NettyTransport > - Remote connection to [null] failed with > java.net.ConnectException: Connection refused: > kens-mbp.hsd1.ca.comcast.net/192.168.3.177:63780 > 2018-04-07 21:59:21,049 WARN akka.remote.transport.netty.NettyTransport > - Remote connection to [null] failed with > java.net.ConnectException: Connection refused: > kens-mbp.hsd1.ca.comcast.net/192.168.3.177:63780 > 2018-04-07 21:59:21,049 WARN akka.remote.ReliableDeliverySupervisor > - Association with remote system [akka.tcp:// > fl...@kens-mbp.hsd1.ca.comcast.net:63780] has failed, address is now > gated for [5000] ms. Reason: [Association failed with [akka.tcp:// > fl...@kens-mbp.hsd1.ca.comcast.net:63780]] Caused by: [Connection > refused: kens-mbp.hsd1.ca.comcast.net/192.168.3.177:63780] > 2018-04-07 21:59:21,056 WARN akka.remote.RemoteWatcher > - Detected unreachable: [akka.tcp://flink@kens-mbp. > hsd1.ca.comcast.net:63780] > 2018-04-07 21:59:21,063 INFO org.apache.flink.runtime.jobmanager.JobManager > - Task manager akka.tcp://flink@kens-mbp. > hsd1.ca.comcast.net:63780/user/taskmanager terminated. > 2018-04-07 21:59:21,064 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph > - FetchUrlsFunction for sitemap -> ParseSiteMapFunction -> > OutlinkToStateUrlFunction (1/1) (3e9374d1bf5fdb359e3a624a4d5d659b) > switched from RUNNING to FAILED. > java.lang.Exception: TaskManager was lost/killed: > c51d3879b6244828eb9fc78c943007ad @ kens-mbp.hsd1.ca.comcast.net > (dataPort=63782) > > — Ken > > > > On Apr 9, 2018, at 12:48 PM, Chesnay Schepler <ches...@apache.org> wrote: > > We will need more information to offer any solution. The exception simply > means that a TaskManager shut down, for which there are a myriad of > possible explanations. > > Please have a look at the TaskManager logs, they may contain a hint as to > why it shut down. > > On 09.04.2018 16:01, Javier Lopez wrote: > > Hi, > > "are you moving the job jar to the ~/flink-1.4.2/lib path ? " -> Yes, > to every node in the cluster. > > On 9 April 2018 at 15:37, miki haiat <miko5...@gmail.com> wrote: > >> Javier >> "adding the jar file to the /lib path of every task manager" >> are you moving the job jar to the* ~/flink-1.4.2/lib path* ? >> >> On Mon, Apr 9, 2018 at 12:23 PM, Javier Lopez <javier.lo...@zalando.de> >> wrote: >> >>> Hi, >>> >>> We had the same metaspace problem, it was solved by adding the jar file >>> to the /lib path of every task manager, as explained here >>> https://ci.apache.org/projects/flink/flink-docs-release >>> -1.4/monitoring/debugging_classloading.html#avoiding-dynamic >>> -classloading. As well we added these java options: >>> "-XX:CompressedClassSpaceSize=100M -XX:MaxMetaspaceSize=300M >>> -XX:MetaspaceSize=200M " >>> >>> From time to time we have the same problem with TaskManagers >>> disconnecting, but the logs are not useful. We are using 1.3.2. >>> >>> On 9 April 2018 at 10:41, Alexander Smirnov < >>> alexander.smirn...@gmail.com> wrote: >>> >>>> I've seen similar problem, but it was not a heap size, but Metaspace. >>>> It was caused by a job restarting in a loop. Looks like for each >>>> restart, Flink loads new instance of classes and very soon in runs out of >>>> metaspace. >>>> >>>> I've created a JIRA issue for this problem, but got no response from >>>> the development team on it: https://issues.apache.org/ >>>> jira/browse/FLINK-9132 >>>> >>>> >>>> On Mon, Apr 9, 2018 at 11:36 AM 王凯 <wangka...@163.com> wrote: >>>> >>>>> thanks a lot,i will try it >>>>> >>>>> 在 2018-04-09 00:06:02,"TechnoMage" <mla...@technomage.com> 写道: >>>>> >>>>> I have seen this when my task manager ran out of RAM. Increase the >>>>> heap size. >>>>> >>>>> flink-conf.yaml: >>>>> taskmanager.heap.mb >>>>> jobmanager.heap.mb >>>>> >>>>> Michael >>>>> >>>>> On Apr 8, 2018, at 2:36 AM, 王凯 <wangka...@163.com> wrote: >>>>> >>>>> <QQ图片20180408163927.png> >>>>> hi all, recently, i found a problem,it runs well when start. But >>>>> after long run,the exception display as above,how can resolve it? >>>>> >>>>> > -------------------------------------------- > http://about.me/kkrugler > +1 530-210-6378 > >