Re: driver OOM due to io.netty.buffer items not getting finalized

fyi after further troubleshooting logging this as  https://issues.apache.org/jira/browse/SPARK-12511 On Tuesday, 22 December 2015, 18:16, Antony Mayi wrote: I narrowed it down to problem described for example here: 

driver OOM due to io.netty.buffer items not getting finalized

I have streaming app (pyspark 1.5.2 on yarn) that's crashing due to driver (jvm part, not python) OOM (no matter how big heap is assigned, eventually runs out). When checking the heap it is all taken by "byte" items of io.netty.buffer.PoolThreadCache. The number of

Re: driver OOM due to io.netty.buffer items not getting finalized

This might be related but the jmap output there looks different: http://stackoverflow.com/questions/32537965/huge-number-of-io-netty-buffer-poolthreadcachememoryregioncacheentry-instances On Tue, Dec 22, 2015 at 2:59 AM, Antony Mayi wrote: > I have streaming app

Re: driver OOM due to io.netty.buffer items not getting finalized

I narrowed it down to problem described for example here:  https://bugs.openjdk.java.net/browse/JDK-6293787 It is the mass finalization of zip Inflater/Deflater objects which can't keep up with the rate of these instances being garbage collected. as the jdk bug report (not being accepted as a

Re: driver OOM due to io.netty.buffer items not getting finalized

I searched code briefly. The following uses ZipEntry, ZipOutputStream : core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala core/src/main/scala/org/apache/spark/deploy/RPackageUtils.scala FYI On Tue, Dec 22, 2015 at 9:16 AM, Antony Mayi