Hi Moon, Thanks for your reply, as for my case: my zeppelin server only have 1 notebook with almost all queries are sparkSql query(30 paragraphs), the result limitation I set is 10000, all query reach the limitation of 10000. The strange thing is: run them one by one won’t cause the OOM error, if I click on run notebooks, it will OOM quickly.
I check on the memory usage, the fastest increasing part is [B and [C, however, it just tells me we do have memory increase, doesn’t hint any other things, I put the table below for reference, the last one is the latest one, you can see only scala.reflect.io.VirtualFile<http://localhost:7000/refsByType/0x616022bc8> got increased when I run it again and again, no obvious clue of memory leak. scala.reflect.io.FileZipArchive$FileEntry$1<http://localhost:7000/refsByType/0x616022ae8> 224886 scala.reflect.io.FileZipArchive$FileEntry$1<http://localhost:7000/refsByType/0x616022ae8> 224886 scala.reflect.io.ZipArchive$DirEntry<http://localhost:7000/refsByType/0x616022ca8> 10123 scala.reflect.io.ZipArchive$DirEntry<http://localhost:7000/refsByType/0x616022ca8> 10123 sun.nio.cs.UTF_8$Encoder<http://localhost:7000/refsByType/0x61668d160> 1090 sun.nio.cs.UTF_8$Encoder<http://localhost:7000/refsByType/0x61668d160> 1091 java.util.zip.Inflater<http://localhost:7000/refsByType/0x615fb8d20> 960 java.util.zip.Inflater<http://localhost:7000/refsByType/0x615fb8d20> 960 org.apache.derby.iapi.services.io.FormatIdInputStream<http://localhost:7000/refsByType/0x61b38f2a0> 884 org.apache.derby.iapi.services.io.FormatIdInputStream<http://localhost:7000/refsByType/0x61b38f2a0> 884 sun.security.util.DerInputBuffer<http://localhost:7000/refsByType/0x61684fc40> 745 sun.security.util.DerInputBuffer<http://localhost:7000/refsByType/0x61684fc40> 745 org.apache.derby.iapi.services.io.FormatIdOutputStream<http://localhost:7000/refsByType/0x61b3f5f20> 479 org.apache.derby.iapi.services.io.FormatIdOutputStream<http://localhost:7000/refsByType/0x61b3f5f20> 479 sun.security.util.ObjectIdentifier<http://localhost:7000/refsByType/0x615d02320> 466 sun.security.util.ObjectIdentifier<http://localhost:7000/refsByType/0x615d02320> 466 org.apache.derby.iapi.services.io.ArrayInputStream<http://localhost:7000/refsByType/0x61b215910> 442 org.apache.derby.iapi.services.io.ArrayInputStream<http://localhost:7000/refsByType/0x61b215910> 442 org.apache.derby.iapi.services.io.ArrayOutputStream<http://localhost:7000/refsByType/0x61b3f6000> 442 org.apache.derby.iapi.services.io.ArrayOutputStream<http://localhost:7000/refsByType/0x61b3f6000> 442 org.apache.derby.iapi.services.io.FormatableBitSet<http://localhost:7000/refsByType/0x617ae17e8> 364 org.apache.derby.iapi.services.io.FormatableBitSet<http://localhost:7000/refsByType/0x617ae17e8> 364 scala.reflect.internal.pickling.UnPickler$Scan<http://localhost:7000/refsByType/0x6166d6e20> 282 scala.reflect.internal.pickling.UnPickler$Scan<http://localhost:7000/refsByType/0x6166d6e20> 282 org.apache.derby.impl.store.raw.data.StoredPage<http://localhost:7000/refsByType/0x61ba57d88> 279 org.apache.derby.impl.store.raw.data.StoredPage<http://localhost:7000/refsByType/0x61ba57d88> 279 java.util.jar.JarFile$JarFileEntry<http://localhost:7000/refsByType/0x615e24618> 276 java.util.jar.JarFile$JarFileEntry<http://localhost:7000/refsByType/0x615e24618> 276 sun.security.x509.X509CertImpl<http://localhost:7000/refsByType/0x616083cf0> 166 sun.security.x509.X509CertImpl<http://localhost:7000/refsByType/0x616083cf0> 166 org.apache.derby.impl.store.raw.data.AllocPage<http://localhost:7000/refsByType/0x61b15ef28> 162 org.apache.derby.impl.store.raw.data.AllocPage<http://localhost:7000/refsByType/0x61b15ef28> 162 org.spark-project.jetty.io.BufferCache$CachedBuffer<http://localhost:7000/refsByType/0x616325310> 155 org.spark-project.jetty.io.BufferCache$CachedBuffer<http://localhost:7000/refsByType/0x616325310> 155 [[B<http://localhost:7000/refsByType/0x616897178> 150 [[B<http://localhost:7000/refsByType/0x616897178> 150 org.spark-project.jetty.io.ByteArrayBuffer<http://localhost:7000/refsByType/0x616309b60> 138 org.spark-project.jetty.io.ByteArrayBuffer<http://localhost:7000/refsByType/0x616309b60> 138 org.apache.thrift.protocol.TBinaryProtocol<http://localhost:7000/refsByType/0x615f91410> 128 org.apache.thrift.protocol.TBinaryProtocol<http://localhost:7000/refsByType/0x615f91410> 128 scala.reflect.io.VirtualFile<http://localhost:7000/refsByType/0x616022bc8> 115 scala.reflect.io.VirtualFile<http://localhost:7000/refsByType/0x616022bc8> 121 org.apache.derby.impl.store.raw.data.RAFContainer4<http://localhost:7000/refsByType/0x61b974dc0> 100 org.apache.derby.impl.store.raw.data.RAFContainer4<http://localhost:7000/refsByType/0x61b974dc0> 100 sun.security.x509.KeyIdentifier<http://localhost:7000/refsByType/0x616856000> 95 sun.security.x509.KeyIdentifier<http://localhost:7000/refsByType/0x616856000> 95 java.lang.Class<http://localhost:7000/refsByType/0x615e60220> 94 java.lang.Class<http://localhost:7000/refsByType/0x615e60220> 94 sun.security.util.Cache$EqualByteArray<http://localhost:7000/refsByType/0x61688c258> 84 sun.security.util.Cache$EqualByteArray<http://localhost:7000/refsByType/0x61688c258> 84 sun.security.x509.X509CertInfo<http://localhost:7000/refsByType/0x6163a8398> 83 sun.security.util.BitArray<http://localhost:7000/refsByType/0x61683bdb0> 83 sun.security.util.BitArray<http://localhost:7000/refsByType/0x61683bdb0> 83 sun.security.x509.X509CertInfo<http://localhost:7000/refsByType/0x6163a8398> 83 sun.security.rsa.RSAPublicKeyImpl<http://localhost:7000/refsByType/0x6160e75d0> 80 sun.security.rsa.RSAPublicKeyImpl<http://localhost:7000/refsByType/0x6160e75d0> 80 sun.security.x509.BasicConstraintsExtension<http://localhost:7000/refsByType/0x616568910> 72 sun.security.x509.BasicConstraintsExtension<http://localhost:7000/refsByType/0x616568910> 72 sun.security.x509.SubjectKeyIdentifierExtension<http://localhost:7000/refsByType/0x616568890> 65 sun.security.x509.SubjectKeyIdentifierExtension<http://localhost:7000/refsByType/0x616568890> 65 sun.security.x509.KeyUsageExtension<http://localhost:7000/refsByType/0x615e96ec0> 62 sun.security.x509.KeyUsageExtension<http://localhost:7000/refsByType/0x615e96ec0> 62 sun.security.ec.NamedCurve<http://localhost:7000/refsByType/0x615e28ef0> 46 sun.security.ec.NamedCurve<http://localhost:7000/refsByType/0x615e28ef0> 46 io.netty.buffer.PoolChunk<http://localhost:7000/refsByType/0x61c0559a0> 40 io.netty.buffer.PoolChunk<http://localhost:7000/refsByType/0x61c0559a0> 40 scala.tools.nsc.symtab.classfile.Pickler$Pickle<http://localhost:7000/refsByType/0x61637b1a0> 34 scala.tools.nsc.symtab.classfile.Pickler$Pickle<http://localhost:7000/refsByType/0x61637b1a0> 36 java.nio.HeapByteBuffer<http://localhost:7000/refsByType/0x61672c5b0> 32 java.nio.HeapByteBuffer<http://localhost:7000/refsByType/0x61672c5b0> 32 sun.security.x509.AuthorityKeyIdentifierExtension<http://localhost:7000/refsByType/0x615f917c8> 30 sun.security.x509.AuthorityKeyIdentifierExtension<http://localhost:7000/refsByType/0x615f917c8> 30 java.security.cert.PolicyQualifierInfo<http://localhost:7000/refsByType/0x61672f090> 28 java.security.cert.PolicyQualifierInfo<http://localhost:7000/refsByType/0x61672f090> 28 java.io.ByteArrayOutputStream<http://localhost:7000/refsByType/0x6162c3f98> 24 java.io.ByteArrayOutputStream<http://localhost:7000/refsByType/0x6162c3f98> 24 java.util.jar.JarVerifier<http://localhost:7000/refsByType/0x615fc7838> 22 java.util.jar.JarVerifier<http://localhost:7000/refsByType/0x615fc7838> 22 sun.security.x509.CRLDistributionPointsExtension<http://localhost:7000/refsByType/0x616568810> 17 sun.security.x509.CRLDistributionPointsExtension<http://localhost:7000/refsByType/0x616568810> 17 java.io.BufferedOutputStream<http://localhost:7000/refsByType/0x61623db10> 13 java.io.BufferedOutputStream<http://localhost:7000/refsByType/0x61623db10> 13 java.net.SocketOutputStream<http://localhost:7000/refsByType/0x61665ded8> 12 java.net.SocketOutputStream<http://localhost:7000/refsByType/0x61665ded8> 12 sun.security.provider.SHA<http://localhost:7000/refsByType/0x615e756c0> 12 sun.security.x509.CertificatePoliciesExtension<http://localhost:7000/refsByType/0x616568710> 12 sun.security.x509.CertificatePoliciesExtension<http://localhost:7000/refsByType/0x616568710> 12 sun.security.provider.SHA<http://localhost:7000/refsByType/0x615e756c0> 12 java.io.BufferedInputStream<http://localhost:7000/refsByType/0x61669a630> 10 java.io.BufferedInputStream<http://localhost:7000/refsByType/0x61669a630> 10 sun.security.provider.SecureRandom<http://localhost:7000/refsByType/0x615f128c8> 10 sun.security.provider.SecureRandom<http://localhost:7000/refsByType/0x615f128c8> 10 From: moon soo Lee [mailto:m...@apache.org] Sent: Thursday, February 25, 2016 10:56 PM To: users@zeppelin.incubator.apache.org Subject: Re: OOM error when run all paragraphs Hi Dafeng, Zeppelin keeps notebooks in memory. That may cause OOM when notebooks has a lot of contents (code + result) inside. Unfortunately, unless modify Zeppelin to not keep notebooks in memory, there're no good way to reduce memory usage except for keep result small. Most of notebooks i've seen is less than 2MB and i've seen people usually has ~300 notebooks in their Zeppelin instance. Do you mind sharing your case? How big your notebook is and how many notebooks your Zeppelin instance have? That might help zeppelin developers to get idea how notebook need to be handled in ZeppelinServer. Thanks, moon On Wed, Feb 24, 2016 at 8:58 PM Dafeng Wang <dafe...@microsoft.com<mailto:dafe...@microsoft.com>> wrote: It turns out zeppelin server itself oom since it cache too many results, are there any methods/work arounds/fixes to get this issue done? Regards, Dafeng From: Dafeng Wang [mailto:dafe...@microsoft.com<mailto:dafe...@microsoft.com>] Sent: Thursday, February 25, 2016 10:31 AM To: users@zeppelin.incubator.apache.org<mailto:users@zeppelin.incubator.apache.org> Subject: OOM error when run all paragraphs Hi, Today when I try to run the zeppelin with a complicated notebook, if I run them one by one, it works well, however, when I click on running all paragraphs, it will throw OOM errors in a short time, I set zeppelin’s interpreter’s memory as 6G. any idea? Regards, Dafeng
