Thanks for sharing your use case and the memory usage table. I could able to reproduce the problem. Do you mind create an issue for it on our jira? I might have sometime next week to dig into this problem.
Thanks, moon On Thu, Feb 25, 2016 at 5:49 PM Dafeng Wang <dafe...@microsoft.com> wrote: > Hi Moon, > > > > Thanks for your reply, as for my case: my zeppelin server only have 1 > notebook with almost all queries are sparkSql query(30 paragraphs), the > result limitation I set is 10000, all query reach the limitation of 10000. > > The strange thing is: run them one by one won’t cause the OOM error, if I > click on run notebooks, it will OOM quickly. > > > > I check on the memory usage, the fastest increasing part is [B and [C, > however, it just tells me we do have memory increase, doesn’t hint any > other things, I put the table below for reference, the last one is the > latest one, you can see only *scala.reflect.io.VirtualFile > <http://localhost:7000/refsByType/0x616022bc8> *got increased when I run > it again and again, no obvious clue of memory leak. > > > > *scala.reflect.io.FileZipArchive$FileEntry$1 > <http://localhost:7000/refsByType/0x616022ae8>* > > 224886 > > *scala.reflect.io.FileZipArchive$FileEntry$1 > <http://localhost:7000/refsByType/0x616022ae8>* > > 224886 > > *scala.reflect.io.ZipArchive$DirEntry > <http://localhost:7000/refsByType/0x616022ca8>* > > 10123 > > *scala.reflect.io.ZipArchive$DirEntry > <http://localhost:7000/refsByType/0x616022ca8>* > > 10123 > > *sun.nio.cs.UTF_8$Encoder <http://localhost:7000/refsByType/0x61668d160>* > > 1090 > > *sun.nio.cs.UTF_8$Encoder <http://localhost:7000/refsByType/0x61668d160>* > > 1091 > > *java.util.zip.Inflater <http://localhost:7000/refsByType/0x615fb8d20>* > > 960 > > *java.util.zip.Inflater <http://localhost:7000/refsByType/0x615fb8d20>* > > 960 > > *org.apache.derby.iapi.services.io.FormatIdInputStream > <http://localhost:7000/refsByType/0x61b38f2a0>* > > 884 > > *org.apache.derby.iapi.services.io.FormatIdInputStream > <http://localhost:7000/refsByType/0x61b38f2a0>* > > 884 > > *sun.security.util.DerInputBuffer > <http://localhost:7000/refsByType/0x61684fc40>* > > 745 > > *sun.security.util.DerInputBuffer > <http://localhost:7000/refsByType/0x61684fc40>* > > 745 > > *org.apache.derby.iapi.services.io.FormatIdOutputStream > <http://localhost:7000/refsByType/0x61b3f5f20>* > > 479 > > *org.apache.derby.iapi.services.io.FormatIdOutputStream > <http://localhost:7000/refsByType/0x61b3f5f20>* > > 479 > > *sun.security.util.ObjectIdentifier > <http://localhost:7000/refsByType/0x615d02320>* > > 466 > > *sun.security.util.ObjectIdentifier > <http://localhost:7000/refsByType/0x615d02320>* > > 466 > > *org.apache.derby.iapi.services.io.ArrayInputStream > <http://localhost:7000/refsByType/0x61b215910>* > > 442 > > *org.apache.derby.iapi.services.io.ArrayInputStream > <http://localhost:7000/refsByType/0x61b215910>* > > 442 > > *org.apache.derby.iapi.services.io.ArrayOutputStream > <http://localhost:7000/refsByType/0x61b3f6000>* > > 442 > > *org.apache.derby.iapi.services.io.ArrayOutputStream > <http://localhost:7000/refsByType/0x61b3f6000>* > > 442 > > *org.apache.derby.iapi.services.io.FormatableBitSet > <http://localhost:7000/refsByType/0x617ae17e8>* > > 364 > > *org.apache.derby.iapi.services.io.FormatableBitSet > <http://localhost:7000/refsByType/0x617ae17e8>* > > 364 > > *scala.reflect.internal.pickling.UnPickler$Scan > <http://localhost:7000/refsByType/0x6166d6e20>* > > 282 > > *scala.reflect.internal.pickling.UnPickler$Scan > <http://localhost:7000/refsByType/0x6166d6e20>* > > 282 > > *org.apache.derby.impl.store.raw.data.StoredPage > <http://localhost:7000/refsByType/0x61ba57d88>* > > 279 > > *org.apache.derby.impl.store.raw.data.StoredPage > <http://localhost:7000/refsByType/0x61ba57d88>* > > 279 > > *java.util.jar.JarFile$JarFileEntry > <http://localhost:7000/refsByType/0x615e24618>* > > 276 > > *java.util.jar.JarFile$JarFileEntry > <http://localhost:7000/refsByType/0x615e24618>* > > 276 > > *sun.security.x509.X509CertImpl > <http://localhost:7000/refsByType/0x616083cf0>* > > 166 > > *sun.security.x509.X509CertImpl > <http://localhost:7000/refsByType/0x616083cf0>* > > 166 > > *org.apache.derby.impl.store.raw.data.AllocPage > <http://localhost:7000/refsByType/0x61b15ef28>* > > 162 > > *org.apache.derby.impl.store.raw.data.AllocPage > <http://localhost:7000/refsByType/0x61b15ef28>* > > 162 > > *org.spark-project.jetty.io.BufferCache$CachedBuffer > <http://localhost:7000/refsByType/0x616325310>* > > 155 > > *org.spark-project.jetty.io.BufferCache$CachedBuffer > <http://localhost:7000/refsByType/0x616325310>* > > 155 > > *[[B <http://localhost:7000/refsByType/0x616897178>* > > 150 > > *[[B <http://localhost:7000/refsByType/0x616897178>* > > 150 > > *org.spark-project.jetty.io.ByteArrayBuffer > <http://localhost:7000/refsByType/0x616309b60>* > > 138 > > *org.spark-project.jetty.io.ByteArrayBuffer > <http://localhost:7000/refsByType/0x616309b60>* > > 138 > > *org.apache.thrift.protocol.TBinaryProtocol > <http://localhost:7000/refsByType/0x615f91410>* > > 128 > > *org.apache.thrift.protocol.TBinaryProtocol > <http://localhost:7000/refsByType/0x615f91410>* > > 128 > > *scala.reflect.io.VirtualFile > <http://localhost:7000/refsByType/0x616022bc8>* > > 115 > > *scala.reflect.io.VirtualFile > <http://localhost:7000/refsByType/0x616022bc8>* > > 121 > > *org.apache.derby.impl.store.raw.data.RAFContainer4 > <http://localhost:7000/refsByType/0x61b974dc0>* > > 100 > > *org.apache.derby.impl.store.raw.data.RAFContainer4 > <http://localhost:7000/refsByType/0x61b974dc0>* > > 100 > > *sun.security.x509.KeyIdentifier > <http://localhost:7000/refsByType/0x616856000>* > > 95 > > *sun.security.x509.KeyIdentifier > <http://localhost:7000/refsByType/0x616856000>* > > 95 > > *java.lang.Class <http://localhost:7000/refsByType/0x615e60220>* > > 94 > > *java.lang.Class <http://localhost:7000/refsByType/0x615e60220>* > > 94 > > *sun.security.util.Cache$EqualByteArray > <http://localhost:7000/refsByType/0x61688c258>* > > 84 > > *sun.security.util.Cache$EqualByteArray > <http://localhost:7000/refsByType/0x61688c258>* > > 84 > > *sun.security.x509.X509CertInfo > <http://localhost:7000/refsByType/0x6163a8398>* > > 83 > > *sun.security.util.BitArray <http://localhost:7000/refsByType/0x61683bdb0>* > > 83 > > *sun.security.util.BitArray <http://localhost:7000/refsByType/0x61683bdb0>* > > 83 > > *sun.security.x509.X509CertInfo > <http://localhost:7000/refsByType/0x6163a8398>* > > 83 > > *sun.security.rsa.RSAPublicKeyImpl > <http://localhost:7000/refsByType/0x6160e75d0>* > > 80 > > *sun.security.rsa.RSAPublicKeyImpl > <http://localhost:7000/refsByType/0x6160e75d0>* > > 80 > > *sun.security.x509.BasicConstraintsExtension > <http://localhost:7000/refsByType/0x616568910>* > > 72 > > *sun.security.x509.BasicConstraintsExtension > <http://localhost:7000/refsByType/0x616568910>* > > 72 > > *sun.security.x509.SubjectKeyIdentifierExtension > <http://localhost:7000/refsByType/0x616568890>* > > 65 > > *sun.security.x509.SubjectKeyIdentifierExtension > <http://localhost:7000/refsByType/0x616568890>* > > 65 > > *sun.security.x509.KeyUsageExtension > <http://localhost:7000/refsByType/0x615e96ec0>* > > 62 > > *sun.security.x509.KeyUsageExtension > <http://localhost:7000/refsByType/0x615e96ec0>* > > 62 > > *sun.security.ec.NamedCurve <http://localhost:7000/refsByType/0x615e28ef0>* > > 46 > > *sun.security.ec.NamedCurve <http://localhost:7000/refsByType/0x615e28ef0>* > > 46 > > *io.netty.buffer.PoolChunk <http://localhost:7000/refsByType/0x61c0559a0>* > > 40 > > *io.netty.buffer.PoolChunk <http://localhost:7000/refsByType/0x61c0559a0>* > > 40 > > *scala.tools.nsc.symtab.classfile.Pickler$Pickle > <http://localhost:7000/refsByType/0x61637b1a0>* > > 34 > > *scala.tools.nsc.symtab.classfile.Pickler$Pickle > <http://localhost:7000/refsByType/0x61637b1a0>* > > 36 > > *java.nio.HeapByteBuffer <http://localhost:7000/refsByType/0x61672c5b0>* > > 32 > > *java.nio.HeapByteBuffer <http://localhost:7000/refsByType/0x61672c5b0>* > > 32 > > *sun.security.x509.AuthorityKeyIdentifierExtension > <http://localhost:7000/refsByType/0x615f917c8>* > > 30 > > *sun.security.x509.AuthorityKeyIdentifierExtension > <http://localhost:7000/refsByType/0x615f917c8>* > > 30 > > *java.security.cert.PolicyQualifierInfo > <http://localhost:7000/refsByType/0x61672f090>* > > 28 > > *java.security.cert.PolicyQualifierInfo > <http://localhost:7000/refsByType/0x61672f090>* > > 28 > > *java.io.ByteArrayOutputStream > <http://localhost:7000/refsByType/0x6162c3f98>* > > 24 > > *java.io.ByteArrayOutputStream > <http://localhost:7000/refsByType/0x6162c3f98>* > > 24 > > *java.util.jar.JarVerifier <http://localhost:7000/refsByType/0x615fc7838>* > > 22 > > *java.util.jar.JarVerifier <http://localhost:7000/refsByType/0x615fc7838>* > > 22 > > *sun.security.x509.CRLDistributionPointsExtension > <http://localhost:7000/refsByType/0x616568810>* > > 17 > > *sun.security.x509.CRLDistributionPointsExtension > <http://localhost:7000/refsByType/0x616568810>* > > 17 > > *java.io.BufferedOutputStream > <http://localhost:7000/refsByType/0x61623db10>* > > 13 > > *java.io.BufferedOutputStream > <http://localhost:7000/refsByType/0x61623db10>* > > 13 > > *java.net.SocketOutputStream > <http://localhost:7000/refsByType/0x61665ded8>* > > 12 > > *java.net.SocketOutputStream > <http://localhost:7000/refsByType/0x61665ded8>* > > 12 > > *sun.security.provider.SHA <http://localhost:7000/refsByType/0x615e756c0>* > > 12 > > *sun.security.x509.CertificatePoliciesExtension > <http://localhost:7000/refsByType/0x616568710>* > > 12 > > *sun.security.x509.CertificatePoliciesExtension > <http://localhost:7000/refsByType/0x616568710>* > > 12 > > *sun.security.provider.SHA <http://localhost:7000/refsByType/0x615e756c0>* > > 12 > > *java.io.BufferedInputStream > <http://localhost:7000/refsByType/0x61669a630>* > > 10 > > *java.io.BufferedInputStream > <http://localhost:7000/refsByType/0x61669a630>* > > 10 > > *sun.security.provider.SecureRandom > <http://localhost:7000/refsByType/0x615f128c8>* > > 10 > > *sun.security.provider.SecureRandom > <http://localhost:7000/refsByType/0x615f128c8>* > > 10 > > > > > > *From:* moon soo Lee [mailto:m...@apache.org] > *Sent:* Thursday, February 25, 2016 10:56 PM > *To:* users@zeppelin.incubator.apache.org > *Subject:* Re: OOM error when run all paragraphs > > > > Hi Dafeng, > > > > Zeppelin keeps notebooks in memory. > > That may cause OOM when notebooks has a lot of contents (code + result) > inside. Unfortunately, unless modify Zeppelin to not keep notebooks in > memory, there're no good way to reduce memory usage except for keep result > small. > > > > Most of notebooks i've seen is less than 2MB and i've seen people usually > has ~300 notebooks in their Zeppelin instance. > > > > Do you mind sharing your case? How big your notebook is and how many > notebooks your Zeppelin instance have? That might help zeppelin developers > to get idea how notebook need to be handled in ZeppelinServer. > > > > Thanks, > > moon > > > > On Wed, Feb 24, 2016 at 8:58 PM Dafeng Wang <dafe...@microsoft.com> wrote: > > It turns out zeppelin server itself oom since it cache too many results, > are there any methods/work arounds/fixes to get this issue done? > > > > Regards, > > Dafeng > > > > *From:* Dafeng Wang [mailto:dafe...@microsoft.com] > *Sent:* Thursday, February 25, 2016 10:31 AM > *To:* users@zeppelin.incubator.apache.org > *Subject:* OOM error when run all paragraphs > > > > Hi, > > Today when I try to run the zeppelin with a complicated notebook, if I run > them one by one, it works well, however, when I click on running all > paragraphs, it will throw OOM errors in a short time, I set zeppelin’s > interpreter’s memory as 6G. any idea? > > Regards, > > Dafeng > >
