I didn't go into the details of the failures, but the exception was caused by netty's not being able to allocate a new chunk of direct memory.
On Tue, Sep 15, 2015 at 10:21 AM, Jacques Nadeau <[email protected]> wrote: > I've seen sporadic failures with those tests. I'm wondering whether they > need to be redesigned. Limiting direct memory to 8gb shouldn't fail the > test. > > -- > Jacques Nadeau > CTO and Co-Founder, Dremio > > On Tue, Sep 15, 2015 at 9:39 AM, Abdel Hakim Deneche < > [email protected]> > wrote: > > > I found what was causing those errors on my VM. I exported a MAVEN_OPTS > > that limits the direct memory to 8GB and forget to unset it, this was > > causing the build to fail because it couldn't allocate more direct > memory, > > for some reason it was always failing on those TestValueVector unit > tests. > > > > On Wed, Sep 2, 2015 at 10:10 AM, Abdel Hakim Deneche < > > [email protected]> > > wrote: > > > > > I am still seeing the TestValueVector unit tests fail on my linux > machine > > > (2/3 of my builds) > > > > > > On Fri, Aug 14, 2015 at 2:40 PM, Abdel Hakim Deneche < > > > [email protected]> wrote: > > > > > >> You can safely ignore the TestImpersonationMetadata error, it's most > > >> likely caused by my change > > >> > > >> On Fri, Aug 14, 2015 at 12:36 PM, Abdel Hakim Deneche < > > >> [email protected]> wrote: > > >> > > >>> In addition to test*VectorReallocation errors I am seeing frequently > on > > >>> my linux VM, today I saw the following error multiple times on my VM > > and > > >>> once on my Macbook: > > >>> > > >>> Tests in error: > > >>>> TestImpersonationMetadata>BaseTestQuery.closeClient:239 ยป > > >>>> IllegalState Failure... > > >>>> java.lang.IllegalStateException: Failure while closing accountor. > > >>>> Expected private and shared pools to be set to initial values. > > However, > > >>>> one or more were not. Stats are > > >>>> zone init allocated delta > > >>>> private 0 0 0 > > >>>> shared 3221225472 3214166779 7058693. > > >>>> at > > >>>> > > > org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:200) > > >>>> ~[classes/:na] > > >>>> at org.apache.drill.exec.memory.Accountor.close(Accountor.java:390) > > >>>> ~[classes/:na] > > >>>> at > > >>>> > > > org.apache.drill.exec.memory.TopLevelAllocator.close(TopLevelAllocator.java:185) > > >>>> ~[classes/:na] > > >>>> at > > >>>> > > > org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:75) > > >>>> ~[classes/:na] > > >>>> at com.google.common.io.Closeables.close(Closeables.java:77) > > >>>> ~[guava-14.0.1.jar:na] > > >>>> at com.google.common.io.Closeables.closeQuietly(Closeables.java:108) > > >>>> ~[guava-14.0.1.jar:na] > > >>>> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:294) > > >>>> ~[classes/:na] > > >>>> at > > >>>> > > > org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:332) > > >>>> ~[classes/:na] > > >>> > > >>> > > >>> On Wed, Aug 5, 2015 at 2:52 PM, Chris Westin < > [email protected]> > > >>> wrote: > > >>> > > >>>> Given that the difference is just > > >>>> > > >>>> > java.lang.Exception: Unexpected exception, > > >>>> > > > >>>> > > expected<org.apache.drill.exec.exception.OversizedAllocationException> > but > > >>>> > was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> > > >>>> > > >>>> The question of "what constitutes an oversized allocation?" comes to > > >>>> mind. > > >>>> Is this test fragile relative to being run in different > environments? > > >>>> I haven't seen the test so how is the determination that something > is > > >>>> oversized made? It seems like that criterion sometimes fails, and we > > >>>> get an > > >>>> OOM because whatever the request is is still very large. > > >>>> > > >>>> > > >>>> On Wed, Aug 5, 2015 at 2:26 PM, Hanifi Gunes <[email protected]> > > >>>> wrote: > > >>>> > > >>>> > I don't seem to be able to re-prod this. Let me look at this and > > >>>> update you > > >>>> > all. > > >>>> > > > >>>> > On Thu, Aug 6, 2015 at 12:03 AM, Abdel Hakim Deneche < > > >>>> > [email protected]> > > >>>> > wrote: > > >>>> > > > >>>> > > I didn't make any change, I'm running 2 forks (the default). I > got > > >>>> those > > >>>> > > errors 3 times now, 2 on a linux VM and 1 on a linux physical > node > > >>>> > > > > >>>> > > On Wed, Aug 5, 2015 at 1:03 PM, Hanifi Gunes < > [email protected] > > > > > >>>> > wrote: > > >>>> > > > > >>>> > > > Did you tighten your memory settings? How many forks are you > > >>>> running > > >>>> > > with? > > >>>> > > > I bet you are truly running out of memory while executing this > > >>>> > particular > > >>>> > > > test case. > > >>>> > > > > > >>>> > > > -H+ > > >>>> > > > > > >>>> > > > On Wed, Aug 5, 2015 at 8:56 PM, Sudheesh Katkam < > > >>>> [email protected]> > > >>>> > > > wrote: > > >>>> > > > > > >>>> > > > > b2bbd99 committed on July 6th introduced the test. > > >>>> > > > > > > >>>> > > > > > On Aug 5, 2015, at 10:21 AM, Jinfeng Ni < > > >>>> [email protected]> > > >>>> > > wrote: > > >>>> > > > > > > > >>>> > > > > > In that case, we probably need do binary search to figure > > out > > >>>> > which > > >>>> > > > > recent > > >>>> > > > > > patch is causing this problem. > > >>>> > > > > > > > >>>> > > > > > On Wed, Aug 5, 2015 at 10:03 AM, Abdel Hakim Deneche < > > >>>> > > > > [email protected]> > > >>>> > > > > > wrote: > > >>>> > > > > > > > >>>> > > > > >> Just got those errors on master too > > >>>> > > > > >> > > >>>> > > > > >> On Wed, Aug 5, 2015 at 9:07 AM, Abdel Hakim Deneche < > > >>>> > > > > [email protected] > > >>>> > > > > >>> > > >>>> > > > > >> wrote: > > >>>> > > > > >> > > >>>> > > > > >>> I'm seeing those errors intermittently when building my > > >>>> private > > >>>> > > > > branch, I > > >>>> > > > > >>> don't believe I made any change that would have caused > > them. > > >>>> > Anyone > > >>>> > > > > seen > > >>>> > > > > >>> them too ? > > >>>> > > > > >>> > > >>>> > > > > >>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > testBitVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector) > > >>>> > > > > >>>> Time elapsed: 2.043 sec <<< ERROR! > > >>>> > > > > >>>> java.lang.Exception: Unexpected exception, > > >>>> > > > > >>>> > > >>>> > > > > > >>>> > expected<org.apache.drill.exec.exception.OversizedAllocationException> > > >>>> > > > > >> but > > >>>> > > > > >>>> > > >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> > > >>>> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658) > > >>>> > > > > >>>> at > > >>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > > >>>> > > > > >>>> at > > java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116) > > >>>> > > > > >>>> at > > >>>> > > > > > org.apache.drill.exec.vector.BitVector.reAlloc(BitVector.java:139) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > org.apache.drill.exec.record.vector.TestValueVector.testBitVectorReallocation(TestValueVector.java:125) > > >>>> > > > > >>> > > >>>> > > > > >>> > > >>>> > > > > >>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > testFixedVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector) > > >>>> > > > > >>>> Time elapsed: 0.436 sec <<< ERROR! > > >>>> > > > > >>>> java.lang.Exception: Unexpected exception, > > >>>> > > > > >>>> > > >>>> > > > > > >>>> > expected<org.apache.drill.exec.exception.OversizedAllocationException> > > >>>> > > > > >> but > > >>>> > > > > >>>> > > >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> > > >>>> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658) > > >>>> > > > > >>>> at > > >>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > > >>>> > > > > >>>> at > > java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > org.apache.drill.exec.vector.UInt4Vector.allocateBytes(UInt4Vector.java:187) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > org.apache.drill.exec.vector.UInt4Vector.allocateNew(UInt4Vector.java:177) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:85) > > >>>> > > > > >>> > > >>>> > > > > >>> > > >>>> > > > > >>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > testVariableVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector) > > >>>> > > > > >>>> Time elapsed: 0.788 sec <<< ERROR! > > >>>> > > > > >>>> java.lang.Exception: Unexpected exception, > > >>>> > > > > >>>> > > >>>> > > > > > >>>> > expected<org.apache.drill.exec.exception.OversizedAllocationException> > > >>>> > > > > >> but > > >>>> > > > > >>>> > > >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> > > >>>> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658) > > >>>> > > > > >>>> at > > >>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > > >>>> > > > > >>>> at > > java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:372) > > >>>> > > > > >>>> at > > >>>> > > > > >>>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:142) > > >>>> > > > > >>> > > >>>> > > > > >>> > > >>>> > > > > >>> Thanks > > >>>> > > > > >>> -- > > >>>> > > > > >>> > > >>>> > > > > >>> Abdelhakim Deneche > > >>>> > > > > >>> > > >>>> > > > > >>> Software Engineer > > >>>> > > > > >>> > > >>>> > > > > >>> <http://www.mapr.com/> > > >>>> > > > > >>> > > >>>> > > > > >>> > > >>>> > > > > >>> Now Available - Free Hadoop On-Demand Training > > >>>> > > > > >>> < > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > >>>> > > > > >>> > > >>>> > > > > >>> > > >>>> > > > > >> > > >>>> > > > > >> > > >>>> > > > > >> > > >>>> > > > > >> -- > > >>>> > > > > >> > > >>>> > > > > >> Abdelhakim Deneche > > >>>> > > > > >> > > >>>> > > > > >> Software Engineer > > >>>> > > > > >> > > >>>> > > > > >> <http://www.mapr.com/> > > >>>> > > > > >> > > >>>> > > > > >> > > >>>> > > > > >> Now Available - Free Hadoop On-Demand Training > > >>>> > > > > >> < > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > >>>> > > > > >>> > > >>>> > > > > >> > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > -- > > >>>> > > > > >>>> > > Abdelhakim Deneche > > >>>> > > > > >>>> > > Software Engineer > > >>>> > > > > >>>> > > <http://www.mapr.com/> > > >>>> > > > > >>>> > > > > >>>> > > Now Available - Free Hadoop On-Demand Training > > >>>> > > < > > >>>> > > > > >>>> > > > >>>> > > > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>> > > >>> > > >>> > > >>> -- > > >>> > > >>> Abdelhakim Deneche > > >>> > > >>> Software Engineer > > >>> > > >>> <http://www.mapr.com/> > > >>> > > >>> > > >>> Now Available - Free Hadoop On-Demand Training > > >>> < > > > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > > > >>> > > >> > > >> > > >> > > >> -- > > >> > > >> Abdelhakim Deneche > > >> > > >> Software Engineer > > >> > > >> <http://www.mapr.com/> > > >> > > >> > > >> Now Available - Free Hadoop On-Demand Training > > >> < > > > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > > > >> > > > > > > > > > > > > -- > > > > > > Abdelhakim Deneche > > > > > > Software Engineer > > > > > > <http://www.mapr.com/> > > > > > > > > > Now Available - Free Hadoop On-Demand Training > > > < > > > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > > > > > > > > > > > > -- > > > > Abdelhakim Deneche > > > > Software Engineer > > > > <http://www.mapr.com/> > > > > > > Now Available - Free Hadoop On-Demand Training > > < > > > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > > > > -- Abdelhakim Deneche Software Engineer <http://www.mapr.com/> Now Available - Free Hadoop On-Demand Training <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
