I've seen sporadic failures with those tests. I'm wondering whether they need to be redesigned. Limiting direct memory to 8gb shouldn't fail the test.
-- Jacques Nadeau CTO and Co-Founder, Dremio On Tue, Sep 15, 2015 at 9:39 AM, Abdel Hakim Deneche <[email protected]> wrote: > I found what was causing those errors on my VM. I exported a MAVEN_OPTS > that limits the direct memory to 8GB and forget to unset it, this was > causing the build to fail because it couldn't allocate more direct memory, > for some reason it was always failing on those TestValueVector unit tests. > > On Wed, Sep 2, 2015 at 10:10 AM, Abdel Hakim Deneche < > [email protected]> > wrote: > > > I am still seeing the TestValueVector unit tests fail on my linux machine > > (2/3 of my builds) > > > > On Fri, Aug 14, 2015 at 2:40 PM, Abdel Hakim Deneche < > > [email protected]> wrote: > > > >> You can safely ignore the TestImpersonationMetadata error, it's most > >> likely caused by my change > >> > >> On Fri, Aug 14, 2015 at 12:36 PM, Abdel Hakim Deneche < > >> [email protected]> wrote: > >> > >>> In addition to test*VectorReallocation errors I am seeing frequently on > >>> my linux VM, today I saw the following error multiple times on my VM > and > >>> once on my Macbook: > >>> > >>> Tests in error: > >>>> TestImpersonationMetadata>BaseTestQuery.closeClient:239 ยป > >>>> IllegalState Failure... > >>>> java.lang.IllegalStateException: Failure while closing accountor. > >>>> Expected private and shared pools to be set to initial values. > However, > >>>> one or more were not. Stats are > >>>> zone init allocated delta > >>>> private 0 0 0 > >>>> shared 3221225472 3214166779 7058693. > >>>> at > >>>> > org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:200) > >>>> ~[classes/:na] > >>>> at org.apache.drill.exec.memory.Accountor.close(Accountor.java:390) > >>>> ~[classes/:na] > >>>> at > >>>> > org.apache.drill.exec.memory.TopLevelAllocator.close(TopLevelAllocator.java:185) > >>>> ~[classes/:na] > >>>> at > >>>> > org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:75) > >>>> ~[classes/:na] > >>>> at com.google.common.io.Closeables.close(Closeables.java:77) > >>>> ~[guava-14.0.1.jar:na] > >>>> at com.google.common.io.Closeables.closeQuietly(Closeables.java:108) > >>>> ~[guava-14.0.1.jar:na] > >>>> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:294) > >>>> ~[classes/:na] > >>>> at > >>>> > org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:332) > >>>> ~[classes/:na] > >>> > >>> > >>> On Wed, Aug 5, 2015 at 2:52 PM, Chris Westin <[email protected]> > >>> wrote: > >>> > >>>> Given that the difference is just > >>>> > >>>> > java.lang.Exception: Unexpected exception, > >>>> > > >>>> > expected<org.apache.drill.exec.exception.OversizedAllocationException> but > >>>> > was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> > >>>> > >>>> The question of "what constitutes an oversized allocation?" comes to > >>>> mind. > >>>> Is this test fragile relative to being run in different environments? > >>>> I haven't seen the test so how is the determination that something is > >>>> oversized made? It seems like that criterion sometimes fails, and we > >>>> get an > >>>> OOM because whatever the request is is still very large. > >>>> > >>>> > >>>> On Wed, Aug 5, 2015 at 2:26 PM, Hanifi Gunes <[email protected]> > >>>> wrote: > >>>> > >>>> > I don't seem to be able to re-prod this. Let me look at this and > >>>> update you > >>>> > all. > >>>> > > >>>> > On Thu, Aug 6, 2015 at 12:03 AM, Abdel Hakim Deneche < > >>>> > [email protected]> > >>>> > wrote: > >>>> > > >>>> > > I didn't make any change, I'm running 2 forks (the default). I got > >>>> those > >>>> > > errors 3 times now, 2 on a linux VM and 1 on a linux physical node > >>>> > > > >>>> > > On Wed, Aug 5, 2015 at 1:03 PM, Hanifi Gunes <[email protected] > > > >>>> > wrote: > >>>> > > > >>>> > > > Did you tighten your memory settings? How many forks are you > >>>> running > >>>> > > with? > >>>> > > > I bet you are truly running out of memory while executing this > >>>> > particular > >>>> > > > test case. > >>>> > > > > >>>> > > > -H+ > >>>> > > > > >>>> > > > On Wed, Aug 5, 2015 at 8:56 PM, Sudheesh Katkam < > >>>> [email protected]> > >>>> > > > wrote: > >>>> > > > > >>>> > > > > b2bbd99 committed on July 6th introduced the test. > >>>> > > > > > >>>> > > > > > On Aug 5, 2015, at 10:21 AM, Jinfeng Ni < > >>>> [email protected]> > >>>> > > wrote: > >>>> > > > > > > >>>> > > > > > In that case, we probably need do binary search to figure > out > >>>> > which > >>>> > > > > recent > >>>> > > > > > patch is causing this problem. > >>>> > > > > > > >>>> > > > > > On Wed, Aug 5, 2015 at 10:03 AM, Abdel Hakim Deneche < > >>>> > > > > [email protected]> > >>>> > > > > > wrote: > >>>> > > > > > > >>>> > > > > >> Just got those errors on master too > >>>> > > > > >> > >>>> > > > > >> On Wed, Aug 5, 2015 at 9:07 AM, Abdel Hakim Deneche < > >>>> > > > > [email protected] > >>>> > > > > >>> > >>>> > > > > >> wrote: > >>>> > > > > >> > >>>> > > > > >>> I'm seeing those errors intermittently when building my > >>>> private > >>>> > > > > branch, I > >>>> > > > > >>> don't believe I made any change that would have caused > them. > >>>> > Anyone > >>>> > > > > seen > >>>> > > > > >>> them too ? > >>>> > > > > >>> > >>>> > > > > >>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > testBitVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector) > >>>> > > > > >>>> Time elapsed: 2.043 sec <<< ERROR! > >>>> > > > > >>>> java.lang.Exception: Unexpected exception, > >>>> > > > > >>>> > >>>> > > > > >>>> expected<org.apache.drill.exec.exception.OversizedAllocationException> > >>>> > > > > >> but > >>>> > > > > >>>> > >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> > >>>> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658) > >>>> > > > > >>>> at > >>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > >>>> > > > > >>>> at > java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116) > >>>> > > > > >>>> at > >>>> > > > > org.apache.drill.exec.vector.BitVector.reAlloc(BitVector.java:139) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > org.apache.drill.exec.record.vector.TestValueVector.testBitVectorReallocation(TestValueVector.java:125) > >>>> > > > > >>> > >>>> > > > > >>> > >>>> > > > > >>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > testFixedVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector) > >>>> > > > > >>>> Time elapsed: 0.436 sec <<< ERROR! > >>>> > > > > >>>> java.lang.Exception: Unexpected exception, > >>>> > > > > >>>> > >>>> > > > > >>>> expected<org.apache.drill.exec.exception.OversizedAllocationException> > >>>> > > > > >> but > >>>> > > > > >>>> > >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> > >>>> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658) > >>>> > > > > >>>> at > >>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > >>>> > > > > >>>> at > java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > org.apache.drill.exec.vector.UInt4Vector.allocateBytes(UInt4Vector.java:187) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > org.apache.drill.exec.vector.UInt4Vector.allocateNew(UInt4Vector.java:177) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:85) > >>>> > > > > >>> > >>>> > > > > >>> > >>>> > > > > >>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > testVariableVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector) > >>>> > > > > >>>> Time elapsed: 0.788 sec <<< ERROR! > >>>> > > > > >>>> java.lang.Exception: Unexpected exception, > >>>> > > > > >>>> > >>>> > > > > >>>> expected<org.apache.drill.exec.exception.OversizedAllocationException> > >>>> > > > > >> but > >>>> > > > > >>>> > >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> > >>>> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658) > >>>> > > > > >>>> at > >>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > >>>> > > > > >>>> at > java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:372) > >>>> > > > > >>>> at > >>>> > > > > >>>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:142) > >>>> > > > > >>> > >>>> > > > > >>> > >>>> > > > > >>> Thanks > >>>> > > > > >>> -- > >>>> > > > > >>> > >>>> > > > > >>> Abdelhakim Deneche > >>>> > > > > >>> > >>>> > > > > >>> Software Engineer > >>>> > > > > >>> > >>>> > > > > >>> <http://www.mapr.com/> > >>>> > > > > >>> > >>>> > > > > >>> > >>>> > > > > >>> Now Available - Free Hadoop On-Demand Training > >>>> > > > > >>> < > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > >>>> > > > > >>> > >>>> > > > > >>> > >>>> > > > > >> > >>>> > > > > >> > >>>> > > > > >> > >>>> > > > > >> -- > >>>> > > > > >> > >>>> > > > > >> Abdelhakim Deneche > >>>> > > > > >> > >>>> > > > > >> Software Engineer > >>>> > > > > >> > >>>> > > > > >> <http://www.mapr.com/> > >>>> > > > > >> > >>>> > > > > >> > >>>> > > > > >> Now Available - Free Hadoop On-Demand Training > >>>> > > > > >> < > >>>> > > > > >> > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > >>>> > > > > >>> > >>>> > > > > >> > >>>> > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > -- > >>>> > > > >>>> > > Abdelhakim Deneche > >>>> > > > >>>> > > Software Engineer > >>>> > > > >>>> > > <http://www.mapr.com/> > >>>> > > > >>>> > > > >>>> > > Now Available - Free Hadoop On-Demand Training > >>>> > > < > >>>> > > > >>>> > > >>>> > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > >>>> > > > > >>>> > > > >>>> > > >>>> > >>> > >>> > >>> > >>> -- > >>> > >>> Abdelhakim Deneche > >>> > >>> Software Engineer > >>> > >>> <http://www.mapr.com/> > >>> > >>> > >>> Now Available - Free Hadoop On-Demand Training > >>> < > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > >>> > >> > >> > >> > >> -- > >> > >> Abdelhakim Deneche > >> > >> Software Engineer > >> > >> <http://www.mapr.com/> > >> > >> > >> Now Available - Free Hadoop On-Demand Training > >> < > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > >> > > > > > > > > -- > > > > Abdelhakim Deneche > > > > Software Engineer > > > > <http://www.mapr.com/> > > > > > > Now Available - Free Hadoop On-Demand Training > > < > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > > > > > > -- > > Abdelhakim Deneche > > Software Engineer > > <http://www.mapr.com/> > > > Now Available - Free Hadoop On-Demand Training > < > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > >
