Hi Weijie It would also help to understand the flow of control that your design uses. I've put a screenshot of a query profile here : https://docs.google.com/document/d/1DgAbGovEWV6rZ4GvioJz5Twe_m5o1ADoBpbqAzqa_aU/edit?usp=sharing
Looking at the subset under [ Hash Join 06-01 ], can you annotate and/or explain how you see the control messages flowing? Also, are you using the control channel to send the bloom filter? Parth On Wed, Jun 20, 2018 at 3:28 PM, Parth Chandra <[email protected]> wrote: > Hi Weijie, > This is a tricky problem. So let me first summarize how this should be > behaving - > > Thread 1 | Thread 2 > -----------------------------------+------------------------ > ------------------ > 1) Allocate DrillBuf | > 2) Pass memory to RPC layer | get reference to DrillBuf > (refcount == 2) > 3) | queue up the send (async) > 4) | Send and release DrillBuf > (refcount == 1) > 5) Continue to end of query | > 6) Cleanup (release DrillBuf) | > 7) Close Allocator (refcount of | > DrillBuf *must* be zero) | > > In your case, steps 3 and 4 are occurring after step 7 which is natural > since the RPC send is async, but that is what we have to prevent. The only > way to do that is to have some synchronization between steps (3), (4), and > (6) such that (6) only happens after (4). With RPC the way to do so is to > require an ack. > > > Thread 1 | Thread 2 (Netty) > | Thread 3 (foreman) > -----------------------------------+------------------------ > --------------------+------------------------ > 1) Allocate DrillBuf | > | > 2) Pass memory to RPC layer | get reference to DrillBuf > (refcount == 2)| > 3) | queue up the send (async) > | > 4) | Send and release DrillBuf > (refcount == 1)| > 4.1) | > | Recv msg, send back Ack (The RPC layer > | > | automatically does this) > 4.2) Check if Ack received | > 5) Continue to end of query | > 6) Cleanup (release DrillBuf) | > 7) Close Allocator (refcount of | > DrillBuf *must* be zero) | > > Note that (4.2) does not have to complete before (5), only before (6) for > the memory to be released. > > One question I have is how the query completed without the Bloom Filter > reaching its destination. How does the destination fragment know when it > has to wait for the Bloom Filter? I suspect this may be more > complicated than it appears at first glance. > > Not sure if this helps narrow it down. If you can share a dev branch we > can help take a look. > > > > On Tue, Jun 19, 2018 at 8:35 PM, weijie tong <[email protected]> > wrote: > >> HI: >> I faced a complicated problem by releasing the BloomFilter's direct >> memory at some special cases. Hope someone could give some advices. >> >> Say, one join node sends out BloomFilter to the foreman >> node(TestHashJoin.simpleEqualityJoin() ) . The sending thread is netty's >> BitClient. The BloomFilter's direct memory is allocated by another thread >> allocator (i.e. the HashJoin fragment's allocator). Once the fragment >> completes quickly. Then its corresponding close logic will check the >> allocator's memory assignment. But the async sender thread has not sent >> out >> the BloomFilter to release the corresponding direct ByteBuffer as the >> query >> has completed quickly , the wire has closed. Then the corresponding >> fragment's close logic will throw exception to complain about the memory >> leak. >> >> So I want to know how to release the allocated direct ByteBuffer at >> such case . >> >> >> The exception is : >> >> [Error Id: 0042b168-8728-4367-b461-653837c3a276 on 10.15.235.86:31010] >> at >> org.apache.drill.common.exceptions.UserException$Builder. >> build(UserException.java:633) >> ~[classes/:na] >> at >> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFin >> alState(FragmentExecutor.java:359) >> [classes/:na] >> at >> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup >> (FragmentExecutor.java:214) >> [classes/:na] >> at >> org.apache.drill.exec.work.fragment.FragmentExecutor.run(Fra >> gmentExecutor.java:325) >> [classes/:na] >> at >> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleanin >> gRunnable.java:38) >> [classes/:na] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1149) >> [na:1.8.0_161] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:624) >> [na:1.8.0_161] >> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] >> Caused by: java.lang.IllegalStateException: Allocator[frag:0:0] closed >> with >> outstanding buffers allocated (1). >> Allocator(frag:0:0) 4000000/16777216/47664448/30357913941 >> (res/actual/peak/limit) >> child allocators: 0 >> ledgers: 1 >> ledger[6268] allocator: frag:0:0), isOwning: true, size: 16777216, >> references: 1, life: 505919927431943..0, allocatorManager: [6050, life: >> 505919927378312..0] holds 2 buffers. >> DrillBuf[10198], udle: [6051 0..16777216] >> DrillBuf[10208], udle: [6051 0..16777216] >> reservations: 0 >> >> at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocat >> or.java:503) >> ~[classes/:na] >> at >> org.apache.drill.exec.ops.FragmentContextImpl.suppressingClo >> se(FragmentContextImpl.java:484) >> ~[classes/:na] >> at >> org.apache.drill.exec.ops.FragmentContextImpl.close(Fragment >> ContextImpl.java:478) >> ~[classes/:na] >> at >> org.apache.drill.exec.work.fragment.FragmentExecutor.closeOu >> tResources(FragmentExecutor.java:382) >> [classes/:na] >> at >> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup >> (FragmentExecutor.java:209) >> [classes/:na] >> ... 5 common frames omitted >> =====================fragment:0 : 0done! >> *************************receive a bloom filter********** >> ************received a bloom filter >> 11:00:41.587 [main] ERROR o.a.d.exec.server.BootStrapContext - Error >> while >> closing >> java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding >> child allocators. >> Allocator(ROOT) 0/16777216/55640064/4294967296 (res/actual/peak/limit) >> child allocators: 1 >> Allocator(frag:0:0) 4000000/16777216/47664448/30357913941 >> (res/actual/peak/limit) >> child allocators: 0 >> ledgers: 1 >> ledger[6268] allocator: frag:0:0), isOwning: true, size: 16777216, >> references: 1, life: 505919927431943..0, allocatorManager: [6050, life: >> 505919927378312..0] holds 2 buffers. >> DrillBuf[10198], udle: [6051 0..16777216] >> DrillBuf[10208], udle: [6051 0..16777216] >> reservations: 0 >> ledgers: 0 >> reservations: 0 >> >> at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocat >> or.java:496) >> ~[classes/:na] >> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81) >> [classes/:na] >> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69) >> [classes/:na] >> at >> org.apache.drill.exec.server.BootStrapContext.close(BootStra >> pContext.java:259) >> ~[classes/:na] >> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81) >> [classes/:na] >> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69) >> [classes/:na] >> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:263) >> [classes/:na] >> at >> org.apache.drill.exec.physical.impl.join.TestHashJoin.simple >> EqualityJoin(TestHashJoin.java:147) >> [test-classes/:na] >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> ~[na:1.8.0_161] >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >> ssorImpl.java:62) >> ~[na:1.8.0_161] >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> thodAccessorImpl.java:43) >> ~[na:1.8.0_161] >> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] >> at >> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall( >> FrameworkMethod.java:50) >> [junit-4.12.jar:4.12] >> at >> org.junit.internal.runners.model.ReflectiveCallable.run(Refl >> ectiveCallable.java:12) >> [junit-4.12.jar:4.12] >> at >> org.junit.runners.model.FrameworkMethod.invokeExplosively(Fr >> ameworkMethod.java:47) >> [junit-4.12.jar:4.12] >> at >> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator >> .executeTestMethod(JUnit4TestRunnerDecorator.java:154) >> [jmockit-1.39.jar:1.39] >> at >> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator >> .invokeExplosively(JUnit4TestRunnerDecorator.java:70) >> [jmockit-1.39.jar:1.39] >> at >> mockit.integration.junit4.internal.FakeFrameworkMethod.invok >> eExplosively(FakeFrameworkMethod.java:34) >> [jmockit-1.39.jar:1.39] >> at >> org.junit.runners.model.FrameworkMethod.invokeExplosively(Fr >> ameworkMethod.java) >> [junit-4.12.jar:4.12] >> at >> org.junit.internal.runners.statements.InvokeMethod.evaluate( >> InvokeMethod.java:17) >> [junit-4.12.jar:4.12] >> at >> org.junit.internal.runners.statements.RunBefores.evaluate( >> RunBefores.java:26) >> [junit-4.12.jar:4.12] >> at >> org.junit.internal.runners.statements.RunAfters.evaluate(Run >> Afters.java:27) >> [junit-4.12.jar:4.12] >> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) >> [junit-4.12.jar:4.12] >> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) >> [junit-4.12.jar:4.12] >> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) >> [junit-4.12.jar:4.12] >> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) >> [junit-4.12.jar:4.12] >> at >> org.junit.rules.ExpectedException$ExpectedExceptionStatement >> .evaluate(ExpectedException.java:239) >> [junit-4.12.jar:4.12] >> at org.junit.rules.RunRules.evaluate(RunRules.java:20) >> [junit-4.12.jar:4.12] >> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) >> [junit-4.12.jar:4.12] >> at >> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit >> 4ClassRunner.java:78) >> [junit-4.12.jar:4.12] >> at >> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit >> 4ClassRunner.java:57) >> [junit-4.12.jar:4.12] >> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) >> [junit-4.12.jar:4.12] >> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) >> [junit-4.12.jar:4.12] >> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) >> [junit-4.12.jar:4.12] >> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) >> [junit-4.12.jar:4.12] >> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) >> [junit-4.12.jar:4.12] >> at >> org.junit.internal.runners.statements.RunBefores.evaluate( >> RunBefores.java:26) >> [junit-4.12.jar:4.12] >> at >> org.junit.internal.runners.statements.RunAfters.evaluate(Run >> Afters.java:27) >> [junit-4.12.jar:4.12] >> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) >> [junit-4.12.jar:4.12] >> at org.junit.rules.RunRules.evaluate(RunRules.java:20) >> [junit-4.12.jar:4.12] >> at org.junit.runners.ParentRunner.run(ParentRunner.java:363) >> [junit-4.12.jar:4.12] >> at org.junit.runner.JUnitCore.run(JUnitCore.java:137) >> [junit-4.12.jar:4.12] >> at >> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs >> (JUnit4IdeaTestRunner.java:68) >> [junit-rt.jar:na] >> at >> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.star >> tRunnerWithArgs(IdeaTestRunner.java:47) >> [junit-rt.jar:na] >> at >> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsA >> ndStart(JUnitStarter.java:242) >> [junit-rt.jar:na] >> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStart >> er.java:70) >> [junit-rt.jar:na] >> 11:00:41.593 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: >> 0 >> B(1 B), h: -360.8 MiB(52.3 MiB), nh: 3.2 MiB(88.4 MiB)): >> simpleEqualityJoin(org.apache.drill.exec.physical.impl.join.TestHashJoin) >> org.apache.drill.exec.rpc.RpcException: >> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: >> IllegalStateException: Allocator[frag:0:0] closed with outstanding buffers >> allocated (1). >> Allocator(frag:0:0) 4000000/16777216/47664448/30357913941 >> (res/actual/peak/limit) >> child allocators: 0 >> ledgers: 1 >> ledger[6268] allocator: frag:0:0), isOwning: true, size: 16777216, >> references: 1, life: 505919927431943..0, allocatorManager: [6050, life: >> 505919927378312..0] holds 2 buffers. >> DrillBuf[10198], udle: [6051 0..16777216] >> DrillBuf[10208], udle: [6051 0..16777216] >> reservations: 0 >> >> >> Fragment 0:0 >> >> [Error Id: 0042b168-8728-4367-b461-653837c3a276 on 10.15.235.86:31010] >> at >> org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60) >> ~[classes/:na] >> at >> org.apache.drill.exec.client.DrillClient$ListHoldingResultsL >> istener.getResults(DrillClient.java:881) >> ~[classes/:na] >> at org.apache.drill.exec.client.DrillClient.runQuery(DrillClien >> t.java:583) >> ~[classes/:na] >> at >> org.apache.drill.exec.physical.impl.join.TestHashJoin.simple >> EqualityJoin(TestHashJoin.java:119) >> ~[test-classes/:na] >> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: >> IllegalStateException: Allocator[frag:0:0] closed with outstanding buffers >> allocated (1). >> Allocator(frag:0:0) 4000000/16777216/47664448/30357913941 >> (res/actual/peak/limit) >> child allocators: 0 >> ledgers: 1 >> ledger[6268] allocator: frag:0:0), isOwning: true, size: 16777216, >> references: 1, life: 505919927431943..0, allocatorManager: [6050, life: >> 505919927378312..0] holds 2 buffers. >> DrillBuf[10198], udle: [6051 0..16777216] >> DrillBuf[10208], udle: [6051 0..16777216] >> reservations: 0 >> >> >> Fragment 0:0 >> >> [Error Id: 0042b168-8728-4367-b461-653837c3a276 on 10.15.235.86:31010] >> at >> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArri >> ved(QueryResultHandler.java:123) >> ~[classes/:na] >> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422) >> ~[classes/:na] >> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96) >> ~[classes/:na] >> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBu >> s.java:275) >> ~[classes/:na] >> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBu >> s.java:245) >> ~[classes/:na] >> at >> io.netty.handler.codec.MessageToMessageDecoder.channelRead(M >> essageToMessageDecoder.java:88) >> ~[netty-codec-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:356) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:342) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRe >> ad(AbstractChannelHandlerContext.java:335) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleSt >> ateHandler.java:287) >> ~[netty-handler-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:356) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:342) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRe >> ad(AbstractChannelHandlerContext.java:335) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.handler.codec.MessageToMessageDecoder.channelRead(M >> essageToMessageDecoder.java:102) >> ~[netty-codec-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:356) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:342) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRe >> ad(AbstractChannelHandlerContext.java:335) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead( >> ByteToMessageDecoder.java:312) >> ~[netty-codec-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.handler.codec.ByteToMessageDecoder.channelRead(Byte >> ToMessageDecoder.java:286) >> ~[netty-codec-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:356) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:342) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRe >> ad(AbstractChannelHandlerContext.java:335) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(Ch >> annelInboundHandlerAdapter.java:86) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:356) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:342) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRe >> ad(AbstractChannelHandlerContext.java:335) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.DefaultChannelPipeline$HeadContext. >> channelRead(DefaultChannelPipeline.java:1294) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:356) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:342) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.DefaultChannelPipeline.fireChannelRead(Defa >> ultChannelPipeline.java:911) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe. >> read(AbstractNioByteChannel.java:131) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEven >> tLoop.java:645) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimiz >> ed(NioEventLoop.java:580) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEve >> ntLoop.java:497) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) >> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] >> at >> io.netty.util.concurrent.SingleThreadEventExecutor$2.run( >> SingleThreadEventExecutor.java:131) >> ~[netty-common-4.0.48.Final.jar:4.0.48.Final] >> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161] >> >> org.apache.drill.exec.rpc.RpcException: >> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: >> IllegalStateException: Allocator[frag:0:0] closed with outstanding buffers >> allocated (1). >> Allocator(frag:0:0) 4000000/16777216/47664448/30357913941 >> (res/actual/peak/limit) >> child allocators: 0 >> ledgers: 1 >> ledger[6268] allocator: frag:0:0), isOwning: true, size: 16777216, >> references: 1, life: 505919927431943..0, allocatorManager: [6050, life: >> 505919927378312..0] holds 2 buffers. >> DrillBuf[10198], udle: [6051 0..16777216] >> DrillBuf[10208], udle: [6051 0..16777216] >> reservations: 0 >> >> >> Fragment 0:0 >> >> [Error Id: 0042b168-8728-4367-b461-653837c3a276 on 10.15.235.86:31010] >> >> at org.apache.drill.exec.rpc.RpcException.mapException(RpcExcep >> tion.java:60) >> at >> org.apache.drill.exec.client.DrillClient$ListHoldingResultsL >> istener.getResults(DrillClient.java:881) >> at org.apache.drill.exec.client.DrillClient.runQuery(DrillClien >> t.java:583) >> at >> org.apache.drill.exec.physical.impl.join.TestHashJoin.simple >> EqualityJoin(TestHashJoin.java:119) >> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM >> ERROR: IllegalStateException: Allocator[frag:0:0] closed with outstanding >> buffers allocated (1). >> Allocator(frag:0:0) 4000000/16777216/47664448/30357913941 >> (res/actual/peak/limit) >> child allocators: 0 >> ledgers: 1 >> ledger[6268] allocator: frag:0:0), isOwning: true, size: 16777216, >> references: 1, life: 505919927431943..0, allocatorManager: [6050, life: >> 505919927378312..0] holds 2 buffers. >> DrillBuf[10198], udle: [6051 0..16777216] >> DrillBuf[10208], udle: [6051 0..16777216] >> reservations: 0 >> >> >> Fragment 0:0 >> >> [Error Id: 0042b168-8728-4367-b461-653837c3a276 on 10.15.235.86:31010] >> at >> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArri >> ved(QueryResultHandler.java:123) >> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422) >> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96) >> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBu >> s.java:275) >> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBu >> s.java:245) >> at >> io.netty.handler.codec.MessageToMessageDecoder.channelRead(M >> essageToMessageDecoder.java:88) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRe >> ad(AbstractChannelHandlerContext.java:335) >> at >> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleSt >> ateHandler.java:287) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRe >> ad(AbstractChannelHandlerContext.java:335) >> at >> io.netty.handler.codec.MessageToMessageDecoder.channelRead(M >> essageToMessageDecoder.java:102) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRe >> ad(AbstractChannelHandlerContext.java:335) >> at >> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead( >> ByteToMessageDecoder.java:312) >> at >> io.netty.handler.codec.ByteToMessageDecoder.channelRead(Byte >> ToMessageDecoder.java:286) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRe >> ad(AbstractChannelHandlerContext.java:335) >> at >> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(Ch >> annelInboundHandlerAdapter.java:86) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRe >> ad(AbstractChannelHandlerContext.java:335) >> at >> io.netty.channel.DefaultChannelPipeline$HeadContext. >> channelRead(DefaultChannelPipeline.java:1294) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannel >> Read(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.DefaultChannelPipeline.fireChannelRead(Defa >> ultChannelPipeline.java:911) >> at >> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe. >> read(AbstractNioByteChannel.java:131) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEven >> tLoop.java:645) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimiz >> ed(NioEventLoop.java:580) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEve >> ntLoop.java:497) >> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) >> at >> io.netty.util.concurrent.SingleThreadEventExecutor$2.run( >> SingleThreadEventExecutor.java:131) >> at java.lang.Thread.run(Thread.java:748) >> > >
