Looking up the stacktrace on Google points towards an issue with HBase. May be you could first confirm that.
Also, on a different note, i see you are using Drill 0.9.0. An updated version (v1.0.0) is out if you'd like to try it out. On Wed, Jun 10, 2015 at 10:12 PM, Abhishek Girish <[email protected] > wrote: > Are you able to successfully run queries such as select * with limit on > Drill? Also can you do a full table scan via HBase shell? I'm not sure > what's the issue here - may be someone else can help. But my guess would be > an issue with HBase. > > On Wed, Jun 10, 2015 at 8:50 PM, George Lu <[email protected]> wrote: > >> OK, My queries is count(*) >> >> Error log below: >> >> Fragment 1:2 >> >> [6171962e-ffa7-4355-b1d2-21d569e1bfe3 on prod8:31010] >> at >> >> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:465) >> ~[drill-common-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:262) >> [drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:232) >> [drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) >> [drill-common-0.9.0-rebuffed.jar:0.9.0] >> at >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> [na:1.8.0_25] >> at >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> [na:1.8.0_25] >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25] >> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: >> org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of >> OutOfOrderScannerNextException: was there a rpc timeout? >> at >> >> org.apache.drill.exec.store.hbase.HBaseRecordReader.next(HBaseRecordReader.java:191) >> ~[drill-storage-hbase-0.9.0.jar:0.9.0] >> at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:170) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:101) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:91) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:144) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:101) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:91) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:96) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:144) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:101) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:91) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:130) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:144) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:74) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:91) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:64) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:199) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:193) >> ~[drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at java.security.AccessController.doPrivileged(Native Method) >> ~[na:1.8.0_25] >> at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_25] >> at >> >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) >> ~[hadoop-common-2.4.1.jar:na] >> at >> >> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:193) >> [drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> ... 4 common frames omitted >> Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: Failed after >> retry of OutOfOrderScannerNextException: was there a rpc timeout? >> at >> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:410) >> ~[hbase-client-0.98.7-hadoop2.jar:0.98.7-hadoop2] >> at >> >> org.apache.drill.exec.store.hbase.HBaseRecordReader.next(HBaseRecordReader.java:184) >> ~[drill-storage-hbase-0.9.0.jar:0.9.0] >> ... 32 common frames omitted >> Caused by: >> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: >> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: >> Expected >> nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: >> 2199 number_of_rows: 4000 close_scanner: false next_call_seq: 0 >> at >> >> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3195) >> at >> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29941) >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2029) >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) >> at >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:112) >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) >> at java.lang.Thread.run(Thread.java:745) >> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >> ~[na:1.8.0_25] >> at >> >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >> ~[na:1.8.0_25] >> at >> >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> ~[na:1.8.0_25] >> at java.lang.reflect.Constructor.newInstance(Constructor.java:408) >> ~[na:1.8.0_25] >> at >> >> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) >> ~[hadoop-common-2.4.1.jar:na] >> at >> >> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) >> ~[hadoop-common-2.4.1.jar:na] >> at >> >> org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:284) >> ~[hbase-client-0.98.7-hadoop2.jar:0.98.7-hadoop2] >> at >> >> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:204) >> ~[hbase-client-0.98.7-hadoop2.jar:0.98.7-hadoop2] >> at >> >> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:59) >> ~[hbase-client-0.98.7-hadoop2.jar:0.98.7-hadoop2] >> at >> >> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114) >> ~[hbase-client-0.98.7-hadoop2.jar:0.98.7-hadoop2] >> at >> >> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90) >> ~[hbase-client-0.98.7-hadoop2.jar:0.98.7-hadoop2] >> at >> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:355) >> ~[hbase-client-0.98.7-hadoop2.jar:0.98.7-hadoop2] >> ... 33 common frames omitted >> Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException: >> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: >> Expected >> nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: >> 2199 number_of_rows: 4000 close_scanner: false next_call_seq: 0 >> at >> >> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3195) >> at >> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29941) >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2029) >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) >> at >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:112) >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) >> at java.lang.Thread.run(Thread.java:745) >> >> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) >> ~[hbase-client-0.98.7-hadoop2.jar:0.98.7-hadoop2] >> at >> >> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) >> ~[hbase-client-0.98.7-hadoop2.jar:0.98.7-hadoop2] >> at >> >> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) >> ~[hbase-client-0.98.7-hadoop2.jar:0.98.7-hadoop2] >> at >> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:30328) >> ~[hbase-protocol-0.98.7-hadoop2.jar:0.98.7-hadoop2] >> at >> >> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:174) >> ~[hbase-client-0.98.7-hadoop2.jar:0.98.7-hadoop2] >> ... 37 common frames omitted >> 2015-06-11 11:42:35,517 [BitServer-5] INFO >> o.a.drill.exec.work.foreman.Foreman - State change requested. RUNNING >> --> >> FAILED >> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: >> org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of >> OutOfOrderScannerNextException: was there a rpc timeout? >> >> Fragment 1:2 >> >> [6171962e-ffa7-4355-b1d2-21d569e1bfe3 on prod8:31010] >> at >> >> org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:409) >> [drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.rpc.control.WorkEventBus.statusUpdate(WorkEventBus.java:71) >> [drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.work.batch.ControlHandlerImpl.handle(ControlHandlerImpl.java:81) >> [drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.rpc.control.ControlServer.handle(ControlServer.java:60) >> [drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> org.apache.drill.exec.rpc.control.ControlServer.handle(ControlServer.java:38) >> [drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:57) >> [drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:194) >> [drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:173) >> [drill-java-exec-0.9.0-rebuffed.jar:0.9.0] >> at >> >> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) >> [netty-codec-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) >> [netty-transport-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) >> [netty-transport-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) >> [netty-codec-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) >> [netty-transport-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) >> [netty-transport-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:161) >> [netty-codec-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) >> [netty-transport-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) >> [netty-transport-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) >> [netty-transport-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) >> [netty-transport-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) >> [netty-transport-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) >> [netty-transport-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) >> [netty-transport-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) >> [netty-transport-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) >> [netty-transport-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) >> [netty-transport-4.0.24.Final.jar:4.0.24.Final] >> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) >> [netty-transport-4.0.24.Final.jar:4.0.24.Final] >> at >> >> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) >> [netty-common-4.0.24.Final.jar:4.0.24.Final] >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25] >> 2015-06-11 11:42:35,547 [BitServer-5] INFO >> o.a.drill.exec.work.foreman.Foreman - foreman cleaning up. >> 2015-06-11 11:42:35,548 [BitServer-5] INFO >> o.a.drill.exec.work.foreman.Foreman - State change requested. FAILED --> >> COMPLETED >> 2015-06-11 11:42:35,548 [BitServer-5] WARN >> o.a.drill.exec.work.foreman.Foreman - Dropping request to move to >> COMPLETED state as query is already at FAILED state (which is terminal). >> >> On Thu, Jun 11, 2015 at 11:42 AM, Abhishek Girish < >> [email protected] >> > wrote: >> >> > It does look like an issue with HBase timeout. Does is occur >> consistently? >> > >> > Can you share more details about the queries you ran and error messages >> in >> > logs? >> > >> > -Abhishek >> > >> > On Wed, Jun 10, 2015 at 8:32 PM, George Lu <[email protected]> >> wrote: >> > >> > > Query failed: SYSTEM ERROR: >> > org.apache.hadoop.hbase.DoNotRetryIOException: >> > > Failed after retry of OutOfOrderScannerNextException: was there a rpc >> > > timeout? >> > > >> > >> > >
