Khurram Faraaz created DRILL-5561:
-------------------------------------
Summary: IllegalArgumentException: Self-suppression not permitted,
seen when foreman Drillbit is killed
Key: DRILL-5561
URL: https://issues.apache.org/jira/browse/DRILL-5561
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Environment: 3 node Centos cluster
Reporter: Khurram Faraaz
Priority: Minor
Drill 1.11.0 running on three nodes
kill foreman Drillbit, [root@centos-01 ~]# kill -9 15852; while a long running
SQL query (TPCDS Query 11) is under execution
drillbit.log from centos-01.qa.lab (foreman Drillbit) has no Exceptions or
Errors logged in it.
snippet of drillbit.log from centos-02.qa.lab ( non foreman Drillbit )
{noformat}
2017-05-30 08:21:32,834 [BitClient-1] INFO o.a.d.e.w.fragment.FragmentExecutor
- 26d2d481-d949-1de8-9497-c1492f086501:12:0: State change requested RUNNING -->
FAILED
2017-05-30 08:21:32,834 [BitServer-9] WARN o.a.d.exec.rpc.RpcExceptionHandler
- Exception occurred with closed channel. Connection: /10.10.100.202:31012
<--> /10.10.100.201:50773 (data server)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_65]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
~[na:1.8.0_65]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
~[na:1.8.0_65]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_65]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
~[na:1.8.0_65]
at
io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311)
~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at io.netty.buffer.WrappedByteBuf.setBytes(WrappedByteBuf.java:407)
~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.buffer.UnsafeDirectLittleEndian.setBytes(UnsafeDirectLittleEndian.java:30)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at io.netty.buffer.DrillBuf.setBytes(DrillBuf.java:770)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at
io.netty.buffer.MutableWrappedByteBuf.setBytes(MutableWrappedByteBuf.java:280)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at
io.netty.buffer.ExpandableByteBuf.setBytes(ExpandableByteBuf.java:26)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
2017-05-30 08:21:32,848 [BitClient-1] INFO o.a.d.e.w.fragment.FragmentExecutor
- 26d2d481-d949-1de8-9497-c1492f086501:16:0: State change requested RUNNING -->
FAILED
2017-05-30 08:21:32,850 [BitClient-1] INFO o.a.d.e.w.fragment.FragmentExecutor
- 26d2d481-d949-1de8-9497-c1492f086501:14:0: State change requested RUNNING -->
FAILED
2017-05-30 08:21:32,851 [BitClient-1] WARN
o.apache.drill.exec.rpc.RequestIdMap - Failure while attempting to fail rpc
response.
java.lang.IllegalArgumentException: Self-suppression not permitted
at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[na:1.8.0_65]
at
org.apache.drill.common.DeferredException.addException(DeferredException.java:88)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:409)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.access$700(FragmentExecutor.java:55)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$ExecutorStateImpl.fail(FragmentExecutor.java:423)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.ops.FragmentContext.fail(FragmentContext.java:214)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.ops.FragmentContext$1.accept(FragmentContext.java:96)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.ops.FragmentContext$1.accept(FragmentContext.java:93)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.ops.StatusHandler.failed(StatusHandler.java:42)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.ListeningCommand$DeferredRpcOutcome.failed(ListeningCommand.java:50)
~[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.data.DataTunnel$ThrottlingOutcomeListener.failed(DataTunnel.java:127)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:134)
~[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:74)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:64)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692)
[hppc-0.7.1.jar:na]
at
org.apache.drill.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:58)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.AbstractRemoteConnection.channelClosed(AbstractRemoteConnection.java:192)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.AbstractClientConnection.channelClosed(AbstractClientConnection.java:97)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:167)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:144)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:205)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:192)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: org.apache.drill.exec.rpc.ChannelClosedException: Channel closed
/10.10.100.202:46969 <--> centos-01.qa.lab/10.10.100.201:31012.
at
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:164)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
... 18 common frames omitted
2017-05-30 08:21:32,852 [BitClient-1] INFO o.a.d.e.w.fragment.FragmentExecutor
- 26d2d481-d949-1de8-9497-c1492f086501:10:0: State change requested RUNNING -->
FAILED
2017-05-30 08:21:32,853 [BitClient-1] WARN
o.apache.drill.exec.rpc.RequestIdMap - Failure while attempting to fail rpc
response.
java.lang.IllegalArgumentException: Self-suppression not permitted
at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[na:1.8.0_65]
at
org.apache.drill.common.DeferredException.addException(DeferredException.java:88)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:409)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.access$700(FragmentExecutor.java:55)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$ExecutorStateImpl.fail(FragmentExecutor.java:423)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.ops.FragmentContext.fail(FragmentContext.java:214)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.ops.FragmentContext$1.accept(FragmentContext.java:96)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.ops.FragmentContext$1.accept(FragmentContext.java:93)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.ops.StatusHandler.failed(StatusHandler.java:42)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.ListeningCommand$DeferredRpcOutcome.failed(ListeningCommand.java:50)
~[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.data.DataTunnel$ThrottlingOutcomeListener.failed(DataTunnel.java:127)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:134)
~[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:74)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:64)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692)
[hppc-0.7.1.jar:na]
at
org.apache.drill.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:58)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.AbstractRemoteConnection.channelClosed(AbstractRemoteConnection.java:192)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.AbstractClientConnection.channelClosed(AbstractClientConnection.java:97)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:167)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:144)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:205)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:192)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
...
...
2017-05-30 08:22:14,210 [BitServer-3] ERROR o.apache.drill.exec.rpc.BasicClient
- Failed to establish connection
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection
refused: centos-01.qa.lab/10.10.100.201:31011
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:47)
~[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
org.apache.drill.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionHandler.operationComplete(BasicClient.java:225)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionHandler.operationComplete(BasicClient.java:212)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:268)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:284)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: java.net.ConnectException: Connection refused:
centos-01.qa.lab/10.10.100.201:31011
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
~[na:1.8.0_65]
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
~[na:1.8.0_65]
at
io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:281)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
... 6 common frames omitted
{noformat}
snippet of drillbit.log from centos-04.qa.lab (another non foreman Drillbit)
{noformat}
2017-05-30 08:21:19,892 [26d2d481-d949-1de8-9497-c1492f086501:frag:8:2] INFO
o.a.d.e.w.f.FragmentStatusReporter - 26d2d481-d949-1de8-9497-c1492f086501:8:2:
State to report: RUNNING
2017-05-30 08:21:35,723 [BitServer-10] ERROR
o.apache.drill.exec.rpc.BasicClient - Failed to establish connection
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection
refused: centos-01.qa.lab/10.10.100.201:31011
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:47)
~[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
org.apache.drill.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionHandler.operationComplete(BasicClient.java:225)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionHandler.operationComplete(BasicClient.java:212)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:268)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:284)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: java.net.ConnectException: Connection refused:
centos-01.qa.lab/10.10.100.201:31011
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
~[na:1.8.0_65]
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
~[na:1.8.0_65]
at
io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:281)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
... 6 common frames omitted
{noformat}
Adding details from [~paul-rogers] email about this problem
{noformat}
The message appears to come from Netty, it is not a Drill string.
I think what this is saying is that we did something wrong in Netty and Netty
is reporting a problem that we’ve violated its API rules.
We should figure out what we did wrong and fix it so that we don’t use Netty
wrong.
Since we killed a Drillbit, the message should be “Connection to a.b.c.d
dropped” — something the user can understand.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
