Hi Team,
I have successfully launched a Drill cluster on 3 docker containers with Zoo
keeper Quorum and accessing data from a remote node on which Drill is not
installed . I am able to list all the drill bits from each node and run
queries on a single drill bit , But I am facing issue when I am trying to run a
query which distributes the query to the other drill bits by creating the
intermediate fragments. I have gone through the documentation but It couldn't
help. I got the below error message in Drill shell and the detailed error log
mentioned below. Please help me on resolving this.
Error in Drill Shell:
Error: CONNECTION ERROR: Error setting up remote intermediate fragment execution
Nodes with failures drillslave-7dccb67657-9qv4l, drillslave-7dccb67657-rhfgt
[Error Id: 5d701a0d-988b-4845-83f6-b029ad6825a8 on
drillmaster-76d6885c59-wpksz:31010] (state=,code=0)
Error Log:
2018-04-11 07:26:17,391 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,401 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,404 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Execute
d 1 out of 1 using 1 threads. Time: 2ms total, 2.798853ms avg, 2ms max.
2018-04-11 07:26:17,404 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Execute
d 1 out of 1 using 1 threads. Earliest start: 0.699000 ?s, Latest start:
0.699000 ?s, Average start: 0.699000 ?s .
2018-04-11 07:26:17,408 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO
o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pr
uning, pruning class:
org.apache.drill.exec.planner.logical.partition.PruneScanRule$DirPruneScanFilterOnScanRule
2018-04-11 07:26:17,408 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO
o.a.d.e.p.l.partition.PruneScanRule - No partition columns a
re projected from the scan..continue. Total pruning elapsed time: 0 ms
2018-04-11 07:26:17,611 [BitServer-3] ERROR o.a.d.e.rpc.ConnectionMultiListener
- Failed to establish connection
java.util.concurrent.ExecutionException:
java.nio.channels.UnresolvedAddressException
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54)
~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89)
[drill-
rpc-1.12.0.jar:1.12.0]
at
org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76)
[drill-
rpc-1.12.0.jar:1.12.0]
at
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
[netty-common-4.0.48.Final.jar:4.0.48.Fina
l]
at
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
[netty-transport-4.0.48.Final.ja
r:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
[netty-transport-4.0.48.F
inal.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
at
io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
[netty-transport-4.0.48.Final.ja
r:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
[netty-transport-4.0.48.F
inal.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
at
io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
[netty-transport-4.0.48.F
inal.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:506)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
at
io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970)
[netty-transport-4.0.48.Final.jar:4.0.48.Fina
l]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:214)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
[netty-common-4.0.48.Final.
at
io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
[netty-common-4.0.48.Final.
jar:4.0.48.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
[netty-common-4.0.48.Final.jar:4.
0.48.Final]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.nio.channels.UnresolvedAddressException: null
at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_161]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
~[na:1.8.0_161]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83)
~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80)
~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at java.security.AccessController.doPrivileged(Native Method)
~[na:1.8.0_161]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80)
~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:243)
~[netty-transport-4.0.48.Final.jar:4.0.48.Fi
nal]
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
... 17 common frames omitted
2018-04-11 07:26:17,693 [BitServer-4] ERROR o.a.d.e.rpc.ConnectionMultiListener
- Failed to establish connection
java.util.concurrent.ExecutionException:
java.nio.channels.UnresolvedAddressException
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54)
~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89)
[drill-
rpc-1.12.0.jar:1.12.0]
at
org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76)
[drill-
rpc-1.12.0.jar:1.12.0]
at
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
[netty-common-4.0.48.Final.jar:4.0.48.Fina
l]
at
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
[netty-transport-4.0.48.Final.ja
r:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
[netty-transport-4.0.48.F
inal.jar:4.0.48.Final]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.nio.channels.UnresolvedAddressException: null
at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_161]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
~[na:1.8.0_161]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83)
~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80)
~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at java.security.AccessController.doPrivileged(Native Method)
~[na:1.8.0_161]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80)
~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:243)
~[netty-transport-4.0.48.Final.jar:4.0.48.Fi
nal]
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
... 17 common frames omitted
2018-04-11 07:26:17,694 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - User Error Occurred: E
rror setting up remote intermediate fragment execution (Command failed while
establishing connection. Failure type CONNECTION.)
org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Error
setting up remote intermediate fragment execution
Nodes with failures drillslave-7dccb67657-9qv4l, drillslave-7dccb67657-rhfgt
[Error Id: 7e6e629d-1558-4a76-b9a9-6d34f86015a2 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
~[drill-common-1.12.0.jar:1.12.0]
at
org.apache.drill.exec.work.foreman.Foreman.scheduleRemoteIntermediateFragments(Foreman.java:1126)
[drill-java-exec-1.12.0.jar:
1.12.0]
at
org.apache.drill.exec.work.foreman.Foreman.setupNonRootFragments(Foreman.java:1217)
[drill-java-exec-1.12.0.jar:1.12.0]
at
org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan(Foreman.java:454)
[drill-java-exec-1.12.0.jar:1.12.0]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1019)
[drill-java-exec-1.12.0.jar:1.12.0]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:289)
[drill-java-exec-1.12.0.jar:1.12.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: org.apache.drill.exec.rpc.RpcException: Command failed while
establishing connection. Failure type CONNECTION.
at
org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:67)
~[drill-rpc-1.12.0.jar:1.12.0]
at
org.apache.drill.exec.rpc.ListeningCommand.connectionFailed(ListeningCommand.java:66)
~[drill-rpc-1.12.0.jar:1.12.0]
at
org.apache.drill.exec.rpc.ReconnectingConnection$ConnectionListeningFuture.connectionFailed(ReconnectingConnection.java:152)
~
[drill-rpc-1.12.0.jar:1.12.0]
at
org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:118)
~[dril
l-rpc-1.12.0.jar:1.12.0]
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
~[netty-transport-4.0.48.Final.
jar:4.0.48.Final]
... 17 common frames omitted
2018-04-11 07:26:17,702 [BitServer-9] INFO o.a.d.e.w.fragment.FragmentExecutor
- 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0: State change
requested AWAITING_ALLOCATION --> CANCELLATION_REQUESTED
2018-04-11 07:26:17,702 [BitServer-9] INFO o.a.d.e.w.f.FragmentStatusReporter
- 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0: State to repor
t: CANCELLATION_REQUESTED
2018-04-11 07:26:17,702 [BitServer-9] INFO o.a.d.e.w.fragment.FragmentExecutor
- 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0: State change
requested CANCELLATION_REQUESTED --> FINISHED
2018-04-11 07:26:17,702 [BitServer-9] INFO o.a.d.e.w.f.FragmentStatusReporter
- 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0: State to repor
t: CANCELLED
2018-04-11 07:26:17,703 [BitServer-5] ERROR o.a.d.e.rpc.ConnectionMultiListener
- Failed to establish connection
java.util.concurrent.ExecutionException:
java.nio.channels.UnresolvedAddressException
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54)
~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89)
[drill-
rpc-1.12.0.jar:1.12.0]
at
org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76)
[drill-
rpc-1.12.0.jar:1.12.0]
at
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
[netty-common-4.0.48.Final.jar:4.0.48.Fina
l]
at
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
[netty-transport-4.0.48.Final.ja
r:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
[netty-transport-4.0.48.F
inal.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
... 17 common frames omitted
2018-04-11 07:26:17,704 [BitServer-5] ERROR
o.a.d.exec.work.foreman.QueryManager - Failure while attempting to CANCEL
fragment query_id {
part1: 2680283233054685576
part2: 2356620362624167161
}
major_fragment_id: 1
minor_fragment_id: 0
on endpoint address: "drillslave-7dccb67657-9qv4l"
user_port: 31010
control_port: 31011
data_port: 31012
version: "1.12.0"
state: ONLINE
with org.apache.drill.exec.rpc.RpcException: Command failed while establishing
connection. Failure type CONNECTION..
2018-04-11 07:26:17,705 [BitServer-6] ERROR o.a.d.e.rpc.ConnectionMultiListener
- Failed to establish connection
java.util.concurrent.ExecutionException:
java.nio.channels.UnresolvedAddressException
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54)
~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89)
[drill-
rpc-1.12.0.jar:1.12.0]
at
org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76)
[drill-
rpc-1.12.0.jar:1.12.0]
at
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
[netty-common-4.0.48.Final.jar:4.0.48.Fina
l]
Please respond back ASAP with your findings.
Best regards,
_________________________
Tilak
