Jonarzz commented on issue #7099:
URL: https://github.com/apache/druid/issues/7099#issuecomment-630721470
We're experiancing a similar issue on Druid 0.17.0:
```
2020-05-13T16:38:59,797 INFO [ServerInventoryView-0]
org.apache.druid.client.BatchServerInventoryView - Server
Disappeared[DruidServerMetadata{name='hostname:8101',
hostAndPort='hostname:8101', hostAndTlsPort='null', maxSize=0,
tier='_default_tier', type=indexer-executor, priority=0}]
2020-05-13T16:38:59,797 INFO [ServerInventoryView-0]
org.apache.druid.client.BatchServerInventoryView - Server
Disappeared[DruidServerMetadata{name='hostname:8101',
hostAndPort='hostname:8101', hostAndTlsPort='null', maxSize=0,
tier='_default_tier', type=indexer-executor, priority=0}]
2020-05-13T16:43:03,020 WARN [HttpClient-Netty-Worker-70]
org.apache.druid.java.util.http.client.NettyHttpClient - [POST
http://hostname2:8101/druid/v2/] Channel disconnected before response complete
2020-05-13T16:43:03,020 WARN [HttpClient-Netty-Worker-71]
org.apache.druid.java.util.http.client.NettyHttpClient - [POST
http://hostname2:8101/druid/v2/] Channel disconnected before response complete
2020-05-13T16:43:03,020 WARN [HttpClient-Netty-Worker-69]
org.apache.druid.java.util.http.client.NettyHttpClient - [POST
http://hostname2:8101/druid/v2/] Channel disconnected before response complete
2020-05-13T16:43:03,020 WARN [HttpClient-Netty-Worker-72]
org.apache.druid.java.util.http.client.NettyHttpClient - [POST
http://hostname2:8101/druid/v2/] Channel disconnected before response complete
2020-05-13T16:43:03,029 WARN
[qtp1952473604-259[groupBy_[eff_test]_30788e21-c073-41c0-8ad4-843f1b8f2f77]]
org.apache.druid.server.QueryLifecycle - Exception while processing queryId
[30788e21-c073-41c0-8ad4-843f1b8f2f77]
(QueryInterruptedException{msg=org.jboss.netty.channel.ChannelException:
Channel disconnected, code=Unknown exception,
class=java.util.concurrent.ExecutionException, host=hostname2:8101})
2020-05-13T16:43:03,029 WARN
[qtp1952473604-296[groupBy_[eff_test]_224dba66-3d1d-41cd-b83a-7d8b345f44ad]]
org.apache.druid.server.QueryLifecycle - Exception while processing queryId
[224dba66-3d1d-41cd-b83a-7d8b345f44ad]
(QueryInterruptedException{msg=org.jboss.netty.channel.ChannelException:
Channel disconnected, code=Unknown exception,
class=java.util.concurrent.ExecutionException, host=hostname2:8101})
2020-05-13T16:43:03,029 WARN [DruidSchema-Cache-0]
org.apache.druid.server.QueryLifecycle - Exception while processing queryId
[361a9aa2-7ed7-4363-a64c-65b1813b66a5]
(QueryInterruptedException{msg=org.jboss.netty.channel.ChannelException:
Channel disconnected, code=Unknown exception,
class=java.util.concurrent.ExecutionException, host=hostname2:8101})
...
2020-05-13T16:43:03,029 WARN [DruidSchema-Cache-0]
org.apache.druid.sql.calcite.schema.DruidSchema - Metadata refresh failed,
trying again soon.
org.apache.druid.query.QueryInterruptedException:
org.jboss.netty.channel.ChannelException: Channel disconnected
at
org.apache.druid.client.JsonParserIterator.interruptQuery(JsonParserIterator.java:189)
~[druid-server-0.17.0.jar:0.17.0]
at
org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:173)
~[druid-server-0.17.0.jar:0.17.0]
at
org.apache.druid.client.JsonParserIterator.hasNext(JsonParserIterator.java:95)
~[druid-server-0.17.0.jar:0.17.0]
at
org.apache.druid.java.util.common.guava.BaseSequence.makeYielder(BaseSequence.java:89)
~[druid-core-0.17.0.jar:0.17.0]
at
org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:69)
~[druid-core-0.17.0.jar:0.17.0]
at
org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
~[druid-core-0.17.0.jar:0.17.0]
at
org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$ResultBatch.fromSequence(ParallelMergeCombiningSequence.java:847)
~[druid-core-0.17.0.jar:0.17.0]
at
org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$SequenceBatcher.block(ParallelMergeCombiningSequence.java:897)
~[druid-core-0.17.0.jar:0.17.0]
at
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
~[?:1.8.0_242]
at
org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$SequenceBatcher.getBatchYielder(ParallelMergeCombiningSequence.java:886)
~[druid-core-0.17.0.jar:0.17.0]
at
org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$YielderBatchedResultsCursor.initialize(ParallelMergeCombiningSequence.java:993)
~[druid-core-0.17.0.jar:0.17.0]
at
org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$PrepareMergeCombineInputsAction.compute(ParallelMergeCombiningSequence.java:702)
~[druid-core-0.17.0.jar:0.17.0]
at
java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
~[?:1.8.0_242]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
~[?:1.8.0_242]
at
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
~[?:1.8.0_242]
at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
~[?:1.8.0_242]
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
~[?:1.8.0_242]
Caused by: java.util.concurrent.ExecutionException:
org.jboss.netty.channel.ChannelException: Channel disconnected
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:272)
~[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)
~[guava-16.0.1.jar:?]
at
org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:138)
~[druid-server-0.17.0.jar:0.17.0]
... 15 more
Caused by: org.jboss.netty.channel.ChannelException: Channel disconnected
at
org.apache.druid.java.util.http.client.NettyHttpClient$1.channelDisconnected(NettyHttpClient.java:351)
~[druid-core-0.17.0.jar:0.17.0]
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.cleanup(ReplayingDecoder.java:570)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:365)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
~[netty-3.10.6.Final.jar:?]
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
~[netty-3.10.6.Final.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_242]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_242]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
2020-05-13T16:43:10,235 WARN [ServerInventoryView-0]
org.apache.druid.client.BatchServerInventoryView - Not adding or running
callbacks for existing
segment[eff_test_2020-05-13T00:00:00.000Z_2020-05-14T00:00:00.000Z_2020-05-12T22:38:57.174Z_3]
on server[hostname2:8101]
```
Nothing special going on the aforementioned machines - at least not in logs:
```
2020-05-13T16:37:48,563 INFO [forking-task-runner-0]
org.apache.druid.indexing.overlord.ForkingTaskRunner - Logging task
index_kafka_eff_test_269717372a47df9_hippkgkp output to:
var/druid/task/index_kafka_eff_test_269717372a47df9_hippkgkp/log
2020-05-13T16:44:56,880 INFO
[forking-task-runner-0-[index_kafka_eff_test_269717372a47df9_hippkgkp]]
org.apache.druid.indexing.overlord.ForkingTaskRunner - Process exited with
status[3] for task: index_kafka_eff_test_269717372a47df9_hippkgkp
```
Is there a way to find the cause of this issue? Seems to occur randomly
(happened twice in 20 hours span).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
