tanisdlj edited a comment on issue #10866:
URL: https://github.com/apache/druid/issues/10866#issuecomment-814167420
Ok, another crash. Supervisor task:
```
Indexer log:
{"ingestionStatsAndErrors":{"taskId":"index_kafka_ssp-events-hourly_4e62ca9663a43dc_cojlbdpc","payload":{"ingestionState":"BUILD_SEGMENTS","unparseableEvents":{},"rowStats":{"buildSegments":{"processed":184276930,"processedWithError":0,"thrownAway":78487,"unparseable":0}},"errorMsg":"org.apache.kafka.common.errors.InterruptException:
java.lang.InterruptedException\n\tat
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeThrowInterruptException(ConsumerNetworkClient.java:520)\n\tat
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281)\n\tat
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)\n\tat
org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1301)\n\tat
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237)\n\tat
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)\n\tat
org.apach
e.druid.indexing.kafka.KafkaRecordSupplier.poll(KafkaRecordSupplier.java:124)\n\tat
org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner.getRecords(IncrementalPublishingKafkaIndexTaskRunner.java:98)\n\tat
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:603)\n\tat
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:267)\n\tat
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:145)\n\tat
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:451)\n\tat
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:423)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoo
lExecutor.java:1149)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat
java.lang.Thread.run(Thread.java:748)\n\tSuppressed:
java.lang.InterruptedException\n\t\tat
java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)\n\t\tat
java.util.concurrent.FutureTask.get(FutureTask.java:191)\n\t\tat
org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver.persist(StreamAppenderatorDriver.java:231)\n\t\tat
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:756)\n\t\t...
8 more\nCaused by: java.lang.InterruptedException\n\t... 17
more\n"},"type":"ingestionStatsAndErrors"}}
```
```
2021-04-06T13:36:36,486 INFO [qtp1718695429-151]
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner -
Stopping gracefully (status: [PUBLISHING])
2021-04-06T13:36:36,490 ERROR [task-runner-0-priority-0]
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner -
Encountered exception in run() before persisting.
org.apache.kafka.common.errors.InterruptException:
java.lang.InterruptedException
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeThrowInterruptException(ConsumerNetworkClient.java:520)
~[?:?]
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281)
~[?:?]
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
~[?:?]
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1301)
~[?:?]
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237)
~[?:?]
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
~[?:?]
at
org.apache.druid.indexing.kafka.KafkaRecordSupplier.poll(KafkaRecordSupplier.java:124)
~[?:?]
at
org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner.getRecords(IncrementalPublishingKafkaIndexTaskRunner.java:98)
~[?:?]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:603)
[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:267)
[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:145)
[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:451)
[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:423)
[druid-indexing-service-0.20.1.jar:0.20.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: java.lang.InterruptedException
... 17 more
2021-04-06T13:36:36,527 ERROR [task-runner-0-priority-0]
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Error
while publishing segments for sequenceNumber[SequenceMetadata{sequenceId=12,
sequenceName='index_kafka_ssp-events-hourly_4e62ca9663a43dc_12',
assignments=[], startOffsets={70=4049765636, 10=97308684185},
exclusiveStartPartitions=[], endOffsets={70=4051651832, 10=97310341952},
sentinel=false, checkpointed=true}]
java.util.concurrent.CancellationException: Task was cancelled.
at
com.google.common.util.concurrent.AbstractFuture.cancellationExceptionWithCause(AbstractFuture.java:392)
~[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306)
~[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)
~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1170)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.cancel(AbstractFuture.java:134)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.Futures$ChainingListenableFuture.cancel(Futures.java:826)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.Futures$CombinedFuture$1.run(Futures.java:1505)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.cancel(AbstractFuture.java:134)
[guava-16.0.1.jar:?]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:867)
[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:267)
[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:145)
[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:451)
[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:423)
[druid-indexing-service-0.20.1.jar:0.20.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: java.util.concurrent.CancellationException: Future.cancel() was
called.
at
com.google.common.util.concurrent.AbstractFuture$Sync.complete(AbstractFuture.java:378)
~[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture$Sync.cancel(AbstractFuture.java:355)
~[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.cancel(AbstractFuture.java:131)
~[guava-16.0.1.jar:?]
... 15 more
2021-04-06T13:36:36,545 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[ssp-events-hourly_2021-04-06T13:00:00.000Z_2021-04-06T14:00:00.000Z_2021-04-06T13:00:01.025Z_103]
at
path[/druid/segments/druid-dataserver-11:8100/druid-dataserver-11:8100_indexer-executor__default_tier_2021-04-06T09:24:09.621Z_11bf1d0b81054d63a80702952d9e13700]
2021-04-06T13:36:36,546 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[ssp-events-hourly_2021-04-06T12:00:00.000Z_2021-04-06T13:00:00.000Z_2021-04-06T12:00:00.048Z_283]
at
path[/druid/segments/druid-dataserver-11:8100/druid-dataserver-11:8100_indexer-executor__default_tier_2021-04-06T09:24:09.621Z_11bf1d0b81054d63a80702952d9e13700]
2021-04-06T13:36:36,546 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[ssp-events-hourly_2021-04-06T13:00:00.000Z_2021-04-06T14:00:00.000Z_2021-04-06T13:00:01.025Z_30]
at
path[/druid/segments/druid-dataserver-11:8100/druid-dataserver-:8100_indexer-executor__default_tier_2021-04-06T09:24:09.621Z_11bf1d0b81054d63a80702952d9e13700]
2021-04-06T13:36:36,546 INFO [task-runner-0-priority-0]
org.apache.druid.curator.announcement.Announcer - Unannouncing
[/druid/segments/druid-dataserver-11:8100/druid-dataserver-11:8100_indexer-executor__default_tier_2021-04-06T09:24:09.621Z_11bf1d0b81054d63a80702952d9e13700]
2021-04-06T13:36:37,104 WARN
[[index_kafka_ssp-events-hourly_4e62ca9663a43dc_cojlbdpc]-appenderator-merge]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Failed to
push merged index for
segment[ssp-events-hourly_2021-04-06T12:00:00.000Z_2021-04-06T13:00:00.000Z_2021-04-06T12:00:00.048Z_283].
java.nio.channels.ClosedByInterruptException: null
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
~[?:1.8.0_275]
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
~[?:1.8.0_275]
at org.apache.druid.io.Channels.writeFully(Channels.java:32)
~[druid-core-0.20.1.jar:0.20.1]
at
org.apache.druid.segment.writeout.FileWriteOutBytes.flush(FileWriteOutBytes.java:62)
~[druid-processing-0.20.1.jar:0.20.1]
at
org.apache.druid.segment.writeout.FileWriteOutBytes.flushIfNeeded(FileWriteOutBytes.java:54)
~[druid-processing-0.20.1.jar:0.20.1]
at
org.apache.druid.segment.writeout.FileWriteOutBytes.write(FileWriteOutBytes.java:86)
~[druid-processing-0.20.1.jar:0.20.1]
at
org.apache.druid.segment.data.GenericIndexedWriter$1.writeTo(GenericIndexedWriter.java:123)
~[druid-processing-0.20.1.jar:0.20.1]
at
org.apache.druid.segment.data.GenericIndexedWriter$1.writeTo(GenericIndexedWriter.java:96)
~[druid-processing-0.20.1.jar:0.20.1]
at
org.apache.druid.segment.data.GenericIndexedWriter.write(GenericIndexedWriter.java:237)
~[druid-processing-0.20.1.jar:0.20.1]
at
org.apache.druid.segment.data.BlockLayoutColumnarDoublesSerializer.add(BlockLayoutColumnarDoublesSerializer.java:95)
~[druid-processing-0.20.1.jar:0.20.1]
at
org.apache.druid.segment.DoubleColumnSerializer.serialize(DoubleColumnSerializer.java:83)
~[druid-processing-0.20.1.jar:0.20.1]
at
org.apache.druid.segment.IndexMergerV9.mergeIndexesAndWriteColumns(IndexMergerV9.java:573)
~[druid-processing-0.20.1.jar:0.20.1]
at
org.apache.druid.segment.IndexMergerV9.makeIndexFiles(IndexMergerV9.java:267)
~[druid-processing-0.20.1.jar:0.20.1]
at org.apache.druid.segment.IndexMergerV9.merge(IndexMergerV9.java:999)
~[druid-processing-0.20.1.jar:0.20.1]
at
org.apache.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:917)
~[druid-processing-0.20.1.jar:0.20.1]
at
org.apache.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:895)
~[druid-processing-0.20.1.jar:0.20.1]
at
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:769)
~[druid-server-0.20.1.jar:0.20.1]
at
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:660)
~[druid-server-0.20.1.jar:0.20.1]
at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861)
[guava-16.0.1.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
2021-04-06T13:36:37,133 ERROR
[[index_kafka_ssp-events-hourly_4e62ca9663a43dc_cojlbdpc]-appenderator-persist]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Incremental
persist failed:
{class=org.apache.druid.segment.realtime.appenderator.AppenderatorImpl,
segment=ssp-events-hourly_2021-04-06T13:00:00.000Z_2021-04-06T14:00:00.000Z_2021-04-06T13:00:01.025Z_103,
dataSource=ssp-events-hourly, count=9}
2021-04-06T13:36:37,157 INFO [task-runner-0-priority-0]
org.apache.druid.curator.announcement.Announcer - Unannouncing
[/druid/internal-discovery/PEON/druid-dataserver-11:8100]
2021-04-06T13:36:37,164 INFO [task-runner-0-priority-0]
org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Unannounced self
[{"druidNode":{"service":"druid/middleManager","host":"druid-dataserver-11","bindOnHost":false,"plaintextPort":8100,"port":-1,"tlsPort":-1,"enablePlaintextPort":true,"enableTlsPort":false},"nodeType":"peon","services":{"dataNodeService":{"type":"dataNodeService","tier":"_default_tier","maxSize":0,"type":"indexer-executor","priority":0},"lookupNodeService":{"type":"lookupNodeService","lookupTier":"__default"}}}].
2021-04-06T13:36:37,165 INFO [task-runner-0-priority-0]
org.apache.druid.curator.announcement.Announcer - Unannouncing
[/druid/announcements/druid-dataserver-11:8100]
2021-04-06T13:36:37,179 ERROR [task-runner-0-priority-0]
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner -
Encountered exception while running task.
org.apache.kafka.common.errors.InterruptException:
java.lang.InterruptedException
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeThrowInterruptException(ConsumerNetworkClient.java:520)
~[?:?]
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281)
~[?:?]
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
~[?:?]
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1301)
~[?:?]
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237)
~[?:?]
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
~[?:?]
at
org.apache.druid.indexing.kafka.KafkaRecordSupplier.poll(KafkaRecordSupplier.java:124)
~[?:?]
at
org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner.getRecords(IncrementalPublishingKafkaIndexTaskRunner.java:98)
~[?:?]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:603)
~[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:267)
[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:145)
[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:451)
[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:423)
[druid-indexing-service-0.20.1.jar:0.20.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Suppressed: java.lang.InterruptedException
at
java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) ~[?:1.8.0_275]
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
~[?:1.8.0_275]
at
org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver.persist(StreamAppenderatorDriver.java:231)
~[druid-server-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:756)
~[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:267)
[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:145)
[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:451)
[druid-indexing-service-0.20.1.jar:0.20.1]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:423)
[druid-indexing-service-0.20.1.jar:0.20.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: java.lang.InterruptedException
... 17 more
2021-04-06T13:36:37,217 INFO [task-runner-0-priority-0]
org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed
with status: {
"id" : "index_kafka_ssp-events-hourly_4e62ca9663a43dc_cojlbdpc",
"status" : "FAILED",
"duration" : 15153268,
"errorMsg" : "org.apache.kafka.common.errors.InterruptException:
java.lang.InterruptedException\n\tat org.apache.kaf...",
"location" : {
"host" : null,
"port" : -1,
"tlsPort" : -1
}
}
2021-04-06T13:36:37,229 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [ANNOUNCEMENTS]
2021-04-06T13:36:37,230 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [SERVER]
2021-04-06T13:37:07,254 INFO [main]
org.eclipse.jetty.server.AbstractConnector - Stopped
ServerConnector@73dbe25{HTTP/1.1, (http/1.1)}{0.0.0.0:8100}
2021-04-06T13:37:07,255 INFO [main] org.eclipse.jetty.server.session - node0
Stopped scavenging
2021-04-06T13:37:07,259 INFO [main]
org.eclipse.jetty.server.handler.ContextHandler - Stopped
o.e.j.s.ServletContextHandler@6d35ff7b{/,null,STOPPED}
2021-04-06T13:37:07,266 ERROR [main]
org.apache.druid.server.initialization.jetty.JettyServerModule - Jetty
lifecycle event failed [class org.eclipse.jetty.server.Server]
java.util.concurrent.TimeoutException: null
at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:130)
~[jetty-util-9.4.34.v20201102.jar:9.4.34.v20201102]
at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30)
~[jetty-util-9.4.34.v20201102.jar:9.4.34.v20201102]
at
org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:175)
~[jetty-server-9.4.34.v20201102.jar:9.4.34.v20201102]
at org.eclipse.jetty.server.Server.doStop(Server.java:447)
~[jetty-server-9.4.34.v20201102.jar:9.4.34.v20201102]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
[jetty-util-9.4.34.v20201102.jar:9.4.34.v20201102]
at
org.apache.druid.server.initialization.jetty.JettyServerModule$2.stop(JettyServerModule.java:454)
[druid-server-0.20.1.jar:0.20.1]
at
org.apache.druid.java.util.common.lifecycle.Lifecycle.stop(Lifecycle.java:368)
[druid-core-0.20.1.jar:0.20.1]
at org.apache.druid.cli.CliPeon.run(CliPeon.java:306)
[druid-services-0.20.1.jar:0.20.1]
at org.apache.druid.cli.Main.main(Main.java:113)
[druid-services-0.20.1.jar:0.20.1]
2021-04-06T13:37:07,270 WARN [main]
org.apache.druid.server.initialization.jetty.JettyServerModule - Unable to stop
Jetty server.
java.util.concurrent.TimeoutException: null
at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:130)
~[jetty-util-9.4.34.v20201102.jar:9.4.34.v20201102]
at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30)
~[jetty-util-9.4.34.v20201102.jar:9.4.34.v20201102]
at
org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:175)
~[jetty-server-9.4.34.v20201102.jar:9.4.34.v20201102]
at org.eclipse.jetty.server.Server.doStop(Server.java:447)
~[jetty-server-9.4.34.v20201102.jar:9.4.34.v20201102]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
~[jetty-util-9.4.34.v20201102.jar:9.4.34.v20201102]
at
org.apache.druid.server.initialization.jetty.JettyServerModule$2.stop(JettyServerModule.java:454)
[druid-server-0.20.1.jar:0.20.1]
at
org.apache.druid.java.util.common.lifecycle.Lifecycle.stop(Lifecycle.java:368)
[druid-core-0.20.1.jar:0.20.1]
at org.apache.druid.cli.CliPeon.run(CliPeon.java:306)
[druid-services-0.20.1.jar:0.20.1]
at org.apache.druid.cli.Main.main(Main.java:113)
[druid-services-0.20.1.jar:0.20.1]
2021-04-06T13:37:07,271 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [NORMAL]
2021-04-06T13:37:07,272 INFO [main]
org.apache.druid.server.coordination.ZkCoordinator - Stopping ZkCoordinator for
[DruidServerMetadata{name='druid-dataserver-11:8100',
hostAndPort='druid-dataserver-1:8100', hostAndTlsPort='null', maxSize=0,
tier='_default_tier', type=indexer-executor, priority=0}]
2021-04-06T13:37:07,272 INFO [main]
org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopping...
2021-04-06T13:37:07,272 INFO [main]
org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopped.
2021-04-06T13:37:07,277 INFO [main]
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting
graceful shutdown of
task[index_kafka_ssp-events-hourly_4e62ca9663a43dc_cojlbdpc].
2021-04-06T13:37:07,277 INFO [main]
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner -
Stopping gracefully (status: [PUBLISHING])
2021-04-06T13:37:07,285 INFO [main]
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Graceful
shutdown of task[index_kafka_ssp-events-hourly_4e62ca9663a43dc_cojlbdpc]
finished in 0ms.
2021-04-06T13:37:07,353 INFO
[LookupExtractorFactoryContainerProvider-MainThread]
org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop
exited. Lookup notices are not handled anymore.
2021-04-06T13:37:07,354 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/publishers.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='token',
valueFieldName='IIN'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@59ffeff7
2021-04-06T13:37:07,356 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[publisherProductIIN].
2021-04-06T13:37:07,356 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/adtagConfigs.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='id',
valueFieldName='creativePublicationId'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@3cc22ab4
2021-04-06T13:37:07,356 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[adtagCreativePublicationId].
2021-04-06T13:37:07,356 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/publishers.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='token',
valueFieldName='tier'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@13bfbdc2
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[publisherTier].
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/adtagConfigs.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='id',
valueFieldName='creativePublicationName'}, fileRegex='null', pollPeriod=PT30M}]
:
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@4d3fea62
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[adtagCreativePublicationName].
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/adtagConfigs.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='id',
valueFieldName='videoDuration'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@2860a0af
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[adtagVideoDuration].
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/adtagConfigs.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='id',
valueFieldName='adtagName'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@6a3ae078
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[adtagName].
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/adtagConfigs.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='id',
valueFieldName='creativeId'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@4ff4c694
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[adtagCreativeId].
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/publishers.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='token',
valueFieldName='IID'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@32a8e87a
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[publisherProductIID].
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/publishers.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='token',
valueFieldName='FID'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@57380ab8
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[publisherProductFID].
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/deals.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='dealId',
valueFieldName='name'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@31c4da0d
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[dealName].
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/publishers.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='token',
valueFieldName='country'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@53078487
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[publisherCountry].
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/publishers.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='token',
valueFieldName='id'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@3c6f005f
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[publisherId].
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/publishers.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='token',
valueFieldName='editorialGroupName'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@64fd134f
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[editorialGroupName].
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/publishers.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='token',
valueFieldName='PIV'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@16050083
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[publisherProductPIV].
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/publishers.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='token',
valueFieldName='categoryName'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@3850830a
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[publisherCategory].
2021-04-06T13:37:07,357 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/publishers.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='token',
valueFieldName='name'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@60c28b08
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[publisherName].
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/adtagConfigs.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='id',
valueFieldName='productFamily'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@1107325c
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[adtagProductFamily].
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/adtagConfigs.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='id',
valueFieldName='size'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@25c01974
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[adtagSize].
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/adtagConfigs.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='id',
valueFieldName='extAdserver'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@43f81db5
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[adtagExtAdserver].
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/publishers.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='token',
valueFieldName='IVD'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@30a3d5e3
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[publisherProductIVD].
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/adtagConfigs.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='id',
valueFieldName='channel'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@79c0a348
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[adtagChannel].
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/publishers.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='token',
valueFieldName='IIV'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@6021c46e
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[publisherProductIIV].
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/publishers.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='token',
valueFieldName='FIV'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@47c78112
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[publisherProductFIV].
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Closing
namespace
[UriExtractionNamespace{uri=file:///usr/share/druid/lookups/publishers.json,
uriPrefix=null, namespaceParseSpec=JSONFlatDataParser{keyFieldName='token',
valueFieldName='IIT'}, fileRegex='null', pollPeriod=PT30M}] :
org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@7d520a64
2021-04-06T13:37:07,358 INFO [main]
org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup
[publisherProductIIT].
2021-04-06T13:37:07,416 INFO [Curator-Framework-0]
org.apache.curator.framework.imps.CuratorFrameworkImpl -
backgroundOperationsLoop exiting
2021-04-06T13:37:07,422 INFO [main] org.apache.zookeeper.ZooKeeper -
Session: 0x1034ca014c22804 closed
2021-04-06T13:37:07,422 INFO [main-EventThread]
org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
0x1034ca014c22804
2021-04-06T13:37:07,422 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle$CloseableHandler -
Closing object[org.asynchttpclient.DefaultAsyncHttpClient@7be3abaa]
2021-04-06T13:37:07,434 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [INIT]
Finished peon task
```
[index_kafka_ssp-events-hourly_4e62ca9663a43dc_cojlbdpc
.txt](https://github.com/apache/druid/files/6265609/index_kafka_ssp-events-hourly_4e62ca9663a43dc_cojlbdpc.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
