silverflow opened a new issue #10607:
URL: https://github.com/apache/druid/issues/10607
Please provide a detailed title (e.g. "Broker crashes when using TopN query
with Bound filter" instead of just "Broker crashes").
### Affected Version
**apache druid 0.20.0**
### Description
I try to load the data using kafka. but Task was failed. It was reset to
task after almost 10,442,979 row.
There is an task error log.
```
2020-11-26T02:01:04,048 INFO [parent-monitor-0]
org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Triggering JVM
shutdown.
2020-11-26T02:01:04,051 INFO [Thread-25] org.apache.druid.cli.CliPeon -
Running shutdown hook
2020-11-26T02:01:04,051 INFO [Thread-25]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [ANNOUNCEMENTS]
2020-11-26T02:01:04,054 INFO [Thread-25]
org.apache.druid.curator.announcement.Announcer - Unannouncing
[/druid/announcements/localhost:8101]
2020-11-26T02:01:04,079 INFO [Thread-25]
org.apache.druid.curator.announcement.Announcer - Unannouncing
[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:06:50.068Z_8e88ca0552d247ae96b6fbbec1328d635]
2020-11-26T02:01:04,090 INFO [Thread-25]
org.apache.druid.curator.announcement.Announcer - Unannouncing
[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:28:12.297Z_36d5d67f7e9d466b9b38c1a7e29716af8]
2020-11-26T02:01:04,092 INFO [Thread-25]
org.apache.druid.curator.announcement.Announcer - Unannouncing
[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:38:20.561Z_1fa942f9e8d8412ca44d227d08565ab61]
2020-11-26T02:01:04,093 INFO [Thread-25]
org.apache.druid.curator.announcement.Announcer - Unannouncing
[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:14:09.148Z_0bcc28e53a85461d81d683f5ace4f4ed6]
2020-11-26T02:01:04,097 INFO [Thread-25]
org.apache.druid.curator.announcement.Announcer - Unannouncing
[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:45:10.147Z_42363f4e23754b69ad78fc9f1750e30b2]
2020-11-26T02:01:04,098 INFO [Thread-25]
org.apache.druid.curator.announcement.Announcer - Unannouncing
[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:21:34.428Z_fad4b4fb98374ea4a7b4f8ef565687ed7]
2020-11-26T02:01:04,101 INFO [Thread-25]
org.apache.druid.curator.announcement.Announcer - Unannouncing
[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:30:46.379Z_5b6fd3bc23a14a6096c14a38097a50200]
2020-11-26T02:01:04,102 INFO [Thread-25]
org.apache.druid.curator.announcement.Announcer - Unannouncing
[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:00:01.340Z_7b3f1a77464849f9850116421dbe7bca4]
2020-11-26T02:01:04,104 INFO [Thread-25]
org.apache.druid.curator.announcement.Announcer - Unannouncing
[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:52:49.096Z_d2bbb9966c5d42249fd6484d8ff4170b3]
2020-11-26T02:01:04,112 INFO [Thread-25]
org.apache.druid.curator.announcement.Announcer - Unannouncing
[/druid/internal-discovery/PEON/localhost:8101]
2020-11-26T02:01:04,114 INFO [Thread-25]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [SERVER]
2020-11-26T02:01:04,125 INFO [Thread-25]
org.eclipse.jetty.server.AbstractConnector - Stopped
ServerConnector@5315266{HTTP/1.1, (http/1.1)}{0.0.0.0:8101}
2020-11-26T02:01:04,126 INFO [Thread-25] org.eclipse.jetty.server.session -
node0 Stopped scavenging
2020-11-26T02:01:04,135 INFO [Thread-25]
org.eclipse.jetty.server.handler.ContextHandler - Stopped
o.e.j.s.ServletContextHandler@1a0ffb9b{/,null,UNAVAILABLE}
2020-11-26T02:01:04,150 INFO [Thread-25]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [NORMAL]
2020-11-26T02:01:04,150 INFO [Thread-25]
org.apache.druid.server.coordination.ZkCoordinator - Stopping ZkCoordinator for
[DruidServerMetadata{name='localhost:8101', hostAndPort='localhost:8101',
hostAndTlsPort='null', maxSize=0, tier='_default_tier', type=indexer-executor,
priority=0}]
2020-11-26T02:01:04,151 INFO [Thread-25]
org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopping...
2020-11-26T02:01:04,151 INFO [Thread-25]
org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopped.
2020-11-26T02:01:04,152 INFO [Thread-25]
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting
graceful shutdown of task[index_kafka_news_29e33becd2f8188_gpkgficc].
2020-11-26T02:01:04,152 INFO [Thread-25]
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner -
Stopping forcefully (status: [PUBLISHING])
2020-11-26T02:01:04,162 ERROR [task-runner-0-priority-0]
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Error
while publishing segments for sequenceNumber[SequenceMetadata{sequenceId=0,
sequenceName='index_kafka_news_29e33becd2f8188_0', assignments=[],
startOffsets={0=0}, exclusiveStartPartitions=[], endOffsets={0=10451319},
sentinel=false, checkpointed=true}]
java.util.concurrent.CancellationException: Task was cancelled.
at
com.google.common.util.concurrent.AbstractFuture.cancellationExceptionWithCause(AbstractFuture.java:392)
~[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306)
~[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)
~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1170)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.cancel(AbstractFuture.java:134)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.Futures$ChainingListenableFuture.cancel(Futures.java:826)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.Futures$CombinedFuture$1.run(Futures.java:1505)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.cancel(AbstractFuture.java:134)
[guava-16.0.1.jar:?]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:841)
[druid-indexing-service-0.20.0.jar:0.20.0]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:267)
[druid-indexing-service-0.20.0.jar:0.20.0]
at
org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:145)
[druid-indexing-service-0.20.0.jar:0.20.0]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:451)
[druid-indexing-service-0.20.0.jar:0.20.0]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:423)
[druid-indexing-service-0.20.0.jar:0.20.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: java.util.concurrent.CancellationException: Future.cancel() was
called.
at
com.google.common.util.concurrent.AbstractFuture$Sync.complete(AbstractFuture.java:378)
~[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture$Sync.cancel(AbstractFuture.java:355)
~[guava-16.0.1.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.cancel(AbstractFuture.java:131)
~[guava-16.0.1.jar:?]
... 15 more
2020-11-26T02:01:04,189 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[news_2019-10-08T00:00:00.000Z_2019-10-09T00:00:00.000Z_2020-11-25T07:03:21.633Z_2]
at
path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:38:20.561Z_1fa942f9e8d8412ca44d227d08565ab61]
2020-11-26T02:01:04,191 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[news_2019-02-16T00:00:00.000Z_2019-02-17T00:00:00.000Z_2020-11-25T07:06:08.110Z_2]
at
path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:38:20.561Z_1fa942f9e8d8412ca44d227d08565ab61]
2020-11-26T02:01:04,193 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[news_2020-03-11T00:00:00.000Z_2020-03-12T00:00:00.000Z_2020-11-25T07:02:31.447Z_2]
at
path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:38:20.561Z_1fa942f9e8d8412ca44d227d08565ab61]
2020-11-26T02:01:04,194 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[news_2020-04-27T00:00:00.000Z_2020-04-28T00:00:00.000Z_2020-11-25T07:51:45.769Z_2]
at
path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:28:12.297Z_36d5d67f7e9d466b9b38c1a7e29716af8]
2020-11-26T02:01:04,195 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[news_2020-01-02T00:00:00.000Z_2020-01-03T00:00:00.000Z_2020-11-25T07:48:09.421Z_2]
at
path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:21:34.428Z_fad4b4fb98374ea4a7b4f8ef565687ed7]
2020-11-26T02:01:04,195 INFO
[LookupExtractorFactoryContainerProvider-MainThread]
org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop
exited. Lookup notices are not handled anymore.
2020-11-26T02:01:04,197 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[news_2019-01-31T00:00:00.000Z_2019-02-01T00:00:00.000Z_2020-11-25T07:11:34.639Z_2]
at
path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:45:10.147Z_42363f4e23754b69ad78fc9f1750e30b2]
2020-11-26T02:01:04,199 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[news_2019-08-27T00:00:00.000Z_2019-08-28T00:00:00.000Z_2020-11-25T06:57:55.971Z_2]
at
path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:30:46.379Z_5b6fd3bc23a14a6096c14a38097a50200]
2020-11-26T02:01:04,200 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[news_2019-07-02T00:00:00.000Z_2019-07-03T00:00:00.000Z_2020-11-25T07:17:01.648Z_2]
at
path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:52:49.096Z_d2bbb9966c5d42249fd6484d8ff4170b3]
2020-11-26T02:01:04,203 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[news_2020-02-12T00:00:00.000Z_2020-02-13T00:00:00.000Z_2020-11-25T07:22:34.133Z_2]
at
path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T00:52:49.096Z_d2bbb9966c5d42249fd6484d8ff4170b3]
2020-11-26T02:01:04,205 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[news_2019-01-24T00:00:00.000Z_2019-01-25T00:00:00.000Z_2020-11-25T07:26:56.883Z_2]
at
path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:00:01.340Z_7b3f1a77464849f9850116421dbe7bca4]
2020-11-26T02:01:04,208 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[news_2019-01-16T00:00:00.000Z_2019-01-17T00:00:00.000Z_2020-11-25T07:53:19.373Z_2]
at
path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:28:12.297Z_36d5d67f7e9d466b9b38c1a7e29716af8]
2020-11-26T02:01:04,210 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[news_2019-07-20T00:00:00.000Z_2019-07-21T00:00:00.000Z_2020-11-25T07:25:05.005Z_2]
at
path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:00:01.340Z_7b3f1a77464849f9850116421dbe7bca4]
2020-11-26T02:01:04,218 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[news_2019-07-24T00:00:00.000Z_2019-07-25T00:00:00.000Z_2020-11-25T07:29:21.789Z_2]
at
path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:00:01.340Z_7b3f1a77464849f9850116421dbe7bca4]
2020-11-26T02:01:04,220 INFO [Curator-Framework-0]
org.apache.curator.framework.imps.CuratorFrameworkImpl -
backgroundOperationsLoop exiting
2020-11-26T02:01:04,222 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[news_2020-05-01T00:00:00.000Z_2020-05-02T00:00:00.000Z_2020-11-25T07:35:31.105Z_2]
at
path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:06:50.068Z_8e88ca0552d247ae96b6fbbec1328d635]
2020-11-26T02:01:04,224 INFO [Thread-25] org.apache.zookeeper.ZooKeeper -
Session: 0x100037867f60030 closed
2020-11-26T02:01:04,224 INFO [main-EventThread]
org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
0x100037867f60030
2020-11-26T02:01:04,224 INFO [Thread-25]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [INIT]
2020-11-26T02:01:04,223 INFO [task-runner-0-priority-0]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing
segment[news_2019-03-11T00:00:00.000Z_2019-03-12T00:00:00.000Z_2020-11-25T07:47:58.558Z_2]
at
path[/druid/segments/localhost:8101/localhost:8101_indexer-executor__default_tier_2020-11-26T01:21:34.428Z_fad4b4fb98374ea4a7b4f8ef565687ed7]
```
I want to know why this happens.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
