zhangyihong opened a new issue #8321: why cause my Druid cluster crashed during run indexing service URL: https://github.com/apache/incubator-druid/issues/8321 these is middleManager logs : ``` 2019-08-15T06:43:56,234 ERROR [forking-task-runner-68] com.google.common.util.concurrent.ExecutionList - RuntimeException while executing runnable com.google.common.util.concurrent.Futures$4@78cf894 with executor com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService@3fecd319 java.util.concurrent.RejectedExecutionException: Task org.apache.druid.indexing.worker.WorkerTaskManager$$Lambda$38/872018987@1339048e rejected from java.util.concurrent.ThreadPoolExecutor@301f24[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3029] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_211] at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_211] at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_211] at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668) ~[?:1.8.0_211] at org.apache.druid.indexing.worker.WorkerTaskManager.submitNoticeToExec(WorkerTaskManager.java:178) ~[druid-indexing-service-0.14.0-incubating-iap14.jar:0.14.0-incubating-iap14] at org.apache.druid.indexing.worker.WorkerTaskManager.access$000(WorkerTaskManager.java:76) ~[druid-indexing-service-0.14.0-incubating-iap14.jar:0.14.0-incubating-iap14] at org.apache.druid.indexing.worker.WorkerTaskManager$2.onSuccess(WorkerTaskManager.java:242) ~[druid-indexing-service-0.14.0-incubating-iap14.jar:0.14.0-incubating-iap14] at org.apache.druid.indexing.worker.WorkerTaskManager$2.onSuccess(WorkerTaskManager.java:238) ~[druid-indexing-service-0.14.0-incubating-iap14.jar:0.14.0-incubating-iap14] at com.google.common.util.concurrent.Futures$4.run(Futures.java:1181) ~[guava-16.0.1.jar:?] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.1.jar:?] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) [guava-16.0.1.jar:?] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) [guava-16.0.1.jar:?] at com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:91) [guava-16.0.1.jar:?] at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:384) [?:1.8.0_211] at java.util.concurrent.FutureTask.set(FutureTask.java:233) [?:1.8.0_211] at java.util.concurrent.FutureTask.run(FutureTask.java:274) [?:1.8.0_211] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_211] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_211] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211] 2019-08-15T06:43:56,235 INFO [Thread-49] org.apache.druid.indexing.overlord.ForkingTaskRunner - Finished stopping in 14,028ms. 2019-08-15T06:43:56,235 INFO [Thread-49] org.apache.druid.indexing.worker.WorkerTaskManager - Stopped WorkerTaskManager. 2019-08-15T06:43:56,235 INFO [Thread-49] org.apache.druid.indexing.worker.WorkerTaskMonitor - Stopped WorkerTaskMonitor. 2019-08-15T06:43:56,235 INFO [Thread-49] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.discovery.DruidLeaderClient.stop()] on object[org.apache.druid.discovery.DruidLeaderClient@22f4f8f2]. 2019-08-15T06:43:56,235 INFO [Thread-49] org.apache.druid.discovery.DruidLeaderClient - Stopped. 2019-08-15T06:43:56,235 INFO [Thread-49] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[org.apache.druid.curator.discovery.ServerDiscoverySelector@1e1b061]. 2019-08-15T06:43:56,236 INFO [Thread-49] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider.stop()] on object[org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider@3a790e40]. 2019-08-15T06:43:56,236 INFO [Thread-49] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider - stopping 2019-08-15T06:43:56,236 INFO [Thread-49] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider - stopped 2019-08-15T06:43:56,236 INFO [Thread-49] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.java.util.http.client.NettyHttpClient.stop()] on object[org.apache.druid.java.util.http.client.NettyHttpClient@38830ea]. 2019-08-15T06:43:56,246 INFO [Thread-49] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.indexing.worker.WorkerCuratorCoordinator.stop()] on object[org.apache.druid.indexing.worker.WorkerCuratorCoordinator@45cd7bc5]. 2019-08-15T06:43:56,246 INFO [Thread-49] org.apache.druid.indexing.worker.WorkerCuratorCoordinator - Stopping WorkerCuratorCoordinator for worker[172.16.0.180:8091] 2019-08-15T06:43:56,246 INFO [Thread-49] org.apache.druid.curator.announcement.Announcer - Stopping announcer 2019-08-15T06:43:56,246 INFO [Thread-49] org.apache.druid.curator.announcement.Announcer - unannouncing [/druid/indexer/announcements/172.16.0.180:8091] 2019-08-15T06:43:56,247 INFO [Thread-49] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.java.util.metrics.MonitorScheduler.stop()] on object[org.apache.druid.java.util.metrics.MonitorScheduler@12bbfc54]. 2019-08-15T06:43:56,248 INFO [Thread-49] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.java.util.emitter.service.ServiceEmitter.close() throws java.io.IOException] on object[ServiceEmitter{serviceDimensions={service=druid/middlemanager, host=172.16.0.180:8091, version=0.14.0-incubating-iap14}, emitter=LoggingEmitter{log=Logger{name=[org.apache.druid.java.util.emitter.core.LoggingEmitter], class[class org.apache.logging.slf4j.Log4jLogger]}, level=DEBUG}}]. 2019-08-15T06:43:56,248 INFO [Thread-49] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.java.util.emitter.core.LoggingEmitter.close()] on object[LoggingEmitter{log=Logger{name=[org.apache.druid.java.util.emitter.core.LoggingEmitter], class[class org.apache.logging.slf4j.Log4jLogger]}, level=DEBUG}]. 2019-08-15T06:43:56,248 INFO [Thread-49] org.apache.druid.curator.CuratorModule - Stopping Curator 2019-08-15T06:43:56,248 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting 2019-08-15T06:43:56,249 INFO [Thread-49] org.apache.zookeeper.ZooKeeper - Session: 0x1000092abc10fd0 closed 2019-08-15T06:43:56,249 INFO [Thread-49] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT] 2019-08-15T06:43:56,249 INFO [Thread-49] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner.stop()] on object[org.apache.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner@2503ec73]. ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org