zenfenan commented on issue #325: URL: https://github.com/apache/druid/issues/325#issuecomment-643065616
I think it still exists in 0.18.1. We have three brokers. After we run benchmark with some heavy queries, I see one of the broker getting into a stale/unresponsive state, meaning, the broker process is running on that node but it doesn't respond. I tried executing query directly on this broker which times out and even /status check doesn't produce a response. When I stop the broker process to restart the service again, I see a bunch of `WARN` log messages about jetty `QueuedThreadPool` and an `ERROR` saying `JettyServerModule - Jetty lifecycle event failed`. The complete error log is given below. ``` 2020-06-11T08:26:27,889 INFO [Thread-43] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/internal-discovery/BROKER/prod-euclid-druid-queryserver1-1735672:8082] 2020-06-11T08:26:27,899 INFO [NodeRoleWatcher[BROKER]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher - Node[http://prod-euclid-druid-queryserver1-1735672:8082] of role[broker] went offline. 2020-06-11T08:26:27,900 INFO [Thread-43] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Unannounced self [{"druidNode":{"service":"druid/broker","host":"prod-euclid-druid-queryserver1-1735672","bindOnHost":false,"plaintextPort":8082,"port":-1,"tlsPort":-1,"enablePlaintextPort":true,"enableTlsPort":false},"nodeType":"broker","services":{"lookupNodeService":{"type":"lookupNodeService","lookupTier":"__default"}}}]. 2020-06-11T08:26:27,901 INFO [Thread-43] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER] 2020-06-11T08:26:57,908 INFO [Thread-43] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@3909a854{HTTP/1.1,[http/1.1]}{0.0.0.0:8082} 2020-06-11T08:26:57,909 INFO [Thread-43] org.eclipse.jetty.server.session - node0 Stopped scavenging 2020-06-11T08:26:57,913 INFO [Thread-43] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@186dcb05{/,null,UNAVAILABLE} 2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-162[timeseries_[euclid_cdo_live_hourly]_18f82248-c0a4-4a1b-b163-7a9626c9d91b],5,main] 2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-165[topN_[euclid_cdo_live_hourly]_549ddbd3-f42e-4f69-b75f-ee053d5bb0d5],5,main] 2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-177[topN_[euclid_cdo_live_hourly]_c99ddc98-0077-4d09-a53d-efd9f21eb5d9],5,main] 2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-159[topN_[euclid_cdo_live_hourly]_2cbbaac8-171b-4039-b8ec-78439c0ca563],5,main] 2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-175[topN_[euclid_cdo_live_hourly]_748433e0-294f-4df3-837b-3836b20d5ef5],5,main] 2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-163[topN_[euclid_cdo_live_hourly]_e41aeaac-5830-4e3e-9f78-1c5fbfca4656],5,main] 2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-182[topN_[euclid_cdo_live_hourly]_0381d5c9-d858-4c9a-aa57-6353e0334453],5,main] 2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-150[topN_[euclid_cdo_live_hourly]_51dd4cda-9fcf-4533-af6f-71d11568e4ae],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-169[topN_[euclid_cdo_live_hourly]_9a40b9f3-e77c-4393-af88-a949df285d61],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-181[topN_[euclid_cdo_live_hourly]_f3a2af6d-78cd-4ff5-8df6-909edb8040bb],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-184[topN_[euclid_cdo_live_hourly]_4ef89502-6e54-436a-b54e-47776a11ee31],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-155[topN_[euclid_cdo_live_hourly]_a4e2f32b-62a7-40b1-9160-54b36205908d],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-173[topN_[euclid_cdo_live_hourly]_c38ae97c-fb65-4b12-91c2-3fb8511d3939],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-180[topN_[euclid_cdo_live_hourly]_2049db4c-6796-4a69-acbc-f38a4fa1a666],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-149[topN_[euclid_cdo_live_hourly]_85f8f14e-7eda-4176-9408-b723728f2a8e],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-179[topN_[euclid_cdo_live_hourly]_6d83af84-35eb-4ecd-b40a-4e615061e0a0],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-174[topN_[euclid_cdo_live_hourly]_1e84dba4-80d5-4bbf-bc52-7373a3adc2fe],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-186[topN_[euclid_cdo_live_hourly]_2c476c10-7e41-45f7-8437-02bd6907d96d],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-161[topN_[euclid_cdo_live_hourly]_312791d2-704d-4b76-8104-8a87c2baa42a],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-166[topN_[euclid_cdo_live_hourly]_a2a2ddc7-cd92-4f99-85af-77cb4b7d050c],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-168[topN_[euclid_cdo_live_hourly]_b352dcb8-6dbc-4b1c-83bf-54f37427a122],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-171[topN_[euclid_cdo_live_hourly]_65d58a84-b67d-444a-b261-5947dec1def9],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-178[topN_[euclid_cdo_live_hourly]_37b5c479-cb6d-4e9e-9917-4b99d510653a],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-157[topN_[euclid_cdo_live_hourly]_14a9ad98-f3a5-407e-b183-f953f9f93df1],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-154[topN_[euclid_cdo_live_hourly]_aa2697cd-5e1c-419b-891b-1c1563dc07b2],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-167[topN_[euclid_cdo_live_hourly]_848e3e21-5da2-4449-a96f-9a79e8581cfa],5,main] 2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-183[topN_[euclid_cdo_live_hourly]_2b2727fa-bc35-455e-b947-bb90463965bb],5,main] 2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-160[topN_[euclid_cdo_live_hourly]_b3f4badf-f5da-4103-820e-37d97afba7d5],5,main] 2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-170[topN_[euclid_cdo_live_hourly]_714624b8-fd38-46a3-8044-17cb91f54acd],5,main] 2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-164[topN_[euclid_cdo_live_hourly]_1fd07432-53bd-433b-9cda-714660401bcc],5,main] 2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-156[topN_[euclid_cdo_live_hourly]_cf25e506-acd1-4979-bfe4-07c0b67b5547],5,main] 2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-158[topN_[euclid_cdo_live_hourly]_e3c5d7c0-0dc1-4098-a8cb-2d0ee3c04a4f],5,main] 2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-176[timeseries_[euclid_cdo_live_hourly]_7755967f-6778-4064-8a23-0b3bdb8742ac],5,main] 2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-172[topN_[euclid_cdo_live_hourly]_c20848de-6dde-43ae-86e2-4cfe54f8f126],5,main] 2020-06-11T08:27:27,920 ERROR [Thread-43] org.apache.druid.server.initialization.jetty.JettyServerModule - Jetty lifecycle event failed [class org.eclipse.jetty.server.Server] java.util.concurrent.TimeoutException: null at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:129) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830] at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830] at org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:181) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830] at org.eclipse.jetty.server.Server.doStop(Server.java:439) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830] at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) [jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830] at org.apache.druid.server.initialization.jetty.JettyServerModule$2.stop(JettyServerModule.java:450) [druid-server-0.18.1.jar:0.18.1] at org.apache.druid.java.util.common.lifecycle.Lifecycle.stop(Lifecycle.java:368) [druid-core-0.18.1.jar:0.18.1] at org.apache.druid.java.util.common.lifecycle.Lifecycle$1.run(Lifecycle.java:401) [druid-core-0.18.1.jar:0.18.1] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172] 2020-06-11T08:27:27,920 WARN [Thread-43] org.apache.druid.server.initialization.jetty.JettyServerModule - Unable to stop Jetty server. java.util.concurrent.TimeoutException: null at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:129) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830] at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830] at org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:181) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830] at org.eclipse.jetty.server.Server.doStop(Server.java:439) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830] at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830] at org.apache.druid.server.initialization.jetty.JettyServerModule$2.stop(JettyServerModule.java:450) [druid-server-0.18.1.jar:0.18.1] at org.apache.druid.java.util.common.lifecycle.Lifecycle.stop(Lifecycle.java:368) [druid-core-0.18.1.jar:0.18.1] at org.apache.druid.java.util.common.lifecycle.Lifecycle$1.run(Lifecycle.java:401) [druid-core-0.18.1.jar:0.18.1] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172] 2020-06-11T08:27:27,920 INFO [Thread-43] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL] 2020-06-11T08:27:27,924 INFO [Thread-43] org.apache.druid.sql.calcite.schema.MetadataSegmentView - MetadataSegmentView is stopping. 2020-06-11T08:27:27,924 INFO [Thread-43] org.apache.druid.sql.calcite.schema.MetadataSegmentView - MetadataSegmentView Stopped. 2020-06-11T08:27:27,925 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore. 2020-06-11T08:27:27,937 INFO [Thread-43] org.apache.druid.guice.LifecycleForkJoinPoolProvider - Shutting down ForkJoinPool [org.apache.druid.guice.LifecycleForkJoinPoolProvider@35bd2eec] ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
