gocho1 commented on issue #8534: Ingest task succeeded then failed but stays running forever URL: https://github.com/apache/incubator-druid/issues/8534#issuecomment-534893627 Once again, it happens and prevent our ingestion tasks to run since 10h... Here is the stack trace after the success is indicated ``` 2019-09-24T23:15:39,187 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: { "id" : "index_hadoop_enriched_sale_2019-09-24T23:14:44.683Z", "status" : "SUCCESS", "duration" : 49372, "errorMsg" : null } 2019-09-24T23:15:39,194 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS] 2019-09-24T23:15:39,195 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.curator.announcement.Announcer.stop()] on object[org.apache.druid.curator.announcement.Announcer@43b172e3]. 2019-09-24T23:15:39,196 INFO [main] org.apache.druid.curator.announcement.Announcer - Stopping announcer 2019-09-24T23:15:39,196 INFO [main] org.apache.druid.curator.announcement.Announcer - unannouncing [/druid_iods/listeners/lookups/__default/http:ip-xxx:8100] 2019-09-24T23:15:39,207 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER] 2019-09-24T23:15:39,207 INFO [main] org.apache.druid.server.initialization.jetty.JettyServerModule - Stopping Jetty Server... 2019-09-24T23:15:39,209 INFO [main] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@2fdf7da2{HTTP/1.1,[http/1.1]}{0.0.0.0:8100} 2019-09-24T23:15:39,210 INFO [main] org.eclipse.jetty.server.session - node0 Stopped scavenging 2019-09-24T23:15:39,211 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@1dd44289{/,null,UNAVAILABLE} 2019-09-24T23:15:39,220 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL] 2019-09-24T23:15:39,220 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.server.listener.announcer.ListenerResourceAnnouncer.stop()] on object[org.apache.druid.query.lookup.LookupResourceListenerAnnouncer@75dd0f94]. 2019-09-24T23:15:39,221 INFO [main] org.apache.druid.server.listener.announcer.ListenerResourceAnnouncer - Unannouncing start time on [/druid_iods/listeners/lookups/__default/http:ip-xxx:8100] 2019-09-24T23:15:39,221 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.indexing.worker.executor.ExecutorLifecycle.stop() throws java.lang.Exception] on object[org.apache.druid.indexing.worker.executor.ExecutorLifecycle@41143873]. 2019-09-24T23:15:39,221 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner.stop()] on object[org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner@4b508371]. 2019-09-24T23:15:39,221 INFO [main] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting graceful shutdown of task[index_hadoop_enriched_sale_2019-09-24T23:14:44.683Z]. 2019-09-24T23:15:39,221 INFO [main] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_enriched_sale_2019-09-24T23:14:44.683Z] status changed to [FAILED]. 2019-09-24T23:15:39,231 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.client.cache.CaffeineCache.close()] on object[org.apache.druid.client.cache.CaffeineCache@356f20b7]. 2019-09-24T23:15:39,232 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.discovery.DruidLeaderClient.stop()] on object[org.apache.druid.discovery.DruidLeaderClient@592a1882]. 2019-09-24T23:15:39,232 INFO [main] org.apache.druid.discovery.DruidLeaderClient - Stopped. 2019-09-24T23:15:39,232 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[org.apache.druid.curator.discovery.ServerDiscoverySelector@30afb481]. 2019-09-24T23:15:39,233 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.stop()] on object[org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager@e881e46]. 2019-09-24T23:15:39,233 INFO [main] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - DefaultBasicAuthenticatorCacheManager is stopping. 2019-09-24T23:15:39,233 INFO [main] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - DefaultBasicAuthenticatorCacheManager is stopped. 2019-09-24T23:15:39,233 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.java.util.metrics.MonitorScheduler.stop()] on object[org.apache.druid.java.util.metrics.MonitorScheduler@55044b34]. 2019-09-24T23:15:39,233 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.stop()] on object[org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager@37cfda1c]. 2019-09-24T23:15:39,233 INFO [main] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopping. 2019-09-24T23:15:39,233 INFO [main] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopped. 2019-09-24T23:15:39,233 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.query.lookup.LookupReferencesManager.stop()] on object[org.apache.druid.query.lookup.LookupReferencesManager@1b475663]. 2019-09-24T23:15:39,233 INFO [main] org.apache.druid.query.lookup.LookupReferencesManager - LookupExtractorFactoryContainerProvider is stopping. 2019-09-24T23:15:39,233 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited, Lookup notices are not handled anymore. 2019-09-24T23:15:39,234 INFO [main] org.apache.druid.query.lookup.LookupReferencesManager - LookupExtractorFactoryContainerProvider is stopped. 2019-09-24T23:15:39,234 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.discovery.DruidLeaderClient.stop()] on object[org.apache.druid.discovery.DruidLeaderClient@7c3c453b]. 2019-09-24T23:15:39,234 INFO [main] org.apache.druid.discovery.DruidLeaderClient - Stopped. 2019-09-24T23:15:39,234 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[org.apache.druid.curator.discovery.ServerDiscoverySelector@44864536]. 2019-09-24T23:15:39,234 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider.stop()] on object[org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider@1ccdc7aa]. 2019-09-24T23:15:39,234 INFO [main] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider - stopping 2019-09-24T23:15:39,234 INFO [main] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider - stopped 2019-09-24T23:15:39,234 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.java.util.http.client.NettyHttpClient.stop()] on object[org.apache.druid.java.util.http.client.NettyHttpClient@63187d63]. 2019-09-24T23:15:39,262 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void org.apache.druid.java.util.emitter.service.ServiceEmitter.close() throws java.io.IOException] on object[ServiceEmitter{serviceDimensions={service=druid/middleManager, host=ip-xxx, version=0.15.1-incubating}, emitter=HttpPostEmitter{config=HttpEmitterConfig{flushMillis=10000, flushCount=500, flushTimeOut=9223372036854775807, basicAuthentication='null', batchingStrategy=ARRAY, maxBatchSize=5191680, contentEncoding=null, batchQueueSizeLimit=9, httpTimeoutAllowanceFactor=2.0, minHttpTimeoutMillis=0, recipientBaseUrl='https://fluentd.data-common.keringapps.io/druid_iods_prod'}}}]. 2019-09-24T23:15:50,534 WARN [BasicAuthenticatorCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 9) in 892ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:250) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:191) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:189) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$start$0(CoordinatorPollingBasicAuthenticatorCacheManager.java:121) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:15:51,432 WARN [BasicAuthenticatorCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (2 of 9) in 1,430ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:250) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:191) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:189) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$start$0(CoordinatorPollingBasicAuthenticatorCacheManager.java:121) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:15:52,045 WARN [CoordinatorPollingBasicAuthorizerCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 9) in 840ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.tryFetchMapsFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:259) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$fetchUserAndRoleMapFromCoordinator$2(CoordinatorPollingBasicAuthorizerCacheManager.java:230) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.fetchUserAndRoleMapFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:228) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$start$0(CoordinatorPollingBasicAuthorizerCacheManager.java:123) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:15:52,865 WARN [BasicAuthenticatorCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (3 of 9) in 4,472ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:250) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:191) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:189) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$start$0(CoordinatorPollingBasicAuthenticatorCacheManager.java:121) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:15:52,888 WARN [CoordinatorPollingBasicAuthorizerCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (2 of 9) in 1,895ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.tryFetchMapsFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:259) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$fetchUserAndRoleMapFromCoordinator$2(CoordinatorPollingBasicAuthorizerCacheManager.java:230) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.fetchUserAndRoleMapFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:228) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$start$0(CoordinatorPollingBasicAuthorizerCacheManager.java:123) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:15:54,787 WARN [CoordinatorPollingBasicAuthorizerCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (3 of 9) in 4,923ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.tryFetchMapsFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:259) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$fetchUserAndRoleMapFromCoordinator$2(CoordinatorPollingBasicAuthorizerCacheManager.java:230) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.fetchUserAndRoleMapFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:228) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$start$0(CoordinatorPollingBasicAuthorizerCacheManager.java:123) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:15:57,340 WARN [BasicAuthenticatorCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (4 of 9) in 7,617ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:250) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:191) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:189) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$start$0(CoordinatorPollingBasicAuthenticatorCacheManager.java:121) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:15:59,713 WARN [CoordinatorPollingBasicAuthorizerCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (4 of 9) in 5,414ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.tryFetchMapsFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:259) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$fetchUserAndRoleMapFromCoordinator$2(CoordinatorPollingBasicAuthorizerCacheManager.java:230) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.fetchUserAndRoleMapFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:228) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$start$0(CoordinatorPollingBasicAuthorizerCacheManager.java:123) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:16:04,960 WARN [BasicAuthenticatorCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (5 of 9) in 15,805ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:250) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:191) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:189) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$start$0(CoordinatorPollingBasicAuthenticatorCacheManager.java:121) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:16:05,130 WARN [CoordinatorPollingBasicAuthorizerCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (5 of 9) in 18,282ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.tryFetchMapsFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:259) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$fetchUserAndRoleMapFromCoordinator$2(CoordinatorPollingBasicAuthorizerCacheManager.java:230) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.fetchUserAndRoleMapFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:228) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$start$0(CoordinatorPollingBasicAuthorizerCacheManager.java:123) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:16:20,767 WARN [BasicAuthenticatorCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (6 of 9) in 17,549ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:250) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:191) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:189) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$start$0(CoordinatorPollingBasicAuthenticatorCacheManager.java:121) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:16:23,415 WARN [CoordinatorPollingBasicAuthorizerCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (6 of 9) in 33,370ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.tryFetchMapsFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:259) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$fetchUserAndRoleMapFromCoordinator$2(CoordinatorPollingBasicAuthorizerCacheManager.java:230) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.fetchUserAndRoleMapFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:228) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$start$0(CoordinatorPollingBasicAuthorizerCacheManager.java:123) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:16:38,319 WARN [BasicAuthenticatorCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (7 of 9) in 55,039ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:250) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:191) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:189) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$start$0(CoordinatorPollingBasicAuthenticatorCacheManager.java:121) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:16:56,787 WARN [CoordinatorPollingBasicAuthorizerCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (7 of 9) in 48,791ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.tryFetchMapsFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:259) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$fetchUserAndRoleMapFromCoordinator$2(CoordinatorPollingBasicAuthorizerCacheManager.java:230) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.fetchUserAndRoleMapFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:228) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$start$0(CoordinatorPollingBasicAuthorizerCacheManager.java:123) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:17:33,361 WARN [BasicAuthenticatorCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (8 of 9) in 64,623ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:250) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:191) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:189) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$start$0(CoordinatorPollingBasicAuthenticatorCacheManager.java:121) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:17:45,582 WARN [CoordinatorPollingBasicAuthorizerCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (8 of 9) in 67,184ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.tryFetchMapsFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:259) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$fetchUserAndRoleMapFromCoordinator$2(CoordinatorPollingBasicAuthorizerCacheManager.java:230) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.fetchUserAndRoleMapFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:228) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$start$0(CoordinatorPollingBasicAuthorizerCacheManager.java:123) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:18:37,987 WARN [BasicAuthenticatorCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (9 of 9) in 61,129ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:250) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:191) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:189) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$start$0(CoordinatorPollingBasicAuthenticatorCacheManager.java:121) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:18:52,769 WARN [CoordinatorPollingBasicAuthorizerCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (9 of 9) in 65,702ms. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.tryFetchMapsFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:259) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$fetchUserAndRoleMapFromCoordinator$2(CoordinatorPollingBasicAuthorizerCacheManager.java:230) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.fetchUserAndRoleMapFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:228) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$start$0(CoordinatorPollingBasicAuthorizerCacheManager.java:123) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:19:39,122 ERROR [BasicAuthenticatorCacheManager-Exec--0] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - Encountered exception while fetching user map for authenticator [iods]: {class=org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager, exceptionType=class java.lang.IllegalStateException, exceptionMessage=null} java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:250) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:191) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:189) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$start$0(CoordinatorPollingBasicAuthenticatorCacheManager.java:121) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:19:39,125 ERROR [BasicAuthenticatorCacheManager-Exec--0] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - Error occured while polling for cachedUserMaps.: {class=org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager, exceptionType=class java.util.concurrent.RejectedExecutionException, exceptionMessage=Service is closed.} java.util.concurrent.RejectedExecutionException: Service is closed. at org.apache.druid.java.util.emitter.core.HttpPostEmitter.awaitStarted(HttpPostEmitter.java:208) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.core.HttpPostEmitter.emitAndReturnBatch(HttpPostEmitter.java:233) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.core.HttpPostEmitter.emit(HttpPostEmitter.java:226) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.service.ServiceEmitter.emit(ServiceEmitter.java:67) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.service.ServiceEmitter.emit(ServiceEmitter.java:72) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.service.AlertBuilder.emit(AlertBuilder.java:88) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.EmittingLogger$EmittingAlertBuilder.emit(EmittingLogger.java:109) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:198) ~[?:?] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$start$0(CoordinatorPollingBasicAuthenticatorCacheManager.java:121) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:19:39,126 ERROR [BasicAuthenticatorCacheManager-Exec--0] org.apache.druid.java.util.common.concurrent.ScheduledExecutors - Uncaught exception. java.util.concurrent.RejectedExecutionException: Service is closed. at org.apache.druid.java.util.emitter.core.HttpPostEmitter.awaitStarted(HttpPostEmitter.java:208) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.core.HttpPostEmitter.emitAndReturnBatch(HttpPostEmitter.java:233) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.core.HttpPostEmitter.emit(HttpPostEmitter.java:226) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.service.ServiceEmitter.emit(ServiceEmitter.java:67) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.service.ServiceEmitter.emit(ServiceEmitter.java:72) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.service.AlertBuilder.emit(AlertBuilder.java:88) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.EmittingLogger$EmittingAlertBuilder.emit(EmittingLogger.java:109) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$start$0(CoordinatorPollingBasicAuthenticatorCacheManager.java:129) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:19:58,474 ERROR [CoordinatorPollingBasicAuthorizerCacheManager-Exec--0] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - Encountered exception while fetching user and role map for authorizer [iods]: {class=org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager, exceptionType=class java.lang.IllegalStateException, exceptionMessage=null} java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:130) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:139) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.tryFetchMapsFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:259) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$fetchUserAndRoleMapFromCoordinator$2(CoordinatorPollingBasicAuthorizerCacheManager.java:230) ~[?:?] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.fetchUserAndRoleMapFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:228) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$start$0(CoordinatorPollingBasicAuthorizerCacheManager.java:123) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:19:58,477 ERROR [CoordinatorPollingBasicAuthorizerCacheManager-Exec--0] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - Error occured while polling for cachedUserMaps.: {class=org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager, exceptionType=class java.util.concurrent.RejectedExecutionException, exceptionMessage=Service is closed.} java.util.concurrent.RejectedExecutionException: Service is closed. at org.apache.druid.java.util.emitter.core.HttpPostEmitter.awaitStarted(HttpPostEmitter.java:208) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.core.HttpPostEmitter.emitAndReturnBatch(HttpPostEmitter.java:233) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.core.HttpPostEmitter.emit(HttpPostEmitter.java:226) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.service.ServiceEmitter.emit(ServiceEmitter.java:67) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.service.ServiceEmitter.emit(ServiceEmitter.java:72) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.service.AlertBuilder.emit(AlertBuilder.java:88) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.EmittingLogger$EmittingAlertBuilder.emit(EmittingLogger.java:109) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.fetchUserAndRoleMapFromCoordinator(CoordinatorPollingBasicAuthorizerCacheManager.java:237) ~[?:?] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$start$0(CoordinatorPollingBasicAuthorizerCacheManager.java:123) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 2019-09-24T23:19:58,478 ERROR [CoordinatorPollingBasicAuthorizerCacheManager-Exec--0] org.apache.druid.java.util.common.concurrent.ScheduledExecutors - Uncaught exception. java.util.concurrent.RejectedExecutionException: Service is closed. at org.apache.druid.java.util.emitter.core.HttpPostEmitter.awaitStarted(HttpPostEmitter.java:208) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.core.HttpPostEmitter.emitAndReturnBatch(HttpPostEmitter.java:233) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.core.HttpPostEmitter.emit(HttpPostEmitter.java:226) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.service.ServiceEmitter.emit(ServiceEmitter.java:67) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.service.ServiceEmitter.emit(ServiceEmitter.java:72) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.service.AlertBuilder.emit(AlertBuilder.java:88) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.emitter.EmittingLogger$EmittingAlertBuilder.emit(EmittingLogger.java:109) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager.lambda$start$0(CoordinatorPollingBasicAuthorizerCacheManager.java:132) ~[?:?] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) ~[druid-core-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:92) [druid-core-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
