alborotogarcia opened a new issue #11992: URL: https://github.com/apache/druid/issues/11992
Unable to connect router to coordinator service. Templating the helm chart a bit (getting rid of bitnami's amd64 zookeeper, and postgres mainly) and deploying the official helm chart I can't connect the router to any other services apparently.  However the coordinator seems to be connected to other services,   And the druid doctor shows the following  Although all my pods seem to be running smoothly  ``` druid-zookeeper-headless ClusterIP None <none> 2181/TCP,3888/TCP,2888/TCP 3h4m druid-zookeeper ClusterIP 10.43.127.52 <none> 2181/TCP 3h4m druid-broker ClusterIP 10.43.35.55 <none> 8082/TCP 3h4m druid-coordinator ClusterIP 10.43.38.218 <none> 8081/TCP 3h4m druid-historical ClusterIP 10.43.173.83 <none> 8083/TCP 3h4m druid-middle-manager ClusterIP 10.43.146.5 <none> 8091/TCP 3h4m druid-router ClusterIP 10.43.135.124 <none> 8888/TCP 3h4m ``` ### Affected Version arm64 build 0.22.0 @98957be0443b669cc7464886ef9ee21d3d21f762 ### Description Please include as much detailed information about the problem as possible. - Cluster size: 3 nodes 4cpu each - Configurations in use: helm v3.7.1, k3s v1.22.2, longhorn v1.2.2 as storageclass - Steps to reproduce the problem: bumping zookeeper image to v3.5.8+ from the helm dependency, and setting an external psql instance for runtime metadata ``` List of relations Schema | Name | Type | Owner --------+-----------------------+-------+---------- public | druid_audit | table | druid public | druid_config | table | druid public | druid_datasource | table | druid public | druid_pendingsegments | table | druid public | druid_rules | table | druid public | druid_segments | table | druid public | druid_supervisors | table | druid public | druid_tasklocks | table | druid public | druid_tasklogs | table | druid public | druid_tasks | table | druid (10 rows) ``` - The error message or stack traces encountered. Providing more context, such as nearby log messages or even entire logs, can be helpful. - Any debugging that you have already done Router trace ``` druid-router-7f6bcfb79c-9wdh9 druid 2021-11-26T04:07:54,891 ERROR [qtp555864569-122] org.apache.druid.server.router.QueryHostFinder - No server found for serviceName[druid/broker]. Using backup druid-router-7f6bcfb79c-9wdh9 druid 2021-11-26T04:07:54,892 ERROR [qtp555864569-122] org.apache.druid.server.router.QueryHostFinder - No backup found for serviceName[druid/broker]. Using default[druid/broker] druid-router-7f6bcfb79c-9wdh9 druid 2021-11-26T04:07:54,893 ERROR [qtp555864569-124] org.apache.druid.server.router.QueryHostFinder - No server found for serviceName[druid/broker]. Using backup druid-router-7f6bcfb79c-9wdh9 druid 2021-11-26T04:07:54,893 ERROR [qtp555864569-124] org.apache.druid.server.router.QueryHostFinder - No backup found for serviceName[druid/broker]. Using default[druid/broker] druid-router-7f6bcfb79c-9wdh9 druid 2021-11-26T04:07:54,893 ERROR [qtp555864569-124] org.apache.druid.server.router.QueryHostFinder - Catastrophic failure! No servers found at all! Failing request!: {class=org.apache.druid.server.router.QueryHostFinder} druid-router-7f6bcfb79c-9wdh9 druid 2021-11-26T04:07:54,893 ERROR [qtp555864569-122] org.apache.druid.server.router.QueryHostFinder - Catastrophic failure! No servers found at all! Failing request!: {class=org.apache.druid.server.router.QueryHostFinder} druid-router-7f6bcfb79c-9wdh9 druid 2021-11-26T04:07:54,893 WARN [qtp555864569-124] org.eclipse.jetty.server.HttpChannel - /druid/v2/sql druid-router-7f6bcfb79c-9wdh9 druid org.apache.druid.java.util.common.ISE: No default server found! druid-router-7f6bcfb79c-9wdh9 druid at org.apache.druid.server.router.QueryHostFinder.assertServerFound(QueryHostFinder.java:158) ~[druid-services-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-router-7f6bcfb79c-9wdh9 druid at org.apache.druid.server.router.QueryHostFinder.pickDefaultServer(QueryHostFinder.java:109) ~[druid-services-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-router-7f6bcfb79c-9wdh9 druid at org.apache.druid.server.AsyncQueryForwardingServlet.service(AsyncQueryForwardingServlet.java:284) ~[druid-services-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-router-7f6bcfb79c-9wdh9 druid at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:82) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.apache.druid.server.security.AllowHttpMethodsResourceFilter.doFilter(AllowHttpMethodsResourceFilter.java:78) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:75) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.apache.druid.server.security.AllowAllAuthenticator$1.doFilter(AllowAllAuthenticator.java:84) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-router-7f6bcfb79c-9wdh9 druid at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413] druid-router-7f6bcfb79c-9wdh9 druid at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312] ``` Coordinator trace ``` druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:42,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Load Queues: druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:42,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Server[:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served. druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:47,842 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.LogUsedSegments - Found [0] used segments. druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:47,843 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.DruidCoordinator - Caught exception, ignoring so that schedule keeps going.: {class=org.apache.druid.server.coordinator.DruidCoordinator, exceptionType=class java.lang.RuntimeException, exceptionMessage=org.apache.druid.java.util.common.IOE: No known server} druid-coordinator-f4cd776f5-xj6g9 druid java.lang.RuntimeException: org.apache.druid.java.util.common.IOE: No known server druid-coordinator-f4cd776f5-xj6g9 druid at org.apache.druid.client.indexing.HttpIndexingServiceClient.getTasks(HttpIndexingServiceClient.java:266) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-coordinator-f4cd776f5-xj6g9 druid at org.apache.druid.client.indexing.HttpIndexingServiceClient.getActiveTasks(HttpIndexingServiceClient.java:231) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-coordinator-f4cd776f5-xj6g9 druid at org.apache.druid.server.coordinator.KillStalePendingSegments.run(KillStalePendingSegments.java:55) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-coordinator-f4cd776f5-xj6g9 druid at org.apache.druid.server.coordinator.DruidCoordinator$DutiesRunnable.run(DruidCoordinator.java:910) [druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-coordinator-f4cd776f5-xj6g9 druid at org.apache.druid.server.coordinator.DruidCoordinator$2.call(DruidCoordinator.java:720) [druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-coordinator-f4cd776f5-xj6g9 druid at org.apache.druid.server.coordinator.DruidCoordinator$2.call(DruidCoordinator.java:713) [druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-coordinator-f4cd776f5-xj6g9 druid at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$4.run(ScheduledExecutors.java:163) [druid-core-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-coordinator-f4cd776f5-xj6g9 druid at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312] druid-coordinator-f4cd776f5-xj6g9 druid at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_312] druid-coordinator-f4cd776f5-xj6g9 druid at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312] druid-coordinator-f4cd776f5-xj6g9 druid at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_312] druid-coordinator-f4cd776f5-xj6g9 druid at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312] druid-coordinator-f4cd776f5-xj6g9 druid at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312] druid-coordinator-f4cd776f5-xj6g9 druid at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312] druid-coordinator-f4cd776f5-xj6g9 druid Caused by: org.apache.druid.java.util.common.IOE: No known server druid-coordinator-f4cd776f5-xj6g9 druid at org.apache.druid.discovery.DruidLeaderClient.getCurrentKnownLeader(DruidLeaderClient.java:267) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-coordinator-f4cd776f5-xj6g9 druid at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:122) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-coordinator-f4cd776f5-xj6g9 druid at org.apache.druid.client.indexing.HttpIndexingServiceClient.getTasks(HttpIndexingServiceClient.java:251) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-coordinator-f4cd776f5-xj6g9 druid ... 13 more druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:47,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.LogUsedSegments - Found [0] used segments. druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:47,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty. druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:47,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - Metadata segments are not available. Cannot balance. druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:47,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Load Queues: druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:47,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Server[:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served. druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:52,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.LogUsedSegments - Found [0] used segments. druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:52,962 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty. druid-coordinator-f4cd776f5-xj6g9 druid 2021-11-26T04:24:52,963 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - Metadata segments are not available. Cannot balance. ``` Broker trace ``` druid-broker-579ffb6b97-jtc5b druid 2021-11-26T01:24:36,788 INFO [ServerInventoryView-0] org.apache.druid.client.BatchServerInventoryView - Server Disappeared[DruidServerMetadata{name=':8083', hostAndPort=':8083', hostAndTlsPort='null', maxSize=300000000000, tier='_default_tier', type=historical, priority=0}] druid-broker-579ffb6b97-jtc5b druid 2021-11-26T01:24:36,788 INFO [ServerInventoryView-0] org.apache.druid.client.BatchServerInventoryView - Server Disappeared[DruidServerMetadata{name=':8083', hostAndPort=':8083', hostAndTlsPort='null', maxSize=300000000000, tier='_default_tier', type=historical, priority=0}] druid-broker-579ffb6b97-jtc5b druid 2021-11-26T01:24:36,832 ERROR [NodeRoleWatcher[COORDINATOR]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher - Unknown error in node watcher of role[coordinator]. druid-broker-579ffb6b97-jtc5b druid java.lang.RuntimeException: java.net.URISyntaxException: Expected hostname at index 7: http://:8081 druid-broker-579ffb6b97-jtc5b druid at org.apache.druid.server.DruidNode.getUriToUse(DruidNode.java:292) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-broker-579ffb6b97-jtc5b druid at org.apache.druid.discovery.BaseNodeRoleWatcher.childAdded(BaseNodeRoleWatcher.java:131) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-broker-579ffb6b97-jtc5b druid at org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher.childAdded(CuratorDruidNodeDiscoveryProvider.java:271) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-broker-579ffb6b97-jtc5b druid at org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher.handleChildEvent(CuratorDruidNodeDiscoveryProvider.java:237) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-broker-579ffb6b97-jtc5b druid at org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher.lambda$new$0(CuratorDruidNodeDiscoveryProvider.java:205) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-broker-579ffb6b97-jtc5b druid at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:538) [curator-recipes-4.3.0.jar:4.3.0] druid-broker-579ffb6b97-jtc5b druid at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:532) [curator-recipes-4.3.0.jar:4.3.0] druid-broker-579ffb6b97-jtc5b druid at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100) [curator-framework-4.3.0.jar:4.3.0] druid-broker-579ffb6b97-jtc5b druid at org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) [curator-client-4.3.0.jar:?] druid-broker-579ffb6b97-jtc5b druid at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92) [curator-framework-4.3.0.jar:4.3.0] druid-broker-579ffb6b97-jtc5b druid at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:530) [curator-recipes-4.3.0.jar:4.3.0] druid-broker-579ffb6b97-jtc5b druid at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-4.3.0.jar:4.3.0] druid-broker-579ffb6b97-jtc5b druid at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:808) [curator-recipes-4.3.0.jar:4.3.0] druid-broker-579ffb6b97-jtc5b druid at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312] druid-broker-579ffb6b97-jtc5b druid at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_312] druid-broker-579ffb6b97-jtc5b druid at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312] druid-broker-579ffb6b97-jtc5b druid at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_312] druid-broker-579ffb6b97-jtc5b druid at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312] druid-broker-579ffb6b97-jtc5b druid at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312] druid-broker-579ffb6b97-jtc5b druid at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312] druid-broker-579ffb6b97-jtc5b druid Caused by: java.net.URISyntaxException: Expected hostname at index 7: http://:8081 druid-broker-579ffb6b97-jtc5b druid at java.net.URI$Parser.fail(URI.java:2847) ~[?:1.8.0_312] druid-broker-579ffb6b97-jtc5b druid at java.net.URI$Parser.failExpecting(URI.java:2853) ~[?:1.8.0_312] druid-broker-579ffb6b97-jtc5b druid at java.net.URI$Parser.parseHostname(URI.java:3389) ~[?:1.8.0_312] druid-broker-579ffb6b97-jtc5b druid at java.net.URI$Parser.parseServer(URI.java:3235) ~[?:1.8.0_312] druid-broker-579ffb6b97-jtc5b druid at java.net.URI$Parser.parseAuthority(URI.java:3154) ~[?:1.8.0_312] druid-broker-579ffb6b97-jtc5b druid at java.net.URI$Parser.parseHierarchical(URI.java:3096) ~[?:1.8.0_312] druid-broker-579ffb6b97-jtc5b druid at java.net.URI$Parser.parse(URI.java:3052) ~[?:1.8.0_312] druid-broker-579ffb6b97-jtc5b druid at java.net.URI.<init>(URI.java:673) ~[?:1.8.0_312] druid-broker-579ffb6b97-jtc5b druid at org.apache.druid.server.DruidNode.getUriToUse(DruidNode.java:289) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-broker-579ffb6b97-jtc5b druid ... 19 more druid-broker-579ffb6b97-jtc5b druid 2021-11-26T01:24:36,838 INFO [ServerInventoryView-0] org.apache.druid.client.BatchServerInventoryView - New Server[DruidServerMetadata{name=':8083', hostAndPort=':8083', hostAndTlsPort='null', maxSize=300000000000, tier='_default_tier', type=historical, priority=0}] druid-broker-579ffb6b97-jtc5b druid 2021-11-26T01:24:36,855 INFO [ServerInventoryView-0] org.apache.druid.client.BatchServerInventoryView - New Server[DruidServerMetadata{name=':8083', hostAndPort=':8083', hostAndTlsPort='null', maxSize=300000000000, tier='_default_tier', type=historical, priority=0}] ``` Historical ``` druid-historical-0 druid 2021-11-26T01:24:36,660 INFO [Announcer-0] org.apache.druid.curator.announcement.Announcer - Node[/druid/announcements/:8083] dropped, reinstating. druid-historical-0 druid 2021-11-26T01:24:36,799 ERROR [NodeRoleWatcher[COORDINATOR]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher - Unknown error in node watcher of role[coordinator]. druid-historical-0 druid java.lang.RuntimeException: java.net.URISyntaxException: Expected hostname at index 7: http://:8081 druid-historical-0 druid at org.apache.druid.server.DruidNode.getUriToUse(DruidNode.java:292) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-historical-0 druid at org.apache.druid.discovery.BaseNodeRoleWatcher.childAdded(BaseNodeRoleWatcher.java:131) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-historical-0 druid at org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher.childAdded(CuratorDruidNodeDiscoveryProvider.java:271) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-historical-0 druid at org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher.handleChildEvent(CuratorDruidNodeDiscoveryProvider.java:237) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-historical-0 druid at org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher.lambda$new$0(CuratorDruidNodeDiscoveryProvider.java:205) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-historical-0 druid at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:538) [curator-recipes-4.3.0.jar:4.3.0] druid-historical-0 druid at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:532) [curator-recipes-4.3.0.jar:4.3.0] druid-historical-0 druid at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100) [curator-framework-4.3.0.jar:4.3.0] druid-historical-0 druid at org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) [curator-client-4.3.0.jar:?] druid-historical-0 druid at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92) [curator-framework-4.3.0.jar:4.3.0] druid-historical-0 druid at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:530) [curator-recipes-4.3.0.jar:4.3.0] druid-historical-0 druid at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-4.3.0.jar:4.3.0] druid-historical-0 druid at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:808) [curator-recipes-4.3.0.jar:4.3.0] druid-historical-0 druid at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312] druid-historical-0 druid at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_312] druid-historical-0 druid at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312] druid-historical-0 druid at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_312] druid-historical-0 druid at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312] druid-historical-0 druid at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312] druid-historical-0 druid at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312] druid-historical-0 druid Caused by: java.net.URISyntaxException: Expected hostname at index 7: http://:8081 druid-historical-0 druid at java.net.URI$Parser.fail(URI.java:2847) ~[?:1.8.0_312] druid-historical-0 druid at java.net.URI$Parser.failExpecting(URI.java:2853) ~[?:1.8.0_312] druid-historical-0 druid at java.net.URI$Parser.parseHostname(URI.java:3389) ~[?:1.8.0_312] druid-historical-0 druid at java.net.URI$Parser.parseServer(URI.java:3235) ~[?:1.8.0_312] druid-historical-0 druid at java.net.URI$Parser.parseAuthority(URI.java:3154) ~[?:1.8.0_312] druid-historical-0 druid at java.net.URI$Parser.parseHierarchical(URI.java:3096) ~[?:1.8.0_312] druid-historical-0 druid at java.net.URI$Parser.parse(URI.java:3052) ~[?:1.8.0_312] druid-historical-0 druid at java.net.URI.<init>(URI.java:673) ~[?:1.8.0_312] druid-historical-0 druid at org.apache.druid.server.DruidNode.getUriToUse(DruidNode.java:289) ~[druid-server-0.23.0-SNAPSHOT.jar:0.23.0-SNAPSHOT] druid-historical-0 druid ... 19 more ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
