xsf1005 opened a new issue, #19440: URL: https://github.com/apache/druid/issues/19440
broker keeps reporting errors when loading and refreshing segments. After calling the indexer and receiving an error, for some reason, the indexer fails to query the datasource. This causes the broker's segment cache to refresh and retry indefinitely, resulting in abnormal metadata cache. I believe this is an issue of robustness that needs to be addressed. Historical issue : https://github.com/apache/druid/issues/16936 When `indexer.getBundle()` is null, an IAE (IllegalArgumentException) is thrown. After the broker captures this exception, it will be added back to the refresh queue, leading to repeated retries and ultimately causing an infinite loop. @VisibleForTesting <T> DatasourceBundle getBundle(final Query<T> query) { final DataSourceAnalysis analysis = query.getDataSource().getAnalysis(); final TableDataSource table = analysis.getBaseTableDataSource() .orElseThrow(() -> new ISE("Cannot handle datasource: %s", query.getDataSource())); final DatasourceBundle bundle; synchronized (this) { bundle = datasourceBundles.get(table.getName()); } if (bundle == null) { throw new IAE("Could not find segment walker for datasource [%s]", table.getName()); } return bundle; } The following code will result in an infinite loop. refresh(segmentsToRefresh, dataSourcesToRebuild); initialized.countDown(); } catch (InterruptedException e) { // Fall through. throw e; } catch (Exception e) { log.warn(e, "Metadata refresh failed, trying again soon."); synchronized (lock) { // Add our segments and dataSources back to their refresh and rebuild lists. segmentsNeedingRefresh.addAll(segmentsToRefresh); dataSourcesNeedingRebuild.addAll(dataSourcesToRebuild); lastFailure = System.currentTimeMillis(); } } The error stack is as follows: broker.log [DruidSchema-Cache-0][ROOT][org.apache.druid.client.JsonParserIterator] Query [12907933-7fdd-4c09-889c-f0c774a0ec91] to host [xx.xx.xx.xx:26204] interrupted org.apache.druid.query.QueryException: Could not find segment walker for datasource [tbl_data_res_a] at sun.reflect.GeneratedConstructorAccessor411.newInstance(Unknown Source) ~[?:?] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_452] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_452] at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:124) ~[jackson-databind-2.12.7.jar:2.12.7] at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:291) ~[jackson-databind-2.12.7.jar:2.12.7] at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith(ValueInstantiator.java:288) ~[jackson-databind-2.12.7.jar:2.12.7] at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:202) ~[jackson-databind-2.12.7.jar:2.12.7] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:454) ~[jackson-databind-2.12.7.jar:2.12.7] at com.fasterxml.jackson.databind.deser.std.ThrowableDeserializer.deserializeFromObject(ThrowableDeserializer.java:65) ~[jackson-databind-2.12.7.jar:2.12.7] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:195) ~[jackson-databind-2.12.7.jar:2.12.7] at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322) ~[jackson-databind-2.12.7.jar:2.12.7] at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4569) ~[jackson-databind-2.12.7.jar:2.12.7] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2798) ~[jackson-databind-2.12.7.jar:2.12.7] at org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:186) ~[druid-server-28.0.1-htrunk50.jar:?] at org.apache.druid.client.JsonParserIterator.hasNext(JsonParserIterator.java:92) ~[druid-server-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:70) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.MergeSequence.lambda$toYielder$0(MergeSequence.java:66) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.MergeSequence.toYielder(MergeSequence.java:63) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.LazySequence.toYielder(LazySequence.java:46) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.MergeSequence.lambda$toYielder$0(MergeSequence.java:66) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.MergeSequence.toYielder(MergeSequence.java:63) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.query.RetryQueryRunner$1.toYielder(RetryQueryRunner.java:133) ~[druid-server-28.0.1-htrunk50.jar:?] at org.apache.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:78) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:77) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.Yielders.each(Yielders.java:32) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.sql.calcite.schema.SegmentMetadataCache.refreshSegmentsForDataSource(SegmentMetadataCache.java:796) ~[druid-sql-28.0.1-htrunk50.jar:?] at org.apache.druid.sql.calcite.schema.SegmentMetadataCache.refreshSegments(SegmentMetadataCache.java:744) ~[druid-sql-28.0.1-htrunk50.jar:?] at org.apache.druid.sql.calcite.schema.SegmentMetadataCache.refresh(SegmentMetadataCache.java:467) ~[druid-sql-28.0.1-htrunk50.jar:?] at org.apache.druid.sql.calcite.schema.SegmentMetadataCache.lambda$startCacheExec$1(SegmentMetadataCache.java:409) ~[druid-sql-28.0.1-htrunk50.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_452] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_452] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_452] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_452] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_452] 2026-05-06 07:02:37,161 WARN [DruidSchema-Cache-0][ROOT][org.apache.druid.server.QueryLifecycle] Exception while processing queryId [12907933-7fdd-4c09-889c-f0c774a0ec91] org.apache.druid.query.QueryInterruptedException: Could not find segment walker for datasource [tbl_data_res_a] at org.apache.druid.client.JsonParserIterator.convertException(JsonParserIterator.java:276) ~[druid-server-28.0.1-htrunk50.jar:?] at org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:186) ~[druid-server-28.0.1-htrunk50.jar:?] at org.apache.druid.client.JsonParserIterator.hasNext(JsonParserIterator.java:92) ~[druid-server-28.0.1-htrunk50.jar:?] at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:70) ~[druid-processing-28.0.1-htrunk50.jar:?] indexer.log at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.servlets.QoSFilter.doFilter(QoSFilter.java:202) ~[jetty-servlets-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555) ~[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) ~[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) ~[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_452] Caused by: org.apache.druid.query.QueryInterruptedException: Could not find segment walker for datasource [tbl_data_res_a] ... 86 more Caused by: org.apache.druid.java.util.common.IAE: Could not find segment walker for datasource [tbl_data_res_a] at org.apache.druid.segment.realtime.appenderator.UnifiedIndexerAppenderatorsManager.getBundle(UnifiedIndexerAppenderatorsManager.java:376) ~[druid-server-28.0.1-htrunk50.jar:?] at org.apache.druid.segment.realtime.appenderator.UnifiedIndexerAppenderatorsManager.getQueryRunnerForSegments(UnifiedIndexerAppenderatorsManager.java:357) ~[druid-server-28.0.1-htrunk50.jar:?] at org.apache.druid.indexing.overlord.ThreadingTaskRunner.getQueryRunnerForSegments(ThreadingTaskRunner.java:550) ~[druid-indexing-service-28.0.1-htrunk50.jar:?] at org.apache.druid.query.spec.MultipleSpecificSegmentSpec.lookup(MultipleSpecificSegmentSpec.java:77) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.query.BaseQuery.getRunner(BaseQuery.java:119) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.query.QueryPlus.run(QueryPlus.java:149) ~[druid-processing-28.0.1-htrunk50.jar:?] at org.apache.druid.server.QueryLifecycle.execute(QueryLifecycle.java:285) ~[druid-server-28.0.1-htrunk50.jar:?] at org.apache.druid.server.QueryResource$QueryResourceQueryResultPusher$1.start(QueryResource.java:512) ~[druid-server-28.0.1-htrunk50.jar:?] at org.apache.druid.server.QueryResultPusher.push(QueryResultPusher.java:121) ~[druid-server-28.0.1-htrunk50.jar:?] ... 85 more -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
