TessaIO opened a new issue, #18040:
URL: https://github.com/apache/druid/issues/18040
Please provide a detailed title (e.g. "Broker crashes when using TopN query
with Bound filter" instead of just "Broker crashes").
### Affected Version
32.0.0
### Description
We have a cluster of 20 historicals with 4 brokers. There are some queries
when being executed in a short duration twice they just error out with the
following issue:
```
2025-05-27T14:22:08,950 WARN
[qtp92221031-168[groupBy_[prod-campaign-in-app-roas]_c532831b-3598-4fa6-a384-aff92a767770]]
org.apache.druid.query.groupby.GroupByQueryResources - 1 merging query runner
merge buffers are not returned yet
2025-05-27T14:22:08,952 WARN
[qtp92221031-168[groupBy_[prod-campaign-in-app-roas]_c532831b-3598-4fa6-a384-aff92a767770]]
org.apache.druid.server.QueryLifecycle - Exception while processing queryId
[c532831b-3598-4fa6-a384-aff92a767770]
DruidException context: {host=null,
errorClass=java.lang.ArrayIndexOutOfBoundsException, legacyErrorCode=Unknown
exception}
org.apache.druid.error.DruidException: Index 180 out of bounds for length 129
at
org.apache.druid.error.QueryExceptionCompat.makeException(QueryExceptionCompat.java:50)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.error.DruidException.fromFailure(DruidException.java:157)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.server.QueryResultPusher.handleQueryException(QueryResultPusher.java:229)
~[druid-server-32.0.0.jar:32.0.0]
at
org.apache.druid.server.QueryResultPusher.push(QueryResultPusher.java:195)
~[druid-server-32.0.0.jar:32.0.0]
at org.apache.druid.server.QueryResource.doPost(QueryResource.java:222)
~[druid-server-32.0.0.jar:32.0.0]
at jdk.internal.reflect.GeneratedMethodAccessor80.invoke(Unknown
Source) ~[?:?]
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
~[jersey-servlet-1.19.4.jar:1.19.4]
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
~[jersey-servlet-1.19.4.jar:1.19.4]
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
~[jersey-servlet-1.19.4.jar:1.19.4]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
~[javax.servlet-api-3.1.0.jar:3.1.0]
at
com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
~[guice-servlet-5.1.0.jar:?]
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
~[guice-servlet-5.1.0.jar:?]
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
~[guice-servlet-5.1.0.jar:?]
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
~[guice-servlet-5.1.0.jar:?]
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
~[guice-servlet-5.1.0.jar:?]
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
~[guice-servlet-5.1.0.jar:?]
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
~[guice-servlet-5.1.0.jar:?]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:84)
~[druid-server-32.0.0.jar:32.0.0]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.apache.druid.server.initialization.jetty.StandardResponseHeaderFilterHolder$StandardResponseHeaderFilter.doFilter(StandardResponseHeaderFilterHolder.java:164)
~[druid-server-32.0.0.jar:32.0.0]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.apache.druid.server.security.AllowHttpMethodsResourceFilter.doFilter(AllowHttpMethodsResourceFilter.java:78)
~[druid-server-32.0.0.jar:32.0.0]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:74)
~[druid-server-32.0.0.jar:32.0.0]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.apache.druid.security.basic.authentication.BasicHTTPAuthenticator$BasicHTTPAuthenticationFilter.doFilter(BasicHTTPAuthenticator.java:220)
~[?:?]
at
org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59)
~[druid-server-32.0.0.jar:32.0.0]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77)
~[druid-server-32.0.0.jar:32.0.0]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.Server.handle(Server.java:516)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: org.apache.druid.query.QueryInterruptedException: Index 180 out
of bounds for length 129
... 79 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 180 out of bounds
for length 129
at
it.unimi.dsi.fastutil.ints.Int2ObjectOpenHashMap.get(Int2ObjectOpenHashMap.java:284)
~[fastutil-core-8.5.4.jar:?]
at
org.apache.druid.query.aggregation.datasketches.theta.SketchBufferAggregatorHelper.get(SketchBufferAggregatorHelper.java:66)
~[?:?]
at
org.apache.druid.query.aggregation.datasketches.theta.SketchBufferAggregator.get(SketchBufferAggregator.java:64)
~[?:?]
at
org.apache.druid.query.groupby.epinephelinae.StreamingMergeSortedGrouper$1.next(StreamingMergeSortedGrouper.java:438)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.groupby.epinephelinae.StreamingMergeSortedGrouper$1.next(StreamingMergeSortedGrouper.java:387)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.CloseableIterators$1.next(CloseableIterators.java:78)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.groupby.epinephelinae.CloseableGrouperIterator.next(CloseableGrouperIterator.java:53)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.common.guava.CombiningSequence.accumulate(CombiningSequence.java:62)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:77)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.server.QueryResultPusher.push(QueryResultPusher.java:166)
~[druid-server-32.0.0.jar:32.0.0]
... 78 more
Suppressed: java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
Index 0 out of bounds for length 0
at
org.apache.druid.query.groupby.epinephelinae.ParallelCombiner.checkCombineFutures(ParallelCombiner.java:204)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.groupby.epinephelinae.ParallelCombiner.lambda$combine$0(ParallelCombiner.java:174)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.io.Closer.close(Closer.java:170)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.CloseableIterators$1.close(CloseableIterators.java:86)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.io.Closer.close(Closer.java:170)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.groupby.epinephelinae.CloseableGrouperIterator.close(CloseableGrouperIterator.java:72)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunner$1.cleanup(GroupByMergingQueryRunner.java:319)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunner$1.cleanup(GroupByMergingQueryRunner.java:183)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:49)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.common.guava.CombiningSequence.accumulate(CombiningSequence.java:62)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:77)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.server.QueryResultPusher.push(QueryResultPusher.java:166)
~[druid-server-32.0.0.jar:32.0.0]
at
org.apache.druid.server.QueryResource.doPost(QueryResource.java:222)
~[druid-server-32.0.0.jar:32.0.0]
at
jdk.internal.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) ~[?:?]
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
~[?:?]
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
~[jersey-server-1.19.4.jar:1.19.4]
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
~[jersey-servlet-1.19.4.jar:1.19.4]
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
~[jersey-servlet-1.19.4.jar:1.19.4]
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
~[jersey-servlet-1.19.4.jar:1.19.4]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
~[javax.servlet-api-3.1.0.jar:3.1.0]
at
com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
~[guice-servlet-5.1.0.jar:?]
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
~[guice-servlet-5.1.0.jar:?]
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
~[guice-servlet-5.1.0.jar:?]
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
~[guice-servlet-5.1.0.jar:?]
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
~[guice-servlet-5.1.0.jar:?]
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
~[guice-servlet-5.1.0.jar:?]
at
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
~[guice-servlet-5.1.0.jar:?]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:84)
~[druid-server-32.0.0.jar:32.0.0]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.apache.druid.server.initialization.jetty.StandardResponseHeaderFilterHolder$StandardResponseHeaderFilter.doFilter(StandardResponseHeaderFilterHolder.java:164)
~[druid-server-32.0.0.jar:32.0.0]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.apache.druid.server.security.AllowHttpMethodsResourceFilter.doFilter(AllowHttpMethodsResourceFilter.java:78)
~[druid-server-32.0.0.jar:32.0.0]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:74)
~[druid-server-32.0.0.jar:32.0.0]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.apache.druid.security.basic.authentication.BasicHTTPAuthenticator$BasicHTTPAuthenticationFilter.doFilter(BasicHTTPAuthenticator.java:220)
~[?:?]
at
org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59)
~[druid-server-32.0.0.jar:32.0.0]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77)
~[druid-server-32.0.0.jar:32.0.0]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at org.eclipse.jetty.server.Server.handle(Server.java:516)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
[jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
[jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: java.util.concurrent.ExecutionException:
java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at
java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
at
java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
at
org.apache.druid.query.PrioritizedListenableFutureTask.get(PrioritizedExecutorService.java:283)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.groupby.epinephelinae.ParallelCombiner.checkCombineFutures(ParallelCombiner.java:197)
~[druid-processing-32.0.0.jar:32.0.0]
... 95 more
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds
for length 0
at
java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
~[?:?]
at
java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
~[?:?]
at
java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)
~[?:?]
at java.base/java.util.Objects.checkIndex(Objects.java:361)
~[?:?]
at java.base/java.util.ArrayList.get(ArrayList.java:427) ~[?:?]
at
org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper$RowBasedKeySerde$AbstractStringRowBasedKeySerdeHelper.getFromByteBuffer(RowBasedGrouperHelper.java:1896)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper$RowBasedKeySerde.readFromByteBuffer(RowBasedGrouperHelper.java:1339)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper$RowBasedKeySerde.readFromByteBuffer(RowBasedGrouperHelper.java:1156)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.groupby.epinephelinae.AbstractBufferHashGrouper.populateBucketEntryForOffset(AbstractBufferHashGrouper.java:185)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.groupby.epinephelinae.BufferHashGrouper$2.next(BufferHashGrouper.java:234)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.groupby.epinephelinae.BufferHashGrouper$2.next(BufferHashGrouper.java:216)
~[druid-processing-32.0.0.jar:32.0.0]
at
com.google.common.collect.Iterators$PeekingImpl.peek(Iterators.java:1217)
~[guava-32.0.1-jre.jar:?]
at
org.apache.druid.java.util.common.guava.MergeIterator.lambda$new$0(MergeIterator.java:53)
~[druid-processing-32.0.0.jar:32.0.0]
at
java.base/java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:660)
~[?:?]
at
java.base/java.util.PriorityQueue.siftUp(PriorityQueue.java:637) ~[?:?]
at
java.base/java.util.PriorityQueue.offer(PriorityQueue.java:330) ~[?:?]
at
java.base/java.util.PriorityQueue.add(PriorityQueue.java:311) ~[?:?]
at
org.apache.druid.java.util.common.guava.MergeIterator.next(MergeIterator.java:81)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.CloseableIterators$1.next(CloseableIterators.java:78)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.CloseableIterators$1.next(CloseableIterators.java:78)
~[druid-processing-32.0.0.jar:32.0.0]
at
com.google.common.collect.Iterators$PeekingImpl.peek(Iterators.java:1217)
~[guava-32.0.1-jre.jar:?]
at
org.apache.druid.java.util.common.guava.MergeIterator.lambda$new$0(MergeIterator.java:53)
~[druid-processing-32.0.0.jar:32.0.0]
at
java.base/java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:660)
~[?:?]
at
java.base/java.util.PriorityQueue.siftUp(PriorityQueue.java:637) ~[?:?]
at
java.base/java.util.PriorityQueue.offer(PriorityQueue.java:330) ~[?:?]
at
java.base/java.util.PriorityQueue.add(PriorityQueue.java:311) ~[?:?]
at
org.apache.druid.java.util.common.guava.MergeIterator.next(MergeIterator.java:81)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.java.util.common.CloseableIterators$1.next(CloseableIterators.java:78)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.groupby.epinephelinae.ParallelCombiner$2.call(ParallelCombiner.java:414)
~[druid-processing-32.0.0.jar:32.0.0]
at
org.apache.druid.query.groupby.epinephelinae.ParallelCombiner$2.call(ParallelCombiner.java:400)
~[druid-processing-32.0.0.jar:32.0.0]
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:259)
~[druid-processing-32.0.0.jar:32.0.0]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
~[?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
~[?:?]
... 1 more
```
This is pretty flaky, and sometimes it happens immediately, sometimes after
two consecutive runs, and sometimes even after 5 to 6 runs.
It should be noted that the query is a `groupBy` query that goes over 6
datasources whose sizes are in the order of few GBs.
Also I'm disabling the cache when I have the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
