Zeyu-Chen-SFDC commented on issue #17163:
URL: https://github.com/apache/druid/issues/17163#issuecomment-2375976225
> Does this issue begin to happen when the broker is serving a lot of
queries, and there are some that are timing out?
We haven't seen that. All these episodes began when the broker was lightly
loaded, serving at most 1 other "normal" query concurrently. And the "normal"
queries completed successfully. The impact of the stuck joins is as if they
simply reduced the jetty threadpool capacity by a constant.
> search for the query ID in the logs of the broker and the historical
serving the queries and please share if there's something relevant.
The pattern of activities from the logs is as follows:
- in 300ms, our historicals finish all the subqueries successfully
- no broker logs whatsoever
- 15min later, our routers error out on the query:
```
org.apache.druid.server.log.LoggingRequestLogger - 2024-09-20T00:13:47.028Z
127.0.0.1 {"success":false,"exception":"Total timeout 900000 ms
elapsed"} {"query":"SELECT ....
org.apache.druid.server.AsyncQueryForwardingServlet - Exception handling
request: {exceptionType=java.util.concurrent.TimeoutException,
exceptionMessage=Total timeout 900000 ms elapsed,
class=org.apache.druid.server.AsyncQueryForwardingServlet,
exception=java.util.concurrent.TimeoutException: Total timeout 900000 ms
elapsed, sqlQuery=SqlQuery{query='SELECT distinct ...
org.apache.druid.server.log.LoggingRequestLogger - 2024-09-20T00:13:47.029Z
127.0.0.1 {"query/time":900001,"success":false}
{"query":"SELECT distinct ...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
