pengding-stripe commented on issue #17657: URL: https://github.com/apache/pinot/issues/17657#issuecomment-3879503737
Updates: **[PR #16791](https://github.com/apache/pinot/pull/16791)** We originally suspect this is the root cause since it changed the broker routing map update parallelism and we saw timeout errors with brokers on 1.5 and servers on 1.4. But we have ruled this PR out because: 1. we changed the parallelism config to 1 and still see timeouts 2. we found server errors as well so don't believe this is a broker only issue ``` [?|?] ERROR [BaseCombineOperator] [pqr-1:270] Timed out while polling results block, numBlocksMerged: 0 (query: QueryContext{ _aliasList[0]="xxx" _distinct=false _explain="NONE" _filter="(xxx')" ``` **[PR #17106](https://github.com/apache/pinot/pull/17106)** We did server profiling and saw more startree data structure calls so this one was flagged, but we have ruled it out because we don't see any count aggregations in the affected table queries **[PR #16728](https://github.com/apache/pinot/pull/16728)** We noticed that the query latency and scheduler wait time were much more spiky and roughly match the timeout errors, which indicates timeout behavior changes and this PR has large updates on the query lifecycle We tried to deploy the [commit](https://github.com/apache/pinot/commit/a7cad307adf220db6f6369dcdd6bf390a9a99fc4) right before this PR to our sandbox cluster with production loads. We noticed significant improvements of query latency and scheduler wait time <img width="3114" height="1060" alt="Image" src="https://github.com/user-attachments/assets/db977338-e4e4-42f3-83d7-8e876ac834db"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
