a2l007 opened a new issue #10193: URL: https://github.com/apache/druid/issues/10193
When a cluster has thousands of segments to load, it is possible that historicals take more time to process all the requests showing up in the loadQueue. Consequently, the coordinator timeout `druid.coordinator.load.timeout` expires and after that the segment request is removed from the CuratorLoadQueuePeon object for this historical and the balancer attempts to balance this segment again. However, since the zookeeper path for this segment still exists on the earlier historical, this historical will eventually load this segment even though the coordinator has decided to place the segment elsewhere. This behavior can cause the balancer to make incorrect segment placement decisions since `ServerHolder.getSizeUsed()` will now be different from the actual used size on that historical. This is more evident when using the disk Normalized balancer strategy as the usageRatio plays an important role in that strategy. Couple of ways to mitigate this problem would be to increase the `druid.coordinator.load.timeout` property or lower the `maxSegmentsInNodeLoadingQueue` so that the historicals aren't flooded with loadqueue entries when there are a large number of segments to move. Investigating into this issue, it looks like there isn't a direct way to fix this without hampering the balancer performance. Since there are workarounds in place and this issue might not exist with `HttpLoadQueuePeon`, it may not be important to fix this soon but this tracking issue can help other users be aware of this behavior. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
