kfaraz commented on issue #12898: URL: https://github.com/apache/druid/issues/12898#issuecomment-1315454052
@danprince1 , @pchang388 , thanks for reporting the issue. There are a set of segment loading changes in #13197 that are currently in review. That PR addresses this and several other issues with segment loading and balancing. That said, the original issue described here should not really be a problem since newer segments are already prioritized over older segments while loading even in the current code. References: https://github.com/apache/druid/blob/309cae7b65e65628f3b277808717a0c7e0bdaccf/server/src/main/java/org/apache/druid/server/coordinator/HttpLoadQueuePeon.java#L86 https://github.com/apache/druid/blob/309cae7b65e65628f3b277808717a0c7e0bdaccf/server/src/main/java/org/apache/druid/server/coordinator/DruidCoordinatorRuntimeParams.java#L52 My guess is that the prioritization is not the issue, the root cause is perhaps that the coordinator is just trying to load everything that is missing and taking very long to do that. If the coordinator gets stuck in a cycle, it doesn't know of new segments that the tasks are publishing and fails to make them available quickly. This problem is being addressed in the mentioned set of changes. Also, as @abhishekagarwal87 mentioned, #13367 is meant to speed up assignments. Since it is a coordinator dynamic config, you can safely enable and disable it during coordinator operations. The suggested approach would be to enable it before spawning a new historical so that assignments happen quickly. Once normal operation has resumed, you can choose to disable the round-robin or let it run if your historicals don't seem too out-of-balance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
