kfaraz commented on PR #18028: URL: https://github.com/apache/druid/pull/18028#issuecomment-2906351126
> I have not review the changes carefully, but leave one question first, some kill tasks might take very long time(for example, up to dozens of minutes) to delete segments for large data source, if there're several such kill tasks run together, wil these kill tasks block the overlord's other duties? Thanks for the comment, @FrankChen021 ! A couple of clarifications on the above: - When embedded kill tasks are running on the Overlord, it is recommended to NOT launch kill tasks manually or from the coordinator duty. It is currently not possible to validate this in the code itself since the new config `druid.manager.segments.killUnused.enable` is required on the Overlord whereas the old one `druid.coordinator.kill.enabled` is required on the Coordinator. - Embedded kill tasks run on a single thread on the Overlord. So, there can be only one. - Since it is a dedicated thread, it does not interfere with other Overlord activities. - It cannot take a long time even for large datasources since each embedded kill task will only ever kill at most 1000 segments in a single interval. So the total time would be just the time taken to delete 1000 segments from metadata store and deep storage. - WIP: If a single kill task still somehow gets stuck, there is a management ping sent from `OverlordDutyExecutor` that will kill off that task. Thanks for bringing this up, I will also add the above points to the PR description (and perhaps some points to the code javadoc as well) to help future readers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
