kfaraz opened a new pull request, #19772:
URL: https://github.com/apache/druid/pull/19772
### Description
The `UnusedSegmentsKiller` currenly has some limitations:
- The killer sometimes launches embedded kill tasks which do not end up
killing any segments. This is because the kill queue contains jobs for all
intervals which have an unused segment, regardless of whether the unused
segment is eligible for kill or not.
- For intervals with a large number of killable unused segments, a single
embedded kill task is launched which would kill only 1000 segments in one go.
In such cases, the `UnusedSegmentsKiller` would take very long to clear a
backlog of killable unused segments.
- When the kill queue is rebuilt, heavy metadata queries are fired for each
datasource in the DB, even if they don't have any unused segments, killable or
otherwise.
### Changes
- While rebuilding the kill queue, fire a single metadata query to retrieve
the intervals containing killable unused segments of all datasources.
- In the above query, scan upto a maximum of 200k eligible unused segments.
This would also be the maximum number of segments killed when the queue has
been fully processed.
- This number is large enough to make meaningful progress in each cycle of
the `UnusedSegmentsKiller` and small enough to keep the query time within ~5s
(tried on MySQL 8 with 17M unused segments).
- Pass in the max segments to kill in each `KillCandidate`.
- Allow each embedded kill task to kill multiple batches of segments, upto a
maximum of 10 (previously 1).
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not
all of these items apply to every PR. Remove the items which are not done or
not relevant to the PR. None of the items from the checklist below are strictly
necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency
checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md)
(Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in
[licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
