kfaraz opened a new pull request, #13287:
URL: https://github.com/apache/druid/pull/13287
In clusters with a large number of segments, the duty
`MarkAsUnusedOvershadowedSegments`
can take a long very long time to finish. This is because of the costly
invocation of
`timeline.isOvershadowed` which is done for every used segment in every
coordinator run.
This is extravagant as there are only a few overshadowed segments which are
already identified
as part of the datasource snapshot. In clusters with ~500k segments, this
duty can take several
minutes to finish even when no segment is actually overshadowed.
### Changes
- Use `DataSourceSnapshot.getOvershadowedSegments` to get all overshadowed
segments
- Iterate over this set instead of all used segments to identify segments
that can be marked as unused
- Mark segments as unused in the DB in batches rather than one at a time
- Refactor: Add class `SegmentTimeline` for ease of use and readability
while using a
`VersionedIntervalTimeline` of segments.
### Further work
The changes here provide significant improvement in the run time of this
duty.
The number of overshadowed segments in a given coordinator run is expected
to be small.
Even in the very rare worst case scenario (half of all used segments are
overshadowed), this change
would halve the run time of the duty. Subsequent runs would again take
negligible time.
Further improvements can be made to the run time of this duty but that would
require changes to the
logic of `timeline.isOvershadowed` and would provide only minor time
reductions.
<hr>
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency
checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md)
(Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in
[licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
