maytasm opened a new pull request #11019:
URL: https://github.com/apache/druid/pull/11019
Auto-compaction with segment granularity retrieve incomplete segments from
timeline when interval overlap
### Description
Auto-compaction with segment granularity retrieve incomplete segments from
timeline when interval overlap.
For example, if we have segments of YEAR granularity from 2000-01-01 to
2001-01-3001 and a segment of DAY granularity from 2000-01-01 to 2000-01-02.
Let's then say that we set auto compaction with a segmentGranularity of MONTH.
Auto-compaction will bucket all the existing segments into the new bucket using
the configured segmentGranularity (MONTH in this example). When we iterate for
segments to compact we will start at the latest and in this case would be the
December (2000-12-01 to 2000-12-31) bucket. We would get only the segment with
YEAR granularity in this bucket. We would then create a compaction task with
the interval 2000-01-01 to 2001-12-31 from the segment we just retrieved. We
would also include a sha256OfSortedSegmentIds which would only be the YEAR
granularity segment (2000-01-01 to 2001-12-31). However, when the compaction
task runs, it would get all segment from 2000-01-01 to 2001-01-01, which would
include both segments (a segment of YEAR granularity from 2000-01-
01 to 2001-12-31 and a segment of DAY granularity from 2000-01-01 to
2000-01-02)
This PR fixes this problem but keeping the original timeline in the
CompactibleTimelineObjectHolderCursor that can then we use to fetch the
original segments back using the umbrella interval of the transformed segments.
This PR has:
- [x] been self-reviewed.
- [ ] using the [concurrency
checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md)
(Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [x] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in
[licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [x] added integration tests.
- [x] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
