cecemei commented on PR #20027: URL: https://github.com/apache/druid/pull/20027#issuecomment-5318352169
> Thanks for the changes, @cecemei ! > > I was under the impression that the code already ignores segments that cross the skip interval boundaries. > > https://github.com/apache/druid/blob/cea0d9950482b565da93351f887016022cf1d760/server/src/main/java/org/apache/druid/server/compaction/DataSourceCompactibleSegmentIterator.java#L418-L424 > > The above snippet is from the method `DataSourceCompactibleSegmentIterator.findInitialSearchInterval()`. > > This method uses the `SegmentTimeline` (which is already aligned to the target segment granularity) to look up segments that need to be compacted. If any segment does not lie fully within the lookup interval, that segment will not be picked for compaction. > > But since the lookup interval has no overlap with any of the skip intervals (due to the `filterSkipIntervals` step performed earlier), we can be sure that none of the segments that fully lie in the lookup interval have any overlap with any of the skip intervals. > > @cecemei , did you encounter/write up a test case which currently fails due to misaligned granularities? I think it would be worth checking that out first. The test case i added in this PR would fail without the change made. The case here is that we could have a skip interval in the middle of the day (10am - 12pm), and there're segments overlapping (0am - 10am) which get passed into findAndEnqueueSegmentsToCompact, so only the exact skip interval is not compacted but some segments during the day is still being compacted. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
