jihoonson opened a new issue #9768: URL: https://github.com/apache/druid/issues/9768
Each segment has a partitionId which uniquely identifies the segment in a time chunk. Currently, you cannot compact segments with minor compaction (which uses the segment lock) if their partitionIds are not consecutive. For example, you cannot compact the segments of the partitionIds 0, 1, 10 together because the partitionId 1 and the partitionId 10 are not consecutive. This is an expected limitation of the minor compaction by its design. (It is for reducing memory footprint. See https://github.com/apache/druid/issues/7491 for more details.) However, in practice, it would be nice if the minor compaction can compact non-consecutive segments. This will be nice especially if there are some expected but transient task failures in streaming ingestion because those task failures can cause the non-consecutive segment IDs. The minor compaction can support this if it's guaranteed that no new segments have the partitionId which falls in the overlapping root partition range of the existing segments. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
