jihoonson opened a new issue #9383: Performance regression in OvershadowableManager URL: https://github.com/apache/druid/issues/9383 ### Affected Version Since 0.16. ### Description `OvershadowableManager` was added in #7491 to manage all segments in the same time chunk. `addChunk()` is pretty slow when you have 1000+ segments per time chunk. Look at the below flame graph.  There are two slow parts in `addChunk()`. - [It currently determines the state of the new atomicUpdateGroup by calling `overshadows()` against all existing segments](https://github.com/apache/druid/blob/master/core/src/main/java/org/apache/druid/timeline/partition/OvershadowableManager.java#L678-L682). This stream contributes 77% to the total overhead. This should be fixed by navigating overlapping visible groups directly rather than iterating all visible groups. - `findLowestOverlappingEntry()` contributes the remaining 23%. This is to skip non-overlapping entries in the map. This iteration can be reduced by using more accurate indexes: `stateMap` uses `RootPartitionRange` as its key, but this can be split into two indexes of `rangeStart` and `rangeEnd`.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
