jihoonson commented on issue #8846: VersionedIntervalTimeline performance corner case with high number of overlapping segments URL: https://github.com/apache/druid/issues/8846#issuecomment-591730668 Ah, I'm looking at the performance of `VersionedIntervalTimeline` using `VersionedIntervalTimelineBenchmark` and the issue I'm seeing is the `remove()` method is too slow. From what I've seen, it is slow because it needs to determine the new visible segments after a segment is removed and I guess `removeAll()` could be faster if the new visible segments can be determined at once after removing segments is done. > also, there is a caveat (on broker) and hence the approach needed discussion, operations in VersionedIntervalTimeline need to hold a lock to protect all data structures .. currently lock is not held for long as we do things one segment at a time .. this gives incoming queries a chance to lookup segments and execute . With batching, lock could potentially be held for relatively longer and queries might block. This is true, so I think we can keep both `remove()` and `removeAll()` so that callers can use either of them based on their preferences. AFAIT, concurrent access to the timeline is required only in brokers, so maybe the broker can use `remove()` to reduce the lock contention while others use `removeAll()`. Does it make sense?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
