AmatyaAvadhanula commented on PR #17420: URL: https://github.com/apache/druid/pull/17420#issuecomment-2445525099
I've added a TaskLockPosse level cache for segment allocations with the idea that the used segment set and pending segments are covered by an appending lock and cannot change unless altered by tasks holding these locks. There are 24 intervals, each with 2000 datasegments, each with 1000 dimensions. With with a batching of 5, and 128 allocations (2 replicas) Here are the results using a simple unit test: - Batch segment allocation with the flag turned off ``` INFO [main] org.apache.druid.indexing.common.actions.SegmentAllocateActionTest - Total time taken for [3072] allocations is [653233] ms. ``` - Batch segment allocation with the flag turned on : `3x Improvement` ``` INFO [main] org.apache.druid.indexing.common.actions.SegmentAllocateActionTest - Total time taken for [3072] allocations is [236854] ms. ``` - Normal segment allocation with the mentioned caching with the flag: `100x improvement` ``` INFO [main] org.apache.druid.indexing.common.actions.SegmentAllocateActionTest - Total time taken for [3072] allocations is [6302] ms. ``` `TLDR` : Turning the flag on gives a 3x boost over batch segment allocation, while using the cache gives about 100x. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
