LakshSingla commented on PR #16887: URL: https://github.com/apache/druid/pull/16887#issuecomment-2295647935
@kfaraz > buffers are all provided as lazy singletons and would be initialized/allocated only if needed by the task. The pool is created lazily, which is when the various query toolchests/runners/engines are created. The allocation of the buffer can or cannot be lazy depending on the type of the pool. 1. For the blocking pool, all the buffers are allocated [upfront in the constructor](https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/collections/DefaultBlockingPool.java#L62). 2. For the non-blocking pool, the buffers are allocated as you go. There's also an initialization count which determines the number of buffers to [initialize at the beginning](https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/collections/StupidPool.java#L132). This is set to number of processing threads, which means that we still allocate the buffers at the beginning. I have verified the above by looking at one of the controller logs, which shouldn't be using the buffers. ``` 316 2024-08-06T15:21:43,532 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[0] of size[400,000,000] 1 2024-08-06T15:21:43,655 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[1] of size[400,000,000] 2 2024-08-06T15:21:43,775 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[0] of size[400,000,000] 3 2024-08-06T15:21:43,894 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[1] of size[400,000,000] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
