clintropolis commented on issue #18733: URL: https://github.com/apache/druid/issues/18733#issuecomment-3531181847
I think this is worth investigating - I would imagine for workloads/tunings that are more likely to be disk bound that this could potentially be effective, though with sufficient ‘free’ memory space lots of segments can be in memory in page cache since it is using mmap, so those workloads might benefit less, since there is quite a lot of actual work they are doing too. I would think this needs lots of measurement for sure. This is something that is on my radar, but I don't personally plan to look into it until after we have migrated to having dart as the default query engine, so that there is less to focus on measuring and fewer moving parts. Another thing that is really important to think about - with it being a fixed size currently it is also controlling memory usage. This includes both heap usage and also access to the number of ‘processing’ buffers that are active since that is currently a non-blocking pool, so we would need to rework that to be blocking to control memory size i think. I haven't thought much beyond these implications. I think we would want to get to where query engines primarily use off heap memory and the engines also to transition to more granular memory allocations so that usage can grow as required instead of setting maximum bounds immediately up front. If done well, this would allow better usage of virtual threads because of better resource usage. Also, semi related, the virtual storage 'on demand' thread pool would surely benefit from being virtual threads, I have done some experimentation with this and while I haven't measured it yet, it is really easy to wire up (at least if you ignore maintaining compatibility for older java versions... though there are possible solutions for this I've also been playing with like multi-release jars). If it turns out virtual threads are good for the processing pool, then the on demand downloads could potentially just be done inline in the same virtual thread pool instead of being a separate pool dedicated for downloads, hard to say without experimentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
