Rachelint commented on issue #24704: URL: https://github.com/apache/datafusion/issues/24704#issuecomment-5666700893
Thanks for sharing the mmap experiments and findings. I understand the motivation, but I have some concerns about making mmap the default implementation. Its performance may vary due to page faults, TLB pressure and NUMA behavior, while portability, memory accounting and OOM handling become more complicated. Where possible, I believe allocation, reclamation and platform-specific memory behavior are better left to mature allocators. From a user's perspective, bypassing the allocator would make me hesitant to use this in production for quite some time. For this part of DataFusion, predictable behavior and production stability are more important to me than a small performance gain. I would feel more comfortable with an allocator-backed implementation as the default, while keeping mmap experimental until it has been thoroughly validated under production-like workloads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
