alamb commented on PR #21182: URL: https://github.com/apache/datafusion/pull/21182#issuecomment-4194437956
> @alamb wdyt of setting the max buffer so something larger like 1Gi? The point is that even if it hits that limit it will be strictly less memory usage than SortExec. If we set it too small some workloads will be IO starved. I think the rationale state in > > /// This is strictly less memory than the SortExec it replaces, and only > /// inserted when PushdownSort eliminates a SortExec — no impact on other > /// query plans. BufferExec also integrates with MemoryPool, so it respects > /// the global memory limit and won't cause OOM. Makes sense to me If you wanted to be really nice you could make it a config option -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
