andygrove commented on issue #949: URL: https://github.com/apache/datafusion-comet/issues/949#issuecomment-2462668202
> The downside (to using unified memory) is that comet operators cannot trigger the spilling of other memory consumers, which makes it easy to run into issues similar to https://github.com/apache/datafusion-comet/issues/886 due to its greedy/unfair nature. Issue #886 is related to Comet columnar shuffle, which currently has its own memory management, which is separate to the unified or native memory management features. There is a PR https://github.com/apache/datafusion-comet/pull/1063 to make it use unified memory instead. We are now leaning towards always using unified memory, and there is a PR to make this the only approach: https://github.com/apache/datafusion-comet/pull/1062 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
