andygrove opened a new issue, #1558: URL: https://github.com/apache/datafusion-comet/issues/1558
### What is the problem the feature request solves? One of the challenges in having users switch to Comet is that it is typically necessary to specify how much memory Comet can use. Memory tuning will be explained in more detail in the Tuning Guide once https://github.com/apache/datafusion-comet/pull/1525 is merged. It would be nice if there were a way that Comet could be enabled by default and choose sensible memory defaults. Currently, when running in on-heap mode, we allocate an additional 20% of executor memory to Comet, but Comet may need the same amount of memory as Spark in some cases. We could change the default to 100%, but then we are doubling the amount of memory, and this may not be ideal. To summarize, if we allocate too little memory then jobs may fail with OOM and we allocate too much it is wasteful and may prevent jobs from running due to lack of resource in a cluster. NVIDIA faces the same challenge with Spark RAPIDS and just announced [Project Aether](https://blogs.nvidia.com/blog/project-aether-accelerates-apache-spark/) to solve this problem for their customers. Let's use this issue to discuss ideas for an approach for Comet. ### Describe the potential solution _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
