andygrove opened a new issue, #1560: URL: https://github.com/apache/datafusion-comet/issues/1560
### What is the problem the feature request solves? PR https://github.com/apache/datafusion-comet/pull/1525 made improvements to the Tuning Guide but there is still incorrect and/or misleading information, and some of this was only discovered while working on that PR. This is a follow-on issue to make improvements to the memory pool configuration code, documentation, and tests. Here are some initial thoughts on areas that need attention: - When running in off-heap mode we tell users to specify Comet overhead using `spark.memory.offHeap.size`. The original goal of using off-heap memory was to implement "unified memory management," where the native pool will interact with Spark's memory pool so that there is one unified memory pool between Spark and Comet. However, we now also have a "fair_unified" pool that can be used in off-heap mode and this does not use the "unified memory management" approach, so the name is misleading. The pool size can be configured in numerous ways and this could perhaps be simplified. - There are very few tests for any of the code that computes Comet memory overhead or memory pool sizes ### Describe the potential solution _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
