andygrove commented on code in PR #1561:
URL: https://github.com/apache/datafusion-comet/pull/1561#discussion_r2005998677
##########
spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala:
##########
@@ -1334,26 +1334,46 @@ object CometSparkSessionExtensions extends Logging {
CometConf.COMET_NATIVE_SCAN_IMPL.get(conf) ==
CometConf.SCAN_NATIVE_DATAFUSION
}
- /** Calculates required memory overhead in MB per executor process for
Comet. */
+ /**
+ * Whether we should override Spark memory configuration for Comet. This
only returns true when
+ * Comet native execution is enabled and/or Comet shuffle is enabled and
Comet doesn't use
+ * off-heap mode (unified memory manager).
+ */
+ def shouldOverrideMemoryConf(conf: SparkConf): Boolean = {
Review Comment:
This is moved from `CometDriverPlugin` to put it next to the code that
calculates the memory to be overridden and makes it easier to call from unit
tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
