andygrove commented on code in PR #1561:
URL: https://github.com/apache/datafusion-comet/pull/1561#discussion_r2006287257
##########
spark/src/main/scala/org/apache/comet/CometExecIterator.scala:
##########
@@ -63,9 +64,28 @@ class CometExecIterator(
}.toArray
private val plan = {
val conf = SparkEnv.get.conf
- // Only enable unified memory manager when off-heap mode is enabled.
Otherwise,
- // we'll use the built-in memory pool from DF, and initializes with
`memory_limit`
- // and `memory_fraction` below.
+
+ val offHeapMode = CometSparkSessionExtensions.isOffHeapEnabled(conf)
+ val memoryLimit = if (offHeapMode) {
+ // in unified mode we share off-heap memory with Spark
+ ByteUnit.MiB.toBytes(conf.getSizeAsMb("spark.memory.offHeap.size"))
+ } else {
+ // we'll use the built-in memory pool from DF, and initializes with
`memory_limit`
+ // and `memory_fraction` below.
+ CometSparkSessionExtensions.getCometMemoryOverhead(conf)
+ }
Review Comment:
This is the main functional change in this PR. When running in off-heap mode
we now consistently use `spark.memory.offHeap.size` for the overall pool size
that is shared with Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
