viirya commented on code in PR #1561:
URL: https://github.com/apache/datafusion-comet/pull/1561#discussion_r2008032645
##########
spark/src/main/scala/org/apache/comet/CometExecIterator.scala:
##########
@@ -63,9 +64,28 @@ class CometExecIterator(
}.toArray
private val plan = {
val conf = SparkEnv.get.conf
- // Only enable unified memory manager when off-heap mode is enabled.
Otherwise,
- // we'll use the built-in memory pool from DF, and initializes with
`memory_limit`
- // and `memory_fraction` below.
+
+ val offHeapMode = CometSparkSessionExtensions.isOffHeapEnabled(conf)
+ val memoryLimit = if (offHeapMode) {
+ // in unified mode we share off-heap memory with Spark
+ ByteUnit.MiB.toBytes(conf.getSizeAsMb("spark.memory.offHeap.size"))
Review Comment:
Hmm, okay. The pool size is only used in the following check, although it
looks unnecessary. The callback should already do such check. Maybe we can
remove it to make two unified memory pools consistent.
```rust
let limit = self.pool_size.checked_div(num).unwrap();
let size = reservation.size();
if limit < size + additional {
return resources_err!(
"Failed to acquire {additional} bytes where {size} bytes already
reserved and the fair limit is {limit} bytes, {num} registered"
);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
