jackylee-ch commented on code in PR #13094:
URL: https://github.com/apache/gluten/pull/13094#discussion_r4173910477
##########
gluten-core/src/main/scala/org/apache/spark/memory/SparkMemoryUtil.scala:
##########
@@ -50,9 +52,24 @@ object SparkMemoryUtil {
// We assume storage memory can be fully transferred to execution memory so
far
def getCurrentAvailableOffHeapMemory: Long = {
- val mm = SparkEnv.get.memoryManager
- val smp = smpField.get(mm).asInstanceOf[StorageMemoryPool]
- val emp = empField.get(mm).asInstanceOf[ExecutionMemoryPool]
+ val env = SparkEnv.get
+ val mm = env.memoryManager
+ // With dynamic off-heap sizing enabled, Gluten's global reservations are
+ // charged to the ON-heap pools (see GlobalOffHeapMemoryTarget), so the
+ // available figure must be read from the same pools to stay meaningful;
+ // reading the off-heap pools would ignore every reservation.
+ val dynamicSizingEnabled =
Review Comment:
`GlobalOffHeapMemoryTarget.mode` decides where the reservations are charged
from `GlutenCoreConfig.get.dynamicOffHeapSizingEnabled`, but here the flag is
read from `SparkEnv.conf`. If those two sources ever disagree (e.g. a
session-level override vs. the startup conf), the metric would read the wrong
pool family again — the exact bug this patch fixes. Can we read it from the
same source as `mode` so the two can't drift apart?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
