jackylee-ch commented on code in PR #12622:
URL: https://github.com/apache/gluten/pull/12622#discussion_r3645768524
##########
gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala:
##########
@@ -134,6 +135,15 @@ private object GlutenDriverPlugin extends Logging {
((onHeapSize - (300 * 1024 * 1024)) *
conf.getDouble(GlutenCoreConfig.DYNAMIC_OFFHEAP_SIZING_MEMORY_FRACTION.key,
0.6d)).toLong
} else {
+ // Untracked memory mode skips the off-heap size requirement in
checkOffHeapSettings, so
+ // the key may be absent here. Normalize it to 0 (mirroring the
dynamic-sizing branch
+ // above) so downstream readers that read spark.memory.offHeap.size
directly, e.g.
+ // VeloxListenerApi.onDriverStart, don't hit NoSuchElementException.
Normal mode always has
+ // the key set because checkOffHeapSettings enforced it, so this only
affects untracked
+ // mode without an explicit off-heap size.
+ if (!conf.contains(GlutenCoreConfig.SPARK_OFFHEAP_SIZE_KEY)) {
Review Comment:
we have checked the offHeap size while [initing
GlutenPlugin](https://github.com/apache/gluten/blob/main/gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala#L96),
this seems un neccesary? or we have any cases that meet this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
