luomh0910 opened a new issue, #13617:
URL: https://github.com/apache/gluten/issues/13617

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   When `spark.gluten.memory.dynamic.offHeap.sizing.enabled=true`,
   `MemoryTargets.global()` still returns a plain `GlobalOffHeapMemoryTarget`.
   Global reservations therefore update Spark's storage memory accounting,
   but bypass the shared off-heap counter, memory-budget checks, and JVM heap
   shrink logic in `DynamicOffHeapSizingMemoryTarget`.
   
   One affected use case is Velox broadcast hash-table construction, which
   allocates memory through the backend-global pool returned by
   `defaultLeafVeloxMemoryPool()`. These reservations use the global memory
   listener rather than a task-level memory target. Arrow global allocations
   also use the global memory target.
   
   There is also a configuration lookup issue: the global target can be
   initialized during executor plugin startup, when no task SQLConf is
   available. Reading the dynamic-sizing flag through the existing SQLConf
   path can return the default value `false`, even when SparkEnv enables it.
   Additionally, recalculating the global target's memory mode from the
   calling thread's SQLConf can make borrowing and releasing select
   different storage pools.
   
   ### Expected behavior
   
   - When dynamic off-heap sizing is enabled, wrap the global memory target
     with the existing `DynamicOffHeapSizingMemoryTarget`. Global reservations,
     including those used for broadcast hash-table construction, should
     contribute to its shared off-heap counter. Releases should remove the
     corresponding bytes from that counter.
   - Global reservation requests should go through the existing dynamic-sizing
     memory-budget checks and trigger the existing JVM heap shrink logic when
     its conditions are met, rather than bypassing that logic. No changes to
     the existing shrink algorithm or thresholds are intended.
   - Resolve the dynamic-sizing flag from SparkEnv when available, falling
     back to the existing GlutenCoreConfig getter otherwise. Use this lookup
     consistently for dynamic-target wrapping, task memory-pool selection,
     global memory-pool selection, and available-memory reporting.
   
   This issue was written with AI assistance (OpenAI Codex).
   
   ### Gluten version
   
   _No response_
   
   ### Spark version
   
   None
   
   ### Spark configurations
   
   _No response_
   
   ### System information
   
   _No response_
   
   ### Relevant logs
   
   ```bash
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to