LuciferYang opened a new pull request, #13094: URL: https://github.com/apache/gluten/pull/13094
## What changes are proposed in this pull request? Two independent bugs in the dynamic off-heap sizing path (`spark.gluten.memory.dynamic.offHeap.sizing.enabled`, experimental, off by default). First, `DynamicOffHeapSizingMemoryTarget.shrinkOnHeapMemory` lowers both `MinHeapFreeRatio` and `MaxHeapFreeRatio` to push the JVM to release heap, but the `finally` block restores only `MaxHeapFreeRatio`. After one shrink the executor keeps `MinHeapFreeRatio=5` for the rest of its life, so it runs a tighter heap and GCs more often long after the off-heap pressure that triggered the shrink is gone. This patch reverts both ratios, each revert best-effort on its own so a failing `Max` revert cannot skip the `Min` revert. Second, `SparkMemoryUtil.getCurrentAvailableOffHeapMemory` reads the off-heap storage and execution pools. Under dynamic sizing the reservations are charged to the on-heap pools instead (`GlobalOffHeapMemoryTarget` acquires with `ON_HEAP`), and the driver plugin forces `spark.memory.offHeap.size=0`, so the off-heap pools are empty and the method returns 0. That value feeds the Velox columnar shuffle writer as its `memLimit`, which then falls back to the minimum buffer size. The metric now reads the on-heap pools when dynamic sizing is enabled, matching where the reservations actually land. ## How was this patch tested? `DynamicOffHeapSizingMemoryTargetTest` gains a case that drives the shrink path and asserts `MinHeapFreeRatio` is restored afterwards (skipped on JVMs where the flag is not settable). `GlutenDynamicOffHeapSizingSuite` gains a case asserting the availability metric follows the on-heap pool family under dynamic sizing and the off-heap family otherwise. ## Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: ZCode (GLM-5.3). Related issue: #13093. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
