andygrove commented on PR #3536: URL: https://github.com/apache/datafusion-comet/pull/3536#issuecomment-3910056037
> > The changes in this PR are somehow causing large differences in memory usage compared to a recent build from the main branch. > > Metric Baseline DF52 Delta > > peak_JVMHeapMemory 6.38 GB 10.60 GB +4.22 GB (+66%) > > peak_JVMOffHeapMemory 148.01 MB 147.28 MB -0.73 MB (~0%) > > peak_OnHeapExecutionMemory 0 B 0 B -- > > peak_OffHeapExecutionMemory 14.20 GB 5.30 GB -8.90 GB (-63%) > > peak_OnHeapUnifiedMemory 74.86 MB 63.63 MB -11.24 MB (-15%) > > peak_OffHeapUnifiedMemory 14.20 GB 5.30 GB -8.90 GB (-63%) > > DF52 shows a significant shift from off-heap execution memory to JVM heap. Peak off-heap execution dropped from 14.20 GB to 5.30 GB (-63%), while JVM heap rose from 6.38 GB to 10.60 GB (+66%). Net combined peak memory actually decreased (~20.6 GB to ~15.9 GB). > > The total mem used seems to be less? +4 for Heap and -8 for on Heap, so -4 total. The JVM growth looks weird as DF52 has very few JVM changes. Is the baseline for latest main? The baseline was a build from Friday. Claude is suggesting this as the root cause: "The memory shift is primarily driven by DataFusion 52's internal changes to how operators manage buffers (batch coalescing, expression evaluation), combined with these buffers likely not being tracked through Comet's CometUnifiedMemoryPool → JNI → Spark accounting path. The result is that native memory still exists but is "invisible" to Spark's off-heap accounting, while Spark's JVM-side operations expand to fill the perceived available space." -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
