comphead commented on PR #3536: URL: https://github.com/apache/datafusion-comet/pull/3536#issuecomment-3909951581
> The changes in this PR are somehow causing large differences in memory usage compared to a recent build from the main branch. > > Metric Baseline DF52 Delta > peak_JVMHeapMemory 6.38 GB 10.60 GB +4.22 GB (+66%) > peak_JVMOffHeapMemory 148.01 MB 147.28 MB -0.73 MB (~0%) > peak_OnHeapExecutionMemory 0 B 0 B -- > peak_OffHeapExecutionMemory 14.20 GB 5.30 GB -8.90 GB (-63%) > peak_OnHeapUnifiedMemory 74.86 MB 63.63 MB -11.24 MB (-15%) > peak_OffHeapUnifiedMemory 14.20 GB 5.30 GB -8.90 GB (-63%) > DF52 shows a significant shift from off-heap execution memory to JVM heap. Peak off-heap execution dropped from 14.20 GB to 5.30 GB (-63%), while JVM heap rose from 6.38 GB to 10.60 GB (+66%). Net combined peak memory actually decreased (~20.6 GB to ~15.9 GB). The total mem used seems to be less? +4 for Heap and -8 for on Heap, so -4 total. The JVM growth looks weird as DF52 has very few JVM changes. Is the baseline for latest main? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
