rluvaton commented on PR #1511: URL: https://github.com/apache/datafusion-comet/pull/1511#issuecomment-2757951289
> > interleave_record_batch is much slower running Q18, this problem is still under investigation. > > This problem happens on macOS 15.3.1 with Apple M1 Pro chip. I suspect that this is an OS-specific problem. I'll rerun the benchmarks on EC2 instances and see if it still happens. I have some interesting findings when troubleshooting this problem. I'd like to share it here to get more understanding about it from other developers. > > ... > > The results are quite interesting: all the Vec resizes that take long time are always resizing from 4MB to 8MB. > > ... > > I have summarized the reallocation target size from the logs of running Q18 10 times, the allocation size of Vec when running `SumDecimalGroupsAccumulator::merge_batch` will usually grow from 128KB to 16MB for each partition/task. It is strange that the slowness only happens when growing from 4MB to 8MB, but not for other reallocation sizes. I saw that too on my MacBook Pro M3 Max 15.3.2 but with different sizes, maybe it's related to L1/2/3 cache? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
