yjshen commented on issue #9417: URL: https://github.com/apache/arrow-datafusion/issues/9417#issuecomment-1987380309
I agree that the core problem for the issue is accounting and that the most overreported batch slice would come from AggExec's mono output record batch. But I also believe there's a distinction between optimizing AggExec's output pattern and handling memory accounting. To improve AggExec's mono output pattern, #7065 might be similar to the idea of incremental output. Regarding the memory accounting side, I'm curious if you have considered alternatives that allow for more accurate accounting for different batches. The idea of having sliced batches not reporting their memory usage or using `get_slice_memory_size` for reporting is a good starting point. What do you think about this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
