2010YOUY01 commented on PR #23777: URL: https://github.com/apache/datafusion/pull/23777#issuecomment-5202689464
> Or you mean changing `output_bytes` itself from `Count` to `RecordBatchMemoryCounter`? It would make sense if there are other operators which slice a large RecordBatch, otherwise is it worth doing this solely for the current aggregation implementation? Yes, exactly. Even if it's only for aggregation, I think this approach is still overall simpler. The existing approach seems to leak some complexity into the aggregation logic. Also, the memory overcounting in the metrics is also likely in other operators, since now we're using batch concatenation quite commonly in joins/window executions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
