alamb opened a new issue, #24310: URL: https://github.com/apache/datafusion/issues/24310
### Describe the bug While profiling with this command (as part of testing out https://github.com/apache/datafusion/pull/24035) I noticed a non trivial amount of time (1% of query) is now spent measuring the memory used in `get_record_batch_memory_size` <img width="1880" height="994" alt="Image" src="https://github.com/user-attachments/assets/fa0a03dd-048e-40f2-bab5-e49152174467"; /> There are at least two memory allocations (for each record batch for each poll for each operator) that add up: 1. calling Array::to_data() allocates a vec (for buffers): https://github.com/apache/datafusion/blob/d428760d709a375f3d997c84e9c4748a22584149/datafusion/common/src/utils/memory.rs#L170-L169 2. Creating a counted buffer hash table: https://github.com/apache/datafusion/blob/d428760d709a375f3d997c84e9c4748a22584149/datafusion/common/src/utils/memory.rs#L154-L153 ### To Reproduce ```shell cargo bench --profile=profiling --bench window_query_sql -- "window empty over, aggregate functions" ``` And then run samply something like ```shell samply record target/profiling/deps/window_query_sql-950028bf4c5b086c --bench "window empty over, aggregate functions" ``` ### Expected behavior I expect memory accounting to be faster: 1. Not allocate vec's 2. Likely not hash either I think the overhead is relatively small but since this is called on basically every operator work we could o ### Additional context It looks like it was most recently modified as part of this PR from @jordepic and @2010YOUY01 : - #22862 But the problematic allocations existed before that -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
