aman-coder03 commented on issue #19481: URL: https://github.com/apache/datafusion/issues/19481#issuecomment-3694746471
thanks @EmilyMatt , that makes sense. I agree that avoiding row shifting is important, and a helper like emit_all_chunked sounds like a good direction to explore. i will start with some trial-and-error experimentation, likely focusing on a single operator first (e.g. aggregates), and try a cursor-based/chunked emission approach so we can emit `RecordBatches `up to `batch_size `without copying or reshuffling rows. I’ll also keep the changes minimal and report back with observations (including any perf or memory trade-offs) before proposing anything broader. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
