alamb opened a new pull request, #24529: URL: https://github.com/apache/datafusion/pull/24529
## Which issue does this PR close? - Part of https://github.com/apache/datafusion/issues/24462 - Backport of #24384 to `branch-55` (for 55.1.0, tracked in #24462). - Fixes #24383 ## Rationale for this change `UnnestExec` emitted exactly one output batch per input batch, however many rows the unnesting produced, never consulting `datafusion.execution.batch_size`. This let downstream operators receive arbitrarily large batches and made peak memory scale with input batch size times list length instead of `batch_size`. No API change, so it fits the backport criteria. ## What changes are included in this PR? Clean cherry-pick of #24384 (commit 6187d47ec5658190e729d4d3cad2cc10805aecdb). No adaptation required. ## Are these changes tested? Yes. Carries the original regression coverage, all tests pass. ## Are there any user-facing changes? `UnnestExec` output batches now respect `datafusion.execution.batch_size` as an upper bound. No API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
