andygrove commented on PR #25853: URL: https://github.com/apache/datafusion/pull/25853#issuecomment-6223976313
Merged main into the branch. Beyond the conflict, this needed two changes: - `VIEW_SIZE_BYTES` is defined for all builds again (ea8d37dd5). #23565 made it test-only, but `SpilledRowSizes` uses it. - Replay compacts the view arrays of nested columns in merged batches (3289c00a6). With main merged, the Case H query with 14 adjacent large groups failed with `ResourcesExhausted` at 24 MB, which I bisected to #23565. A merged batch references the whole data buffers of the spilled batches its rows come from, and replay keeps the `array_agg` lists of each group until the group completes, so it held about twice the state of each group. Since #23565, spilling no longer compacts view arrays whose buffers are fully used, which was enough to push this query over the limit. Top-level columns are left as they are, since group keys and the accumulators of top-level states typically copy the values they keep. @comphead both came after your approval, in case you want to look at 3289c00a6. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
