Weijun-H commented on issue #25183: URL: https://github.com/apache/datafusion/issues/25183#issuecomment-5715092672
Thanks for exploring this. We have a relevant downstream synthetic case: 9,000 rows with 128 KiB payloads, one task, and a 966,367,616-byte task budget. Our baseline fails during spill-merge admission with an approximately 1 GiB reservation request; our combined downstream changes complete the case. Our approach also sizes output before materialization, reserves memory, and reduces the row count if needed. We include output-construction memory alongside read buffers in merge admission. Some types still use the existing fallback, and we have not isolated the performance cost of these changes. Two open choices are the output byte target (our current 10 MiB floor is not validated) and the single-row case: we return a memory error if it cannot be reserved, whereas your prototype retains best-effort spilling. Our prototype targets DataFusion 54.1; adapting it to current main also requires checking reservation lifetimes across asynchronous spill writes. Would you prefer that we help extend your prototype, or share a draft PR and the reproducer for comparison? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
