Weijun-H commented on issue #25183: URL: https://github.com/apache/datafusion/issues/25183#issuecomment-5771134286
>properly accounting for the already-built batch at construction time? Yes. #25565 reserves an estimated upper bound before interleave, retries with fewer rows if reservation fails, and errors if one row still cannot fit. This covers supported flat types, including Utf8/Binary; nested and view types retain the existing fallback. >account for the batch that would be held There is still a lifetime gap in #25565: finish_record_batch frees the construction reservation before returning the batch. The separate stream-attached reservation is sized for merge inputs and sorting workspace, not the emitted output. We therefore cannot claim that the output remains fully covered during its async spill write. Keeping its reservation through write completion is still needed to meet that requirement. We also have not measured the smaller-batch cost on multi-pass sort_tpch or an Utf8-heavy workload yet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
