andygrove opened a new pull request, #2871: URL: https://github.com/apache/datafusion-ballista/pull/2871
# Which issue does this PR close? Closes #2870. # Rationale for this change `stream_sort_shuffle_block` reads the schema header and the partition through a file handle and its `try_clone()`. A cloned handle shares its file cursor, so seeking to the partition also moves the header read. Every `IO_BLOCK_TRANSPORT` fetch of a sort-shuffle partition came back as the partition followed by `header_end` stray bytes from after it, instead of the header followed by the partition. Decoders happened to tolerate this, because each partition stream carries its own schema and `StreamDecoder` drops a trailing partial message when the input ends. A stricter reader rejects the response, which is how this surfaced in #2809. The bug dates from #1615. # What changes are included in this PR? - `stream_sort_shuffle_block` opens the data file twice, so the header and the partition each have their own cursor. - A test writes a small data file and index and checks the exact bytes served for each partition. It fails on `main`: partition 0 comes back as bytes `10..30` instead of `0..10` followed by `10..20`. # Are there any user-facing changes? No. The bytes served for a sort-shuffle partition over block transport are now exactly what the index describes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
