dayan02dev opened a new pull request, #12334: URL: https://github.com/apache/arrow-rs/pull/12334
# Which issue does this PR close? - Closes #6311 # Rationale for this change An IPC file is `ARROW1`, padding to an 8-byte boundary, then an IPC stream ending in EOS, then the footer ([spec](https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format)), so a reader can skip 8 bytes and read the file as a stream. Since #5554 (52.0.0), `FileWriter` pads the magic to the write alignment instead, 64 bytes by default. A stream reader at byte 8 then sees zeros and stops: `StreamReader` fails with "Expected schema message, found empty stream." and pyarrow 26.0.1 with "Tried reading schema message, was null or length 0". Only files written with alignment 8 embed a valid stream today. hohav/peppi works around this by skipping 64 bytes or by writing with alignment 8. #5554 padded the magic so that every later message, and so every body buffer, starts at a multiple of the alignment in the file, which zero-copy readers rely on (16 bytes for `Decimal128`). This PR keeps that. # What changes are included in this PR? - `FileWriter` pads the magic to 8 bytes at every alignment. - The schema message takes up the difference: `MetadataLayout::new` and the private `write_encoded_data` get the offset the message starts at, and the schema message's metadata padding now ends it on an alignment boundary of the file. Dictionary and record batch messages start at the same alignment as before, and the footer offsets follow. - All other callers pass offset 0, so `StreamWriter`, `StreamEncoder` and `write_message` write the same bytes as before, and no public API changes. bkietz's #6321 fixed the same bug by adding a parameter to the public `write_message`; Jefffrey noted in review that this is a breaking change. It stayed in draft and was closed as stale. This PR reaches the same file layout without changing any public API. # Are these changes tested? Yes. `test_file_embeds_stream` writes a file with a dictionary and a decimal column at alignments 8, 16, 32 and 64, and in the legacy format. It reads `bytes[8..]` with `StreamReader`, compares the batches with `FileReader`'s, and checks that every footer block and body starts at a multiple of the alignment. Without the fix it fails at alignment 16 with "Expected schema message, found empty stream.". Padding only the magic, without the schema change, fails the alignment check and the existing `test_decimal128_alignment16_is_sufficient`. The agent also checked the output with Arrow C++ (pyarrow 26.0.1): for the 37 arrow-testing integration JSON files converted with `arrow-json-integration-test`, the embedded stream now reads and matches the file contents (0 of 37 before), and in memory-mapped test files the record batch buffers are still aligned to 16, 32 and 64 bytes. # Are there any significant user-facing changes? No API changes. Files written with alignment 16, 32 or 64 now have their first message at byte 8 and are the same size or one alignment unit smaller; alignment-8 files are byte-identical. Readers that use the footer are unaffected. Code that skipped `alignment` bytes to reach the stream needs to skip 8, as the format specifies. # AI usage disclosure The fix, the test and the pyarrow and integration-file checks above were produced with an AI coding agent (Claude Code). I read the full diff and ran the new test locally; it fails on main and passes with this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
