dayan02dev opened a new pull request, #12334:
URL: https://github.com/apache/arrow-rs/pull/12334

   # Which issue does this PR close?
   
   - Closes #6311
   
   # Rationale for this change
   
   An IPC file is `ARROW1`, padding to an 8-byte boundary, then an IPC stream 
ending in EOS, then the footer 
([spec](https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format)), 
so a reader can skip 8 bytes and read the file as a stream. Since #5554 
(52.0.0), `FileWriter` pads the magic to the write alignment instead, 64 bytes 
by default. A stream reader at byte 8 then sees zeros and stops: `StreamReader` 
fails with "Expected schema message, found empty stream." and pyarrow 26.0.1 
with "Tried reading schema message, was null or length 0". Only files written 
with alignment 8 embed a valid stream today. hohav/peppi works around this by 
skipping 64 bytes or by writing with alignment 8.
   
   #5554 padded the magic so that every later message, and so every body 
buffer, starts at a multiple of the alignment in the file, which zero-copy 
readers rely on (16 bytes for `Decimal128`). This PR keeps that.
   
   # What changes are included in this PR?
   
   - `FileWriter` pads the magic to 8 bytes at every alignment.
   - The schema message takes up the difference: `MetadataLayout::new` and the 
private `write_encoded_data` get the offset the message starts at, and the 
schema message's metadata padding now ends it on an alignment boundary of the 
file. Dictionary and record batch messages start at the same alignment as 
before, and the footer offsets follow.
   - All other callers pass offset 0, so `StreamWriter`, `StreamEncoder` and 
`write_message` write the same bytes as before, and no public API changes.
   
   bkietz's #6321 fixed the same bug by adding a parameter to the public 
`write_message`; Jefffrey noted in review that this is a breaking change. It 
stayed in draft and was closed as stale. This PR reaches the same file layout 
without changing any public API.
   
   # Are these changes tested?
   
   Yes. `test_file_embeds_stream` writes a file with a dictionary and a decimal 
column at alignments 8, 16, 32 and 64, and in the legacy format. It reads 
`bytes[8..]` with `StreamReader`, compares the batches with `FileReader`'s, and 
checks that every footer block and body starts at a multiple of the alignment. 
Without the fix it fails at alignment 16 with "Expected schema message, found 
empty stream.". Padding only the magic, without the schema change, fails the 
alignment check and the existing `test_decimal128_alignment16_is_sufficient`.
   
   The agent also checked the output with Arrow C++ (pyarrow 26.0.1): for the 
37 arrow-testing integration JSON files converted with 
`arrow-json-integration-test`, the embedded stream now reads and matches the 
file contents (0 of 37 before), and in memory-mapped test files the record 
batch buffers are still aligned to 16, 32 and 64 bytes.
   
   # Are there any significant user-facing changes?
   
   No API changes. Files written with alignment 16, 32 or 64 now have their 
first message at byte 8 and are the same size or one alignment unit smaller; 
alignment-8 files are byte-identical. Readers that use the footer are 
unaffected. Code that skipped `alignment` bytes to reach the stream needs to 
skip 8, as the format specifies.
   
   # AI usage disclosure
   
   The fix, the test and the pyarrow and integration-file checks above were 
produced with an AI coding agent (Claude Code). I read the full diff and ran 
the new test locally; it fails on main and passes with this change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to