kentkwu opened a new pull request, #470: URL: https://github.com/apache/arrow-js/pull/470
## Summary The Arrow columnar format spec requires the eight-byte prefix on each compressed IPC body buffer to hold the *uncompressed* length so a reader can size the decompression destination buffer. > Spec: https://arrow.apache.org/docs/format/Columnar.html#compression the body includes a flat sequence of compressed buffers together with the length of the uncompressed buffer as a 64-bit little-endian signed integer stored in the first 8 bytes of each buffer in the sequence. This uncompressed length can be set to -1 to indicate that that specific buffer is left uncompressed. The writer was emitting the compressed length instead, so PyArrow rejects Arrow JS-produced ZSTD and LZ4_FRAME streams. JS-to-JS round trips missed the bug because the reader sizes buffers from the codec frame, not the prefix. The compression code path is shared between the stream and file writers, so the one-line fix covers both formats. - `src/ipc/writer.ts`: write `byteBuf.length` in the prefix instead of `compressed.length` - New byte-level regression test in both writer suites (two codecs × stream and file) decompresses each emitted body buffer and asserts the prefix equals the decompressed length - Extract `registerCompressionCodecs` and the new prefix inspector into `test/unit/ipc/writer/compression-codecs.ts`, dropping a duplicate copy from `file-writer-tests.ts` Cross-language interop verified against PyArrow 25.0.1. ## Test Plan - [x] `npm test` - [x] `npm run build` - [x] `npm run lint:ci` - [x] All four regression cases (two codecs × stream and file writers) fail against the unfixed writer and pass with the fix ## Related - Arrow columnar format spec, compression section: https://arrow.apache.org/docs/format/Columnar.html#compression - Cross-language reproducer by @ianmcook: https://gist.github.com/ianmcook/cae432c969498adf56f5b3d437eb7d92 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
