jatorre opened a new issue, #867: URL: https://github.com/apache/arrow-nanoarrow/issues/867
## Summary The output from nanoarrow's `to_arrow_ipc()` DuckDB extension does not include the Arrow IPC end-of-stream (EOS) marker (`0xFFFFFFFF 0x00000000`). This causes consumers that expect a well-terminated IPC stream to fail. ## Reproduction When using `to_arrow_ipc()` from the nanoarrow DuckDB extension and concatenating the resulting chunks, the stream lacks the 8-byte EOS sentinel defined by the [Arrow IPC spec](https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format): > The end of a stream is indicated by a 0 length and 0 metadata length. For example, feeding the output to DuckDB-WASM's `insertArrowFromIPCStream` fails with: ``` Header-type of flatbuffer-encoded Message is not RecordBatch ``` ## Workaround Manually appending the EOS marker after concatenating chunks resolves the issue: ```js chunks.push(Buffer.from([0xFF, 0xFF, 0xFF, 0xFF, 0x00, 0x00, 0x00, 0x00])); return Buffer.concat(chunks); ``` ## Expected behavior `to_arrow_ipc()` should produce a complete, spec-compliant IPC stream that includes the EOS marker, so consumers don't need to append it manually. ## Context Discussion originated in duckdb/duckdb-node-neo#45. The [Mosaic project](https://github.com/uwdata/mosaic/pull/993) also uses this `to_arrow_ipc()` + nanoarrow pattern. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
