CurtHagenlocher opened a new pull request, #432: URL: https://github.com/apache/arrow-dotnet/pull/432
Adds IPC `Message.custom_metadata` support to the stream and file readers and writers. ## Credit The original implementation is **@cmettler's** work in #283 (closing #282). **@rustyconover** rebased it onto a `main` that had drifted ~76 commits ahead and addressed the first round of review comments in #424. Both of those PRs are branches on personal forks; this one moves the work onto a branch in the base repository so that any committer can push to it, and supersedes them. The individual commits here retain their original authorship. ## What this adds - `ArrowStreamReader.ReadNextRecordBatchWithCustomMetadata()` and `…Async()` return a `RecordBatchWithMetadata` pairing the batch with its IPC `Message.custom_metadata`, mirroring pyarrow's `read_next_batch_with_custom_metadata()` and the equivalent Arrow C++ struct. The struct deconstructs, so callers can write `var (batch, metadata) = …`. - `ArrowFileReader.ReadRecordBatchWithCustomMetadataAsync(int index)` gives the indexed read the same capability as the sequential one. - `ArrowStreamWriter.WriteRecordBatch(batch, customMetadata)` and the async counterpart attach per-message `custom_metadata` when writing, matching pyarrow's `write_batch(batch, custom_metadata)`. - Cross-language round-trip tests via pythonnet + pyarrow, skipped unless `PYTHONNET_PYDLL` is set, consistent with the existing `CDataSchemaPythonTest` pattern. ## Changes on top of #424 - The read side originally exposed a `LastBatchCustomMetadata` property. A property that has to be read at exactly the right moment is easy to get out of step with the batch in hand, and it had no sensible value at the end of the stream, so it was replaced with the `RecordBatchWithMetadata` return type above. The transient state on `ArrowReaderImplementation` remains, but it is internal and consumed immediately. - The new write overloads went straight to `WriteRecordBatchInternal`, while `ArrowFileWriter` relied on overriding each public `WriteRecordBatch` to call `WriteStart()` first — so the new overload on an `ArrowFileWriter` skipped the ARROW1 magic and silently produced a file that `ArrowFileReader` rejects. `WriteStart()`/`WriteStartAsync()` moved into `WriteRecordBatchInternal`, where every write path must pass through it; both are idempotent, so byte output is unchanged for the stream writer, the file writer and Flight. - Removed the second virtual `WriteMessageAsync` overload. Two virtual overloads where one forwards to the other is the trap that already routed Flight's record batch writes past `FlightDataStream`'s override. - Custom metadata is validated before anything is written rather than part-way through building the message, so a rejected dictionary leaves the writer usable. - `Message.custom_metadata` is read the same way schema and field metadata already are in `MessageSerializer`, rather than skipping null keys and rewriting null values as `""`. ## Verification - `dotnet build Apache.Arrow.sln` — succeeds, 0 warnings, 0 errors. - `dotnet test test/Apache.Arrow.Tests` — 1893 passed / 30 skipped on net8.0, 1849 passed / 30 skipped on net462 and net472, 0 failed. The skips are the pre-existing Python-dependent tests. Closes #282 Supersedes #283 Supersedes #424 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DT86mdkGm3XseKwiUeGUcx Co-Authored-By: Christoph Mettler <[email protected]> Co-Authored-By: Rusty Conover <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
