rustyconover opened a new issue, #923:
URL: https://github.com/apache/arrow-nanoarrow/issues/923
Code I've written use the `custom_metadata` field often RecordBatches, which
is separate form schema or field-level metadata in the schema. Right now the
IPC encoder cannot write these messages.
From the robots:
---
### Current state (as of `b27fd93`)
Encoder: `ArrowIpcEncoderEncodeSimpleRecordBatch()` and
`ArrowIpcEncoderEncodeRecordBatch()` take
no metadata argument, and `ArrowIpcEncoderEncodeRecordBatchImpl()` never
calls
`Message_custom_metadata_add`.
Decoder: `decoder.c` reads `custom_metadata` only for `Schema`
(`decoder.c:1719`) and `Field`
(`decoder.c:1283`). There is no accessor for the decoded
`Message.custom_metadata`, so the
information is dropped even when present in the input stream.
### Why this looks like a small change
The flatcc plumbing is already generated on both sides, so no schema
regeneration should be needed:
- read accessor: `org_apache_arrow_flatbuf_Message_custom_metadata`
(`flatcc_generated.h:11298`)
- builder: `org_apache_arrow_flatbuf_Message_custom_metadata_add`
(`flatcc_generated.h:11525`)
And `ArrowIpcEncodeMetadata()` already exists in `encoder.c` for the
schema/field paths, and is
already parameterized over the flatcc push-start/push-end callbacks, so it
should extend to the
`Message` table by passing the `Message_custom_metadata_push_*` pair. It
currently takes a
`const struct ArrowSchema*` and reads `schema->metadata` internally; taking
the packed
`const char* metadata` directly would make it reusable from the RecordBatch
path without otherwise
changing its behaviour.
### Sketch of an API
Encoder — a setter that applies to the next encoded message, so the existing
`EncodeRecordBatch`
signatures stay unchanged:
```c
ArrowErrorCode ArrowIpcEncoderSetMessageMetadata(struct ArrowIpcEncoder*
encoder,
const char* metadata,
struct ArrowError* error);
```
taking nanoarrow's packed metadata representation (the same format
`ArrowSchema.metadata` uses, so
`ArrowMetadataBuilder*` can produce it and no new representation is
introduced).
Decoder — an accessor for the metadata of the message whose header was last
decoded:
```c
ArrowErrorCode ArrowIpcDecoderGetMessageMetadata(struct ArrowIpcDecoder*
decoder,
const char** metadata_out,
struct ArrowError* error);
```
returning the same packed representation, borrowed from the decoder (valid
until the next
`ArrowIpcDecoderDecodeHeader`), so no ownership question arises.
Whether the higher-level `ArrowIpcArrayStreamReader` / `ArrowIpcWriter`
should surface this too is a
separate question — the `ArrowArrayStream` interface has nowhere to put
per-batch metadata, so
exposing it only at the encoder/decoder layer may be the right scope for a
first pass.
### Precedent
- Arrow C++ writes it via `RecordBatchWriter::WriteRecordBatch(batch,
custom_metadata)` and returns
it from `RecordBatchStreamReader::ReadNext()` as `RecordBatchWithMetadata`.
- arrow-go added the same in apache/arrow-go#669 ("support custom_metadata
on RecordBatch messages").
---
### Why I want this.
I have a DuckDB extension that speaks an RPC protocol over Arrow IPC
streams, where every
control signal (message type, cache-control directives, row provenance,
out-of-band payload
pointers) rides on per-message `custom_metadata`. That protocol is the
reason the extension links
Arrow C++ rather than nanoarrow — it is a ~22 MB static dependency for what
is otherwise IPC
read/write plus the C data interface.
I'm happy to workup a PR if this API seems like the right way to go.
Rusty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
