[
https://issues.apache.org/jira/browse/ARROW-16131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17518114#comment-17518114
]
David Li commented on ARROW-16131:
----------------------------------
Alright. I don't think there's anything Flight-specific, as indicated in the
other ticket, but we can use it to double-check or add some tests on that side.
> [C++] Record batch specific metadata is not saved in IPC file
> -------------------------------------------------------------
>
> Key: ARROW-16131
> URL: https://issues.apache.org/jira/browse/ARROW-16131
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Affects Versions: 7.0.0
> Reporter: Yue Ni
> Assignee: Yue Ni
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When writing an IPC file having multiple record batches, the schema provided
> to `IpcFormatWriter` is correctly written to IPC file's footer, however, if
> the record batch written has its batch specific metadata associated with it,
> this metadata is not written.
> This can be reproduced with the following test case (using pyarrow):
> {code:java}
> def test_chunked_record_batch_meta():
> num_batches = 2
> ipc_file = "/tmp/batches_with_metadata.arrow"
> int_array = pa.array([i for i in range(chunk_size)])
> schema = pa.schema(
> [
> ("values", pa.int64()),
> ],
> metadata={"foo": "bar"},
> )
> writer = pa.RecordBatchFileWriter(
> ipc_file, schema
> )
> for i in range(num_batches):
> # follow examples here:
> #
> https://github.com/apache/arrow/blob/master/python/pyarrow/tests/test_table.py
> batch = pa.record_batch(
> [int_array],
> names=["values"],
> metadata={"batch_id": str},
> )
> writer.write_batch(batch)
> writer.close()
> mmapped_file = pa.memory_map(ipc_file)
> reader = pa.ipc.open_file(mmapped_file)
> batch_0 = reader.get_record_batch(0)
> assert batch_0.schema.metadata {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
