[
https://issues.apache.org/jira/browse/ARROW-16131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17518080#comment-17518080
]
David Li commented on ARROW-16131:
----------------------------------
Is this different from ARROW-6940 as mentioned on the mailing list?
> record batch specific metadata is not saved in IPC file
> -------------------------------------------------------
>
> Key: ARROW-16131
> URL: https://issues.apache.org/jira/browse/ARROW-16131
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Affects Versions: 7.0.0
> Reporter: Yue Ni
> Priority: Major
>
> When writing an IPC file having multiple record batches, the schema provided
> to `IpcFormatWriter` is correctly written to IPC file's footer, however, if
> the record batch written has its batch specific metadata associated with it,
> this metadata is not written.
> This can be reproduced with the following test case (using pyarrow):
> {code:java}
> def test_chunked_record_batch_meta():
> num_batches = 2
> ipc_file = "/tmp/batches_with_metadata.arrow"
> int_array = pa.array([i for i in range(chunk_size)])
> schema = pa.schema(
> [
> ("values", pa.int64()),
> ],
> metadata={"foo": "bar"},
> )
> writer = pa.RecordBatchFileWriter(
> ipc_file, schema
> )
> for i in range(num_batches):
> # follow examples here:
> #
> https://github.com/apache/arrow/blob/master/python/pyarrow/tests/test_table.py
> batch = pa.record_batch(
> [int_array],
> names=["values"],
> metadata={"batch_id": str},
> )
> writer.write_batch(batch)
> writer.close()
> mmapped_file = pa.memory_map(ipc_file)
> reader = pa.ipc.open_file(mmapped_file)
> batch_0 = reader.get_record_batch(0)
> assert batch_0.schema.metadata {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
