lidavidm commented on issue #13535:
URL: https://github.com/apache/arrow/issues/13535#issuecomment-1177580038
The metadata is actually part of the schema:
```
metadata : dict or Mapping, default None
Optional metadata for the schema (if schema not passed).
```
So when the writer is created with an explicit schema, the metadata is
dropped.
For example, this works:
```python
import tempfile
import pyarrow as pa
print("PyArrow version:", pa.__version__)
table = pa.table([[1, 2, 3, 4]], names=["ints"], metadata={"foo": "bar"})
print("Before")
print(table.to_string(show_metadata=True))
print()
with tempfile.NamedTemporaryFile() as path:
with pa.ipc.new_file(path, schema=table.schema) as sink:
sink.write_table(table)
with pa.ipc.open_file(path) as reader:
table2 = reader.read_all()
print("After")
print(table2.to_string(show_metadata=True))
```
```
PyArrow version: 8.0.0
Before
pyarrow.Table
ints: int64
-- schema metadata --
foo: 'bar'
After
pyarrow.Table
ints: int64
-- schema metadata --
foo: 'bar'
```
But if you give it a schema without the metadata, you lose the data:
```python
import tempfile
import pyarrow as pa
print("PyArrow version:", pa.__version__)
table = pa.table([[1, 2, 3, 4]], names=["ints"], metadata={"foo": "bar"})
print("Before")
print(table.to_string(show_metadata=True))
print()
with tempfile.NamedTemporaryFile() as path:
# Explicit schema here!
with pa.ipc.new_file(path, schema=pa.schema([("ints", pa.int64())])) as
sink:
sink.write_table(table)
with pa.ipc.open_file(path) as reader:
table2 = reader.read_all()
print("After")
print(table2.to_string(show_metadata=True))
```
```
PyArrow version: 8.0.0
Before
pyarrow.Table
ints: int64
-- schema metadata --
foo: 'bar'
After
pyarrow.Table
ints: int64
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
