orlp opened a new issue, #47893:
URL: https://github.com/apache/arrow/issues/47893
### Describe the bug, including details regarding any error messages,
version, and platform.
I'm not sure whether the Parquet spec is overly restrictive or pyarrow
overly permissive, but pyarrow allows writing arbitrary bytestrings as
metadata. But in Parquet the metadata is a `string:string` mapping. A modified
extension type example which will write illegal bytes into the Parquet metadata:
```python
import pyarrow as pa
import pyarrow.parquet as pq
class PeriodType(pa.ExtensionType):
def __init__(self, freq):
self._freq = freq
super().__init__(pa.int64(), "my_package.period")
@property
def freq(self):
return self._freq
def __arrow_ext_serialize__(self):
return b"\xff" + "freq={}".format(self.freq).encode() + b"\xff"
@classmethod
def __arrow_ext_deserialize__(cls, storage_type, serialized):
print("deserializing:", serialized)
assert serialized[0] == 0xff and serialized[-1] == 0xff
serialized = serialized[1:-1].decode()
assert serialized.startswith("freq=")
freq = serialized.split("=")[1]
return PeriodType(freq)
pa.register_extension_type(PeriodType(0.0))
storage = pa.array([1, 2, 3], pa.int64())
arr = pa.ExtensionArray.from_storage(PeriodType(0.0), storage)
tab = pa.Table.from_arrays([arr], names=["periods"])
pq.write_table(tab, 'example.parquet')
print(pq.read_table('example.parquet').schema)
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
