willfrey opened a new issue, #53595:
URL: https://github.com/apache/arrow/issues/53595
### Describe the bug, including details regarding any error messages,
version, and platform.
`pyarrow.parquet.write_table` refuses a table with a sparse union column,
whether the union is at the top level or nested in a list or struct:
```python
import io
import pyarrow as pa
import pyarrow.parquet as pq
union = pa.UnionArray.from_sparse(
pa.array([0, 1], type=pa.int8()),
[pa.array([1, None]), pa.array([None, "x"])],
field_names=["a", "b"],
)
pq.write_table(pa.table({"u": union}), io.BytesIO())
```
```
pyarrow.lib.ArrowNotImplementedError: Unhandled type for Arrow to Parquet
schema conversion: sparse_union<a: int64=0, b: string=1>
```
`pa.ListArray.from_arrays([0, 2], union)` and
`pa.StructArray.from_arrays([union], names=["f"])` raise the same error.
pyarrow 26.0.0 and 25.0.1, Python 3.14, macOS arm64.
#28288 reported the same error for dense unions and was closed as stale
without a resolution.
Parquet has no union type, but a sparse union's children are each as long as
the union, so it maps onto a group of its children plus a type-code column. The
stored Arrow schema (`ARROW:schema`) could then restore the union on read, as
it already restores `duration` written as `INT64`. If write support is out of
scope, documenting the limitation would help.
This issue was drafted with an AI assistant, which also ran the reproducer
on pyarrow 26.0.0 and 25.0.1. I reviewed it before filing.
### Component(s)
Parquet, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
