jiayuasu opened a new issue, #26331:
URL: https://github.com/apache/datafusion/issues/26331
### Describe the bug
Unnesting a list drops the list item field's metadata. When the item field
carries an Arrow extension type (or any other field metadata), the unnested
column comes back as the plain storage type. Unnesting a struct keeps each
child field's metadata, so the two cases behave differently.
### To Reproduce
```python
import datafusion
import pyarrow as pa
# A list whose item field carries metadata, here an Arrow extension type
item = pa.field(
"item",
pa.binary(),
metadata={"ARROW:extension:name": "geoarrow.wkb",
"ARROW:extension:metadata": "{}"},
)
table = pa.table({"values": pa.array([[b"\x01", b"\x02"]],
type=pa.list_(item))})
ctx = datafusion.SessionContext()
ctx.register_record_batches("t", [table.to_batches()])
listed = ctx.sql("SELECT values FROM t")
print(listed.schema().field("values").type.value_field.metadata)
# {b'ARROW:extension:metadata': b'{}', b'ARROW:extension:name':
b'geoarrow.wkb'}
unnested = ctx.sql("SELECT unnest(values) AS value FROM t")
print(unnested.schema().field("value").metadata)
# None: the item field's metadata is dropped
```
(datafusion-python 55.0.0; the same happens in Rust with DataFusion 54.1 and
55.1.)
### Expected behavior
The unnested column keeps the item field's metadata, as the struct case
already does.
### Additional context
In `get_unnested_columns` (`datafusion/expr/src/logical_plan/plan.rs`), the
list case builds a new field from the item's data type:
```rust
let new_field = Arc::new(Field::new(col_name, data_type, true));
```
while the struct case clones each child field
(`f.as_ref().clone().with_name(new_name)`), which keeps its metadata. For a
depth-1 unnest, reusing the item field with the new name (and `nullable =
true`) would keep it.
We hit this in [Apache SedonaDB](https://github.com/apache/sedona-db), where
geometry columns are a GeoArrow extension type: `UNNEST` of a list of
geometries returns plain binary rather than geometry, losing the spatial type
and CRS (apache/sedona-db#1429).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
