hamarshehmhmd commented on issue #55298: URL: https://github.com/apache/arrow/issues/55298#issuecomment-6474844678
<!-- https://github.com/apache/arrow/issues/55298 --> I can reproduce this on pyarrow 26.0.0 (pip wheel). All five value types fail inside a list with `ArrowNotImplementedError: Not implemented type for Arrow list to pandas: ...`, both through `Array.to_pandas()` and through `Table.to_pandas()`. Here is a workaround until the `ListTypeSupported` fix ships. Cast the nested value types to their supported equivalents first. The pandas result is identical: object arrays of `str`/`bytes`/`Decimal`, with the same nulls. ```python import pyarrow as pa import pyarrow.types as pt def _compat(t): if pt.is_string_view(t): return pa.large_string() if pt.is_binary_view(t) or pt.is_fixed_size_binary(t): return pa.large_binary() if pt.is_decimal32(t) or pt.is_decimal64(t): return pa.decimal128(t.precision, t.scale) if pt.is_list(t) or pt.is_large_list(t): v = t.value_field return (pa.list_ if pt.is_list(t) else pa.large_list)(v.with_type(_compat(v.type))) if pt.is_struct(t): return pa.struct([f.with_type(_compat(f.type)) for f in t]) return t def pandas_compatible(tbl: pa.Table) -> pa.Table: """Cast only the nested columns that to_pandas() can't handle yet.""" schema = pa.schema( [f.with_type(_compat(f.type)) if pt.is_nested(f.type) else f for f in tbl.schema], metadata=tbl.schema.metadata, ) return tbl.cast(schema) df = pandas_compatible(tbl).to_pandas() ``` It only rewrites nested columns (lists, including lists inside structs), so top-level `string_view` columns still go through the normal conversion. I tested it on a table with `list<string_view>`, `large_list<binary_view>`, `list<decimal32(5, 1)>`, `list<fixed_size_binary[2]>` and `struct<x: list<string_view>>` columns. `to_pandas()` raises on the original table, and after the cast it returns the same values, nulls included, as the `string`/`binary`/`decimal128` versions. For a single array, `arr.cast(pa.list_(pa.string())).to_pandas()` is enough. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
