DG47 opened a new pull request, #4121: URL: https://github.com/apache/iceberg-python/pull/4121
Closes #4120 # Rationale for this change `string_view` and `binary_view` columns are accepted on write, and appending them to an unpartitioned table works. Appending them to a partitioned table fails in `_determine_partitions`, because PyArrow has no group by, filter or take kernels for view types, and the pyiceberg-core bucket and truncate transforms do not accept them either. `pa.table(polars_df)` produces exactly this shape, since Polars exports strings and binaries as view types over the PyCapsule interface. Depending on the spec, the append fails with `Keys of type string_view`, `Function 'array_filter' has no kernel matching input types (string_view, bool)`, or `Unsupported data type for bucket transform: Utf8View`. This change casts view types, including ones nested in structs, lists and maps, to `large_string` / `large_binary` before the table is split into partitions. Tables without view types are not cast. The data files are written the same as for `large_string` / `large_binary` input. One remaining gap is outside this repo: truncate on a binary column already fails for `large_binary` input in pyiceberg-core (apache/iceberg-rust#3119), and now `binary_view` input hits that same error. ## Are these changes tested? Yes: - `test_determine_partitions_with_view_types`: identity, bucket and truncate partitions on a `string_view` column, with `binary_view` and nested view columns alongside. - `test_append_view_types_to_partitioned_table`: end-to-end append to a partitioned in-memory table, then read back. Both fail on `main` and pass with this change, on pyarrow 25.0.1 and on the minimum supported pyarrow 18.1.0. ## Are there any user-facing changes? Yes. Appending, overwriting, and dynamically overwriting partitioned tables with Arrow view type columns now works where it used to raise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
