hedger9487 opened a new pull request, #3849: URL: https://github.com/apache/iceberg-python/pull/3849
### Description Fixes #3260. When an Iceberg table contains mixed data files where some files contain dictionary-encoded string columns (e.g. written by clients using dictionary encoding) and other files contain plain strings (e.g. after Athena/Trino `OPTIMIZE` data compaction), `ArrowScan.to_table()` fails during batch concatenation with: ```text pyarrow.lib.ArrowTypeError: Unable to merge: Field col has incompatible types: string vs dictionary<values=string, indices=int32, ordered=0> ``` This occurs because `ArrowProjectionVisitor._cast_if_needed()` did not cast primitive dictionary-encoded arrays to the target schema type when projecting batches unless explicitly requested in `dictionary_columns`. This PR updates `_cast_if_needed()` in `pyiceberg/io/pyarrow.py` to cast `DictionaryType` arrays to the target Iceberg primitive schema type when the column is not in `dictionary_columns`. ### Testing - Added regression test `test_arrow_scan_mixed_dict_encoded_and_plain_strings` in `tests/io/test_pyarrow.py`. - All dictionary and scan projection unit tests pass locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
