hedger9487 opened a new pull request, #3843: URL: https://github.com/apache/iceberg-python/pull/3843
### Description Fixes #3835. When appending, overwriting, or upserting dataframes containing dictionary-encoded columns (`pa.DictionaryArray`), `ArrowProjectionVisitor._cast_if_needed` did not cast dictionary arrays to the target schema type (e.g. `pa.string()`). As a result, newly written data files retain dictionary types while existing files contain plain strings, causing subsequent `table.scan().to_arrow()` or upsert table concats to fail with: ```text pyarrow.lib.ArrowTypeError: Unable to merge: Field name has incompatible types: dictionary<values=string, indices=int32, ordered=0> vs string ``` This PR updates `ArrowProjectionVisitor._cast_if_needed` in `pyiceberg/io/pyarrow.py` to cast `DictionaryArray` columns to the target type when the target is not a dictionary type. ### Testing - Added unit test `test_upsert_dictionary_encoded_columns` in `tests/table/test_upsert.py`. - All 24 upsert tests and full test suites pass locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
