Anurag-S1ngh opened a new pull request, #4342: URL: https://github.com/apache/iceberg-python/pull/4342
# Rationale for this change A table can hold files with plain string columns and files with dictionary-encoded string columns, for example after a writer started passing dictionary-encoded Arrow tables. `ArrowScan.to_table` concatenates one table per batch with `promote_options="permissive"`, which promotes large types but cannot merge a plain string column with a dictionary-encoded one, so the scan failed with: ``` pyarrow.lib.ArrowTypeError: Unable to merge: Field col has incompatible types: string vs dictionary<values=string, indices=int32, ordered=0> ``` Batches are now normalized before concatenation. Dictionary-encoded columns that were not requested through `ArrowScan(dictionary_columns=...)` are decoded to their value type, so every batch shares one logical schema. Columns that were explicitly requested as dictionary-encoded keep their dictionary type, which the existing `dictionary_columns` behavior and its test rely on. # Are these changes tested? Yes. `tests/catalog/test_catalog_behaviors.py::test_scan_mixed_dictionary_and_plain_string_batches` creates a table, appends a dictionary-encoded file and then a plain file, and asserts the scan succeeds, returns the plain string type, and returns both rows. It runs against the memory, sql, and sql_without_rowcount catalogs and fails on main. The existing `tests/io/test_pyarrow.py::test_dictionary_columns_produces_dict_encoded_output` still passes, which guards that requested dictionary columns are not decoded. Full unit suite: `4563 passed, 5 skipped`. `prek run -a` passes. # Are there any user-facing changes? Yes. Scanning a table whose files mix dictionary-encoded and plain columns now works instead of raising `ArrowTypeError`. Scans that request `dictionary_columns` still return dictionary-encoded arrays for those columns. Closes #3260 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
