hzmsh commented on issue #3260:
URL:
https://github.com/apache/iceberg-python/issues/3260#issuecomment-6198820440
This still reproduces on `main` (b32392dc) and 0.12.0. A common way to hit
it is a pandas `Categorical` column, followed by a normal append:
```python
import pandas as pd, pyarrow as pa
# table has a string column `status`
tbl.append(pa.Table.from_pandas(pd.DataFrame({"id": [1, 2], "status":
pd.Categorical(["new", "done"])}), preserve_index=False))
tbl.append(pa.Table.from_pandas(pd.DataFrame({"id": [3], "status":
["new"]}), preserve_index=False))
tbl.scan().to_arrow() # ArrowTypeError: Unable to merge: Field status has
incompatible types: large_string vs dictionary<values=string, indices=int8, ...>
tbl.scan().to_pandas() # same
tbl.scan().to_polars() # same
```
`to_arrow_batch_reader()` works because it casts each batch to the table
schema, but `to_table` concatenates the per-file tables as they are.
@Fokko, would you be open to a fix in `ArrowScan.to_table` that decodes
dictionary columns to plain strings before concatenating, unless the column is
listed in `dictionary_columns`? This matches `to_arrow_batch_reader()`, and
Iceberg has no dictionary type, so `string` is what the schema says. It is a
small change (about 10 lines plus tests).
One behavior change to flag: a table where every file was written from
categorical data currently comes back as `category` in `to_pandas()`. With this
fix it would come back as a plain string dtype unless the column is passed in
`dictionary_columns`.
The same error came up in #2250, where you asked what outcome users expect;
the answer there was the type from the table schema, which is what this fix
returns. Three earlier PRs for this (#3444, #3667, #3849) were closed as stale
without review. If this direction works for you, I'm happy to open a PR and
credit those authors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
