JooyoungJeong opened a new pull request, #3850: URL: https://github.com/apache/iceberg-python/pull/3850
Closes #2467 # Rationale for this change `Transaction.upsert()` pins the target branch with `use_ref(branch)`, which sets a snapshot id, so `TableScan.projection()` resolves the schema recorded on that snapshot rather than the table's current schema. A schema-only update creates no data snapshot, so the branch tip can still carry the pre-update schema, and the matched rows are then read with a schema that no longer matches the input dataframe. An added column surfaces as `ValueError: Target schema's field names are not matching the table's field names` in `get_rows_to_update()`. A renamed non-key column fails the same way, and a renamed join column fails earlier still, in `has_duplicate_rows()`, with `KeyError`. The change is one call site in `pyiceberg/table/__init__.py`: the scan that reads the matched rows now projects the current table schema. File planning is untouched and still uses the pinned snapshot, so time-travel and ref scan semantics do not change. `ArrowScan` already binds the row filter against `table_metadata.schema()`, so the projection was the only part still tied to the snapshot's historical schema. ## Are these changes tested? Yes. Four cases added to `tests/table/test_upsert.py`, each of which fails on `main`: - `test_upsert_after_adding_column` - `test_upsert_after_renaming_column` - `test_upsert_after_renaming_join_column` - `test_upsert_after_adding_column_in_transaction` `uv run pytest tests/table/test_upsert.py -q` → 27 passed. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
