rahulsmahadev opened a new pull request, #4455:
URL: https://github.com/apache/iceberg-python/pull/4455
# Rationale for this change
Bug: when every field of an Arrow table carries a field ID, `append` drops
the columns whose field ID is not in the table schema, without an error:
```python
schema = Schema(NestedField(1, "id", LongType()), NestedField(2, "x",
LongType()))
tbl = catalog.create_table("db.t", schema=schema)
with_z = Schema(*schema.fields, NestedField(99, "z", LongType()))
tbl.append(pa.Table.from_pylist([{"id": 1, "x": 2, "z": 3}],
schema=with_z.as_arrow()))
tbl.scan().to_arrow().to_pylist()
# [{'id': 1, 'x': 2}]
```
The same happens to a column that was dropped and added back: an Arrow
schema taken before `delete_column("x")` and `add_column("x", LongType())` has
`x` with field ID 2, the new `x` has field ID 3, and the appended values are
stored as null. `overwrite` and `dynamic_partition_overwrite` lose these
columns too, and `upsert` can lose them on inserts or when updating with a
stale ID. Without field IDs, the extra `z` is already rejected (`PyArrow table
contains more columns: z`). Fokko raised the same concern on #4419
([review](https://github.com/apache/iceberg-python/pull/4419#pullrequestreview-5966591141)):
silently dropping a column on append loses data without any signal.
Fix: the four write paths now reject input field IDs that are not in the
table schema, before anything is written:
```
ValueError: PyArrow table contains columns whose field IDs are not in the
table schema: z (field ID 99).
```
`add_files` is unchanged (#4359). Note that #4444 currently tests that
`upsert` accepts such a stale ID; with this change it raises.
Java: Spark's write builder gives the input the table's field IDs by name
(`SparkSchemaUtil.convert` -> `TypeUtil.reassignIds`) and fails on unknown
columns (`Field z not found in source schema`) unless merge-schema adds them
first. PyIceberg has no merge-schema option on write and keeps the input's own
IDs when every field has one, so it rejects the stale `x` instead of writing it
to the new `x`.
## Are these changes tested?
Yes, `test_write_rejects_field_ids_not_in_table_schema` in
`tests/catalog/test_catalog_behaviors.py` runs both cases through the four
write APIs on an evolved table, checks that nothing is committed, and that
appends with a renamed column's old name, reordered, promoted, missing optional
and ID-less columns still work. It fails on main with `DID NOT RAISE
ValueError` and passes with this change.
`test_schema_mismatch_unknown_field_ids` in `tests/io/test_pyarrow.py` covers
struct fields, list elements and map values.
## Are there any user-facing changes?
Yes, writes now reject unknown field IDs, including dropped columns kept in
Arrow data read from older Parquet files.
Please add the `changelog` label.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]