rahulsmahadev opened a new pull request, #4455:
URL: https://github.com/apache/iceberg-python/pull/4455

   # Rationale for this change
   
   Bug: when every field of an Arrow table carries a field ID, `append` drops 
the columns whose field ID is not in the table schema, without an error:
   
   ```python
   schema = Schema(NestedField(1, "id", LongType()), NestedField(2, "x", 
LongType()))
   tbl = catalog.create_table("db.t", schema=schema)
   with_z = Schema(*schema.fields, NestedField(99, "z", LongType()))
   tbl.append(pa.Table.from_pylist([{"id": 1, "x": 2, "z": 3}], 
schema=with_z.as_arrow()))
   tbl.scan().to_arrow().to_pylist()
   # [{'id': 1, 'x': 2}]
   ```
   
   The same happens to a column that was dropped and added back: an Arrow 
schema taken before `delete_column("x")` and `add_column("x", LongType())` has 
`x` with field ID 2, the new `x` has field ID 3, and the appended values are 
stored as null. `overwrite` and `dynamic_partition_overwrite` lose these 
columns too, and `upsert` can lose them on inserts or when updating with a 
stale ID. Without field IDs, the extra `z` is already rejected (`PyArrow table 
contains more columns: z`). Fokko raised the same concern on #4419 
([review](https://github.com/apache/iceberg-python/pull/4419#pullrequestreview-5966591141)):
 silently dropping a column on append loses data without any signal.
   
   Fix: the four write paths now reject input field IDs that are not in the 
table schema, before anything is written:
   
   ```
   ValueError: PyArrow table contains columns whose field IDs are not in the 
table schema: z (field ID 99).
   ```
   
   `add_files` is unchanged (#4359). Note that #4444 currently tests that 
`upsert` accepts such a stale ID; with this change it raises.
   
   Java: Spark's write builder gives the input the table's field IDs by name 
(`SparkSchemaUtil.convert` -> `TypeUtil.reassignIds`) and fails on unknown 
columns (`Field z not found in source schema`) unless merge-schema adds them 
first. PyIceberg has no merge-schema option on write and keeps the input's own 
IDs when every field has one, so it rejects the stale `x` instead of writing it 
to the new `x`.
   
   ## Are these changes tested?
   
   Yes, `test_write_rejects_field_ids_not_in_table_schema` in 
`tests/catalog/test_catalog_behaviors.py` runs both cases through the four 
write APIs on an evolved table, checks that nothing is committed, and that 
appends with a renamed column's old name, reordered, promoted, missing optional 
and ID-less columns still work. It fails on main with `DID NOT RAISE 
ValueError` and passes with this change. 
`test_schema_mismatch_unknown_field_ids` in `tests/io/test_pyarrow.py` covers 
struct fields, list elements and map values.
   
   ## Are there any user-facing changes?
   
   Yes, writes now reject unknown field IDs, including dropped columns kept in 
Arrow data read from older Parquet files.
   
   Please add the `changelog` label.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to