rahulsmahadev opened a new pull request, #4389:
URL: https://github.com/apache/iceberg-python/pull/4389

   # Rationale for this change
   
   A case-insensitive lookup on a schema with two columns whose names differ 
only by case uses the last one. `Schema._lazy_name_to_id_lower` builds the 
lower-case index with a dict comprehension, so `id` overwrites `ID`, and 
`Schema.find_field` and `Schema.select` use that index for case-insensitive 
name lookups:
   
   ```python
   # columns ID (field 1) and id (field 2); rows (ID, id): (5, 7) and (6, 5)
   tbl.scan(row_filter=EqualTo("ID", 5), case_sensitive=False).to_arrow()  # 
returns (6, 5), the row whose id is 5
   tbl.delete("ID = 5", case_sensitive=False)  # removes (6, 5); (5, 7) is kept
   ```
   
   The same wrong column is used by sort order updates and by schema updates 
(`delete_column("ID")` drops `id`, `rename_column("ID", "key")` renames `id`).
   
   Java does not pick a column here. `TypeUtil.indexByLowerCaseName` throws 
`Cannot build lower case index: ID and id collide`, and 
`Schema.caseInsensitiveFindField` and `SchemaUpdate` use that index, so the 
lookup fails.
   
   This change makes the index raise `ValueError: Cannot build lower case 
index: ID and id collide` (the same message as Java) when two different field 
IDs share a lower-case name. As in Java's `Schema` lookup, building the 
lower-case index fails for any requested name when the schema contains a 
collision, including when the requested name is unambiguous. Names that map to 
the same field ID, such as the short name of a list element struct field, are 
not collisions. Case-sensitive lookups do not change. 
`StructType.field_by_name` uses a separate lookup and is outside this change; 
it can still return the first matching field (it serves `union_by_name`).
   
   Callers that treated any `ValueError` from a lookup as "not found" 
(`UpdateSchema.add_column`, the move methods, the identifier field check on 
commit, and the metadata column check in scan projection) now use a private 
lookup that returns `None` for a missing name and still raises for an ambiguous 
one. Without updating the add-column check, an index-only fix would let 
`table.update_schema(case_sensitive=False).add_column("Id", 
LongType()).commit()` add a third case variant.
   
   ## Are these changes tested?
   
   Yes. `test_case_insensitive_operations_raise_on_case_variant_columns` in 
`tests/table/test_init.py` runs a delete, a scan, a sort order update, and 
schema updates with `case_sensitive=False` on a table with `ID` and `id`, for 
the memory, sql and sql_without_rowcount catalogs. Each must raise, and the 
table must be unchanged afterwards. It fails on main, where the delete does not 
raise. `test_case_sensitive_and_unambiguous_operations_are_unchanged` checks 
exact-case lookups on the same table and a case-insensitive delete on a table 
without a collision; it passes on main and with this change. In 
`tests/test_schema.py`, 
`test_schema_case_insensitive_lookup_of_case_variant_names_raises` and 
`test_schema_case_insensitive_lookup_of_nested_case_variant_names_raises` fail 
on main with `DID NOT RAISE`, and 
`test_schema_case_insensitive_lookup_of_list_element_struct_field` covers the 
same-ID short names and passes on main and with this change. 
`test_schema_case_insensitive_lookup_rejects_co
 llision_for_any_name`, 
`test_metadata_column_projection_propagates_case_collision` and 
`test_identifier_field_lookup_propagates_case_collision` cover unambiguous and 
missing names, the metadata column projection and the identifier field check; 
they fail on main.
   
   ## Are there any user-facing changes?
   
   Yes: case-insensitive name lookups through `Schema.find_field` and 
`Schema.select` now raise `ValueError` when the schema has names that differ 
only by case, instead of silently using the last column. Case-sensitive lookups 
are unchanged.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to