rahulsmahadev opened a new pull request, #4389:
URL: https://github.com/apache/iceberg-python/pull/4389
# Rationale for this change
A case-insensitive lookup on a schema with two columns whose names differ
only by case uses the last one. `Schema._lazy_name_to_id_lower` builds the
lower-case index with a dict comprehension, so `id` overwrites `ID`, and
`Schema.find_field` and `Schema.select` use that index for case-insensitive
name lookups:
```python
# columns ID (field 1) and id (field 2); rows (ID, id): (5, 7) and (6, 5)
tbl.scan(row_filter=EqualTo("ID", 5), case_sensitive=False).to_arrow() #
returns (6, 5), the row whose id is 5
tbl.delete("ID = 5", case_sensitive=False) # removes (6, 5); (5, 7) is kept
```
The same wrong column is used by sort order updates and by schema updates
(`delete_column("ID")` drops `id`, `rename_column("ID", "key")` renames `id`).
Java does not pick a column here. `TypeUtil.indexByLowerCaseName` throws
`Cannot build lower case index: ID and id collide`, and
`Schema.caseInsensitiveFindField` and `SchemaUpdate` use that index, so the
lookup fails.
This change makes the index raise `ValueError: Cannot build lower case
index: ID and id collide` (the same message as Java) when two different field
IDs share a lower-case name. As in Java's `Schema` lookup, building the
lower-case index fails for any requested name when the schema contains a
collision, including when the requested name is unambiguous. Names that map to
the same field ID, such as the short name of a list element struct field, are
not collisions. Case-sensitive lookups do not change.
`StructType.field_by_name` uses a separate lookup and is outside this change;
it can still return the first matching field (it serves `union_by_name`).
Callers that treated any `ValueError` from a lookup as "not found"
(`UpdateSchema.add_column`, the move methods, the identifier field check on
commit, and the metadata column check in scan projection) now use a private
lookup that returns `None` for a missing name and still raises for an ambiguous
one. Without updating the add-column check, an index-only fix would let
`table.update_schema(case_sensitive=False).add_column("Id",
LongType()).commit()` add a third case variant.
## Are these changes tested?
Yes. `test_case_insensitive_operations_raise_on_case_variant_columns` in
`tests/table/test_init.py` runs a delete, a scan, a sort order update, and
schema updates with `case_sensitive=False` on a table with `ID` and `id`, for
the memory, sql and sql_without_rowcount catalogs. Each must raise, and the
table must be unchanged afterwards. It fails on main, where the delete does not
raise. `test_case_sensitive_and_unambiguous_operations_are_unchanged` checks
exact-case lookups on the same table and a case-insensitive delete on a table
without a collision; it passes on main and with this change. In
`tests/test_schema.py`,
`test_schema_case_insensitive_lookup_of_case_variant_names_raises` and
`test_schema_case_insensitive_lookup_of_nested_case_variant_names_raises` fail
on main with `DID NOT RAISE`, and
`test_schema_case_insensitive_lookup_of_list_element_struct_field` covers the
same-ID short names and passes on main and with this change.
`test_schema_case_insensitive_lookup_rejects_co
llision_for_any_name`,
`test_metadata_column_projection_propagates_case_collision` and
`test_identifier_field_lookup_propagates_case_collision` cover unambiguous and
missing names, the metadata column projection and the identifier field check;
they fail on main.
## Are there any user-facing changes?
Yes: case-insensitive name lookups through `Schema.find_field` and
`Schema.select` now raise `ValueError` when the schema has names that differ
only by case, instead of silently using the last column. Case-sensitive lookups
are unchanged.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]