rahulsmahadev opened a new pull request, #4424:
URL: https://github.com/apache/iceberg-python/pull/4424

   # Rationale for this change
   
   Follow-up to #4389.
   
   Bug: `StructType.field_by_name(name, case_sensitive=False)` returns the 
first match when a struct has two fields that differ only by case, like `ID` 
and `id`. So a case-insensitive `union_by_name` can silently pick the wrong 
column:
   
   ```python
   table = catalog.create_table("default.t", Schema(NestedField(1, "ID", 
LongType()), NestedField(2, "id", LongType())))
   with table.update_schema(case_sensitive=False) as update:
       update.union_by_name(Schema(NestedField(1, "id", LongType())))
   ```
   
   Fix: raise `ValueError: Cannot build lower case index: ID and id collide`, 
the same message as #4389. Java fails here too: 
`Types.StructType.caseInsensitiveField` builds its index with 
`ImmutableMap.Builder`, which throws on duplicate keys.
   
   Remaining differences from Java:
   - Java caches the lower-case index per struct. This PR rebuilds it on every 
lookup, which is slower on very wide schemas. Caching can be a follow-up.
   - Java also validates the whole schema at the end of a schema update 
(`SchemaUpdate.applyChanges`). PyIceberg doesn't, so for example a union that 
adds a new struct with both `ID` and `id` still succeeds.
   
   ## Are these changes tested?
   
   Yes, added `test_struct_field_by_name_case_insensitive_duplicates` in 
`tests/test_types.py` and `test_union_by_name_case_insensitive_duplicates` in 
`tests/catalog/test_catalog_behaviors.py`, which runs on the memory, sql and 
sql_without_rowcount catalogs. Both fail on main with `DID NOT RAISE 
ValueError` and pass with this change.
   
   ## Are there any user-facing changes?
   
   Yes, case-insensitive lookups on such a struct now raise `ValueError` 
instead of returning the first field, and so does a case-insensitive 
`union_by_name` that looks up a field in it. Case-sensitive lookups don't 
change.
   
   Please add the `changelog` label.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to