rahulsmahadev opened a new pull request, #4424:
URL: https://github.com/apache/iceberg-python/pull/4424
# Rationale for this change
Follow-up to #4389.
Bug: `StructType.field_by_name(name, case_sensitive=False)` returns the
first match when a struct has two fields that differ only by case, like `ID`
and `id`. So a case-insensitive `union_by_name` can silently pick the wrong
column:
```python
table = catalog.create_table("default.t", Schema(NestedField(1, "ID",
LongType()), NestedField(2, "id", LongType())))
with table.update_schema(case_sensitive=False) as update:
update.union_by_name(Schema(NestedField(1, "id", LongType())))
```
Fix: raise `ValueError: Cannot build lower case index: ID and id collide`,
the same message as #4389. Java fails here too:
`Types.StructType.caseInsensitiveField` builds its index with
`ImmutableMap.Builder`, which throws on duplicate keys.
Remaining differences from Java:
- Java caches the lower-case index per struct. This PR rebuilds it on every
lookup, which is slower on very wide schemas. Caching can be a follow-up.
- Java also validates the whole schema at the end of a schema update
(`SchemaUpdate.applyChanges`). PyIceberg doesn't, so for example a union that
adds a new struct with both `ID` and `id` still succeeds.
## Are these changes tested?
Yes, added `test_struct_field_by_name_case_insensitive_duplicates` in
`tests/test_types.py` and `test_union_by_name_case_insensitive_duplicates` in
`tests/catalog/test_catalog_behaviors.py`, which runs on the memory, sql and
sql_without_rowcount catalogs. Both fail on main with `DID NOT RAISE
ValueError` and pass with this change.
## Are there any user-facing changes?
Yes, case-insensitive lookups on such a struct now raise `ValueError`
instead of returning the first field, and so does a case-insensitive
`union_by_name` that looks up a field in it. Case-sensitive lookups don't
change.
Please add the `changelog` label.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]