rahulsmahadev opened a new pull request, #4410:
URL: https://github.com/apache/iceberg-python/pull/4410
# Rationale for this change
`update_column` checks a type change with `promote()`, the read-side rule
that the Avro and Parquet readers use. That rule is more lenient than the table
spec: it also accepts `string -> binary`, `binary -> string` and `fixed(16) ->
uuid`. So a populated `string` column can be changed to `binary` without any
warning, and the commit succeeds:
```python
tbl.update_schema().update_column("name", field_type=BinaryType()).commit()
# name: string
```
The spec's "Valid primitive type promotions" table does not list these
pairs. Java's `SchemaUpdate.updateColumn` rejects them through
`TypeUtil.isPromotionAllowed`, which allows equal types, `int -> long`, `float
-> double` and decimal widening with the same scale.
This PR validates primitive type changes using Java's v1/v2 promotion rules,
while keeping PyIceberg's existing `unknown -> primitive` behaviour. It does
not add v3 `date -> timestamp` or `date -> timestamp_ns` support, and it does
not broaden unknown-type conversions. Validation runs when `field_type` is
supplied and `allow_incompatible_changes` is false. `promote()` is unchanged,
because the Avro resolver, the Arrow projection, the `union_by_name` no-op
check and append compatibility (`_check_schema_compatible`) still rely on its
leniency.
Behaviour change: without `allow_incompatible_changes=True`, `update_column`
warns before applying `string -> binary`, `binary -> string` or `fixed[16] ->
uuid`. The warning is a `DeprecationWarning` that says the change is deprecated
in 0.13.0 and will be removed in 0.14.0. A subsequent release will require the
explicit opt-in. Every other invalid change still raises `ValidationError`, as
on `main`. `allow_incompatible_changes=True` applies these three changes
without a warning.
`union_by_name` warns when an incoming UUID field replaces an existing
`fixed[16]` field, including nested struct fields, list elements and map
values. String/binary unions remain unchanged, because `union_by_name` checks
them with `promote()` before it calls `update_column`. An incoming `fixed[16]`
field over an existing UUID field also remains a no-op.
Precedent: Fokko added the string <-> binary allowed-update tests in
apache/iceberg#8374 (September 2023), and apache/iceberg-python#207 only moved
that test file. #986 added `test_table_scan_default_to_large_types`, which
changed a string column to binary with `update_column`. #1669 later changed
that test to create the column as binary, so the update became a no-op.
Alternative for maintainers: reject these three changes immediately instead
of warning first. apache/iceberg-python#3613 rejected decimal promotions that
change the scale without a deprecation cycle, and Fokko approved it. This PR
chooses a warning cycle, because these changes were tested as supported
(apache/iceberg#8374), and AGENTS.md asks for a deprecation cycle before
breaking public API. Choosing immediate rejection means replacing the warning
branch in `update_column` with a `ValidationError`.
Other engines: in a local DuckDB 2.0.0 `iceberg_scan` reproduction,
promoting a populated string column to binary made a non-ASCII value fail to
read, and changed the four-character string `\x41` to the single byte `A`.
PyIceberg continued to read the original UTF-8 bytes.
## Are these changes tested?
Yes. `tests/test_schema.py` covers the following.
- `test_update_column_rejects_invalid_promotions`: the exact error message
for each invalid change (for example `int -> float`, a decimal scale change and
`fixed[15] -> uuid`), unchanged table metadata after the rejections, and the
valid promotions committed and read back after a reload.
- `test_update_column_warns_for_non_spec_promotions`: the three deprecated
changes warn with the exact message, commit, and read back after a reload.
- `test_update_column_no_warning_with_allow_incompatible_changes`: the
explicit opt-in applies the same three changes without a warning.
- `test_union_by_name_fixed_uuid_and_string_binary`: `fixed(16)` and UUID in
both directions, string/binary unchanged, and a list element.
- `test_update_column_promotes_unknown_to_primitive_only`: the existing
`unknown` behaviour, on a v3 table.
On `main`, the three warning tests fail with `DID NOT WARN` and the union
case fails because no warning is emitted. With this change they all pass. In
`tests/integration/test_rest_schema.py`, `test_allowed_updates` and
`test_disallowed_updates` expect the deprecation warning for string <-> binary,
and `test_alter_map_key` still opts in with `allow_incompatible_changes=True`
to reach the map key check.
## Are there any user-facing changes?
Yes. Without `allow_incompatible_changes=True`, `update_column` emits a
`DeprecationWarning` before applying `string -> binary`, `binary -> string` or
`fixed[16] -> uuid`. Without this opt-in, `union_by_name` warns when incoming
UUID fields replace existing fixed[16] fields. String/binary unions and
incoming fixed[16] fields over existing UUID fields remain type no-ops. The
deprecated implicit changes are scheduled to require the explicit opt-in in
0.14.0. Please add the `changelog` label.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]