Rahul-cs50 opened a new pull request, #4266:
URL: https://github.com/apache/iceberg-python/pull/4266
Closes #4198
# Rationale for this change
`UpdateSpec` checks each new partition name against the schema it was loaded
with, using the check from #2305. Its commit asserts only the last assigned
partition id and the default spec id. Suppose another writer adds a column `xb`
in between. A stale handle can then add a partition field named `xb`, and the
commit goes through. Java then refuses to load that metadata (`Cannot create
identity partition sourced from different field in schema: xb`).
This change runs `validate_partition_name` again when
`AddPartitionSpecUpdate` is applied, against the schema of the metadata the
spec is being added to. On a catalog that applies updates to the latest
metadata at commit time, the stale `update_spec` now raises the same
`ValueError` it raises without the concurrent change. The schema-side check in
`UpdateSchema` already validates every partition spec in this way.
The issue also mentions the opposite order: a stale
`update_schema().add_column("xb")` after another writer added partition field
`xb`. That is the same problem, but on the schema update path. I kept it out of
this PR so this one stays small, and can follow up if wanted.
## Are these changes tested?
Yes. `test_concurrent_update_spec_checks_names_against_current_schema` in
`tests/catalog/test_catalog_behaviors.py` follows the issue's steps:
- One handle adds column `xb`.
- A stale handle then adds a bucket partition field named `xb`.
- The test expects the `ValueError`, and checks that the table is still
unpartitioned.
It fails on `main` (no error is raised) and passes with this change, for all
catalog fixtures. Unit tests (`tests/` without integration and cloud suites):
4501 passed; the 4 failures are Kerberos tests in `tests/catalog/test_hive.py`
that fail the same way on `main` in my environment. `ruff check` and `ruff
format --check` are clean.
## Are there any user-facing changes?
A stale `update_spec()` that adds a partition field whose name now clashes
with a column in the current schema raises `ValueError` at commit. Before, it
wrote metadata that Java refuses to load.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
