JingsongLi opened a new pull request, #1374: URL: https://github.com/apache/paimon-rust/pull/1374
### Purpose Data Evolution updates to existing row IDs can commit successfully after their written fields are dropped and recreated, leaving the update invisible under the current schema. Internal snapshot retries reload the schema ID without revalidating the staged field identities. Align Rust's commit and logical conflict checks with the field-identity policy in [apache/paimon#11048](https://github.com/apache/paimon/pull/11048), while allowing compatible schema evolution. ### Brief change log - Validate staged existing-row field IDs against the exact schema recorded by every APPEND or OVERWRITE publication attempt, including internal CAS retries. Reject dropped written fields, including descendants in ROW, ARRAY, MAP, and MULTISET columns. - Allow logical row-ID conflict checks across compatible schema changes. Resolve and cache catalog-only REST schemas before conflict detection, so independent stale handles can check and commit without a filesystem copy of the current schema. - Interpret omitted `write_cols` using the file's historical physical schema, preventing normal-only writes from falsely conflicting with dedicated BLOB/vector updates. - Preserve ordinary append and compaction schema policies, and allow partial updates when only unwritten fields were dropped. ### Tests - `cargo test --locked -p paimon --lib table::table_commit -- --nocapture`: 211 passed. - `cargo test --locked -p paimon --test historical_table_update_test --test rest_catalog_test`: 182 passed. - `cargo +1.98.0 clippy --locked -p paimon --lib --tests -- -D warnings`: passed. - `cargo +1.98.0 fmt --all -- --check` and `git diff --check`: passed. Regression coverage includes deterministic APPEND/OVERWRITE CAS failures, compatible renames, dropped/recreated fields, catalog-only DDL with and without snapshot progress, concurrent updates to different columns, nested field identity, and dedicated-column conflict checks. The initial regressions were confirmed failing before their fixes. ### API and Format Public signatures and storage formats are unchanged. Existing-row updates targeting removed field IDs now fail instead of publishing an update that is invisible under the current schema. ### Documentation Document the existing-row update validation policy in the Python binding guide and update the Rust logical conflict check documentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
