JingsongLi opened a new pull request, #1374:
URL: https://github.com/apache/paimon-rust/pull/1374

   ### Purpose
   
   Data Evolution updates to existing row IDs can commit successfully after 
their written fields are dropped and recreated, leaving the update invisible 
under the current schema. Internal snapshot retries reload the schema ID 
without revalidating the staged field identities.
   
   Align Rust's commit and logical conflict checks with the field-identity 
policy in [apache/paimon#11048](https://github.com/apache/paimon/pull/11048), 
while allowing compatible schema evolution.
   
   ### Brief change log
   
   - Validate staged existing-row field IDs against the exact schema recorded 
by every APPEND or OVERWRITE publication attempt, including internal CAS 
retries. Reject dropped written fields, including descendants in ROW, ARRAY, 
MAP, and MULTISET columns.
   - Allow logical row-ID conflict checks across compatible schema changes. 
Resolve and cache catalog-only REST schemas before conflict detection, so 
independent stale handles can check and commit without a filesystem copy of the 
current schema.
   - Interpret omitted `write_cols` using the file's historical physical 
schema, preventing normal-only writes from falsely conflicting with dedicated 
BLOB/vector updates.
   - Preserve ordinary append and compaction schema policies, and allow partial 
updates when only unwritten fields were dropped.
   
   ### Tests
   
   - `cargo test --locked -p paimon --lib table::table_commit -- --nocapture`: 
211 passed.
   - `cargo test --locked -p paimon --test historical_table_update_test --test 
rest_catalog_test`: 182 passed.
   - `cargo +1.98.0 clippy --locked -p paimon --lib --tests -- -D warnings`: 
passed.
   - `cargo +1.98.0 fmt --all -- --check` and `git diff --check`: passed.
   
   Regression coverage includes deterministic APPEND/OVERWRITE CAS failures, 
compatible renames, dropped/recreated fields, catalog-only DDL with and without 
snapshot progress, concurrent updates to different columns, nested field 
identity, and dedicated-column conflict checks. The initial regressions were 
confirmed failing before their fixes.
   
   ### API and Format
   
   Public signatures and storage formats are unchanged. Existing-row updates 
targeting removed field IDs now fail instead of publishing an update that is 
invisible under the current schema.
   
   ### Documentation
   
   Document the existing-row update validation policy in the Python binding 
guide and update the Rust logical conflict check documentation.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to