junmuz opened a new pull request, #11017:
URL: https://github.com/apache/paimon/pull/11017

   ### Purpose
   
   Prevent out-of-order updates from recreating deleted rows after primary-key 
compaction prunes delete records. A delete at version `4` continues blocking an 
insert at version `2`, while an insert at version `5` is accepted.
   
   Add the boolean option `delete-ordering-guards.enabled` (**default: 
`false`**) for fixed-bucket primary-key tables using `deduplicate` or 
`partial-update`. Guards retain keys and ordering metadata without deleted 
payloads. Disabling capture preserves existing guards.
   
   - Publish guards atomically with compaction, with retry recovery, failure 
cleanup, and generation fencing across ordering-history resets.
   - Resolve guards in native snapshot, incremental, streaming, and changelog 
paths, including partial-update sequence groups and delete before-images.
   - Consolidate guard segments and protect retained snapshot/tag/branch 
references during cleanup. Regular consolidation defaults to four segments; 
full compaction consolidates two or more. Newer inserts do not automatically 
retire guards.
   - Preserve guards through native Flink/Spark rescaling and partition 
expiration, including guard-only partitions. Use native serialization for Spark 
rescale tasks under Java and Kryo.
   - Validate ordering-policy changes, bootstrap legacy bucket layouts, and 
document configuration and maintenance behavior.
   
   Scope excludes append/Data Evolution tables, dynamic/postpone buckets, field 
aggregates, deletion vectors, primary-key indexes and chain tables. TTL expiry 
creates no guards; overwrite/truncate explicitly resets ordering history. This 
doesn't impact Iceberg compatibility as it's only considered during write paths.
   
   Fixes the issue https://github.com/apache/paimon/issues/5387
   
   ### Tests
   - **475 core tests across 17 classes passed**, with normal Maven checks and 
no failures or skips. Coverage includes guard storage, recovery, reads, 
maintenance, and existing commit behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to