junmuz commented on PR #11017:
URL: https://github.com/apache/paimon/pull/11017#issuecomment-6924005739

   > Please break down the PR so that it is around 1,000 lines long.
   
   I am thinking of  splitting this into six smaller PRs:
   
   1. **Guard storage primitives:** payload-free delete bounds, serialization, 
immutable SST readers/writers, and storage tests.
   2. **Metadata and compatibility:** guard references in splits, 
generation/source fields in commit messages, checkpoint/recovery metadata, 
Flink message formats, and compatibility fixtures. Runtime discovery stays in 
PR 4.
   3. **Merge and compaction primitives:** guard capture/resolution, reducer 
and changelog hooks, compaction index deltas, and output ownership/cleanup. 
Keep normal table capture disconnected until PR 4.
   4. **Initial safe feature:** enable guards for fixed-bucket primary-key 
tables using `deduplicate`, with ingestion, compaction, and batch 
snapshot/time-travel reads. Include atomic publication, retry/source-set and 
generation fencing, recovery, sequence watermarks, retention/partition 
expiration, policy validation, and client/operation restrictions. Initially 
reject guarded incremental and streaming reads, including when capture is 
disabled but guards remain.
   5. **Consolidation and guard-only compaction:** segment consolidation, 
thresholds, guard-only bucket discovery/scheduling, and replacement/retention 
tests. Until this lands, existing segments accumulate and remain effective.
   6. **Incremental and streaming reads:** endpoint-aware incremental diffs, 
streaming overwrite/all-delta reads, visible before-images, and 
expired-predecessor handling. Preserve raw input changelog semantics and remove 
the corresponding temporary restrictions from PR 4.
   
   PRs 1 and 2 can merge independently. PR 3 depends on PR 1; PR 4 depends on 
PRs 1–3. PRs 5 and 6 depend on PR 4 and can merge in either order.
   
   The metadata PR needs an explicit checkpoint compatibility review: the 
current format change also affects ordinary jobs with guard capture disabled.
   
   Each PR should carry its relevant tests and documentation, incorporate the 
later correctness fixes from the final diff, and pass validation independently.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to