junmuz commented on PR #11017: URL: https://github.com/apache/paimon/pull/11017#issuecomment-6924005739
> Please break down the PR so that it is around 1,000 lines long. I am thinking of splitting this into six smaller PRs: 1. **Guard storage primitives:** payload-free delete bounds, serialization, immutable SST readers/writers, and storage tests. 2. **Metadata and compatibility:** guard references in splits, generation/source fields in commit messages, checkpoint/recovery metadata, Flink message formats, and compatibility fixtures. Runtime discovery stays in PR 4. 3. **Merge and compaction primitives:** guard capture/resolution, reducer and changelog hooks, compaction index deltas, and output ownership/cleanup. Keep normal table capture disconnected until PR 4. 4. **Initial safe feature:** enable guards for fixed-bucket primary-key tables using `deduplicate`, with ingestion, compaction, and batch snapshot/time-travel reads. Include atomic publication, retry/source-set and generation fencing, recovery, sequence watermarks, retention/partition expiration, policy validation, and client/operation restrictions. Initially reject guarded incremental and streaming reads, including when capture is disabled but guards remain. 5. **Consolidation and guard-only compaction:** segment consolidation, thresholds, guard-only bucket discovery/scheduling, and replacement/retention tests. Until this lands, existing segments accumulate and remain effective. 6. **Incremental and streaming reads:** endpoint-aware incremental diffs, streaming overwrite/all-delta reads, visible before-images, and expired-predecessor handling. Preserve raw input changelog semantics and remove the corresponding temporary restrictions from PR 4. PRs 1 and 2 can merge independently. PR 3 depends on PR 1; PR 4 depends on PRs 1–3. PRs 5 and 6 depend on PR 4 and can merge in either order. The metadata PR needs an explicit checkpoint compatibility review: the current format change also affects ordinary jobs with guard capture disabled. Each PR should carry its relevant tests and documentation, incorporate the later correctness fixes from the final diff, and pass validation independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
