goutamadwant opened a new pull request, #12881: URL: https://github.com/apache/seatunnel/pull/12881
### Purpose of this pull request This is second Iceberg v3 implementation slice to SeaTunnel, following #12777 and as per plan discussed in #12763. Related to #12763. This does not close the remaining v3 roadmap items. This PR: - Enables v3 CDC/upsert writes using equality deletes for previously committed rows and Puffin deletion vectors for invalidated rows in the current writer's new files. - Supports Parquet, ORC and Avro data files without scanning existing files or rewriting their deletion vectors. - Makes ordinary and restore commits recognize previously published writer results, preventing duplicate publication during checkpoint recovery. - Uses immutable writer-result fingerprints rather than checkpoint numbers alone, allowing different jobs to reuse checkpoint IDs safely. - Adds nanosecond equality-delete reader support and a regression for reused Avro binary buffers. - Adds native Flink and Zeta recovery fixtures, existing-sink E2E coverage, English/Chinese documentation and changelog entries. ### Does this PR introduce _any_ user-facing change? Yes. Current dev rejects v3 CDC/upsert configurations. This PR enables them. Format version 2 remains the default, and its existing writer selection is preserved. No new dependency, configuration option, public API or checkpoint serialization format is introduced. Recovery checks are scoped to the configured table branch. Legacy snapshots require complete file-publication evidence; partial matches and manifest-read failures fail closed. Recovery requires retaining the relevant snapshot history, manifests and pending files, plus engine fencing against overlapping attempts publishing the same payload. Recovery after that evidence expires is not covered. Partitioned upserts retain existing key-affinity, ordering and partition-key requirements. These constraints are documented. Variant/default values, row-lineage exposure, incremental v3 reads and multi-argument transforms remain separate roadmap slices. ### How was this patch tested? - Final connector suite on Java 11 and Java 17: 179 tests per JDK, zero failures/errors and one existing disabled metastore test. - Final Zeta recovery: v2/v3 both passed after actual server-JVM termination and explicit checkpoint restore, using the unchanged upstream harness. - Final targeted Flink recovery: the previously failing v2 pre-publication case passed on Flink 1.18 and 1.20 with local harness corrections. - The corrected Flink fixture waits for a different worker registration, verifies exactly one restore from the original checkpoint, completes fresh checkpoints and checks exact rows/files/snapshots. - Earlier broader Flink testing exercised v2/v3 crashes before and after external publication: seven cases passed; the remaining worker-restart fixture failure was corrected and rerun as described above. The complete eight-case matrix was not rerun on the final artifact. - A real Avro regression reproduced a deleted row surviving when nanosecond equality keys reused binary buffers. Copying records before retaining normalized keys fixes it; the regression passes on both JDKs. - Unit coverage includes repeated updates, deletes, partition/key changes, file rolling, cleanup failures, checkpoint-ID reuse, regrouped committables, branch isolation, legacy publication checks and deletion-vector identity. - Formatting, whitespace checks and independent correctness/security reviews completed. Local harness limitation: the unchanged harness passed Flink 1.18 but could not build the Flink 1.20 JDK image on an ARM host. Flink 1.20 passed with a local platform correction; the successful corrected-harness run also used a local native-G1-thread classification correction. Neither shared-harness change is included here. #12855 separately tracks the image-platform fix. Zeta coverage is single-node crash/manual restore, not HA failover. Full-reactor, Spark recovery, native Windows and cloud-catalog validation are not claimed. ### Check list * [ ] New Jar binary package and License/NOTICE updates: not applicable; no dependency is added. * [x] English and Chinese documentation updated. * [ ] `incompatible-changes.md`: not applicable; existing configuration defaults and public contracts are preserved. * [x] Connector integration reviewed: E2E coverage added; existing plugin mapping, distribution registration, CI labels and plugin configuration remain unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
