danny0405 commented on issue #20401: URL: https://github.com/apache/hudi/issues/20401#issuecomment-6999039998
**Status update — November 19, 2026**, following the [November 12 update](https://github.com/apache/hudi/issues/20401#issuecomment-6813893501) and the [four-phase plan](https://github.com/apache/hudi/issues/20401#issuecomment-6485534890). **Newly merged** - **#20453 merged November 16:** Spark Connect / real Python CLI harness and CI coverage are now on master. Both layers run within the existing Scala-other / Scala-DML jobs; this closes the previously outstanding harness integration. - **#20622 merged November 18:** SDP truncate/full-refresh limitations and the corresponding MV acceptance notes are documented. This implements the documentation portion of the agreed Phase 4 deferral. The previously merged MVP (#20493), MV refresh (#20494), keyed flows (#20496), cleanup/recovery (#20495), and incremental chaining (#20515) remain the baseline. **Phase 2 — remaining work now has additional reviewable patches** | Work | Latest status / evidence | | --- | --- | | Keyless aggregate MV — #20675 / #20405 | Open; received approval. Reported Spark Connect 4.1.1 acceptance passed four refreshes and twelve exact-result checks, including changed aggregates, removed/new groups and an empty result. | | MOR keyed flows — #20676 / #20408 | Open; received approvals. November 19 review fixes assert the exact pre-update read-optimized rows and include both commit lists in restart diagnostics. Full COW/MOR CLI matrices were rerun successfully, including unchanged restart and five rejected configurations. | | Post-commit failures inside the sink retry loop — #20677 / #20406 | Open; received approval with non-blocking follow-ups. Prevents rewriting an already committed batch after a post-commit error. Reported 13 selected regressions passed, including real metadata-sync failures for new/existing COW/MOR targets and same-checkpoint restart. Review follow-ups cover config/release documentation, terminal error messages, inline clean/archive and unreadable-timeline fault coverage, and MOR async-compaction recovery. | | Async table-service retry after client swap — #20445 | Open; received approval and GitHub currently reports no merge conflict. Review fixes strengthen concurrent-writer takeover checks and verification that the specific failed instant completes on retry. Reported local results: 23 async-service tests and 29 structured-streaming tests passed. | | Reject unsupported fan-in before any write — #20406 | Still outstanding; no implementation PR found in the linked work or current search. #20530 tracks future fan-in support and does not satisfy this rejection requirement. | The keyless/MOR CLI results above used metadata indexing disabled. They do not establish full-refresh failure acceptance or hard-delete propagation through a chain. **Phase 2 remains incomplete.** #20491 remains a general metadata-writer follow-up outside Phase 2. **Phase 3 — recovery and bounded source consumption** - **#20516 remains open and currently has merge conflicts.** It proposes default `FAIL` plus explicit snapshot-replay / skip recovery. November 17 fixes add partition-overwrite coverage and clarify overwrite/TTL behavior; Spark 3.5 source suites reported **29 passed / 2 expected version skips**. Spark 4.x and CLI/Connect acceptance were **not rerun for that latest follow-up**. - **#20674 / #20679 is new release-targeted throttling work**, separate from and not gating #20516. The open PR adds an opt-in commits-per-trigger cap and native AvailableNow end pinning. Reported COW/MOR tests cover both offset versions on Spark 3.5/4.1, including 2/2/1 backlog draining and restart. It currently has merge conflicts; integration with #20516 and CLI/Connect throttling acceptance have not been run. Snapshot recovery remains unbounded. - The merged chaining scenario does not establish hard-delete CDC. The remaining projection/reader-option and delete-behavior acceptance under #20407 still needs to be reconciled with the final contract. **Phase 4 remains deferred:** timeline-preserving truncate and checkpoint-generation isolation move together to later work; #20378 remains a draft. This release retains purge-style full-table truncate and fresh-checkpoint refresh, with coordinated rebuilding of affected downstream tables/checkpoints. #20622 documents the non-transactional operation and loss of history/savepoints/metadata; its merge is not new runtime or failure-injection acceptance. **Next:** land the approved Phase 2 patches with their applicable follow-ups/checks, resolve fan-in rejection, rebase/integrate source recovery and throttling and complete applicable acceptance, then align RFC #19997 and the tracking criteria with the agreed release scope. RFC #19997 remains open. Azure CI is still pending on all six open implementation/test PRs listed above; reported local results are not a claim of fully green CI. No replacement delivery date or overall completion is established by this update. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
