danny0405 commented on issue #20401:
URL: https://github.com/apache/hudi/issues/20401#issuecomment-6999039998

   **Status update — November 19, 2026**, following the [November 12 
update](https://github.com/apache/hudi/issues/20401#issuecomment-6813893501) 
and the [four-phase 
plan](https://github.com/apache/hudi/issues/20401#issuecomment-6485534890).
   
   **Newly merged**
   - **#20453 merged November 16:** Spark Connect / real Python CLI harness and 
CI coverage are now on master. Both layers run within the existing Scala-other 
/ Scala-DML jobs; this closes the previously outstanding harness integration.
   - **#20622 merged November 18:** SDP truncate/full-refresh limitations and 
the corresponding MV acceptance notes are documented. This implements the 
documentation portion of the agreed Phase 4 deferral.
   
   The previously merged MVP (#20493), MV refresh (#20494), keyed flows 
(#20496), cleanup/recovery (#20495), and incremental chaining (#20515) remain 
the baseline.
   
   **Phase 2 — remaining work now has additional reviewable patches**
   
   | Work | Latest status / evidence |
   | --- | --- |
   | Keyless aggregate MV — #20675 / #20405 | Open; received approval. Reported 
Spark Connect 4.1.1 acceptance passed four refreshes and twelve exact-result 
checks, including changed aggregates, removed/new groups and an empty result. |
   | MOR keyed flows — #20676 / #20408 | Open; received approvals. November 19 
review fixes assert the exact pre-update read-optimized rows and include both 
commit lists in restart diagnostics. Full COW/MOR CLI matrices were rerun 
successfully, including unchanged restart and five rejected configurations. |
   | Post-commit failures inside the sink retry loop — #20677 / #20406 | Open; 
received approval with non-blocking follow-ups. Prevents rewriting an already 
committed batch after a post-commit error. Reported 13 selected regressions 
passed, including real metadata-sync failures for new/existing COW/MOR targets 
and same-checkpoint restart. Review follow-ups cover config/release 
documentation, terminal error messages, inline clean/archive and 
unreadable-timeline fault coverage, and MOR async-compaction recovery. |
   | Async table-service retry after client swap — #20445 | Open; received 
approval and GitHub currently reports no merge conflict. Review fixes 
strengthen concurrent-writer takeover checks and verification that the specific 
failed instant completes on retry. Reported local results: 23 async-service 
tests and 29 structured-streaming tests passed. |
   | Reject unsupported fan-in before any write — #20406 | Still outstanding; 
no implementation PR found in the linked work or current search. #20530 tracks 
future fan-in support and does not satisfy this rejection requirement. |
   
   The keyless/MOR CLI results above used metadata indexing disabled. They do 
not establish full-refresh failure acceptance or hard-delete propagation 
through a chain. **Phase 2 remains incomplete.** #20491 remains a general 
metadata-writer follow-up outside Phase 2.
   
   **Phase 3 — recovery and bounded source consumption**
   - **#20516 remains open and currently has merge conflicts.** It proposes 
default `FAIL` plus explicit snapshot-replay / skip recovery. November 17 fixes 
add partition-overwrite coverage and clarify overwrite/TTL behavior; Spark 3.5 
source suites reported **29 passed / 2 expected version skips**. Spark 4.x and 
CLI/Connect acceptance were **not rerun for that latest follow-up**.
   - **#20674 / #20679 is new release-targeted throttling work**, separate from 
and not gating #20516. The open PR adds an opt-in commits-per-trigger cap and 
native AvailableNow end pinning. Reported COW/MOR tests cover both offset 
versions on Spark 3.5/4.1, including 2/2/1 backlog draining and restart. It 
currently has merge conflicts; integration with #20516 and CLI/Connect 
throttling acceptance have not been run. Snapshot recovery remains unbounded.
   - The merged chaining scenario does not establish hard-delete CDC. The 
remaining projection/reader-option and delete-behavior acceptance under #20407 
still needs to be reconciled with the final contract.
   
   **Phase 4 remains deferred:** timeline-preserving truncate and 
checkpoint-generation isolation move together to later work; #20378 remains a 
draft. This release retains purge-style full-table truncate and 
fresh-checkpoint refresh, with coordinated rebuilding of affected downstream 
tables/checkpoints. #20622 documents the non-transactional operation and loss 
of history/savepoints/metadata; its merge is not new runtime or 
failure-injection acceptance.
   
   **Next:** land the approved Phase 2 patches with their applicable 
follow-ups/checks, resolve fan-in rejection, rebase/integrate source recovery 
and throttling and complete applicable acceptance, then align RFC #19997 and 
the tracking criteria with the agreed release scope. RFC #19997 remains open. 
Azure CI is still pending on all six open implementation/test PRs listed above; 
reported local results are not a claim of fully green CI. No replacement 
delivery date or overall completion is established by this update.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to