danny0405 commented on issue #20401: URL: https://github.com/apache/hudi/issues/20401#issuecomment-6813893501
**Status update — November 12**, following the [four-phase plan](https://github.com/apache/hudi/issues/20401#issuecomment-6485534890) and [November 4 update](https://github.com/apache/hudi/issues/20401#issuecomment-6628676171). **Merged progress** - **Phase 1:** the runtime foundations and acceptance scenario (#20493) are merged; the previously reported local MVP acceptance remains established. - **Phase 2:** repeated MV refresh **#20494 merged November 5**, and keyed ordering/delete/reinsertion plus partition validation **#20496 merged November 6**. These join the already-merged append/catalog prerequisites (#20423, #20492) and cleanup/commit-checkpoint recovery coverage (#20495). - **Phase 3:** incremental chaining acceptance **#20515 merged November 11**. Its final COW/MOR CLI runs used **metadata enabled**, checked exact rows across two partitions, and verified checkpoint resume and quiescent restart without business-record replay. It uses explicit business-column projection; it does not establish hard-delete CDC or bounded source admission control. **Open implementation and CI work** | Item | Latest status | | --- | --- | | Harness / CI — #20453 | Open, ready for review. Both test layers now run within existing Scala-other / Scala-DML jobs, with no new workflow, job or matrix entry. The [reported added runtime](https://github.com/apache/hudi/pull/20453#issuecomment-6813070944) is **7m28s / 7m04s**, each below the 30-minute condition. The Maven-resolution failure has since passed on rerun: [Java CI is now green](https://github.com/apache/hudi/actions/runs/42975502397/job/130954860718), as is Java CI Engines. Azure CI remains pending. | | Async service retry after client swap — #20445 | Open, with merge conflicts. Latest review still raises the heartbeat-deletion race and responsiveness of graceful shutdown during heartbeat polling; review fixes/rebase remain needed. | | Streaming-source recovery — #20516 | Open. The proposal now uses `FAIL` by default plus explicit snapshot-replay / skip strategies, replacing the earlier opt-in boolean guard. Latest revision preserves healthy scan optimization and documents that FAIL requires sink error propagation. Reported local source-suite results: Spark 3.5 **25 passed / 2 expected skips**, Spark 4.2 **27 passed**. Current-head remote CI is still running; CLI/Connect acceptance was not rerun for these latest fixes. | | Fan-in rejection — #20406 | Still outstanding in the published implementation work: reject multiple flows targeting one table before any write. **#20530 separately tracks future fan-in support**; filing it does not implement rejection or safe multi-flow writes. | Phase 2 therefore remains incomplete under the agreed acceptance criteria. Remaining coverage includes the previously identified keyless aggregate MV case, MOR log-merge keyed-flow coverage, and the post-commit failure window inside the sink retry loop. Phase 3 also remains open beyond the merged incremental scenario: source-recovery review/acceptance and the remaining projection/delete-behavior contract must be completed. #20491 remains a general metadata-writer follow-up, outside Phase 2. **Release scope change: defer Phase 4** The [proposal in #20404](https://github.com/apache/hudi/issues/20404#issuecomment-6762486498) has received agreement to defer **timeline-preserving truncate and checkpoint-generation isolation together** from this SDP release. The release keeps today's purge-style full-table truncate: history, savepoints, metadata and stored sink batch IDs are removed; full refresh rebuilds from a fresh checkpoint. The operation remains non-transactional, and downstream tables/checkpoints require coordinated rebuilding. **#20622 is open and ready for review** to document these limitations and update the MV acceptance notes. **#20378 remains an open draft** for the deferred implementation. This scope decision is not evidence of new full-refresh or failure-injection acceptance, and the old November 13 Phase 4 target no longer describes the release plan. **Next steps:** land #20453 after review/remaining checks, address #20445's review findings and conflicts, finish the #20516 review/CI and applicable acceptance, resolve the outstanding fan-in rejection requirement, and land #20622. RFC **#19997 remains open** and needs to stay aligned with the revised release scope. The earlier November 5/10 targets have passed; this update does not establish replacement delivery dates or mark the overall issue complete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
