danny0405 commented on issue #20401:
URL: https://github.com/apache/hudi/issues/20401#issuecomment-6813893501

   **Status update — November 12**, following the [four-phase 
plan](https://github.com/apache/hudi/issues/20401#issuecomment-6485534890) and 
[November 4 
update](https://github.com/apache/hudi/issues/20401#issuecomment-6628676171).
   
   **Merged progress**
   - **Phase 1:** the runtime foundations and acceptance scenario (#20493) are 
merged; the previously reported local MVP acceptance remains established.
   - **Phase 2:** repeated MV refresh **#20494 merged November 5**, and keyed 
ordering/delete/reinsertion plus partition validation **#20496 merged November 
6**. These join the already-merged append/catalog prerequisites (#20423, 
#20492) and cleanup/commit-checkpoint recovery coverage (#20495).
   - **Phase 3:** incremental chaining acceptance **#20515 merged November 
11**. Its final COW/MOR CLI runs used **metadata enabled**, checked exact rows 
across two partitions, and verified checkpoint resume and quiescent restart 
without business-record replay. It uses explicit business-column projection; it 
does not establish hard-delete CDC or bounded source admission control.
   
   **Open implementation and CI work**
   
   | Item | Latest status |
   | --- | --- |
   | Harness / CI — #20453 | Open, ready for review. Both test layers now run 
within existing Scala-other / Scala-DML jobs, with no new workflow, job or 
matrix entry. The [reported added 
runtime](https://github.com/apache/hudi/pull/20453#issuecomment-6813070944) is 
**7m28s / 7m04s**, each below the 30-minute condition. The Maven-resolution 
failure has since passed on rerun: [Java CI is now 
green](https://github.com/apache/hudi/actions/runs/42975502397/job/130954860718),
 as is Java CI Engines. Azure CI remains pending. |
   | Async service retry after client swap — #20445 | Open, with merge 
conflicts. Latest review still raises the heartbeat-deletion race and 
responsiveness of graceful shutdown during heartbeat polling; review 
fixes/rebase remain needed. |
   | Streaming-source recovery — #20516 | Open. The proposal now uses `FAIL` by 
default plus explicit snapshot-replay / skip strategies, replacing the earlier 
opt-in boolean guard. Latest revision preserves healthy scan optimization and 
documents that FAIL requires sink error propagation. Reported local 
source-suite results: Spark 3.5 **25 passed / 2 expected skips**, Spark 4.2 
**27 passed**. Current-head remote CI is still running; CLI/Connect acceptance 
was not rerun for these latest fixes. |
   | Fan-in rejection — #20406 | Still outstanding in the published 
implementation work: reject multiple flows targeting one table before any 
write. **#20530 separately tracks future fan-in support**; filing it does not 
implement rejection or safe multi-flow writes. |
   
   Phase 2 therefore remains incomplete under the agreed acceptance criteria. 
Remaining coverage includes the previously identified keyless aggregate MV 
case, MOR log-merge keyed-flow coverage, and the post-commit failure window 
inside the sink retry loop. Phase 3 also remains open beyond the merged 
incremental scenario: source-recovery review/acceptance and the remaining 
projection/delete-behavior contract must be completed. #20491 remains a general 
metadata-writer follow-up, outside Phase 2.
   
   **Release scope change: defer Phase 4**
   
   The [proposal in 
#20404](https://github.com/apache/hudi/issues/20404#issuecomment-6762486498) 
has received agreement to defer **timeline-preserving truncate and 
checkpoint-generation isolation together** from this SDP release. The release 
keeps today's purge-style full-table truncate: history, savepoints, metadata 
and stored sink batch IDs are removed; full refresh rebuilds from a fresh 
checkpoint. The operation remains non-transactional, and downstream 
tables/checkpoints require coordinated rebuilding.
   
   **#20622 is open and ready for review** to document these limitations and 
update the MV acceptance notes. **#20378 remains an open draft** for the 
deferred implementation. This scope decision is not evidence of new 
full-refresh or failure-injection acceptance, and the old November 13 Phase 4 
target no longer describes the release plan.
   
   **Next steps:** land #20453 after review/remaining checks, address #20445's 
review findings and conflicts, finish the #20516 review/CI and applicable 
acceptance, resolve the outstanding fan-in rejection requirement, and land 
#20622. RFC **#19997 remains open** and needs to stay aligned with the revised 
release scope. The earlier November 5/10 targets have passed; this update does 
not establish replacement delivery dates or mark the overall issue complete.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to