danny0405 opened a new issue, #20752:
URL: https://github.com/apache/hudi/issues/20752

   ### Task Description
   
   **What needs to be done:**
   
   Add deterministic failure-injection and retry acceptance for Hudi 
materialized-view refresh through the real Spark pipelines CLI / Spark Connect 
harness. This carries forward the unfinished failure-testing requirement from 
#20405.
   
   Use the current purge-style full-table truncate contract agreed in #20404 
and documented by #20622. A refresh purges/reinitializes the target, reconciles 
catalog metadata, and writes the recomputed result. The sequence is 
non-transactional: there is no cross-step rollback or guarantee that the old 
snapshot survives after purge begins.
   
   **Why this task is needed:**
   
   Merged #20494 and #20675 establish successful repeated refreshes and exact 
keyed/keyless results. They do not inject failures around truncate, 
reconciliation, or result writing. Documenting the empty window does not 
establish recovery acceptance.
   
   ### Acceptance criteria
   
   - [ ] Inject a failure before any destructive truncate work and verify the 
previous target contents remain intact.
   - [ ] Inject a failure after purge/truncate completes but before result 
writing; assert the observed empty/reinitialized target and catalog state, 
without assuming rollback to the previous snapshot.
   - [ ] Inject failures during catalog reconciliation and result writing. 
Assert the actual target/catalog state and completed/incomplete write evidence 
at each boundary; do not assume that a failed client operation implies no data 
committed.
   - [ ] Remove each injected fault and rerun the refresh through the CLI. 
Assert exact recomputed business rows, with no stale or duplicate output; 
include an empty recomputation.
   - [ ] Verify failures reach the CLI with useful diagnostics and that cleanup 
leaves the Connect server usable for the subsequent retry.
   - [ ] Retain deterministic fault settings, generated definitions, CLI/server 
logs, and state assertions. Checks must remain effective under Python 
optimization.
   - [ ] Update the runnable acceptance documentation with the tested failure 
boundaries, empty window, retry procedure, and lack of cross-step rollback.
   
   ### Scope
   
   Test the current purge-style MV refresh. Timeline-preserving truncate, 
replace-commit/history/savepoint assertions, and streaming 
checkpoint-generation isolation remain together under #20404 / #20378. Atomic 
refresh and streaming-table full-refresh acceptance are not established by this 
task.
   
   ### Task Type
   
   Test enhancement
   
   ### Related Issues
   
   Parent feature issue: #20401
   Extracted from: #20405
   Existing acceptance: #20494, #20675
   Harness: #20402 / #20453
   Current refresh contract/documentation: #20404, #20622
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to