danny0405 opened a new pull request, #20756:
URL: https://github.com/apache/hudi/pull/20756

   ### Describe the issue this Pull Request addresses
   
   Closes #20408. Merge after #20755, which completes the #20406 prerequisite; 
#20403 is already closed.
   
   The keyed-flow fixtures merged in #20496 and #20676 cover ordering, 
delete/reinsertion and MOR log merging. This completes acceptance for existing 
targets loading persisted settings and keyed batch replay after Hudi commit but 
before Spark checkpoint completion, with metadata enabled.
   
   ### Summary and Changelog
   
   - Seed the existing target with an unrelated row in partition `c`. Omit 
`TBLPROPERTIES` from its SDP declaration so table type, record key, ordering 
field and merger are loaded from persisted settings.
   - Preserve that seed row throughout the six input stages, retry, restart and 
five rejected configurations.
   - Enable metadata indexing and require initialized metadata tables with 
completed metadata writes for both targets.
   - Model the commit/checkpoint interruption by removing each stopped query's 
latest Spark commit-log entry while retaining source offsets, checkpoint 
identities and Hudi timelines. Retry must restore Spark's entries, preserve 
exact rows and add no Hudi data commits.
   - Retain retry timeline evidence and assert that rejected configurations 
also leave valid target timelines unchanged.
   - Default the harness directory to the fixture's own directory and update 
the runnable instructions.
   
   Validation: a fresh Spark 4.1 / Scala 2.13 bundle build on JDK 17 passed, 
including style and license checks; unit-test execution was skipped for the 
bundle build. Both full COW and MOR CLI/Connect matrices passed with 
`PYTHONOPTIMIZE=1`, including verification subprocesses. Each exercised new and 
pre-existing targets, six input stages, retry, ordinary restart and five 
negative cases. Both MOR targets retained the expected old base rows while 
snapshots reflected updated log records. Python compilation and `git diff 
--check` passed.
   
   ```sh
   PYTHONOPTIMIZE=1 python3 hudi-integ-test/spark-pipelines/keyed_flows.py \
     --bundle 
packaging/hudi-spark-bundle/target/hudi-spark4.1-bundle_2.13-1.3.0-SNAPSHOT.jar 
\
     --table-type cow
   # Repeat with --table-type mor.
   ```
   
   This fixture models the checkpoint gap directly; real sink failure injection 
and lifecycle coverage remain in the existing `TestStreamingSinkRecovery` suite.
   
   ### Impact
   
   Completes the existing keyed SDP acceptance matrix using the production 
metadata-enabled default. The fixture verifies persisted target configuration 
and idempotent keyed replay through the real CLI/Connect path.
   
   ### Risk Level
   
   low — Test and documentation changes only. Checkpoint-log removal is 
restricted to the fixture's isolated storage after its AvailableNow queries 
stop. Both table-type matrices passed with exact-row and timeline assertions.
   
   ### Documentation Update
   
   Updated `hudi-integ-test/spark-pipelines/KEYED_FLOWS.md` with 
persisted-setting reuse, the seed row, metadata checks, the checkpoint replay 
model and runnable commands. No new runtime feature or configuration requires a 
website change.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to