danny0405 opened a new pull request, #20515:
URL: https://github.com/apache/hudi/pull/20515

   ### Describe the issue this Pull Request addresses
   
   Issue: #20407; parent #20401. The incremental Hudi streaming-table chaining 
path needs executable acceptance through the real Spark pipelines CLI and 
Connect server. This PR covers that path only and does not close #20407.
   
   ### Summary and Changelog
   
   - Add a standalone two-stage scenario adapted from the existing 
phase1_acceptance.py: external JSON input -> keyed Hudi upsert table -> Hudi 
insert table via FROM STREAM.
   - Assert exact business rows after duplicate-key input and a later update. 
Using insert downstream makes replay visible as extra rows rather than masking 
it with upserts.
   - Verify checkpoint resume and unchanged completed timelines after 
quiescence for COW and MOR. For MOR, require compaction and separately drain 
its maintenance instant while asserting no business-record replay.
   - Explicitly project business columns and retain generated SQL, pipeline 
configuration, and timeline evidence in the logs.
   
   This requires the #20453 harness checkout through --harness-dir. 
Reset/recreation detection, expired-history errors, reader-option conflicts, 
and coordinated rebuild acceptance remain outstanding under #20407. No new 
production source behavior is claimed.
   
   ### Impact
   
   Adds opt-in integration acceptance and runnable examples. No production 
behavior, public API, configuration default, or storage-format changes. The 
source remains latest-state incremental, without full hard-delete propagation 
or bounded admission control.
   
   ### Risk Level
   
   low: test and documentation changes only. The MOR scenario explicitly 
accounts for async compaction completing after AvailableNow captures its end 
offset; that catch-up can produce an empty downstream commit.
   
   ### Documentation Update
   
   Updated hudi-integ-test/spark-pipelines/ACCEPTANCE.md with invocation, 
assertions, compaction behavior, source limitations, and outstanding 
reset-safety work. No new public feature or configuration requiring a website 
change.
   
   ### Testing
   
   - Built the Spark 4.1 bundle from master c58c54e068b with JDK 17: mvn 
package -Pjava17 -Dspark4.1 -Dscala-2.13 -DskipTests -Dgpg.skip -Djacoco.skip 
-am -pl packaging/hudi-spark-bundle.
   - Real Spark 4.1.1 CLI/Connect scenario passed with --table-type cow and 
--table-type mor, both with metadata disabled. Each verifies exact rows across 
runs and unchanged quiescent restart timelines; MOR additionally verifies 
compaction catch-up.
   - Python syntax compilation and git diff --check passed.
   - Metadata-enabled chaining and CI have not been run. The Maven package 
command skipped unit-test execution.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to