chrevanthreddy opened a new pull request, #19801:
URL: https://github.com/apache/hudi/pull/19801
## What changed
Spark streaming metadata writes now defer RLI/SI metadata generation until
commit finalization and rebuild those partitions from the successful
task-attempt write stats captured in `HoodieCommitMetadata`.
The existing coalescing shuffle is retained as the materialization boundary,
but raw pre-commit `WriteStatus` attempts are no longer unioned with metadata
generation. This prevents failed/speculative Spark task attempts from leaking
stale file locations into committed metadata indexes.
The generic completion path still preserves partial streaming updates from
engines that emit them: only metadata partitions absent from
`partialWriteStats` are regenerated from committed metadata.
## Why
A Spark task can write a `WriteStatus` and then fail or lose to another
attempt. Generating RLI/SI records directly from that pre-commit lineage can
commit a location from the losing attempt. Commit metadata already contains the
scheduler-selected successful outputs, so this change reuses that authoritative
seam instead of adding a retry protocol.
## Verification
- JDK 17, Spark 4.0 / Scala 2.13 targeted reactor:
- `TestSparkStreamingMetadataWriteHandler`: **8 passed, 0 failed**
- full selected reactor: **BUILD SUCCESS**
- Regression test forces task 0 attempt 0 to fail under `local[8,2]` and
verifies only attempt 1's output is collected.
- `git diff --check`: pass
## Compatibility
- Spark defers metadata generation to committed stats.
- Other engines may continue returning partial MDT statuses through the
existing abstract contract.
- No timeline protocol or metadata schema changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
