chrevanthreddy opened a new pull request, #19801: URL: https://github.com/apache/hudi/pull/19801
### Describe the issue this Pull Request addresses Spark can produce a data-table `WriteStatus` from a task attempt that later fails or loses to a retry/speculative attempt. The streaming metadata path previously derived RLI/SI records from that raw pre-shuffle lineage, allowing a losing attempt's stale file location to leak into committed metadata. This is a core metadata-index correctness issue discovered while validating RFC-109, but the fix is independent of vector indexing. ### Summary and Changelog Spark streaming metadata writes now place the data-table statuses behind the existing coalescing shuffle **before** deriving RLI/SI metadata records. - Build scheduler-selected data statuses at the coalescing shuffle boundary. - Feed that successful-attempt lineage into the existing streaming RLI/SI mappers. - Preserve the existing metadata-status persistence and data/metadata union. - Add a regression that forces partition 0 attempt 0 to fail under `local[8,2]` and verifies both collected data output and metadata input contain only attempt 1. - Add coverage that metadata statuses remain unioned into the returned DAG. No code was copied. ### Impact Correctness fix for Spark writers using streaming metadata writes, particularly RLI and SI under task retry/speculation. No storage-format, public API, timeline-protocol, metadata-schema, or configuration change. ### Risk Level medium — this moves the existing coalescing shuffle ahead of streaming metadata generation. Mitigation: - JDK 17, Spark 4.0 / Scala 2.13 targeted reactor: **8 tests passed, 0 failed**. - Forced Spark retry regression verifies metadata consumes only scheduler-selected output. - Existing streaming metadata mappers and commit-completion flow are retained. - `git diff --check` passes. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
