chrevanthreddy opened a new pull request, #19801:
URL: https://github.com/apache/hudi/pull/19801

   ### Describe the issue this Pull Request addresses
   
   Spark can produce a data-table `WriteStatus` from a task attempt that later 
fails or loses to a retry/speculative attempt. The streaming metadata path 
previously derived RLI/SI records from that raw pre-shuffle lineage, allowing a 
losing attempt's stale file location to leak into committed metadata.
   
   This is a core metadata-index correctness issue discovered while validating 
RFC-109, but the fix is independent of vector indexing.
   
   ### Summary and Changelog
   
   Spark streaming metadata writes now place the data-table statuses behind the 
existing coalescing shuffle **before** deriving RLI/SI metadata records.
   
   - Build scheduler-selected data statuses at the coalescing shuffle boundary.
   - Feed that successful-attempt lineage into the existing streaming RLI/SI 
mappers.
   - Preserve the existing metadata-status persistence and data/metadata union.
   - Add a regression that forces partition 0 attempt 0 to fail under 
`local[8,2]` and verifies both collected data output and metadata input contain 
only attempt 1.
   - Add coverage that metadata statuses remain unioned into the returned DAG.
   
   No code was copied.
   
   ### Impact
   
   Correctness fix for Spark writers using streaming metadata writes, 
particularly RLI and SI under task retry/speculation. No storage-format, public 
API, timeline-protocol, metadata-schema, or configuration change.
   
   ### Risk Level
   
   medium — this moves the existing coalescing shuffle ahead of streaming 
metadata generation. Mitigation:
   
   - JDK 17, Spark 4.0 / Scala 2.13 targeted reactor: **8 tests passed, 0 
failed**.
   - Forced Spark retry regression verifies metadata consumes only 
scheduler-selected output.
   - Existing streaming metadata mappers and commit-completion flow are 
retained.
   - `git diff --check` passes.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to