yihua opened a new pull request, #19940:
URL: https://github.com/apache/hudi/pull/19940

   ### Describe the issue this Pull Request addresses
   
   With `hoodie.errortable.write.unification.enabled=true`, `StreamSync` 
commits the error table and afterwards sums the error-table write statuses for 
the record counts. That ordering duplicates every error record.
   
   The error-table write-status RDD is kept alive only by a Spark cache. The 
bulk insert persists it in `BaseSparkCommitActionExecutor.updateIndex` under 
`HoodieDataCacheKey.of(basePath, instantTime)`, and the error table's own 
`writeClient.commit` releases it through `releaseResources` -> 
`SparkReleaseResources.releaseCachedData`. A real error-table writer typically 
also drops its cache of the upstream error events inside `commit`. So by the 
time the count runs, nothing backs the RDD, the `aggregate` re-evaluates its 
lineage, and the executors run the bulk insert a second time under an instant 
that has already completed. The re-run allocates fresh file ids, so the 
duplicates land in new file groups stamped with that instant and a snapshot 
read returns two rows per error record.
   
   The counts themselves stay correct, which is why this is invisible from the 
outside: the recomputed statuses still describe one write's worth of records.
   
   This arrived with #18750. That PR's first cut counted before committing the 
error table. A later revision hoisted the error-table commit ahead of the 
validators so error records would survive a validator abort, and the count 
moved behind it as a consequence. In the same round the data-table RDD was 
given a cache and a driver collect while the error-table RDD was left as a lazy 
`aggregate`, so the asymmetry went unnoticed.
   
   ### Summary and Changelog
   
   - `StreamSync.writeToSinkAndDoMetaSync` counts records before committing the 
error table, restoring the ordering the pre-#18750 
`HoodieStreamerWriteStatusValidator` used. The error-table commit still runs 
ahead of the validators, so the property the reorder was introduced for is 
preserved. Steps renumbered accordingly.
   - `ErrorTableCommitter` and `SuccessfulRecordCounter` are unchanged. The 
error-table statuses are still summed on the executors with a single 
`aggregate`; nothing is materialized on the driver.
   - Tests: `RddBackedErrorTableWriter` is an error-table writer backed by a 
real `SparkRDDWriteClient` over a real Hudi table, whose `commit` releases the 
write statuses it is given and the upstream error events. The only 
`BaseErrorTableWriter` implementation in the tree is a stub that stores the RDD 
and releases nothing, so no existing test could observe this. 
`TestErrorTableWriteOnce` asserts the record keys the error table actually 
holds: one case on the fixed ordering, one pinning the previous ordering that 
wrote every record twice.
   
   #### Ordering, for review
   
   The whole change is the position of the counting block. 
`ErrorTableCommitter` and `SuccessfulRecordCounter` are byte-identical to the 
base branch.
   
   Before, in `writeToSinkAndDoMetaSync`:
   
   ```
       writeStatusRDD.collect()                 data-table write runs
   1   ErrorTableCommitter.commit(...)          error table commits, its 
write-status cache is released
   2   runValidators(...)
   3   SuccessfulRecordCounter.compute(...)     error-table statuses read HERE, 
after the release  <-- bug
   4   write-error gate
   5   writeClient.commit(...)
   ```
   
   After:
   
   ```
       writeStatusRDD.collect()                 data-table write runs
   1   SuccessfulRecordCounter.compute(...)     error-table statuses read while 
still cached
   2   ErrorTableCommitter.commit(...)          error table commits, cache 
released, nothing reads it again
   3   runValidators(...)
   4   write-error gate
   5   writeClient.commit(...)
   ```
   
   This restores the order `HoodieStreamerWriteStatusValidator` used before 
#18750, where the same two operations ran inside the `writeClient.commit` 
callback rather than ahead of it. On `release-1.2.0` that is 
`StreamSync.HoodieStreamerWriteStatusValidator.validate`: the error-table sum, 
then `errorTableWriter.commit(...)`, then the write-error gate, and only then 
does `SparkRDDWriteClient.commit` proceed to `commitStats`.
   
   | | before #18750 | this PR |
   | ---| ---| --- |
   | error-table statuses summed with one distributed `aggregate` | yes | yes |
   | summed on executors, not collected to the driver | yes | yes |
   | sum happens before the error-table commit | yes | yes |
   | error-table commit before the write-error gate | yes | yes |
   | data-table commit last | yes | yes |
   
   Step 2 stays ahead of the validators, so the property #18750 introduced the 
reorder for (error records survive a validator-driven abort) is unchanged.
   
   Two things that are deliberately *not* restored here, both introduced by 
#18750 and out of scope: the sequence runs before `writeClient.commit` rather 
than inside its callback, so failures roll the inflight instant back 
explicitly; and the data-table statuses are collected in `StreamSync` as full 
`WriteStatus` objects rather than inside the write client as `SlimWriteStats`.
   
   The error table is a separate table with its own write client, instant and 
`HoodieDataCacheKey`, so its commit releases only its own write statuses. The 
data-table RDD is unaffected either way, which is why only error-table rows 
were ever duplicated.
   
   ### Impact
   
   Error-table writes land once under write unification. No config, API or 
public behaviour change.
   
   ### Risk Level
   
   low. The change is a move of the counting block within one method; no helper 
signatures change. Red-then-green verified: with the count after the commit the 
new test fails with each of the 20 expected record keys appearing twice, and 
passes with the count before it.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to