yihua opened a new pull request, #19940: URL: https://github.com/apache/hudi/pull/19940
### Describe the issue this Pull Request addresses closes #19938 With error table write unification enabled, the streamer commits the error table and afterwards sums the error-table write statuses for the record counts. The error table's commit releases the write-status RDD it was given, so that later sum re-evaluates the RDD's lineage, which is the bulk insert itself: the executors write the same records again under the instant that already completed, and the error table reads two rows per error record. ### Summary and Changelog - `ErrorTableCommitter.collectAndCommit` materializes the error-table write statuses on the driver before delegating to the existing `commit`, and returns them with the outcome. An RDD the writer did not persist is cached for the duration of the call, so the collect and the commit share one evaluation. - `StreamSync` uses that entry point and counts from the returned statuses. - `SuccessfulRecordCounter.compute` takes the error-table statuses as a list instead of an RDD, so nothing downstream of the commit can touch the RDD again. - Tests: `TestErrorTableCommitter` gains cases on a real local Spark context where the writer's commit consumes and then unpersists a cached RDD, asserting each partition is computed exactly once and the statuses are returned; `TestSuccessfulRecordCounter` is adapted to the list-based signature. ### Impact Error table writes land once. No public API change; `SuccessfulRecordCounter` is internal to the streamer. ### Risk Level low. The collect happens while the RDD is still cached by the write, so it adds no recomputation. Verified with the new unit tests, which fail on the previous ordering with two evaluations per partition. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
