yihua opened a new issue, #19938: URL: https://github.com/apache/hudi/issues/19938
### Problem With `hoodie.errortable.write.unification.enabled` on, the streamer's pre-commit flow commits the error table first (`ErrorTableCommitter.commit`) and only afterwards sums the error-table write statuses for the record counts (`SuccessfulRecordCounter.compute`). The write client's commit releases the write-status RDD it was handed (`releaseResources` unpersists what the write persisted), so the later aggregation re-evaluates the RDD's lineage. That lineage is the bulk insert itself, so the executors run the write a second time and produce a second base file under the instant that has already completed. The error table then reads two rows for every error record, while the commit metadata reports the right count. Observed with a `BaseErrorTableWriter` backed by `SparkRDDWriteClient.bulkInsert` + `commit`: the log shows a second `HoodieCreateHandle` for the error table's partition right after "Result of error table commit ... is true", writing a new file id under the same instant time. ### Expected The error-table write statuses are materialized once, before the error table commit releases them, and the counting step reads from that materialized set. One base file per error-table write, one row per error record. ### Environment Hudi master (1.3.0-SNAPSHOT), Spark 3.5. Affects any deployment using the unified error table write path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
