deepakpanda93 commented on issue #11527: URL: https://github.com/apache/hudi/issues/11527#issuecomment-4846088166
Thanks for the detailed root-cause analysis. Status on the two points you raised: - Per-task bucket fileid generation (Spark): still the case — SparkBucketIndexPartitioner assigns the bucket fileid in the task, so a retried/speculative task can create a second fileid for the same bucket. The dedicated fix for this so far (HUDI-5373, [#7433](https://github.com/apache/hudi/pull/7433)) was Flink-only; the Spark side isn't addressed yet. Your suggestion to assign it deterministically/driver-side remains a valid enhancement (the memory concern @beyond1920 raised is the open design question). - Filtering failed-commit file slices in loadBucketIdToFileIdMappingForPartition: not implemented — the index still throws on a bucket conflict. What has improved: - On 1.2.0, a guardrail rejects writes when spark.speculation=true ([#18045](https://github.com/apache/hudi/pull/18045)), removing one common trigger. Set spark.speculation=false for Hudi bucket-index writes regardless of version. - On 1.0.0+, the error now identifies the conflicting instants and you can clean them with the rollback_to_instant procedure (HUDI-7522, [#10898](https://github.com/apache/hudi/pull/10898)) to recover the table. Keeping this open to track the Spark-side prevention (driver-side/deterministic bucket fileid and/or filtering uncommitted file slices). In the meantime: disable Spark speculation, and use rollback_to_instant to clear the duplicate-bucket files left by the failed attempt. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
