deepakpanda93 commented on issue #11567: URL: https://github.com/apache/hudi/issues/11567#issuecomment-4850563197
Closing this out — the trigger you identified (spark.speculation) is now guarded against in Hudi 1.2.0, and the related MDT-deletion behavior is now configurable. To recap: the record_index purge/rebuild happened when the same streaming program was started twice with spark.speculation enabled (as you concluded on 2024-08-06). This is a concurrent-duplicate-writer + speculation scenario, and @ad1happy2go wasn't able to reproduce it with a single writer. In 1.2.0: - A guardrail now fails fast if spark.speculation=true ([#18045](https://github.com/apache/hudi/pull/18045)) — removing the trigger you identified. - The automatic deletion of MDT partitions can be disabled via config ([#18181](https://github.com/apache/hudi/pull/18181)), so the record_index isn't silently purged. - Compaction now fails if another active writer is executing the same plan ([#18012](https://github.com/apache/hudi/pull/18012)), tightening multi-writer safety. Recommendations: - Disable spark.speculation for Hudi write jobs (regardless of version). - Ensure a single writer per table — don't start the same streaming job twice; if you genuinely need concurrent writers, configure OCC/NBCC with a lock provider. - Retest on 1.2.0+. Closing as mitigated in 1.2.0. If you still see record_index deleted with spark.speculation=false and a single writer on 1.2.0+, please reopen with the timeline and configs. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
