rangareddy commented on issue #8178: URL: https://github.com/apache/hudi/issues/8178#issuecomment-4834997602
The two symptoms have two causes: "different commit time" dupes come from non-global `BLOOM` being partition-scoped — if a `guid` moves across `collectionName`, the index can't see the old copy and re-inserts it; "same commit time" dupes come from `ROLLBACK_USING_MARKERS_ENABLE=false`, which leaves partial files from failed streaming batches uncleaned. Fix: re-enable marker-based rollback, and if `guid` must be unique across partitions switch to a global index — but only after upgrading to 0.14.0+, since pre-0.14 global index had its own dup bug (HUDI-5968 / #8490). To clean existing dupes, don't use the CLI (unreliable on MOR) — snapshot-query the dupes, stage to parquet, DELETE by key, then re-upsert one deduped version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
