rangareddy commented on issue #10882: URL: https://github.com/apache/hudi/issues/10882#issuecomment-4852587890
Since this started only after the 0.11.1 → 0.13.1 upgrade with the same config that was clean before, and both dups are in the same file group/base file, this looks like an intermittent index-tagging miss — the upsert occasionally fails to tag an existing key as an update and inserts a duplicate, which compaction then merges into the base file. Given nsivabalan's note, the metadata table is the prime suspect: a jar-swap upgrade re-bootstraps MDT over existing data, and a stale MDT index (esp. bloom filters) can cause false-negative lookups. Quick way to confirm: run with `hoodie.metadata.enable=false` and see if dups stop. If so, rebuild the MDT cleanly (remove `<tablePath>/.hoodie/metadata` and let it re-bootstrap), and consider switching the index type (SIMPLE/GLOBAL_SIMPLE, or RECORD_INDEX on 0.14+) to bypass the MDT bloom path. Multiple MDT/index consistency bugs were fixed after 0.13.1, so upgrading to 0.14.x/1.x is the durable fix. For existing dups, use snapshot-query � �� stage to parquet → delete by key → re-upsert (deduped on max ts), since the CLI dedup is unreliable on MOR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
