rangareddy commented on issue #9674: URL: https://github.com/apache/hudi/issues/9674#issuecomment-4842035880
This isn't a Hudi bug — it's the multi-writer safety contract. You're running a standalone `HoodieCleaner` concurrently with an active writer but neither process sets `hoodie.write.concurrency.mode` or a `hoodie.write.lock.provider`, so the cleaner builds its `KEEP_LATEST_COMMITS` plan from a timeline that doesn't see the in-flight upsert/compaction and deletes file slices that are still live → data loss. `failed.writes=LAZY` only addresses the eager-rollback race, which is why loss persisted. Fix: enable OCC + a distributed lock provider (DynamoDB on S3/EC2) with `failed.writes=LAZY`, and apply the **identical** lock config to BOTH the writer and the cleaner so they share the same lock table. Better still, drop the external cleaner and let the writer clean inline (`clean.automatic=true`, `clean.async=true`) so a single process owns the timeline. Also widen the archival window — `retained=48/keep.min=49/keep.max=50` is too tight; use something like 10/20/30. Lost slices must be re-ingested from your CDC source (or restored from a savepoint). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
