deepakpanda93 commented on issue #11767: URL: https://github.com/apache/hudi/issues/11767#issuecomment-4850586016
Closing this out — it's on Hudi 0.10.0 (well past EOL), it points to a concurrency setup, and the marker/rollback/cleaning subsystem has been substantially rewritten since. The FileAlreadyExistsException creating the rollback marker is a concurrent cleaning/rollback collision: with LAZY failed-writes cleaning (which, as @danny0405 noted, isn't lock-guarded on that version), two rollbacks race to create the same .marker.APPEND. That typically means more than one job/cleaner is operating on the table. Guidance / status: - Run cleaning from a single writer only, or configure OCC/NBCC with a lock provider so concurrent table services are coordinated. Don't run duplicate jobs against the same table. - Marker creation now tolerates a pre-existing marker (logs "already exists, cancel creation" and returns empty rather than failing), and marker-based rollback has been hardened (HUDI-5862 [#8077], HUDI-4550 [#6313]). - The timeline, markers, and concurrency control (NBCC) have all been reworked in the 1.x line. 0.10.0 is very old; please retest on a recent 1.x release with a single cleaning writer (or proper lock-guarded concurrency). If you still hit a rollback marker collision on 1.x, please reopen with the full writer configs and whether multiple jobs run against the table. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
