Jason-liujc commented on issue #9728: URL: https://github.com/apache/hudi/issues/9728#issuecomment-1726647147
Thanks. We've tried the newest update from `DynamoDBBasedLockProvider` and `DynamoDbBasedLockConfig` but we are still seeing jobs fail pretty soon if encountering conflicting writes. Just going through the source code, I'm seeing `ZookeeperBasedLockProvider` is using retry under the hood: https://github.com/apache/hudi/blob/9e9f768434333380640ba5e452b168f9e92ee110/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/ZookeeperBasedLockProvider.java#L58 Whereas DynamoDB is not: https://github.com/apache/hudi/blob/9e9f768434333380640ba5e452b168f9e92ee110/hudi-aws/src/main/java/org/apache/hudi/aws/transaction/lock/DynamoDBBasedLockProvider.java#L62 My theory right now is when we are using DynamoDB Lock Provider, the retry parameters are not used, hence the job fails pretty fast. (i.e. `hoodie.write.lock.num_retries`). Since going through the Hudi created DynamoDB entry, I don't see any state regarding how many retries a given lock was used. Let me know if that's the case or an expected behavior from Hudi. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
