SamarthRaval commented on issue #9728: URL: https://github.com/apache/hudi/issues/9728#issuecomment-1765275004
> Thanks. We've tried the newest update from `DynamoDBBasedLockProvider` and `DynamoDbBasedLockConfig` but we are still seeing jobs fail pretty soon if encountering conflicting writes. > > Just going through the source code, I'm seeing `ZookeeperBasedLockProvider` is using retry under the hood: > > https://github.com/apache/hudi/blob/9e9f768434333380640ba5e452b168f9e92ee110/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/ZookeeperBasedLockProvider.java#L58 > > Whereas DynamoDB is not: > > https://github.com/apache/hudi/blob/9e9f768434333380640ba5e452b168f9e92ee110/hudi-aws/src/main/java/org/apache/hudi/aws/transaction/lock/DynamoDBBasedLockProvider.java#L62 > > My theory right now is when we are using DynamoDB Lock Provider, the retry parameters are not used, hence the job fails pretty fast. (i.e. `hoodie.write.lock.num_retries`). Since going through the Hudi created DynamoDB entry, I don't see any state regarding how many retries a given lock was used. > > Let me know if that's the case or an expected behavior from Hudi. Can we please confirm this issue if it persists ? Should we start using zookeeper rather then DynamoDB lock ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
