gtwuser commented on issue #6925: URL: https://github.com/apache/hudi/issues/6925#issuecomment-1303982961
> > oh btw, one issue I see w/ you configs is, you are not setting the configs properly for multi-writer. https://hudi.apache.org/docs/concurrency_control#enabling-multi-writing > > ``` > > hoodie.write.concurrency.mode=optimistic_concurrency_control > > hoodie.cleaner.policy.failed.writes=LAZY > > hoodie.write.lock.provider=<lock-provider-classname> > > ``` > > > > > > > > > > > > > > > > > > > > > > > > you were setting the last one, but not the first one. > > Also, can you first try out using just 1 thread along w/ setting above configs. we can go from there if basic set up is not working. > > Sorry for the delay. But now after adding above changes we are getting this error now : `Caused by: java.lang.ClassNotFoundException: org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider` [link to full stack of error ](https://gist.github.com/gtwuser/cd23f4851aab035774717f08ce1fbfc2) I feel we are close to fix it, so please let me know whats that im missing now. I checked the classpath of the `DynamoDBBasedLockProvider` it seems correct. > > PLEASE NOTE THE JARS USED ARE FROM HUDI `v0.13.0` **jars used**: > > ```shell > hudi-utilities-bundle_2.12-0.13.0-SNAPSHOT.jar, > hudi-spark3.1-bundle_2.12-0.13.0-SNAPSHOT.jar, > calcite-core-1.30.0.jar > ``` > > **config used for locking**: > > ```shell > 'className': 'org.apache.hudi', > 'hoodie.datasource.hive_sync.use_jdbc': 'false', > 'hoodie.datasource.write.precombine.field': 'payload.recordedAt', > 'hoodie.datasource.write.recordkey.field': 'metadata.msgID,metadata.topic', > 'hoodie.write.lock.provider': 'org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider', > 'hoodie.write.lock.dynamodb.table': 'hudi_db_lock', > 'hoodie.write.lock.dynamodb.partition_key':'hudi_db_lock', > 'hoodie.write.concurrency.mode': 'optimistic_concurrency_control', > 'hoodie.write.lock.dynamodb.endpoint_url':'dynamodb.us-east-1.amazonaws.com', > 'hoodie.cleaner.policy.failed.writes': 'LAZY', > ``` This same issue if ClassNotFoundException is reproducible with below jars as well which is from hudi build `0.12.1`: ```bash hudi-utilities_2.12-0.12.1.jar, hudi-spark3.1-bundle_2.12-0.12.1.jar, calcite-core-1.30.0.jar ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
