NewtonXu opened a new issue, #9018:
URL: https://github.com/apache/hudi/issues/9018
**Describe the problem you faced**
Trying to enable DynamoDB concurrency controls but unable to instantiate due
to builder error.
```
Caused by: org.apache.hudi.exception.HoodieException: Unable to instantiate
class org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:81)
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:108)
at
org.apache.hudi.client.transaction.lock.LockManager.getLockProvider(LockManager.java:118)
at
org.apache.hudi.client.transaction.lock.LockManager.lock(LockManager.java:71)
at
org.apache.hudi.client.transaction.TransactionManager.beginTransaction(TransactionManager.java:58)
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:226)
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:209)
at
org.apache.hudi.internal.DataSourceInternalWriterHelper.commit(DataSourceInternalWriterHelper.java:84)
... 125 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:79)
... 132 more
Caused by: java.lang.NoSuchMethodError:
com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions.builder(Lcom/amazonaws/services/dynamodbv2/AmazonDynamoDB;Ljava/lang/String;)Lcom/amazonaws/services/dynamodbv2/AmazonDynamoDBLockClientOptions$AmazonDynamoDBLockClientOptionsBuilder;
at
org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:91)
at
org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:77)
... 137 more
```
**To Reproduce**
Steps to reproduce the behavior:
Executing the Spark job with these packages:
```
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf
spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
--packages
org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.1,com.amazonaws:dynamodb-lock-client:1.2.0,com.amazonaws:aws-java-sdk-dynamodb:1.12.490,com.amazonaws:aws-java-sdk-core:1.12.490,org.apache.hudi:hudi-aws:0.13.1
--conf
spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
```
Enabling concurrency control with these settings
```
'hoodie.write.lock.provider':
"org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider",
'hoodie.write.lock.dynamodb.table': "LockTableName",
'hoodie.write.concurrency.mode': 'optimistic_concurrency_control',
'hoodie.cleaner.policy.failed.writes': "LAZY",
'hoodie.write.lock.dynamodb.region': 'us-east-1',
'hoodie.write.lock.dynamodb.partition_key': "TableName",
```
This job is created on AWS EMR Serverless
**Expected behavior**
DynamoDB lock is created
**Environment Description**
* Hudi version : 13.1
* Spark version : Spark version 3.3.1-amzn-0
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
I'm running this job on Amazon EMR serverless, but I've decided to use the
open-source Hudi bundle instead.
Previously when using the AWS bundle, it could not find
DynamoDBBasedLockProvider.
**Stacktrace**
```Add the stacktrace of the error.```
```
Caused by: org.apache.hudi.exception.HoodieException: Unable to instantiate
class org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:81)
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:108)
at
org.apache.hudi.client.transaction.lock.LockManager.getLockProvider(LockManager.java:118)
at
org.apache.hudi.client.transaction.lock.LockManager.lock(LockManager.java:71)
at
org.apache.hudi.client.transaction.TransactionManager.beginTransaction(TransactionManager.java:58)
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:226)
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:209)
at
org.apache.hudi.internal.DataSourceInternalWriterHelper.commit(DataSourceInternalWriterHelper.java:84)
... 125 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:79)
... 132 more
Caused by: java.lang.NoSuchMethodError:
com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions.builder(Lcom/amazonaws/services/dynamodbv2/AmazonDynamoDB;Ljava/lang/String;)Lcom/amazonaws/services/dynamodbv2/AmazonDynamoDBLockClientOptions$AmazonDynamoDBLockClientOptionsBuilder;
at
org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:91)
at
org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:77)
... 137 more
```
I am wondering if there is some incompatibility with the SDK's I've chosen?
org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.1
com.amazonaws:dynamodb-lock-client:1.2.0
com.amazonaws:aws-java-sdk-dynamodb:1.12.490
com.amazonaws:aws-java-sdk-core:1.12.490
org.apache.hudi:hudi-aws:0.13.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
