Limess opened a new issue, #8129: URL: https://github.com/apache/hudi/issues/8129
**Describe the problem you faced** When using `org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider` as a lock provider for Hudi, the documentation suggests that `hoodie.write.lock.dynamodb.partition_key` will by default be set to the table name if not set. https://hudi.apache.org/docs/0.12.2/configurations/ > For DynamoDB based lock provider, the partition key for the DynamoDB lock table. Each Hudi dataset should has it's unique key so concurrent writers could refer to the same partition key. By default we use the Hudi table name specified to be the partition key However this behaviour doesn't seem to work. When `hoodie.write.lock.dynamodb.partition_key` is not specified I see the following error: ``` Caused by: org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider ``` due to `java.lang.IllegalArgumentException: Property hoodie.write.lock.dynamodb.partition_key not found` **To Reproduce** Steps to reproduce the behavior: 1. Create a Hudi job with * `hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider` * `hoodie.write.lock.dynamodb.table=test_dynamodb_table` * `hoodie.write.lock.dynamodb.region=us-east-1`, * `hoodie.write.lock.dynamodb.endpoint_url=dynamodb.us-east-1.amazonaws.com` * `hoodie.write.lock.dynamodb.billing_mode=PAY_PER_REQUEST` * `hoodie.table.name=test_table` Do not set `hoodie.write.lock.dynamodb.partition_key` 3. Observe that it fails with above exception **Expected behavior** I'd expect `hoodie.write.lock.dynamodb.partition_key` to use `hoodie.table.name` here. **Environment Description** * Hudi version : 0.12.2 * Spark version : 3.3.1 * Hive version : N/A * Hadoop version : N/A * Storage (HDFS/S3/GCS..) : S3 * Running on Docker? (yes/no) : No **Additional context** AWS EMR 6.10.0 **Stacktrace** ``` Caused by: org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider at org.apache.hudi.internal.DataSourceInternalWriterHelper.commit(DataSourceInternalWriterHelper.java:87) at org.apache.hudi.spark3.internal.HoodieDataSourceInternalBatchWrite.commit(HoodieDataSourceInternalBatchWrite.java:93) at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2(WriteToDataSourceV2Exec.scala:392) ... 88 more Suppressed: org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:91) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:118) at org.apache.hudi.client.transaction.lock.LockManager.getLockProvider(LockManager.java:116) at org.apache.hudi.client.transaction.lock.LockManager.lock(LockManager.java:67) at org.a pache.hudi.client.transaction.TransactionManager.beginTransaction(TransactionManager.java:53) at org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1473) at org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1508) at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:750) at org.apache.hudi.internal.DataSourceInternalWriterHelper.abort(DataSourceInternalWriterHelper.java:95) at org.apache.hudi.spark3.internal.HoodieDataSourceInternalBatchWrite.abort(HoodieDataSourceInternalBatchWrite.java:98) at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2(WriteToDataSourceV2Exec.scala:399) ... 88 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl .newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:89) ... 98 more Caused by: java.lang.IllegalArgumentException: Property hoodie.write.lock.dynamodb.partition_key not found at org.apache.hudi.common.config.TypedProperties.checkKey(TypedProperties.java:67) at org.apache.hudi.common.config.TypedProperties.getString(TypedProperties.java:72) at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.checkRequiredProps(DynamoDBBasedLockProvider.java:206) at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:81) at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:77) ... 103 more -- Caused by: org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:91) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:118) at org.apache.hudi.client.transaction.lock.LockManager.getLockProvider(LockManager.java:116) at org.apache.hudi.client.transaction.lock.LockManager.lock(LockManager.java:67) at org.apache.hudi.client.transaction.TransactionManager.beginTransaction(TransactionManager.java:53) at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:239) at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:223) at org.apache.hudi.internal.DataSourceInternalWriterHelper.commit(DataSourceInternalWriterHelper.java:84) ... 90 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:89) ... 97 more Caused by: java.lang.IllegalArgumentException: Property hoodie.write.lock.dynamodb.partition_key not found at org.apache.hudi.common.config.TypedProperties.checkKey(TypedProperties.java:67) at org.apache.hudi.common.config.TypedProperties.getString(TypedProperties.java:72) at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.checkRequiredProps(DynamoDBBasedLockProvider.java:206) at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:81) at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:77) ... 102 more ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
