[GitHub] [hudi] jdattani commented on issue #5451: [SUPPORT] Hudi 0.10.1 raises exception java.lang.NoClassDefFoundError: com/amazonaws/services/dynamodbv2/model/LockNotGrantedException

[GitBox]

jdattani commented on issue #5451:
URL: https://github.com/apache/hudi/issues/5451#issuecomment-1113001589

   @yihua I tried adding retries as you suggested, still got the same error 
msg:  Unable to acquire lock, lock object null.
   
   Additional detail on the error states : **"The provided key element does not 
match the schema "**
   
   ```
   'hoodie.write.concurrency.mode': 'optimistic_concurrency_control',
   'hoodie.cleaner.policy.failed.writes': 'LAZY',
   'hoodie.write.lock.dynamodb.endpoint_url': 
'dynamodb.us-east-1.amazonaws.com',
   'hoodie.write.lock.provider': 
'org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider',
   'hoodie.write.lock.dynamodb.table': '<TABLE_NAME>',
   'hoodie.write.lock.dynamodb.partition_key': 'tablename',
   'hoodie.write.lock.dynamodb.region': 'us-east-1',
   'hoodie.write.lock.wait_time_ms':'1500',
   'hoodie.write.lock.num_retries':'10'
   ```
   
   I see it retried the operation 3 times.
   
   ```
   
   2022-04-29 07:55:09,998 WARN [Thread-8] dynamodbv2.AmazonDynamoDBLockClient 
(AmazonDynamoDBLockClient.java:acquireLock(534)): Could not acquire lock 
because of a client side failure in talking to DDB
   com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The 
provided key element does not match the schema (Service: AmazonDynamoDBv2; 
Status Code: 400; Error Code: ValidationException; Request ID: 
6EVA0VDQKC3Q7HNI6NNCCL67MJVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)
       at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
       at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
       at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
       at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
   
   2022-04-29 07:55:11,010 WARN [Thread-8] dynamodbv2.AmazonDynamoDBLockClient 
(AmazonDynamoDBLockClient.java:acquireLock(534)): Could not acquire lock 
because of a client side failure in talking to DDB
   com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The 
provided key element does not match the schema (Service: AmazonDynamoDBv2; 
Status Code: 400; Error Code: ValidationException; Request ID: 
M630K6QR1GRQ8UNJKRV1CGNQAJVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)
       at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
       at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
       at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
       at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
       at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
       at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770
   
   
   2022-04-29 07:55:12,018 WARN [Thread-8] dynamodbv2.AmazonDynamoDBLockClient 
(AmazonDynamoDBLockClient.java:acquireLock(534)): Could not acquire lock 
because of a client side failure in talking to DDB
   com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The 
provided key element does not match the schema (Service: AmazonDynamoDBv2; 
Status Code: 400; Error Code: ValidationException; Request ID: 
N1U0BAITEL1G6Q9HH4UJKO2VERVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)
       at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
       at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
       at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
       at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
       at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
   
   ```
   I checked the CloudTrail logs and could see a successful "DescribeTable" 
call from Glue Job to the DynamoDB lock table. So that rules out and IAM 
related issues. 
   
   The table has only one field - partition key "tablename" as seen in 
screenshot below
   
   <img width="991" alt="image" 
src="https://user-images.githubusercontent.com/14809344/165906389-95bf5357-675f-4fea-8e27-7533acbe8558.png";>
   
   I also manually inserted a dummy value in the table. 
   
   <img width="998" alt="image" 
src="https://user-images.githubusercontent.com/14809344/165906550-2d04497c-c8d7-4841-95da-579c4aba969a.png";>
   
   Do we also need to define a sort key on the table? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org