hudi-bot opened a new issue, #14968:
URL: https://github.com/apache/hudi/issues/14968
How reproduce:
# Set up multiple writing
[https://hudi.apache.org/docs/concurrency_control/] for dynamodb (do not forget
to set _hoodie.write.lock.dynamodb.region_ and
{_}hoodie.write.lock.dynamodb.billing_mode{_}). Do not create anty dynamodb
table.
# Run multiple writers to the table
(Tested on aws EMR, so multiple writers is EMR steps)
Expected result - all steps completed.
Actual result: some steps failed with exception
{code:java}
Caused by: com.amazonaws.services.dynamodbv2.model.ResourceInUseException:
Table already exists: truedata_detections (Service: AmazonDynamoDBv2; Status
Code: 400; Error Code: ResourceInUseException; Request ID:; Proxy: null)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:6214)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:6181)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeCreateTable(AmazonDynamoDBClient.java:1160)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:1124)
at
org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.createLockTableInDynamoDB(DynamoDBBasedLockProvider.java:188)
at
org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:99)
at
org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:77)
... 54 more
21/12/19 13:42:06 INFO Yar {code}
This happens because all steps tried to create table at the same time.
Suggested solution:
A catch statment for _Table already exists_ exception should be added into
dynamodb table creation code. May be with delay and additional check that table
is present.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3067
- Type: Bug
- Fix version(s):
- 1.1.0
---
## Comments
30/Jan/22 19:32;shivnarayan;[~wenningd] : Can you please follow up on this.
;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
