Nikita Sheremet created HUDI-3067:
-------------------------------------
Summary: "Table already exists" error with multiple writers
Key: HUDI-3067
URL: https://issues.apache.org/jira/browse/HUDI-3067
Project: Apache Hudi
Issue Type: Bug
Reporter: Nikita Sheremet
How reproduce:
# Set up multiple writing [https://hudi.apache.org/docs/concurrency_control/]
for dynamodb (do not forget to set _hoodie.write.lock.dynamodb.region_ and
{_}hoodie.write.lock.dynamodb.billing_mode{_}). Do not create anty dynamodb
table.
# Run multiple writers to the table
(Tested on aws EMR, so multiple writers is EMR steps)
Expected result - all steps completed.
Actual result: some steps failed with exception
{code:java}
Caused by: com.amazonaws.services.dynamodbv2.model.ResourceInUseException:
Table already exists: truedata_detections (Service: AmazonDynamoDBv2; Status
Code: 400; Error Code: ResourceInUseException; Request ID:; Proxy: null)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:6214)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:6181)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeCreateTable(AmazonDynamoDBClient.java:1160)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:1124)
at
org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.createLockTableInDynamoDB(DynamoDBBasedLockProvider.java:188)
at
org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:99)
at
org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:77)
... 54 more
21/12/19 13:42:06 INFO Yar {code}
This happens because all steps tried to create table at the same time.
Suggested solution:
A catch statment for _Table already exists_ exception should be added into
dynamodb table creation code. May be with delay and additional check that table
is present.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
