Jason-liujc opened a new issue, #9512:
URL: https://github.com/apache/hudi/issues/9512
**Describe the problem you faced**
We have a usecase where we need a table level lock for multiple EMR clusters
writing to the same Hudi table. We've tried the two options provided by Hudi
OCC guide page.
For `single_writer` option, we don't see any lock being created in DynamoDB,
therefore there were no write locks and the EMR Hudi upsert job fails due to
multiple writer writing to the same table.
For `optimistic_concurrency_control`, it's creating and deleting many locks
in DynamoDB entry. And the lock is not on table level. So when two jobs write
to the same file, one of the job will fail. This leads to us having to add a
lot of retries in our cluster, which is not ideal.
Is there any
**To Reproduce**
Steps to reproduce the behavior:
1. Run Hudi upsert jobs in multiple AWS EMR cluster writing to the same
table at the same time.
2. Use DynamoDB as the lock provider, these jobs write to the same
partitions.
3. Use `single_writer` + `hoodie.cleaner.policy.failed.writes=EAGER`or
`optimistic_concurrency_control` + `hoodie.cleaner.policy.failed.writes=LAZY`
as part of the Hudi write option.
4. See each time, the jobs will fail due to different errors.
**Expected behavior**
Hudi would provide a table level lock that allow us to block other writers
from writing until one of the writer is done when writing to the same partition.
**Environment Description**
* Hudi version : 0.13.0 (EMR 6.11)
* Spark version : 3.3.1
* Hive version : 3.1.3
* Hadoop version : 3.3.3
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : No
**Additional context**
We are considering building something our own to enforce sequential
insertion during orchestration. But this can be avoided if Hudi provide a table
level lock.
**Stacktrace**
Single writer:
```
23/08/21 23:19:47 ERROR Client: Application diagnostics message: User class
threw exception: org.apache.hudi.exception.HoodieRollbackException: Failed to
rollback s3://xxxxbucket/xxxkey commits 20230821231625673
at
org.apache.hudi.client.BaseHoodieTableServiceClient.rollback(BaseHoodieTableServiceClient.java:823)
at
org.apache.hudi.client.BaseHoodieTableServiceClient.rollbackFailedWrites(BaseHoodieTableServiceClient.java:727)
at
org.apache.hudi.client.BaseHoodieTableServiceClient.rollbackFailedWrites(BaseHoodieTableServiceClient.java:711)
at
org.apache.hudi.client.BaseHoodieTableServiceClient.rollbackFailedWrites(BaseHoodieTableServiceClient.java:706)
at
org.apache.hudi.client.BaseHoodieWriteClient.lambda$startCommitWithTime$97cdbdca$1(BaseHoodieWriteClient.java:843)
```
Multi writer with `optimistic_concurrency_control`:
```
23/08/21 21:40:31 ERROR Client: Application diagnostics message: User class
threw exception: org.apache.hudi.exception.HoodieWriteConflictException:
java.util.ConcurrentModificationException: Cannot resolve conflicts for
overlapping writes
at
org.apache.hudi.client.transaction.SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict(SimpleConcurrentFileWritesConflictResolutionStrategy.java:108)
at
org.apache.hudi.client.utils.TransactionUtils.lambda$resolveWriteConflictIfAny$0(TransactionUtils.java:85)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at
java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
at
java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
