jtchen-study opened a new issue, #5807:
URL: https://github.com/apache/hudi/issues/5807
HoodieWriteConfig cfg =
HoodieWriteConfig.newBuilder().withPath(tablePath)
.withSchema(tableSchema)
.forTable(tableName)
.withIndexConfig(HoodieIndexConfig.newBuilder().withIndexType(HoodieIndex.IndexType.BLOOM).build())
.withCompactionConfig(HoodieCompactionConfig.newBuilder()
.withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.LAZY)
.build())
.withWriteConcurrencyMode(WriteConcurrencyMode.OPTIMISTIC_CONCURRENCY_CONTROL)
.withLockConfig(HoodieLockConfig.newBuilder()
.withLockProvider(InProcessLockProvider.class)
.build())
.build();
my client config is above, but when i try to write to hudi with multi thread
in a process , it produces the following error , is there any solution ?
org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit
time 20220609103047174
at
org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:64)
~[hudi-client-common-0.11.0.jar:0.11.0]
at
org.apache.hudi.table.action.commit.JavaInsertCommitActionExecutor.execute(JavaInsertCommitActionExecutor.java:47)
~[hudi-java-client-0.11.0.jar:0.11.0]
at
org.apache.hudi.table.HoodieJavaCopyOnWriteTable.insert(HoodieJavaCopyOnWriteTable.java:111)
~[hudi-java-client-0.11.0.jar:0.11.0]
at
org.apache.hudi.table.HoodieJavaCopyOnWriteTable.insert(HoodieJavaCopyOnWriteTable.java:82)
~[hudi-java-client-0.11.0.jar:0.11.0]
at
org.apache.hudi.client.HoodieJavaWriteClient.insert(HoodieJavaWriteClient.java:129)
~[hudi-java-client-0.11.0.jar:0.11.0]
at
com.alibaba.datax.plugin.writer.hudiwriter.HudiWriterProxy.doBatchInsert(HudiWriterProxy.java:119)
~[hudiwriter-0.0.1-SNAPSHOT.jar:na]
at
com.alibaba.datax.plugin.writer.hudiwriter.HudiWriter$Task.startWrite(HudiWriter.java:109)
~[hudiwriter-0.0.1-SNAPSHOT.jar:na]
at
com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:114)
[datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: org.apache.hudi.exception.HoodieWriteConflictException:
java.util.ConcurrentModificationException: Cannot resolve conflicts for
overlapping writes
at
org.apache.hudi.client.transaction.SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict(SimpleConcurrentFileWritesConflictResolutionStrategy.java:102)
~[hudi-client-common-0.11.0.jar:0.11.0]
at
org.apache.hudi.client.utils.TransactionUtils.lambda$resolveWriteConflictIfAny$0(TransactionUtils.java:85)
~[hudi-client-common-0.11.0.jar:0.11.0]
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1380)
~[na:1.8.0_151]
at
java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
~[na:1.8.0_151]
at
java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
~[na:1.8.0_151]
at
java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
~[na:1.8.0_151]
at
org.apache.hudi.client.utils.TransactionUtils.resolveWriteConflictIfAny(TransactionUtils.java:79)
~[hudi-client-common-0.11.0.jar:0.11.0]
at
org.apache.hudi.table.action.commit.BaseCommitActionExecutor.autoCommit(BaseCommitActionExecutor.java:189)
~[hudi-client-common-0.11.0.jar:0.11.0]
at
org.apache.hudi.table.action.commit.BaseCommitActionExecutor.commitOnAutoCommit(BaseCommitActionExecutor.java:175)
~[hudi-client-common-0.11.0.jar:0.11.0]
at
org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor.updateIndexAndCommitIfNeeded(BaseJavaCommitActionExecutor.java:347)
~[hudi-java-client-0.11.0.jar:0.11.0]
at
org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor.execute(BaseJavaCommitActionExecutor.java:126)
~[hudi-java-client-0.11.0.jar:0.11.0]
at
org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor.execute(BaseJavaCommitActionExecutor.java:68)
~[hudi-java-client-0.11.0.jar:0.11.0]
at
org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:57)
~[hudi-client-common-0.11.0.jar:0.11.0]
... 8 common frames omitted
Caused by: java.util.ConcurrentModificationException: Cannot resolve
conflicts for overlapping writes
... 21 common frames omitted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
