Sahil333 opened a new issue, #18266:
URL: https://github.com/apache/hudi/issues/18266
### Describe the problem you faced
I am building a SparkApplication in java that has two Spark queries running
concurrently. First query is an upsert StreamingQuery which is ingesting from a
kafka topic to a hudi table. The other query is a simple spark delete query
that does delete of records in batches from same table after publishing them to
kafka. The application is supposed to have 5 executor instances. From
documentation, it says if the query shares the JVM it shall work which is the
case for driver instance but it doesn't mention if it will work with multiple
executor instances as well. In my testing, I do get a "HoodieConflictException"
after enabling below options on both queries and then, I retry the hudi write
operation again.
.option("hoodie.write.concurrency.mode", "optimistic_concurrency_control")
.option("hoodie.write.concurrency.early.conflict.detection.enable", "true")
.option("hoodie.write.lock.provider",
"org.apache.hudi.client.transaction.lock.InProcessLockProvider")
### To Reproduce
N/A
### Expected behavior
N/A
### Environment Description
* Hudi version: 1.1.0
* Spark version: 3.5.6
* Flink version:
* Hive version:
* Hadoop version: 3.4.6
* Storage (HDFS/S3/GCS..): S3
* Running on Docker? (yes/no): yes
### Additional context
_No response_
### Stacktrace
```shell
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
