[
https://issues.apache.org/jira/browse/HUDI-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447520#comment-17447520
]
sivabalan narayanan commented on HUDI-2567:
-------------------------------------------
Scenarios tested:
1. Two concurrent writers writing to two different partitions. green. validated
metadata file listing.
2. two concurrent writers writing to two different file groups in same
partition. green. validated metadata file listing.
3. 3 concurrent writers. 2 of them overlap wrt file groups, while the other
does not have any over lap. 2 out of 3 succeeded and 1 failed. green. validated
metadata file listing.
4. Deltastreamer and spark data source writer. spark-data source writer failed
when it had overlapping writes wrt delta streamer commits. green. validated
metadata file listing.
5. Deltastreamer in continuous mode and async compaction. concurrent write from
spark-datasource succeeds if started just after compaction got scheduled for
same file group. green.
6. Deltastreamer in continuous mode with async compaction and async clustering.
Used allow updates strategy for clustering updates as I was focused on
validating metadata file listing. Ran it for 20+ commits. all good.
7. same set up as above, but crashed compaction 2 times. ran into the metadata
inconsistent issue. but after bypassing that, things were good. regular delta
commits were succeeding. and concurrent writer from spark-datasource succeeds
if there is not over-lapping delta commits. if not, it fails.
> Verify synchronous metadata patch w/ multi writers end to end
> -------------------------------------------------------------
>
> Key: HUDI-2567
> URL: https://issues.apache.org/jira/browse/HUDI-2567
> Project: Apache Hudi
> Issue Type: Sub-task
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Blocker
> Labels: oct18_2021
> Fix For: 0.10.0
>
>
> Verify synchronous metadata patch w/ multi writers end to end
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
