[
https://issues.apache.org/jira/browse/HUDI-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-2774:
--------------------------------------
Priority: Blocker (was: Major)
> Async Clustering via deltstreamer fails with IllegalStateException: Duplicate
> key [==>20211116123724586__replacecommit__INFLIGHT]
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: HUDI-2774
> URL: https://issues.apache.org/jira/browse/HUDI-2774
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: sivabalan narayanan
> Assignee: Sagar Sumit
> Priority: Blocker
> Fix For: 0.10.0
>
> Attachments: Screen Shot 2021-11-16 at 12.42.20 PM.png
>
>
> Setup:
> Started deltastreamer with parquet dfs source. source folder did not have any
> data as such. Enabled async clustering with below props
> ```
> hoodie.clustering.async.max.commits=2
> hoodie.clustering.plan.strategy.sort.columns=type,id
> ```
> Added 1 file to the source folder. and deltastreamer failed during this.
> commit went through fine. 1st replace commit got scheduled. here is my hunch
> : and I guess in subsequent round of deltastreamer sync, it tries to schedule
> again and fails.
>
> clustering plan seems to be same in both requested meta files
> {code:java}
> grep "2b9b3f9d-f68c-4404-8352-1708089d2cca-0_13-49-202_20211116123721000"
> /tmp/hudi-deltastreamer-gh-mw/.hoodie/* | grep replacecommit
> grep: /tmp/hudi-deltastreamer-gh-mw/.hoodie/archived: Is a directory
> Binary file
> /tmp/hudi-deltastreamer-gh-mw/.hoodie/20211116123724586.replacecommit.requested
> matches
> Binary file
> /tmp/hudi-deltastreamer-gh-mw/.hoodie/20211116123725199.replacecommit.requested
> matches {code}
>
>
> timeline
> !Screen Shot 2021-11-16 at 12.42.20 PM.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
