[
https://issues.apache.org/jira/browse/HUDI-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liujinhui updated HUDI-3765:
----------------------------
Summary: Structured streaming MOR/COW table can not asynchronous clean
(was: Structured streaming MOR table can not asynchronous clean)
> Structured streaming MOR/COW table can not asynchronous clean
> --------------------------------------------------------------
>
> Key: HUDI-3765
> URL: https://issues.apache.org/jira/browse/HUDI-3765
> Project: Apache Hudi
> Issue Type: Bug
> Affects Versions: 0.11.0
> Reporter: liujinhui
> Priority: Major
>
> When writing to the mor table using structured streaming, when the
> asynchronous clean service is enabled, the clean will only be triggered when
> the task is restarted. Through debugging, it is found that in the process of
> continuous operation, the generation of clean will not actually be triggered.
> ...
>
> Although there will be the log, it will not trigger clean. Pretty sure that
> the number of version files that triggers a clean has been reached
> {code:java}
> // Async cleaner has been spawned. Waiting for it to finish |
> org.apache.hudi.client.BaseHoodieWriteClient.autoCleanOnCommit(BaseHoodieWriteClient.java:541)
> 2022-03-31 19:26:06,677 | INFO | [stream execution thread for [id =
> ce830c81-00c6-4d83-8a60-1970b8a6a1c9, runId =
> d57ad29c-c63e-441d-b183-0e5eb028acc5]] | Waiting for async clean service to
> finish |
> org.apache.hudi.async.AsyncCleanerService.waitForCompletion(AsyncCleanerService.java:73)
> 2022-03-31 19:26:06,677 | INFO | [stream execution thread for [id =
> ce830c81-00c6-4d83-8a60-1970b8a6a1c9, runId =
> d57ad29c-c63e-441d-b183-0e5eb028acc5]] | Async cleaner has finished |
> org.apache.hudi.client.BaseHoodieWriteClient.autoCleanOnCommit(BaseHoodieWriteClient.java:543)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
