chenshzh opened a new pull request, #5991: URL: https://github.com/apache/hudi/pull/5991
### Change Logs Add separate control for Flink compaction operation sync/async mode. Details in https://issues.apache.org/jira/projects/HUDI/issues/HUDI-4329 #### Problem Review The compact operation sync/async in CompactionFunction is now controlled by FlinkOptions#COMPACTION_ASYNC_ENABLED ```java public CompactFunction(Configuration conf) { this.conf = conf; this.asyncCompaction = StreamerUtil.needsAsyncCompaction(conf); } ``` While in fact it cannot be switched to sync mode because the pipeline defined by sync compaction will only include the clean but not compact operators. ```java // compaction if (StreamerUtil.needsAsyncCompaction(conf)) { return Pipelines.compact(conf, pipeline); } else { return Pipelines.clean(conf, pipeline); } ``` #### Improvement Add another separate control switch for compaction operation sync/async mode. ### Impact add sync compaction switch for MoR compaction. ### Risk level (write none, low medium or high below) _If medium or high, explain what verification was done to mitigate the risks._ ### Documentation Update Add `compaction.operation.async.enabled`: used to turn on the synchronous compaction operation The original `compaction.async.enabled`: used to turn on the compaction process. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
