[
https://issues.apache.org/jira/browse/HUDI-8661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Y Ethan Guo reassigned HUDI-8661:
---------------------------------
Assignee: Y Ethan Guo
> Support log compaction in HoodieSparkFileGroupReaderBasedMergeHandle
> --------------------------------------------------------------------
>
> Key: HUDI-8661
> URL: https://issues.apache.org/jira/browse/HUDI-8661
> Project: Apache Hudi
> Issue Type: Sub-task
> Reporter: Y Ethan Guo
> Assignee: Y Ethan Guo
> Priority: Blocker
> Fix For: 1.0.1
>
> Original Estimate: 8h
> Time Spent: 1h
> Remaining Estimate: 7h
>
> The new file group reader-based compaction in
> HoodieSparkFileGroupReaderBasedMergeHandle has not supported log compaction
> yet. Right now it's disabled through the code below in HoodieCompactor.
> When enabled, there are a few tests that fail:
> TestHoodieMergeOnReadTable.testLogBlocksCountsAfterLogCompaction,
> TestHoodieClientOnMergeOnReadStorage.testArchivalOnLogCompaction.
> {code:java}
> boolean useFileGroupReaderBasedCompaction = context.supportsFileGroupReader()
> // the engine needs to support fg reader first
> && !metaClient.isMetadataTable()
> &&
> config.getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED)
> && operationType == WriteOperationType.COMPACT
> && !hasBootstrapFile(operations)
> // bootstrap file read for fg reader is not ready
> && StringUtils.isNullOrEmpty(config.getInternalSchema())
> // schema evolution support for fg reader is not ready
> && config.populateMetaFields();
> // Virtual key support by fg reader is not ready
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
