[
https://issues.apache.org/jira/browse/HUDI-8661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17910873#comment-17910873
]
Y Ethan Guo commented on HUDI-8661:
-----------------------------------
We need to introduce a new append handle leveraging fg reader (similar to new
compaction implementation based on fg reader), so we can get rid of the old
logic in compactor in the future.
Punting this for now since log compaction is not used much at this point.
> Support log compaction in HoodieSparkFileGroupReaderBasedMergeHandle
> --------------------------------------------------------------------
>
> Key: HUDI-8661
> URL: https://issues.apache.org/jira/browse/HUDI-8661
> Project: Apache Hudi
> Issue Type: Sub-task
> Reporter: Y Ethan Guo
> Assignee: Y Ethan Guo
> Priority: Blocker
> Fix For: 1.0.1
>
> Original Estimate: 8h
> Time Spent: 1h
> Remaining Estimate: 7h
>
> The new file group reader-based compaction in
> HoodieSparkFileGroupReaderBasedMergeHandle has not supported log compaction
> yet. Right now it's disabled through the code below in HoodieCompactor.
> When enabled, there are a few tests that fail:
> TestHoodieMergeOnReadTable.testLogBlocksCountsAfterLogCompaction,
> TestHoodieClientOnMergeOnReadStorage.testArchivalOnLogCompaction.
> {code:java}
> boolean useFileGroupReaderBasedCompaction = context.supportsFileGroupReader()
> // the engine needs to support fg reader first
> && !metaClient.isMetadataTable()
> &&
> config.getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED)
> && operationType == WriteOperationType.COMPACT
> && !hasBootstrapFile(operations)
> // bootstrap file read for fg reader is not ready
> && StringUtils.isNullOrEmpty(config.getInternalSchema())
> // schema evolution support for fg reader is not ready
> && config.populateMetaFields();
> // Virtual key support by fg reader is not ready
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
