[
https://issues.apache.org/jira/browse/HUDI-8661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Y Ethan Guo updated HUDI-8661:
------------------------------
Description:
The new file group reader-based compaction in
HoodieSparkFileGroupReaderBasedMergeHandle has not supported log compaction
yet. Right now it's disabled through the code below in HoodieCompactor. When
enabled, there are a few tests that fail:
TestHoodieMergeOnReadTable.testLogBlocksCountsAfterLogCompaction,
TestHoodieClientOnMergeOnReadStorage.testArchivalOnLogCompaction.
{code:java}
boolean useFileGroupReaderBasedCompaction = context.supportsFileGroupReader()
// the engine needs to support fg reader first
&& !metaClient.isMetadataTable()
&& config.getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED)
&& operationType == WriteOperationType.COMPACT
&& !hasBootstrapFile(operations)
// bootstrap file read for fg reader is not ready
&& StringUtils.isNullOrEmpty(config.getInternalSchema())
// schema evolution support for fg reader is not ready
&& config.populateMetaFields();
// Virtual key support by fg reader is not ready
{code}
> Support log compaction in the new file group reader
> ---------------------------------------------------
>
> Key: HUDI-8661
> URL: https://issues.apache.org/jira/browse/HUDI-8661
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Y Ethan Guo
> Priority: Critical
> Fix For: 1.0.1
>
>
> The new file group reader-based compaction in
> HoodieSparkFileGroupReaderBasedMergeHandle has not supported log compaction
> yet. Right now it's disabled through the code below in HoodieCompactor.
> When enabled, there are a few tests that fail:
> TestHoodieMergeOnReadTable.testLogBlocksCountsAfterLogCompaction,
> TestHoodieClientOnMergeOnReadStorage.testArchivalOnLogCompaction.
> {code:java}
> boolean useFileGroupReaderBasedCompaction = context.supportsFileGroupReader()
> // the engine needs to support fg reader first
> && !metaClient.isMetadataTable()
> &&
> config.getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED)
> && operationType == WriteOperationType.COMPACT
> && !hasBootstrapFile(operations)
> // bootstrap file read for fg reader is not ready
> && StringUtils.isNullOrEmpty(config.getInternalSchema())
> // schema evolution support for fg reader is not ready
> && config.populateMetaFields();
> // Virtual key support by fg reader is not ready
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
