[
https://issues.apache.org/jira/browse/HUDI-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17892094#comment-17892094
]
Y Ethan Guo commented on HUDI-8420:
-----------------------------------
After reviewing the HUDI-8203 PR, the following API in HoodieRecordMerger
provides the functionality to return all fields:
{code:java}
/**
* If false, whenever we have log files, we will need to read all columns
* If true, mor merging can be done without all columns. The columns required
can be configured
* by overriding getMandatoryFieldsForMerging
*/
default boolean isProjectionCompatible() {
return false;
} {code}
> Return all fields as mandatory for custom merging
> -------------------------------------------------
>
> Key: HUDI-8420
> URL: https://issues.apache.org/jira/browse/HUDI-8420
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Y Ethan Guo
> Priority: Critical
> Fix For: 1.1.0
>
>
> [https://github.com/apache/hudi/pull/11943#discussion_r1805674206]
> The default implementation of {{getMandatoryFieldsForMerging}} should return
> all columns since custom merging may need all columns regardless of the
> request schema. Only the {{DefaultSparkRecordMerger}} and
> {{{}OverwriteWithLatestSparkRecordMerger{}}}(and counterpart classes for
> other engines) should overwrite the implementation to return record key and
> optional preCombine field for better performance with projection.
>
> Right now the user must provide {{getMandatoryFieldsForMerging}} to make sure
> the functionality is correct, which is OK as long as we update the docs
> (HUDI-8421). For better dev experience, we should not require user to
> override this unless the user understands the tradeoff here.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
