[
https://issues.apache.org/jira/browse/HUDI-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Y Ethan Guo updated HUDI-8420:
------------------------------
Description:
[https://github.com/apache/hudi/pull/11943#discussion_r1805674206]
The default implementation of {{getMandatoryFieldsForMerging}} should return
all columns since custom merging may need all columns regardless of the request
schema. Only the {{DefaultSparkRecordMerger}} and
{{{}OverwriteWithLatestSparkRecordMerger{}}}(and counterpart classes for other
engines) should overwrite the implementation to return record key and optional
preCombine field for better performance with projection.
Right now the user must provide {{getMandatoryFieldsForMerging}} to make sure
the functionality is correct, which is OK as long as we update the docs
(HUDI-8421). For better dev experience, we should not require user to override
this unless the user understands the tradeoff here.
was:
https://github.com/apache/hudi/pull/11943#discussion_r1805674206
The default implementation of {{getMandatoryFieldsForMerging}} should return
all columns since custom merging may need all columns regardless of the request
schema. Only the {{DefaultSparkRecordMerger}} and
{{{}OverwriteWithLatestSparkRecordMerger{}}}(and counterpart classes for other
engines) should overwrite the implementation to return record key and optional
preCombine field for better performance with projection.
Right now the user must provide {{getMandatoryFieldsForMerging}} to make sure
the functionality is correct, which is OK as long as we update the docs.
> Return all fields as mandatory for custom merging
> -------------------------------------------------
>
> Key: HUDI-8420
> URL: https://issues.apache.org/jira/browse/HUDI-8420
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Y Ethan Guo
> Priority: Critical
> Fix For: 1.1.0
>
>
> [https://github.com/apache/hudi/pull/11943#discussion_r1805674206]
> The default implementation of {{getMandatoryFieldsForMerging}} should return
> all columns since custom merging may need all columns regardless of the
> request schema. Only the {{DefaultSparkRecordMerger}} and
> {{{}OverwriteWithLatestSparkRecordMerger{}}}(and counterpart classes for
> other engines) should overwrite the implementation to return record key and
> optional preCombine field for better performance with projection.
>
> Right now the user must provide {{getMandatoryFieldsForMerging}} to make sure
> the functionality is correct, which is OK as long as we update the docs
> (HUDI-8421). For better dev experience, we should not require user to
> override this unless the user understands the tradeoff here.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
