[
https://issues.apache.org/jira/browse/HUDI-3365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar updated HUDI-3365:
---------------------------------
Sprint: Hudi-Sprint-Jan-31, Hudi-Sprint-Feb-7, Hudi-Sprint-Feb-14 (was:
Hudi-Sprint-Jan-31, Hudi-Sprint-Feb-7)
> Make sure Metadata Records always bear full file-size
> -----------------------------------------------------
>
> Key: HUDI-3365
> URL: https://issues.apache.org/jira/browse/HUDI-3365
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Alexey Kudinkin
> Assignee: Alexey Kudinkin
> Priority: Blocker
> Labels: HUDI-bug, pull-request-available
> Fix For: 0.11.0
>
>
> Currently, when log-file is appended the Metadata Table record w/ the size of
> appended delta will be submitted to MT.
> MT, in turn, will sum it up with whatever record is currently persisted there
> at the moment (REF:
> [https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java#L227)]
>
> This is problematic in multiple ways:
> # We're abusing FileStatus interface which unequivocally relates the size as
> a full file-size.
> # MT receiving new record can't determine whether it's received a delta or a
> new record that have to override the old one. As such, it stick to the
> protocol, that if record already exists it will treat new one as a delta.
> This behavior is very implicit, and there are currently at least one bug
> where actually full file-size is provided, while it would lead to the
> incorrect file size being stored in MT.
>
> Proposal: Unify the data flow and always provide full, up-to-date file-size
> to the MT. Even in the log-file appending flow, we do have a way to
> reconstruct full file-size (from `AppendResult`, w/o additional
> `getFileStatus` necessary
>
> Currently, when log-file is appended (on FS that supports it), only the
> appended delta will be submitted w/in records to MT
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
