[
https://issues.apache.org/jira/browse/HUDI-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prashant Wason closed HUDI-2013.
--------------------------------
Resolution: Fixed
> Fallback to file listing may lead to data loss
> ----------------------------------------------
>
> Key: HUDI-2013
> URL: https://issues.apache.org/jira/browse/HUDI-2013
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Prashant Wason
> Assignee: Prashant Wason
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> When fallback to file listing mode is enabled
> (hoodie.metadata.fallback.enable, default is true), then if listing from the
> metadata table leads to an exception the normal file-system listing used.
> Metadata table listing may fail if the table is inconsistent or due to bugs.
> Falling back to file listing has the following downsides:
> # It masks the issue as the commit does not fail (only an exception is
> logged).
> # By the time the issue is discovered, logs may have been lost
> # There is no guarantee that all the commits wrote/updated the correct files.
> Since listing from metadata table is per-partition, the issue is further
> complicated when listing for some partitions succeeds (file-list retrieved
> from metadata table) and fails for other partitions (file list retrieved from
> filesystem).
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
