Prashant Wason created HUDI-2013:
------------------------------------
Summary: Fallback to file listing may lead to data loss
Key: HUDI-2013
URL: https://issues.apache.org/jira/browse/HUDI-2013
Project: Apache Hudi
Issue Type: Bug
Reporter: Prashant Wason
Assignee: Prashant Wason
When fallback to file listing mode is enabled (hoodie.metadata.fallback.enable,
default is true), then if listing from the metadata table leads to an exception
the normal file-system listing used.
Metadata table listing may fail if the table is inconsistent or due to bugs.
Falling back to file listing has the following downsides:
# It masks the issue as the commit does not fail (only an exception is logged).
# By the time the issue is discovered, logs may have been lost
# There is no guarantee that all the commits wrote/updated the correct files.
Since listing from metadata table is per-partition, the issue is further
complicated when listing for some partitions succeeds (file-list retrieved from
metadata table) and fails for other partitions (file list retrieved from
filesystem).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
