[
https://issues.apache.org/jira/browse/HUDI-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167843#comment-17167843
]
sivabalan narayanan commented on HUDI-1098:
-------------------------------------------
[~vinoth] : Isn't this automatically taken care of w/ upgrade infra?
Within upgrade, we delete the marker directory and recreate all marker files by
listing. So, if data file is not present, after recreating, corresponding
marker file should not be present.
> Marker file finalizing may block on a data file that was never written
> ----------------------------------------------------------------------
>
> Key: HUDI-1098
> URL: https://issues.apache.org/jira/browse/HUDI-1098
> Project: Apache Hudi
> Issue Type: Bug
> Components: Writer Core
> Reporter: Vinoth Chandar
> Assignee: sivabalan narayanan
> Priority: Blocker
> Fix For: 0.6.0
>
>
> {code:java}
> // Ensure all files in delete list is actually present. This is mandatory for
> an eventually consistent FS. // Otherwise, we may miss deleting such files.
> If files are not found even after retries, fail the commit
> if (consistencyCheckEnabled) {
> // This will either ensure all files to be deleted are present.
> waitForAllFiles(jsc, groupByPartition, FileVisibility.APPEAR);
> }
> {code}
> We need to handle the case where marker file was created, but we crashed
> before the data file was created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
