[
https://issues.apache.org/jira/browse/HUDI-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17375781#comment-17375781
]
ASF GitHub Bot commented on HUDI-2016:
--------------------------------------
vinothchandar merged pull request #3083:
URL: https://github.com/apache/hudi/pull/3083
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> Metadata table bootstrap does not work when there are inflight instances
> ------------------------------------------------------------------------
>
> Key: HUDI-2016
> URL: https://issues.apache.org/jira/browse/HUDI-2016
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Prashant Wason
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> There is a race condition in metadata table bootstrap when there are inflight
> instances.
> Example: Assume a CLEAN is in progress which is planning to delete
> p1/f1.parquet (as per clean plan). If bootstrap is going on at the same time,
> there are two cases possible:
> # bootstrap lists files in partition p1 BEFORE clean deletes them
> ## hence p1/f1.parquet is added to metadata table during bootstrap
> ## When processing the CLEAN, p1/f1.parquet will be deleted from metadata
> table
> # bootstrap lists files in partition p1 AFTER clean deletes them
> ## p1/f1.parquet is not found
> ## When processing the CLEAN, p1/f1.parquet will be deleted from metadata
> table
> We cannot differenciate 2.2 from the case that we missed adding p1/f1.parquet
> to the metadata table.
> There is an exception in the metadata reader code to ensure that that any
> file being deleted was added to the metadata table. This exception is throws
> in case 2.2 above.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
