novisfff commented on issue #6095: URL: https://github.com/apache/hudi/issues/6095#issuecomment-1211483503
> Nope. It will never lose data. If process is crashed mid-way, the commit has also failed mid-way. So, next time when you restart your pipeline, the rollback of that partially failed commit will get triggered. > > Main purpose of the marker files are, during rollbacks, instead of doing fs.listing of entire data directory, we can exactly get hold of data files written as part of the commit of interest. And we have a contract where in, data files will be created only after marker files are created. So, unless the marker creation succeeds, the corresponding data file will not be created. > > So, given how timeline server based markers are designed, you should not see any data loss. > > let me know if you need any more clarification. happy to help. Thank you very much. I understand. I'm sorry I had some problems with my previous understanding of timeline server based markers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
