boneanxs commented on PR #5048: URL: https://github.com/apache/hudi/pull/5048#issuecomment-1118504685
> @boneanxs is this more like a transient issue for readers? There could be cases where empty or partial complete instant file is left in the timeline. There are fixes to it: https://github.com/apache/hudi/pull/4016 https://github.com/apache/hudi/pull/5261 . Do they solve your problem? @yihua sorry for a late reply, It's not the same issue, this will cause data loss when doing incremental reading. > Given that HDFS is append-only, while cloud storage like S3 guarantees atomic file overwrites, it is better to put such different logic in file system abstraction and APIs, instead of leaking them into Hudi core logic like timeline, to make it easier to maintain and understand. Do you mean move this logic in `HoodieWrapperFileSystem`? As there are two steps to write commit data(create file first and then write the data), I don't get any ideas to put it in fie system, can you please give me some hint? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
