JingsongLi commented on code in PR #10853:
URL: https://github.com/apache/paimon/pull/10853#discussion_r4479141244
##########
paimon-python/pypaimon/filesystem/local_file_io.py:
##########
@@ -110,8 +189,10 @@ def new_output_stream(self, path: str):
parent = file_path.parent
if parent and not parent.exists():
parent.mkdir(parents=True, exist_ok=True)
+ _fsync_directory(parent)
Review Comment:
[P1] Sync the directories that contain newly created path components. A
successful local append to a new partition synchronized `t/part=new/bucket-0`,
`t/manifest` and `t/snapshot`, but neither `t/part=new` nor `t`. Syncing
`bucket-0` persists its own file entries. Its link is stored in `part=new`,
whose link is stored in `t`. A power loss can therefore leave a published
snapshot referencing an unreachable partition file, even after the commit
returned successfully. The first `snapshot` directory has the same issue in the
atomic-write path. Please make directory creation durable in both paths,
including format writers that currently precreate the parent, and verify the
ancestor syncs on a real append to a new partition. [Linux fsync
semantics](https://man7.org/linux/man-pages/man2/fsync.2.html#DESCRIPTION)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]