reidkaufmann opened a new issue, #888: URL: https://github.com/apache/arrow-rs-object-store/issues/888
**Describe the bug** With `LocalFileSystem::with_fsync(true)`, [`create_parent_dirs`](https://github.com/apache/arrow-rs-object-store/blob/279572ea60f3a7a6e5237e066a6f4a35eee611e0/src/local.rs#L905-L947) records the deepest existing ancestor, calls `create_dir_all` and fsyncs from the new parent up to that ancestor. Three interleavings return success while a directory entry on the path to the written file is not durable, so a power loss can remove the directory with the acknowledged file: - A write into a directory another call is still creating finds it existing, fsyncs only that directory ([`fsync_parent_dir`](https://github.com/apache/arrow-rs-object-store/blob/279572ea60f3a7a6e5237e066a6f4a35eee611e0/src/local.rs#L1037-L1042)) and returns, possibly before the creator has fsynced the ancestor holding the directory's entry. - With `with_automatic_cleanup(true)`, a concurrent `delete` can remove the ancestor found existing between the `exists()` check and `create_dir_all`; `create_dir_all` recreates it and its new entry in the directory above is never fsynced. - When a creator's directory fsync fails after `create_dir_all`, the directories exist but nothing records that they are not durable; a later write into them fsyncs only its own parent and returns success. **To Reproduce** Tests in the linked PRs interleave the other party with the creation and record which directories get fsynced. **Expected behavior** A successful durable write means every directory entry leading to the file has been fsynced by some call, as the `with_fsync` doc says: "This guarantees that both the file contents and the directory entries pointing to them are durable on stable storage". **Additional context** Coordination can only be per process and by path; other processes are out of scope. The first two interleavings are one PR, the third a follow-up PR on top of it. **AI usage** Written with AI assistance; I reviewed it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
