alamb commented on issue #888: URL: https://github.com/apache/arrow-rs-object-store/issues/888#issuecomment-6747221901
First of all, I found this issue hard to understand as it immediately dives into the implementation details without explaining the usecase. I think the core issue you are hitting (as we have discussed internally, but I am putting here) is that under load / failure scenarios, certain LocalFileSystem operations can leave empty directories and/or zero length files. Fundamentally I don't think we should be trying to implement a durable tranaction layer over the LocalFileSystem that handles all failure scenarios by sprinkling `fsync` around the codebase. My reasoning: 1. I think is treating symptoms rather than root cause and I don't think we will get all the corner cases and we (I/the other maintainers) will be dealing with a string of bugs forever. 2. It will potentially slow things down a lot (how many more fsyncs does it require now for any operation) for cases when durability requirements are not that high If the goal is to implement a durable layer for the `ObjectStore` API that is more resilient to various filesystems issues, I think we would have to also handle things like detecting the file system corrupting a file (which does happen from time to time), and the LocalFileSystem has no way (e.g. checksums, etc) to detect or recover from this For example, a classic design (which is also how real object stores are implemented internally) is to use an embedded database (sqlite, rocksdb, etc) to hold the path metadata, and store the actual data on disk files. The embedded database handles durability and transaction isolation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
