tustvold opened a new issue, #7030: URL: https://github.com/apache/arrow-rs/issues/7030
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** <!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] (This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*) --> Currently object_store treats directories as an implementation detail: * LocalFileSystem automatically creates them on-demand. * There is no mechanism to unambiguously refer to a directory (#7026) * Various implementations filter out directories when listing (e.g. [azure](https://github.com/apache/arrow-rs/blob/main/object_store/src/azure/client.rs#L1013)) This ensures portability but has a couple of downsides: * There is no way to delete directories in LocalFileSystem * There is no way to create or delete objects with trailing / (#7026) which are sometimes used as pseudo directories * Some stores may have more efficient mechanisms for deleting a directory and all contents (e.g. LocalFileSystem / ADLS2) A pattern used by some tools is to create empty objects with a trailing `/` to represent directories on stores that don't natively support directories. However, these pseudo directories do not really behave like normal directories: * A filename can be created without its parent directory existing * A directory can be deleted without deleting its children Additionally, creating such pseudo directories on stores with native directories is at best confusing, if it is even possible. This inherent inconsistency makes it very hard to devise a coherent API **Describe the solution you'd like** <!-- A clear and concise description of what you want to happen. --> I think we should at least do the following: * Better document the current situation w.r.t directories * Skip over pseudo directories when listing (we have no mechanism to create them, so whatever is creating them should manage their lifecycle) I'd also be interested if people have any ideas of a coherent way to support a notion of directories **Describe alternatives you've considered** <!-- A clear and concise description of any alternative solutions or features you've considered. --> **Additional context** <!-- Add any other context or screenshots about the feature request here. --> * https://github.com/apache/arrow-rs/pull/5978 added automatic cleanup of empty directories to LocalFileSystem -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
