orf commented on issue #40589: URL: https://github.com/apache/arrow/issues/40589#issuecomment-2002426869
> > Then we will end up with many tens of thousands of versioned objects with the key some_directory/. > > The solution here may be just checking that the empty dir marker exists instead of re-creating it unconditionally. That’s no guarantee though, and would generate a lot of redundant requests. Another idea would be to stop trying to emulate directories on S3. This would need to be feature gated in some way as there may be conditional code that relies on the existence of a directory, but if the feature is enabled (via a flag or a s3 filesystem subclass) then the “directory exists” function always returns True? I think it’s a fare trade-off if you know you want it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
