westonpace commented on issue #36275: URL: https://github.com/apache/arrow/issues/36275#issuecomment-1632735662
This would be a change in behavior so I think we probably shouldn't change our default here. At the moment S3FileSystem is doing its best to have the S3 storage act like a normal filesystem and this is a part of that. So users may have come to rely on this behavior. However, we (as in arrow-c++) do not rely on this function at all (dataset writer does do some deletes but it only uses DeleteDirContents). I do agree that there should be a way to avoid this. I think there are a few reasonable approaches that could be taken here if someone has time to work on this. > check for other objects in the same path before taking it upon itself to create this file. Perhaps `S3Options` could have a setting to control whether marker directories are created or not (by any operation) Of course, it would also be possible to create a very specific setting (e.g. `S3Options::create_missing_dirs_on_delete`) I don't think adding any new flags to the delete method itself would be a good idea. There are others out there with their own custom implementations of `FileSystem` and so any changes to that interface incur additional work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
