jorisvandenbossche commented on issue #38618: URL: https://github.com/apache/arrow/issues/38618#issuecomment-1821252024
I can reproduce the issue with an actual AWS S3 bucket. Enabling some debug logging, and comparing the output using pyarrow 13.0 vs 14.0 using the example of "Dir with subdir deleted" (`s3fs.delete_dir(s3_bucket + "/" + test_dir)`). With pyarrow 13.0: * Listing with test_dir/ as prefix (GET with list-type), which returns a ListBucketResult * Listing with sub_dir/ as prefix * POST /?delete (passing both test_dir/ as test_dir/sub_dir/ as objects to delete with a `Delete` xml object * POST /?delete with only the test_dir/sub_dir/ (no idea why this is done a second time) * DELETE /test_dir/ With pyarrow 14.0: * HEAD /test_dir * Listing with test_dir/ as prefix * POST /?delete with specifying test_dir/sub_dir as single object to delete (without trailing /, is that the difference?) * DELETE /test_dir/ (and there is no indication that this last DELETE failed, but in practice it did not delete the test_dir, and neither the sub_dir was deleted) In case this is useful for someone more familiar with S3 / the AWS SDK / our bindings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
