h-vetinari opened a new issue, #37692:
URL: https://github.com/apache/arrow/issues/37692

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   Conda-forge very recently added 
[`minio`](https://github.com/conda-forge/minio-server-feedstock), which means 
we could now use that, at least in the conda(-forge) jobs. I'd be interested in 
doing that because I see a bunch of skips in the test suite of the following 
kind (not exhaustive):
   ```
   SKIPPED [2] pyarrow/tests/test_dataset.py:2623: `minio` command cannot be 
located
   SKIPPED [1] pyarrow/tests/test_dataset.py:2638: `minio` command cannot be 
located
   SKIPPED [1] pyarrow/tests/test_dataset.py:2664: `minio` command cannot be 
located
   SKIPPED [1] pyarrow/tests/test_dataset.py:4754: `minio` command cannot be 
located
   SKIPPED [1] pyarrow/tests/test_dataset.py:4821: `minio` command cannot be 
located
   SKIPPED [1] pyarrow/tests/test_fs.py:471: `minio` command cannot be located
   SKIPPED [2] pyarrow/tests/test_fs.py:562: `minio` command cannot be located
   ```
   
   so if all it takes is adding the test dependency - why not?
   
   As it turns out, this blows up pretty badly on main (through a 
[proxy](https://github.com/conda-forge/arrow-cpp-feedstock/pull/1170) PR on our 
infra), which is at least partly, because arrow currently pins to a relatively 
outdated version
   
https://github.com/apache/arrow/blob/4fac528e2ee9c4efce453420275cbaf0b3b6adb0/ci/scripts/install_minio.sh#L54-L56
   while the oldest available version in conda-forge is 2023.08.23.10.07.06.
   
   Aside from 2-3 exceptions, the failures are all from the teardown of the 
`s3fs` and `py_fsspec_s3fs` fixtures, where the code unconditionally does 
`fs.delete_dir(bucket)`, even though the test presumably scribbled stuff in 
there. This leads to errors of the kind:
   ```
   _ ERROR at teardown of 
test_filesystem_is_functional_after_pickling[builtin_pickle-S3FileSystem] _
   
   request = <SubRequest 's3fs' for <Function 
test_filesystem_is_functional_after_pickling[builtin_pickle-S3FileSystem]>>
   s3_server = {'connection': ('localhost', 41707, 'arrow', 'apachearrow'), 
'process': <Popen: returncode: None args: ['minio', '--compat', 'server', 
'--quiet', '-...>, 'tempdir': local('/tmp/pytest-of-conda/pytest-0')}
   
       @pytest.fixture
       def s3fs(request, s3_server):
           [...]
   >       fs.delete_dir(bucket)
   
   pyarrow/tests/test_fs.py:258: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ 
   pyarrow/_fs.pyx:616: in pyarrow._fs.FileSystem.delete_dir
       check_status(self.fs.DeleteDir(directory))
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ 
   
   >   raise convert_status(status)
   E   OSError: When deleting bucket 'pyarrow-filesystem': AWS Error UNKNOWN 
(HTTP status 409) during DeleteBucket operation: Unable to parse ExceptionName: 
BucketNotEmpty Message: The bucket you tried to delete is not empty
   
   pyarrow/error.pxi:91: OSError
   ```
   more specifically, close to ~60 of them:
   ```
   = 4 failed, 7276 passed, 223 skipped, 19 deselected, 28 xfailed, 2 xpassed, 
8 warnings, 57 errors in 177.84s (0:02:57) =
   ```
   
   My first guess would be that:
   * either minio got stricter (less buggier) in when it allows deletion of 
non-empty buckets
   * or something in the conda-forge setup does not yet accurately reproduce 
what the CI is doing here.
   
   I'm quite out of my depths here, but I think a alternative would be to 
somehow pipe through 
[`ForceBucketDelete`](https://min.io/docs/minio/linux/administration/identity-access-management/policy-based-access-control.html#policy-action.s3-ForceDeleteBucket)
 or `--force`. A cheap alternative is the following patch (also doesn't cover 
100%), which just doesn't care about failed bucket deletes:
   ```diff
   --- a/python/pyarrow/tests/test_fs.py
   +++ b/python/pyarrow/tests/test_fs.py
   @@ -256,7 +256,10 @@ def s3fs(request, s3_server):
            allow_move_dir=False,
            allow_append_to_file=False,
        )
   -    fs.delete_dir(bucket)
   +    try:
   +        fs.delete_dir(bucket)
   +    except OSError:
   +        pass
   
   
    @pytest.fixture
   @@ -358,7 +361,10 @@ def py_fsspec_s3fs(request, s3_server):
            allow_move_dir=False,
            allow_append_to_file=True,
        )
   -    fs.delete_dir(bucket)
   +    try:
   +        fs.delete_dir(bucket)
   +    except OSError:
   +        pass
   
   
    @pytest.fixture(params=[
   ```
   
   After that patch, the only remaining errors are then:
   ```
   FAILED 
pyarrow/tests/test_fs.py::test_get_file_info[PyFileSystem(FSSpecHandler(s3fs.S3FileSystem()))]
 - AssertionError: assert <FileType.Directory: 3> == <FileType.NotFound: 0>
    +  where <FileType.Directory: 3> = <FileInfo for 
'pyarrow-filesystem/a/aa/aaa/': type=FileType.Directory>.type
    +  and   <FileType.NotFound: 0> = FileType.NotFound
   FAILED pyarrow/tests/test_fs.py::test_delete_dir[S3FileSystem] - Failed: DID 
NOT RAISE <class 'OSError'>
   FAILED pyarrow/tests/test_fs.py::test_delete_dir_contents[S3FileSystem] - 
Failed: DID NOT RAISE <class 'OSError'>
   FAILED 
pyarrow/tests/test_fs.py::test_move_directory[PyFileSystem(FSSpecHandler(s3fs.S3FileSystem()))]
 - Failed: DID NOT RAISE <class 'OSError'>
   ```
   
   # This issue does not happen on 13.0.0?!
   
   The final kicker is that _**all this is passing with arrow 13**_ - I even 
checked that the tests didn't get skipped. So it appears there are at least two 
things at work here: a change in minio behaviour & a change in pyarrow 
somewhere.
   
   ### Component(s)
   
   Continuous Integration, Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to