[
https://issues.apache.org/jira/browse/ARROW-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17223719#comment-17223719
]
Joris Van den Bossche commented on ARROW-10370:
-----------------------------------------------
Yes, I have also seen it locally the last weeks. I think it's since we
activated to optionally use the new dataset writing functionality in
{{pq.write_to_dataset}}. So the "legacy" implementation correctly raises an
error when passing a pathlib path for a non-local filesystem. But the new
{{ds.write_dataset}} does not do this, it actually works (and then it fails in
some further step that wasn't supposed to be reached in this case).
> [Python] Spurious s3fs-related test failures
> --------------------------------------------
>
> Key: ARROW-10370
> URL: https://issues.apache.org/jira/browse/ARROW-10370
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Antoine Pitrou
> Priority: Major
>
> I frequently get this error when running the Python test suite:
> {code}
> _____________________________________________________________
> test_write_to_dataset_pathlib_nonlocal[False]
> _____________________________________________________________
> Traceback (most recent call last):
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py",
> line 984, in _initiate_upload
> Bucket=self.bucket, Key=self.key, ACL=self.acl)
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py",
> line 971, in _call_s3
> **kwargs)
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py",
> line 189, in _call_s3
> return method(**additional_kwargs)
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/botocore/client.py",
> line 357, in _api_call
> return self._make_api_call(operation_name, kwargs)
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/botocore/client.py",
> line 661, in _make_api_call
> raise error_class(parsed_response, operation_name)
> botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when
> calling the CreateMultipartUpload operation: The specified bucket does not
> exist
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/home/antoine/arrow/dev/python/pyarrow/tests/test_parquet.py", line
> 2978, in test_write_to_dataset_pathlib_nonlocal
> tempdir / "test1", use_legacy_dataset, filesystem=fs)
> File "/home/antoine/arrow/dev/python/pyarrow/tests/test_parquet.py", line
> 2853, in _test_write_to_dataset_with_partitions
> pq.write_metadata(output_table.schema, f)
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1457, in __exit__
> self.close()
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1425, in close
> self.flush(force=True)
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1297, in flush
> self._initiate_upload()
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py",
> line 986, in _initiate_upload
> raise translate_boto_error(e)
> FileNotFoundError: The specified bucket does not exist
> -------------------------------------------------------------------------
> Captured stderr call
> --------------------------------------------------------------------------
> Exception ignored in: Exception ignored in: Exception ignored in: <function
> AbstractBufferedFile.__del__ at 0x7f1b119097a0>
> Traceback (most recent call last):
> <function AbstractBufferedFile.__del__ at 0x7f1b119097a0>
> Exception ignored in: File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1446, in __del__
> <function AbstractBufferedFile.__del__ at 0x7f1b119097a0>
> Traceback (most recent call last):
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1446, in __del__
> <function AbstractBufferedFile.__del__ at 0x7f1b119097a0>
> Exception ignored in: Traceback (most recent call last):
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1446, in __del__
> Traceback (most recent call last):
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1446, in __del__
> <function AbstractBufferedFile.__del__ at 0x7f1b119097a0> Exception
> ignored in: <function AbstractBufferedFile.__del__ at 0x7f1b119097a0>
> self.close()
> self.close()
> Traceback (most recent call last):
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1446, in __del__
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1425, in close
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1425, in close
> self.flush(force=True)
> self.flush(force=True)
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1297, in flush
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1297, in flush
> self.close()
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1425, in close
> self._initiate_upload()
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py",
> line 986, in _initiate_upload
> Traceback (most recent call last):
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1446, in __del__
> self.close()self.flush(force=True) self._initiate_upload()
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py",
> line 986, in _initiate_upload
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1425, in close
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1297, in flush
> raise translate_boto_error(e)self.close()self.close()
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1425, in close
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1425, in close
> raise translate_boto_error(e)
> self._initiate_upload()
> FileNotFoundError:
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py",
> line 986, in _initiate_upload
> The specified bucket does not existFileNotFoundError:
> The specified bucket does not exist
> self.flush(force=True)
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1297, in flush
> self.flush(force=True)
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1297, in flush
> self.flush(force=True)
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py",
> line 1297, in flush
> self._initiate_upload()raise translate_boto_error(e)
> FileNotFoundError: The specified bucket does not exist
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py",
> line 986, in _initiate_upload
> self._initiate_upload()
> self._initiate_upload()
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py",
> line 986, in _initiate_upload
> File
> "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py",
> line 986, in _initiate_upload
> raise translate_boto_error(e)
> FileNotFoundError: The specified bucket does not exist
> raise translate_boto_error(e)raise translate_boto_error(e)
> FileNotFoundErrorFileNotFoundError: : The specified bucket does not existThe
> specified bucket does not exist
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
