jorisvandenbossche commented on a change in pull request #7395:
URL: https://github.com/apache/arrow/pull/7395#discussion_r438129306
##########
File path: python/pyarrow/tests/test_fs.py
##########
@@ -324,6 +350,14 @@ def hdfs(request, hdfs_connection):
pytest.lazy_fixture('py_localfs'),
id='PyFileSystem(ProxyHandler(LocalFileSystem()))'
),
+ pytest.param(
+ pytest.lazy_fixture('py_fsspec_localfs'),
+ id='PyFileSystem(FSSpecHandler(fsspec.LocalFileSystem()))'
+ ),
+ # pytest.param(
+ # pytest.lazy_fixture('py_fsspec_memoryfs'),
+ # id='PyFileSystem(FSSpecHandler(fsspec.filesystem("memory")))'
Review comment:
I still need to clean this up before merging. I added them for testing,
but the problem is that several tests fail because both the in-memory
filesystem as s3fs don't fully follow the spec, so not all "generic" tests work
(like creating or removing nested directories).
Opened issues for the in-memory:
https://github.com/intake/filesystem_spec/issues/314,
https://github.com/intake/filesystem_spec/issues/313, and existing one for
s3fs: https://github.com/dask/s3fs/issues/245
Now I suppose that in practice when it comes to reading files (eg in the
dataset API, or in `parquet.read_table`), those limitations won't necessarily
be a problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
