[ https://issues.apache.org/jira/browse/ARROW-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joris Van den Bossche updated ARROW-7102: ----------------------------------------- Labels: FileSystem dataset-dask-integration (was: FileSystem) > [Python] Make filesystems compatible with fsspec > ------------------------------------------------ > > Key: ARROW-7102 > URL: https://issues.apache.org/jira/browse/ARROW-7102 > Project: Apache Arrow > Issue Type: Improvement > Components: Python > Reporter: Tom Augspurger > Priority: Major > Labels: FileSystem, dataset-dask-integration > > Update: regarding compatibility with {{fsspec}}, there are two directions of > wrapping possible: > * Make a {{fsspec}} wrapper for {{pyarrow.fs}} (-> tracked in ARROW-8780, > this can ensure {{pyarrow.fs}} filesystems can be used where {{fsspec}} > filesytems are expected ) > * Make a {{pyarrow.fs}} wrapper for {{fsspec}} (-> tracked in ARROW-8766 this > can ensure {{fsspec}} filesystems can be used where {{pyarrow.fs}} filesytems > are expected ) > ---- > [fsspec|https://filesystem-spec.readthedocs.io/en/latest] defines a common > API for a variety filesystem implementations. I'm proposing a FSSpecWrapper, > similar to S3FSWrapper, that works with any fsspec implementation. > > Right now, pyarrow has a pyarrow.filesystems.S3FSWrapper, which is specific > to s3fs. > [https://github.com/apache/arrow/blob/21ad7ac1162eab188a1e15923fb1de5b795337ec/python/pyarrow/filesystem.py#L320]. > This implementation could be removed entirely once an FSSPecWrapper is done, > or kept as an alias if it's part of the public API. > > This is realted to ARROW-3717, which requested a GCSFSWrapper for working > with google cloud storage. -- This message was sent by Atlassian Jira (v8.3.4#803005)