[
https://issues.apache.org/jira/browse/ARROW-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom Augspurger updated ARROW-7102:
----------------------------------
Description:
[fsspec|https://filesystem-spec.readthedocs.io/en/latest] defines a common API
for a variety filesystem implementations. I'm proposing a FSSpecWrapper,
similar to S3FSWrapper, that works with any fsspec implementation.
Right now, pyarrow has a pyarrow.filesystems.S3FSWrapper, which is specific to
s3fs.
[https://github.com/apache/arrow/blob/21ad7ac1162eab188a1e15923fb1de5b795337ec/python/pyarrow/filesystem.py#L320].
This implementation could be removed entirely once an FSSPecWrapper is done,
or kept as an alias if it's part of the public API.
This is realted to ARROW-3717, which requested a GCSFSWrapper for working with
google cloud storage.
was:
[fsspec|fsspec: https://filesystem-spec.readthedocs.io/en/latest/] defines a
common API for a variety filesystem implementations. I'm proposing a
FSSpecWrapper, similar to S3FSWrapper, that works with any fsspec
implementation.
Right now, pyarrow has a pyarrow.filesystems.S3FSWrapper, which is specific to
s3fs.
[https://github.com/apache/arrow/blob/21ad7ac1162eab188a1e15923fb1de5b795337ec/python/pyarrow/filesystem.py#L320].
This implementation could be removed entirely once an FSSPecWrapper is done,
or kept as an alias if it's part of the public API.
This is realted to ARROW-3717, which requested a GCSFSWrapper for working with
google cloud storage.
> Make filesystem wrappers compatible with fsspec
> -----------------------------------------------
>
> Key: ARROW-7102
> URL: https://issues.apache.org/jira/browse/ARROW-7102
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Reporter: Tom Augspurger
> Priority: Major
> Labels: FileSystem
>
> [fsspec|https://filesystem-spec.readthedocs.io/en/latest] defines a common
> API for a variety filesystem implementations. I'm proposing a FSSpecWrapper,
> similar to S3FSWrapper, that works with any fsspec implementation.
>
> Right now, pyarrow has a pyarrow.filesystems.S3FSWrapper, which is specific
> to s3fs.
> [https://github.com/apache/arrow/blob/21ad7ac1162eab188a1e15923fb1de5b795337ec/python/pyarrow/filesystem.py#L320].
> This implementation could be removed entirely once an FSSPecWrapper is done,
> or kept as an alias if it's part of the public API.
>
> This is realted to ARROW-3717, which requested a GCSFSWrapper for working
> with google cloud storage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
