[
https://issues.apache.org/jira/browse/ARROW-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joris Van den Bossche updated ARROW-8766:
-----------------------------------------
Labels: dataset-dask-integration filesystem (was: filesystem)
> [Python] A FileSystem implementation based on Python callbacks
> --------------------------------------------------------------
>
> Key: ARROW-8766
> URL: https://issues.apache.org/jira/browse/ARROW-8766
> Project: Apache Arrow
> Issue Type: Sub-task
> Components: Python
> Reporter: Joris Van den Bossche
> Priority: Major
> Labels: dataset-dask-integration, filesystem
>
> The new {{pyarrow.fs}} filesystems are now actual C++ objects, and no longer
> "just" a python interface. So they can't easily be expanded from the Python
> side, and the existing integration with {{fsspec}} filesystems is therefore
> also not working anymore.
> One possible solution is to have a C++ filesystem that calls back into a
> python object for each of its methods (possibly similar to how you can
> implement a flight server in Python, I suppose).
> Such a FileSystem implementation would allow to make a {{pyarrow.fs}} wrapper
> for {{fsspec}} filesystems, and thus allow such filesystems to be used in
> pyarrow where new filesystems are expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
