Joris Van den Bossche created ARROW-8766:
--------------------------------------------
Summary: [Python] A FileSystem implementation based on Python
callbacks
Key: ARROW-8766
URL: https://issues.apache.org/jira/browse/ARROW-8766
Project: Apache Arrow
Issue Type: Improvement
Components: Python
Reporter: Joris Van den Bossche
The new {{pyarrow.fs}} filesystems are now actual C++ objects, and no longer
"just" a python interface. So they can't easily be expanded from the Python
side, and the existing integration with {{fsspec}} filesystems is therefore
also not working anymore.
One possible solution is to have a C++ filesystem that calls back into a
python object for each of its methods (possibly similar to how you can
implement a flight server in Python, I suppose).
Such a FileSystem implementation would allow to make a {{pyarrow.fs}} wrapper
for {{fsspec}} filesystems, and thus allow such filesystems to be used in
pyarrow where new filesystems are expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
