Lawrence Chan created ARROW-9820:
------------------------------------
Summary: Plugin Architecture for Filesystem and File IO
Key: ARROW-9820
URL: https://issues.apache.org/jira/browse/ARROW-9820
Project: Apache Arrow
Issue Type: New Feature
Components: C++
Reporter: Lawrence Chan
Adding a new custom filesystem with corresponding file i/o streams is quite a
process at the moment. Looks like HDFS and S3FS are basically hardcoded in
many places. It would be useful to develop a plugin system to allow users to
interface with other data stores without maintaining a permanent fork with
hardcoded changes.
We can either do runtime plugins or compile-time plugins. Runtime is more
user-friendly, but with C++, ABI compatibility is fairly delicate. So we would
either want to use a C ABI or accept a youre-on-your-own situation where the
user is expected to be very careful with versioning and compiler flags.
With compile-time plugins, maybe there's a way to have the cmake machinery
build third party code and also register those new URI schemes automatically.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
