[
https://issues.apache.org/jira/browse/ARROW-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17182140#comment-17182140
]
Lawrence Chan commented on ARROW-9820:
--------------------------------------
Thinking about this more, I should also ask: are the other language libraries
implemented as bindings to the C++ library, or do they re-implement natively?
If they re-implement, then there's perhaps more reason to do a
language-agnostic runtime plugin system with a C API, so that the filesystem
stuff is implemented once for all languages. Most languages should have a way
to dlopen a library, so we just need to spec out an ABI, and then the user can
load additional filesystem plugins at runtime.
> Plugin Architecture for Filesystem and File IO
> ----------------------------------------------
>
> Key: ARROW-9820
> URL: https://issues.apache.org/jira/browse/ARROW-9820
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Lawrence Chan
> Priority: Minor
>
> Adding a new custom filesystem with corresponding file i/o streams is quite a
> process at the moment. Looks like HDFS and S3FS are basically hardcoded in
> many places. It would be useful to develop a plugin system to allow users to
> interface with other data stores without maintaining a permanent fork with
> hardcoded changes.
> We can either do runtime plugins or compile-time plugins. Runtime is more
> user-friendly, but with C++, ABI compatibility is fairly delicate. So we
> would either want to use a C ABI or accept a youre-on-your-own situation
> where the user is expected to be very careful with versioning and compiler
> flags.
> With compile-time plugins, maybe there's a way to have the cmake machinery
> build third party code and also register those new URI schemes automatically.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
