What about a beam Filesystem impl on top of Vfs as an alternative short-term solution? This would allow Vfs to be used with any IO.
On Mon, Mar 5, 2018 at 11:37 AM Robert Bradshaw <[email protected]> wrote: > > On Mon, Mar 5, 2018 at 11:23 AM Romain Manni-Bucau <[email protected]> > wrote: > >> >> 2018-03-05 20:04 GMT+01:00 Chamikara Jayalath <[email protected]>: >> >>> I assume you mean https://commons.apache.org/proper/commons-vfs/. >>> >>> I'm not sure if we considered this when we originally implemented our >>> own file-system abstraction but based on a quick look seems like this is >>> Java only. >>> >> >> Yes, java only >> >> >>> >>> I think having a similar file-system abstraction for various languages >>> is a plus point for Beam. May be we should consider a Java file-system >>> implementation for VFS ? >>> >> >> Can be an option but when I see the current complexity I'm not sure >> mixing 2 abstractions would help, maybe just a VfsIO for java users would >> be good enough - thinking out loud. >> >> What sounds clear to me is that each language will need its own >> abstraction - which kind of join your proposal. However we can still make >> it smooth and easy on the java side - which >> will likely stay mainstream for still some years - using vfs as our java >> impl instead of reimplementing the full abstraction? This way we keep our >> *API* but we drop beam *impl* to just reuse VFS. >> >> PS: for gcs https://github.com/ltouati/vfs-gcs can be a good example on >> how it can work. >> > > I think a VfsIO makes a lot of sense in the short term, and will give use > the experience needed to decide if we can move solely to VFS (for Java at > least) for implementation, and possibly API in a future major release, in > the long run. >
