terminology is confusing here, since the existing FileIO is a PTransform. VfsFilesystem would be a better name.
On Mon, Mar 5, 2018 at 11:46 AM Robert Bradshaw <rober...@google.com> wrote: > On Mon, Mar 5, 2018 at 11:38 AM Reuven Lax <re...@google.com> wrote: > >> What about a beam Filesystem impl on top of Vfs as an alternative >> short-term solution? This would allow Vfs to be used with any IO. >> > > Yes, I think this is the VfsIO that was proposed. > > >> On Mon, Mar 5, 2018 at 11:37 AM Robert Bradshaw <rober...@google.com> >> wrote: >> >>> >>> On Mon, Mar 5, 2018 at 11:23 AM Romain Manni-Bucau < >>> rmannibu...@gmail.com> wrote: >>> >>>> >>>> 2018-03-05 20:04 GMT+01:00 Chamikara Jayalath <chamik...@google.com>: >>>> >>>>> I assume you mean https://commons.apache.org/proper/commons-vfs/. >>>>> >>>>> I'm not sure if we considered this when we originally implemented our >>>>> own file-system abstraction but based on a quick look seems like this is >>>>> Java only. >>>>> >>>> >>>> Yes, java only >>>> >>>> >>>>> >>>>> I think having a similar file-system abstraction for various languages >>>>> is a plus point for Beam. May be we should consider a Java file-system >>>>> implementation for VFS ? >>>>> >>>> >>>> Can be an option but when I see the current complexity I'm not sure >>>> mixing 2 abstractions would help, maybe just a VfsIO for java users would >>>> be good enough - thinking out loud. >>>> >>>> What sounds clear to me is that each language will need its own >>>> abstraction - which kind of join your proposal. However we can still make >>>> it smooth and easy on the java side - which >>>> will likely stay mainstream for still some years - using vfs as our >>>> java impl instead of reimplementing the full abstraction? This way we keep >>>> our *API* but we drop beam *impl* to just reuse VFS. >>>> >>>> PS: for gcs https://github.com/ltouati/vfs-gcs can be a good example >>>> on how it can work. >>>> >>> >>> I think a VfsIO makes a lot of sense in the short term, and will give >>> use the experience needed to decide if we can move solely to VFS (for Java >>> at least) for implementation, and possibly API in a future major release, >>> in the long run. >>> >>
