On Mon, Mar 5, 2018 at 11:23 AM Romain Manni-Bucau <rmannibu...@gmail.com> wrote:
> > 2018-03-05 20:04 GMT+01:00 Chamikara Jayalath <chamik...@google.com>: > >> I assume you mean https://commons.apache.org/proper/commons-vfs/. >> >> I'm not sure if we considered this when we originally implemented our own >> file-system abstraction but based on a quick look seems like this is Java >> only. >> > > Yes, java only > > >> >> I think having a similar file-system abstraction for various languages is >> a plus point for Beam. May be we should consider a Java file-system >> implementation for VFS ? >> > > Can be an option but when I see the current complexity I'm not sure mixing > 2 abstractions would help, maybe just a VfsIO for java users would be good > enough - thinking out loud. > > What sounds clear to me is that each language will need its own > abstraction - which kind of join your proposal. However we can still make > it smooth and easy on the java side - which > will likely stay mainstream for still some years - using vfs as our java > impl instead of reimplementing the full abstraction? This way we keep our > *API* but we drop beam *impl* to just reuse VFS. > > PS: for gcs https://github.com/ltouati/vfs-gcs can be a good example on > how it can work. > I think a VfsIO makes a lot of sense in the short term, and will give use the experience needed to decide if we can move solely to VFS (for Java at least) for implementation, and possibly API in a future major release, in the long run.
