Java only is not a blocker - we don't expect all language SDKs to look the same. They should all support the same functionality, but should do so in a way that is idiomatically correct for that language.
On Mon, Mar 5, 2018 at 11:23 AM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > > 2018-03-05 20:04 GMT+01:00 Chamikara Jayalath <chamik...@google.com>: > >> I assume you mean https://commons.apache.org/proper/commons-vfs/. >> >> I'm not sure if we considered this when we originally implemented our own >> file-system abstraction but based on a quick look seems like this is Java >> only. >> > > Yes, java only > > >> >> I think having a similar file-system abstraction for various languages is >> a plus point for Beam. May be we should consider a Java file-system >> implementation for VFS ? >> > > Can be an option but when I see the current complexity I'm not sure mixing > 2 abstractions would help, maybe just a VfsIO for java users would be good > enough - thinking out loud. > > What sounds clear to me is that each language will need its own > abstraction - which kind of join your proposal. However we can still make > it smooth and easy on the java side - which > will likely stay mainstream for still some years - using vfs as our java > impl instead of reimplementing the full abstraction? This way we keep our > *API* but we drop beam *impl* to just reuse VFS. > > PS: for gcs https://github.com/ltouati/vfs-gcs can be a good example on > how it can work. > > > >> >> Thanks, >> Cham >> >> >> >> On Mon, Mar 5, 2018 at 10:56 AM Reuven Lax <re...@google.com> wrote: >> >>> Are the filesystem classes marked experimental? If so, precise >>> compatibility is less of a concern. However vfs does need to have better fs >>> support first. >>> >>> Also what about other languages? >>> >>> On Mon, Mar 5, 2018, 3:35 PM Romain Manni-Bucau <rmannibu...@gmail.com> >>> wrote: >>> >>>> I'd say to beam 2.x and to beam 3 to move all IO/extension from the >>>> core to actual IO/extension modules. Sounds compatible this way - in the >>>> sense we can have it eagerly without breaking anything. >>>> >>>> wdyt? >>>> >>>> >>>> Romain Manni-Bucau >>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>> <http://rmannibucau.wordpress.com> | Github >>>> <https://github.com/rmannibucau> | LinkedIn >>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >>>> >>>> 2018-03-05 19:32 GMT+01:00 Reuven Lax <re...@google.com>: >>>> >>>>> Actually FileIO is only somewhat related. >>>>> >>>>> It's an interesting proposal. However a quick look shows that vfs only >>>>> has read-only support for hdfs and I'm not sure it has any support for >>>>> gcs. >>>>> Both are often used with Beam. Once vfs supports these filesystems it's >>>>> worth looking at. >>>>> >>>>> Maybe add to the beam 3.0 hotlidt? >>>>> >>>>> On Mon, Mar 5, 2018, 3:26 PM Romain Manni-Bucau <rmannibu...@gmail.com> >>>>> wrote: >>>>> >>>>>> Yes (FileIO being the visible part of the FileSystems iceberg ;)). >>>>>> >>>>>> >>>>>> Romain Manni-Bucau >>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>> <http://rmannibucau.wordpress.com> | Github >>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >>>>>> >>>>>> 2018-03-05 19:23 GMT+01:00 Reuven Lax <re...@google.com>: >>>>>> >>>>>>> I'm confused, as FileIO doesn't seem the same as vfs. Are you maybe >>>>>>> referring to the filesystem abstraction instead? >>>>>>> >>>>>>> On Mon, Mar 5, 2018, 3:19 PM Romain Manni-Bucau < >>>>>>> rmannibu...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi guys, >>>>>>>> >>>>>>>> What's the rational behind the fileIO impl? >>>>>>>> >>>>>>>> Why not using commons-vfs + a pluggable format? Sounds way more >>>>>>>> open and reusable for end users than a few hardcoded supported >>>>>>>> formats, no? >>>>>>>> What's the blocker? If there is a blocker, can't we contribute to >>>>>>>> [vfs] to >>>>>>>> make it disappear? >>>>>>>> >>>>>>>> Romain Manni-Bucau >>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>>> <http://rmannibucau.wordpress.com> | Github >>>>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >>>>>>>> >>>>>>> >>>>>> >>>> >
