2018-03-05 20:04 GMT+01:00 Chamikara Jayalath <[email protected]>:
> I assume you mean https://commons.apache.org/proper/commons-vfs/. > > I'm not sure if we considered this when we originally implemented our own > file-system abstraction but based on a quick look seems like this is Java > only. > Yes, java only > > I think having a similar file-system abstraction for various languages is > a plus point for Beam. May be we should consider a Java file-system > implementation for VFS ? > Can be an option but when I see the current complexity I'm not sure mixing 2 abstractions would help, maybe just a VfsIO for java users would be good enough - thinking out loud. What sounds clear to me is that each language will need its own abstraction - which kind of join your proposal. However we can still make it smooth and easy on the java side - which will likely stay mainstream for still some years - using vfs as our java impl instead of reimplementing the full abstraction? This way we keep our *API* but we drop beam *impl* to just reuse VFS. PS: for gcs https://github.com/ltouati/vfs-gcs can be a good example on how it can work. > > Thanks, > Cham > > > > On Mon, Mar 5, 2018 at 10:56 AM Reuven Lax <[email protected]> wrote: > >> Are the filesystem classes marked experimental? If so, precise >> compatibility is less of a concern. However vfs does need to have better fs >> support first. >> >> Also what about other languages? >> >> On Mon, Mar 5, 2018, 3:35 PM Romain Manni-Bucau <[email protected]> >> wrote: >> >>> I'd say to beam 2.x and to beam 3 to move all IO/extension from the core >>> to actual IO/extension modules. Sounds compatible this way - in the sense >>> we can have it eagerly without breaking anything. >>> >>> wdyt? >>> >>> >>> Romain Manni-Bucau >>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>> <https://rmannibucau.metawerx.net/> | Old Blog >>> <http://rmannibucau.wordpress.com> | Github >>> <https://github.com/rmannibucau> | LinkedIn >>> <https://www.linkedin.com/in/rmannibucau> | Book >>> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >>> >>> 2018-03-05 19:32 GMT+01:00 Reuven Lax <[email protected]>: >>> >>>> Actually FileIO is only somewhat related. >>>> >>>> It's an interesting proposal. However a quick look shows that vfs only >>>> has read-only support for hdfs and I'm not sure it has any support for gcs. >>>> Both are often used with Beam. Once vfs supports these filesystems it's >>>> worth looking at. >>>> >>>> Maybe add to the beam 3.0 hotlidt? >>>> >>>> On Mon, Mar 5, 2018, 3:26 PM Romain Manni-Bucau <[email protected]> >>>> wrote: >>>> >>>>> Yes (FileIO being the visible part of the FileSystems iceberg ;)). >>>>> >>>>> >>>>> Romain Manni-Bucau >>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>> <http://rmannibucau.wordpress.com> | Github >>>>> <https://github.com/rmannibucau> | LinkedIn >>>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >>>>> >>>>> 2018-03-05 19:23 GMT+01:00 Reuven Lax <[email protected]>: >>>>> >>>>>> I'm confused, as FileIO doesn't seem the same as vfs. Are you maybe >>>>>> referring to the filesystem abstraction instead? >>>>>> >>>>>> On Mon, Mar 5, 2018, 3:19 PM Romain Manni-Bucau < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi guys, >>>>>>> >>>>>>> What's the rational behind the fileIO impl? >>>>>>> >>>>>>> Why not using commons-vfs + a pluggable format? Sounds way more open >>>>>>> and reusable for end users than a few hardcoded supported formats, no? >>>>>>> What's the blocker? If there is a blocker, can't we contribute to >>>>>>> [vfs] to >>>>>>> make it disappear? >>>>>>> >>>>>>> Romain Manni-Bucau >>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>> <http://rmannibucau.wordpress.com> | Github >>>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >>>>>>> >>>>>> >>>>> >>>
