Personally, this is a case where I think forking might be a better option,
even though I'm not generally a fan of duplicating code.

In past projects, depending on internal modules of other projects never
seems lead to good outcomes. FileSystem exists for Beam today, and Beam
might make changes to it that cause problems for your other project. I
would recommend starting by forking if it serves your needs.


On Fri, Apr 6, 2018 at 8:17 AM Romain Manni-Bucau <>

> Hi guys,
> I have a use case where I'd like to be able to expose to a user some file
> system navigation and enable him to visualize the file system (as in beam
> sense)
> Technically it is a matter of being able to use glob pattern to browse the
> file system using match(specs).
> What is important in that use case is to align the visualization and the
> potential runtime to have the same impl/view and not have to split it in 2
> code branches which can lead to inconsistency.
> Therefore i'd like to be able to reuse beam FileSystem but I have a few
> blockers:
> 1. it is nested in sdk-java-core which brings 2 drawbacks
> a. it brings the whole beam sdk which is not desired in that part of the
> app (should not be visible in the classpath)
> b. the dependency stack is just unpractiable (guava, jackson, byte-buddy,
> avro, joda, at least, are not desired at all here) and a shade makes it way
> too fat to be a valid dependency for that usage
> 2. I don't know how to configure the FS from one of its instance (I'd like
> to be able to get its options class like FileSystem#getConfigurationType
> returning a PipelineOptions)
> Do you think it is possible to extract the filesystem API in a dependency
> free beam subproject (or at least submodule) and add the configuration hint
> in the API?
> Romain Manni-Bucau
> @rmannibucau <> |  Blog
> <> | Old Blog
> <> | Github
> <> | LinkedIn
> <> | Book
> <>

Reply via email to