BeamJava uses com.google.auto.service.AutoService which, at the end of the day, is shorthand for Java's standard ServiceLoader mechanisms (e.g. see [1]). I'm not an expert on the details of how this works, but you'll probably have to make sure these filesystem dependencies are in your custom classloader's jar.
[1] https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystemRegistrar.java On Thu, Nov 29, 2018 at 3:57 PM Matt Casters <[email protected]> wrote: > > Hello Beam, > > I've been taking great steps forward in having Kettle generate Beam pipelines > and they actually execute just find in unit testing in IntelliJ. > The problem starts when I collect all the libraries needed for Beam and the > Runners and throw them into the Kettle project as a plugin. > > Caused by: java.lang.IllegalArgumentException: No filesystem found for scheme > gs > at > org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:456) > at org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:526) > at > org.apache.beam.sdk.io.FileBasedSink.convertToFileResourceIfPossible(FileBasedSink.java:213) > at org.apache.beam.sdk.io.TextIO$TypedWrite.to(TextIO.java:700) > at org.apache.beam.sdk.io.TextIO$Write.to(TextIO.java:1028) > at > org.kettle.beam.core.transform.BeamOutputTransform.expand(BeamOutputTransform.java:87) > ... 32 more > > This also happens for local file execution ("scheme file" in that case). > > So the questions are: how is Beam bootstrapped? How does Beam determine which > libraries to use and what is the recommended way for packaging things up > properly? > The Beam plugin is running in a separate URLClassloader so I think something > is going awry there. > > Thanks a lot for any answers or tips you might have! > > Matt > --- > Matt Casters <[email protected]> > Senior Solution Architect, Kettle Project Founder > >
