If you are using Gradle for packaging, make sure your final jar (fat-jar)
contains all the services files merged.
Using the Gradle shadowJar plugin include "*mergeServiceFiles()*"
instruction like:
apply plugin: 'com.github.johnrengelman.shadow'
shadowJar {
mergeServiceFiles()
zip64 true
classifier = 'bundled'
}
If you are using Maven then use the Shade plugin.
On Thu, Nov 29, 2018 at 4:50 PM Robert Bradshaw <[email protected]> wrote:
> BeamJava uses com.google.auto.service.AutoService which, at the end of
> the day, is shorthand for Java's standard ServiceLoader mechanisms
> (e.g. see [1]). I'm not an expert on the details of how this works,
> but you'll probably have to make sure these filesystem dependencies
> are in your custom classloader's jar.
>
> [1]
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystemRegistrar.java
> On Thu, Nov 29, 2018 at 3:57 PM Matt Casters <[email protected]>
> wrote:
> >
> > Hello Beam,
> >
> > I've been taking great steps forward in having Kettle generate Beam
> pipelines and they actually execute just find in unit testing in IntelliJ.
> > The problem starts when I collect all the libraries needed for Beam and
> the Runners and throw them into the Kettle project as a plugin.
> >
> > Caused by: java.lang.IllegalArgumentException: No filesystem found for
> scheme gs
> > at org.apache.beam.sdk.io
> .FileSystems.getFileSystemInternal(FileSystems.java:456)
> > at org.apache.beam.sdk.io
> .FileSystems.matchNewResource(FileSystems.java:526)
> > at org.apache.beam.sdk.io
> .FileBasedSink.convertToFileResourceIfPossible(FileBasedSink.java:213)
> > at org.apache.beam.sdk.io.TextIO$TypedWrite.to(TextIO.java:700)
> > at org.apache.beam.sdk.io.TextIO$Write.to(TextIO.java:1028)
> > at
> org.kettle.beam.core.transform.BeamOutputTransform.expand(BeamOutputTransform.java:87)
> > ... 32 more
> >
> > This also happens for local file execution ("scheme file" in that case).
> >
> > So the questions are: how is Beam bootstrapped? How does Beam determine
> which libraries to use and what is the recommended way for packaging things
> up properly?
> > The Beam plugin is running in a separate URLClassloader so I think
> something is going awry there.
> >
> > Thanks a lot for any answers or tips you might have!
> >
> > Matt
> > ---
> > Matt Casters <[email protected]>
> > Senior Solution Architect, Kettle Project Founder
> >
> >
>
--
JC
