Hi Juan,
Service files need to be merged when creating a fat jar because Beam
uses `ServiceLoader` at various places to register transforms. In your
case, only the special Flink transforms had been registered but not even
the default Beam transforms were picked up because their service files
were ignored.
Glad you were able to solve your problems.
-Max
On 16.10.18 13:27, Juan Carlos Garcia wrote:
I figured it out, i was missing the 'mergeServiceFiles()' for the
shadowJar gradle plugin, it looks like the SparkRunner was not needing
such merge. (Or i was lucky, how the classes were arranged and processed
when using the SparkRunner)
Thanks.
On Tue, Oct 16, 2018 at 1:09 PM Juan Carlos Garcia <[email protected]
<mailto:[email protected]>> wrote:
I haven't yet resolved it but found out that my fat-jar (shadowJar)
may have incomplete entries.
Running the pipeline from within the IDE shows the following
*KNOWN_PAYLOAD_TRANSLATOR*
image.png
But debugging the JVM process when launching the jar from the
command line with
(-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005)
shows only the following *KNOWN_PAYLOAD_TRANSLATOR*
image.png
Will come back once i figure it out what configuration the shadowJar
gradle plugin is missing.
On Tue, Oct 16, 2018 at 12:26 PM Juan Carlos Garcia
<[email protected] <mailto:[email protected]>> wrote:
Hi Folks,
I am switching some pipelines from SparkRunner to the
FlinkRunner on beam 2.7
I started with a very simple pipeline which just reads from
multiple kafka sources, flatten those and then apply a regular DoFn.
On my first try of the pipeline (from command line using a fat
jar) like:
```
java -cp my-fat.jar FQCN --configuration=foo --runner=FlinkRunner
```
complains with:
```
Exception in thread "main" java.lang.IllegalStateException: No
translator known for org.apache.beam.sdk.io.Read$Unbounded
at
org.apache.beam.runners.core.construction.PTransformTranslation.urnForTransform(PTransformTranslation.java:164)
at
org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:100)
...
...
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
```
On my build.gradle i just replaced my spark-runner with this:
``
runtime "org.apache.beam:beam-runners-flink_2.11:2.7.0"
``
Am i missing something?
--
JC
--
JC
--
JC
