The Spark runner currently only supports a limited set of sources because there is no support in the Spark runner implementation which handles the Read PTransform directly.
They are able to support TextIO, AvroIO and a few others by manually providing a few direct implementations. The supported list is here: https://github.com/apache/incubator-beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java#L865 On Mon, Sep 19, 2016 at 5:39 PM, Jesse Anderson <[email protected]> wrote: > I'm running this Beam program > <https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game/utils/Input.java#L38>. > When I run it on Spark, it get this error: > > Exception in thread "main" java.lang.RuntimeException: > java.lang.IllegalStateException: > No TransformEvaluator registered for class org.apache.beam.sdk.io.Read$ > Bounded > at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:201) > at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:71) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:182) > at org.apache.beam.examples.tutorial.game.solution. > Exercise2.main(Exercise2.java:91) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$ > deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.IllegalStateException: No TransformEvaluator > registered for class org.apache.beam.sdk.io.Read$Bounded > at org.apache.beam.runners.spark.translation.TransformTranslator. > getTransformEvaluator(TransformTranslator.java:892) > at org.apache.beam.runners.spark.translation.TransformTranslator$ > Translator.translate(TransformTranslator.java:910) > at org.apache.beam.runners.spark.translation.SparkPipelineEvaluator. > doVisitTransform(SparkPipelineEvaluator.java:49) > at org.apache.beam.runners.spark.SparkRunner$Evaluator. > visitPrimitiveTransform(SparkRunner.java:242) > at org.apache.beam.sdk.runners.TransformTreeNode.visit( > TransformTreeNode.java:224) > at org.apache.beam.sdk.runners.TransformTreeNode.visit( > TransformTreeNode.java:219) > at org.apache.beam.sdk.runners.TransformTreeNode.visit( > TransformTreeNode.java:219) > at org.apache.beam.sdk.runners.TransformHierarchy.visit( > TransformHierarchy.java:101) > at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:292) > at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:179) > ... 12 more > > Looking at the TransformTranslator class, I don't see that > org.apache.beam.sdk.io.Read is added. Is there a call I need to make to > add Read? > > Thanks, > > Jesse > >
