Hi There's a work in progress (on a branch) to support bounded/unbounded read and write.
Amit did it and I'm testing/reviewing it (so far so good). Regards JB On Sep 19, 2016, 20:52, at 20:52, Lukasz Cwik <[email protected]> wrote: >The Spark runner currently only supports a limited set of sources >because >there is no support in the Spark runner implementation which handles >the >Read PTransform directly. > >They are able to support TextIO, AvroIO and a few others by manually >providing a few direct implementations. > >The supported list is here: >https://github.com/apache/incubator-beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java#L865 > > > >On Mon, Sep 19, 2016 at 5:39 PM, Jesse Anderson <[email protected]> >wrote: > >> I'm running this Beam program >> ><https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game/utils/Input.java#L38>. >> When I run it on Spark, it get this error: >> >> Exception in thread "main" java.lang.RuntimeException: >java.lang.IllegalStateException: >> No TransformEvaluator registered for class >org.apache.beam.sdk.io.Read$ >> Bounded >> at >org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:201) >> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:71) >> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:182) >> at org.apache.beam.examples.tutorial.game.solution. >> Exercise2.main(Exercise2.java:91) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at sun.reflect.NativeMethodAccessorImpl.invoke( >> NativeMethodAccessorImpl.java:62) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke( >> DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$ >> deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) >> at >org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> Caused by: java.lang.IllegalStateException: No TransformEvaluator >> registered for class org.apache.beam.sdk.io.Read$Bounded >> at org.apache.beam.runners.spark.translation.TransformTranslator. >> getTransformEvaluator(TransformTranslator.java:892) >> at org.apache.beam.runners.spark.translation.TransformTranslator$ >> Translator.translate(TransformTranslator.java:910) >> at org.apache.beam.runners.spark.translation.SparkPipelineEvaluator. >> doVisitTransform(SparkPipelineEvaluator.java:49) >> at org.apache.beam.runners.spark.SparkRunner$Evaluator. >> visitPrimitiveTransform(SparkRunner.java:242) >> at org.apache.beam.sdk.runners.TransformTreeNode.visit( >> TransformTreeNode.java:224) >> at org.apache.beam.sdk.runners.TransformTreeNode.visit( >> TransformTreeNode.java:219) >> at org.apache.beam.sdk.runners.TransformTreeNode.visit( >> TransformTreeNode.java:219) >> at org.apache.beam.sdk.runners.TransformHierarchy.visit( >> TransformHierarchy.java:101) >> at >org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:292) >> at >org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:179) >> ... 12 more >> >> Looking at the TransformTranslator class, I don't see that >> org.apache.beam.sdk.io.Read is added. Is there a call I need to make >to >> add Read? >> >> Thanks, >> >> Jesse >> >>
