Hi Antony, Unbounded views are not currently supported in the Spark runner. The following ticket tracks the progress for adding support for this: https://issues.apache.org/jira/browse/BEAM-2112
On Tue, May 30, 2017 at 7:11 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi Antony, > > which version of the Spark runner are you using ? > > Regards > JB > > On 05/30/2017 06:04 PM, Antony Mayi wrote: > > Hi, > > > > started getting following error when running on spark. can anyone see > what I > > could be doing wrong? It only happens with unbounded source. Same > pipeline with > > bounded source runs fine. > > > > thx, > > a. > > > > java.lang.IllegalStateException: No TransformEvaluator registered for > UNBOUNDED > > transform class org.apache.beam.sdk.transforms.View$CreatePCollectionView > > at > > > org.apache.beam.runners.spark.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:518) > > at > > > org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$Translator.translateUnbounded(StreamingTransformTranslator.java:529) > > at > > > org.apache.beam.runners.spark.SparkRunner$Evaluator.translate(SparkRunner.java:435) > > at > > > org.apache.beam.runners.spark.SparkRunner$Evaluator.doVisitTransform(SparkRunner.java:405) > > at > > > org.apache.beam.runners.spark.SparkRunner$Evaluator.visitPrimitiveTransform(SparkRunner.java:395) > > at > > > org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:491) > > at > > > org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:486) > > at > > > org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:486) > > at > > > org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:486) > > at > > > org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:486) > > at > > > org.apache.beam.sdk.runners.TransformHierarchy$Node.access$400(TransformHierarchy.java:235) > > at > org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:210) > > at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:409) > > at > > > org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:88) > > at > > > org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:47) > > at > > > org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$10.apply(JavaStreamingContext.scala:776) > > at > > > org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$10.apply(JavaStreamingContext.scala:775) > > at scala.Option.getOrElse(Option.scala:120) > > at > > > org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:864) > > at > > > org.apache.spark.streaming.api.java.JavaStreamingContext$.getOrCreate(JavaStreamingContext.scala:775) > > at > > > org.apache.spark.streaming.api.java.JavaStreamingContext.getOrCreate(JavaStreamingContext.scala) > > at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:155) > > at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:85) > > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:295) > > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:281) > > > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com >
