Amit Sela created BEAM-797:
------------------------------
Summary: A PipelineVisitor that creates a Spark-native pipeline.
Key: BEAM-797
URL: https://issues.apache.org/jira/browse/BEAM-797
Project: Beam
Issue Type: Wish
Components: runner-spark
Reporter: Amit Sela
Assignee: Amit Sela
Priority: Minor
It could be very useful for debugging purposes to have a custom PipelineVisitor
that can tell what's the underlying Spark code that is being called.
One idea:
This could be called with a flag in SparkPipelineOptions and instead of
executing the pipeline, it would print the underlying Spark DAG.
Clearly, DoFn internals would be obfuscated, but the Spark code could note
{{mapPartitions("ExtractWords")}}
Another difference would be Sources as they are a custom implementation for
Beam.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
