[
https://issues.apache.org/jira/browse/BEAM-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905065#comment-15905065
]
ASF GitHub Bot commented on BEAM-797:
-------------------------------------
Github user asfgit closed the pull request at:
https://github.com/apache/beam/pull/1868
> A PipelineVisitor that creates a Spark-native pipeline.
> --------------------------------------------------------
>
> Key: BEAM-797
> URL: https://issues.apache.org/jira/browse/BEAM-797
> Project: Beam
> Issue Type: Wish
> Components: runner-spark
> Reporter: Amit Sela
> Assignee: Aviem Zur
> Priority: Minor
>
> It could be very useful for debugging purposes to have a custom
> PipelineVisitor that can tell what's the underlying Spark code that is being
> called.
> One idea:
> This could be called with a flag in SparkPipelineOptions and instead of
> executing the pipeline, it would print the underlying Spark DAG.
> Clearly, DoFn internals would be obfuscated, but the Spark code could note
> {{mapPartitions("ExtractWords")}}
> Another difference would be Sources as they are a custom implementation for
> Beam.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
