[
https://issues.apache.org/jira/browse/BEAM-470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15689594#comment-15689594
]
Amit Sela commented on BEAM-470:
--------------------------------
Can this be reproduced ? I've managed to run a local history server and see
results there.
> Spark Runner does not send the job execution information into the Spark
> History Server
> --------------------------------------------------------------------------------------
>
> Key: BEAM-470
> URL: https://issues.apache.org/jira/browse/BEAM-470
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
> Affects Versions: 0.2.0-incubating
> Reporter: Ismaël Mejía
> Priority: Minor
>
> If you run a Beam pipeline using the spark runner from spark (via
> spark-submit), the execution is registered in the spark-history-server if it
> is active and configured.
> if you do this directly from a main method with --runner=SparkRunner (the
> beam way) the Beam runner does not report the execution to the history
> server, it seems the issue is the runner does not take into account an
> existing spark configuration file SPARK_HOME/conf/spark-defaults.conf (or
> there is not a way to tell the runner to take such conf into account).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
