[
https://issues.apache.org/jira/browse/BEAM-470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit Sela closed BEAM-470.
--------------------------
Resolution: Not A Bug
Fix Version/s: Not applicable
> Spark Runner does not send the job execution information into the Spark
> History Server
> --------------------------------------------------------------------------------------
>
> Key: BEAM-470
> URL: https://issues.apache.org/jira/browse/BEAM-470
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
> Affects Versions: 0.2.0-incubating
> Reporter: Ismaël Mejía
> Priority: Minor
> Fix For: Not applicable
>
>
> If you run a Beam pipeline using the spark runner from spark (via
> spark-submit), the execution is registered in the spark-history-server if it
> is active and configured.
> if you do this directly from a main method with --runner=SparkRunner (the
> beam way) the Beam runner does not report the execution to the history
> server, it seems the issue is the runner does not take into account an
> existing spark configuration file SPARK_HOME/conf/spark-defaults.conf (or
> there is not a way to tell the runner to take such conf into account).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
