[
https://issues.apache.org/jira/browse/BEAM-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17023657#comment-17023657
]
Enis Nazif edited comment on BEAM-8970 at 1/25/20 10:12 PM:
------------------------------------------------------------
looking at this issue, to run a pipeline on YARN backed spark cluster, a user
should be able to specify runner options of
{code:java}
['--runner=SparkRunner',
'--spark_submit_uber_jar',
'--spark_rest_url=http://spark-rest-api:6066',
'--spark_master_url='yarn']{code}
As it stands, the 'spark_master_url' isn't being passed into the request
created in in
[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/spark_uber_jar_job_server.py#L145]
It seems that this is necessary to support YARN
Failing this, an alternative way may be to bypass the Spark REST API (which
seems like fairly hidden functionality) and instead directly spark-submit the
portable jars that are created.
was (Author: enazif):
looking at this issue, to run a pipeline on YARN backed sparked, a user should
be able to specify runner options of
{code:java}
['--runner=SparkRunner',
'--spark_submit_uber_jar',
'--spark_rest_url=http://spark-rest-api:6066',
'--spark_master_url='yarn']{code}
As it stands, the 'spark_master_url' isn't being passed into the request
created in in
[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/spark_uber_jar_job_server.py#L145]
It seems that this is necessary to support YARN
Failing this, an alternative way may be to bypass the Spark REST API (which
seems like fairly hidden functionality) and instead directly spark-submit the
portable jars that are created.
> Spark portable runner supports Yarn
> -----------------------------------
>
> Key: BEAM-8970
> URL: https://issues.apache.org/jira/browse/BEAM-8970
> Project: Beam
> Issue Type: Wish
> Components: runner-spark
> Reporter: Kyle Weaver
> Assignee: Kyle Weaver
> Priority: Major
> Labels: portability-spark
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
