[
https://issues.apache.org/jira/browse/SPARK-6797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568863#comment-14568863
]
Sun Rui commented on SPARK-6797:
--------------------------------
@shivaram, I can run SparkR in the YARN cluster mode (need to set SPARK_HOME
for the AM). So could you explain in more detail what the problem is. For now,
SparkR package has to be installed into the R environment on each worker node.
Do you mean that we ship SparkR package source to AM and worker nodes and
compile/install it for each application? or Do you mean that we support
something like spark-submit --rfiles (similar to --pyfiles) on which the
primary R script depends?
> Add support for YARN cluster mode
> ---------------------------------
>
> Key: SPARK-6797
> URL: https://issues.apache.org/jira/browse/SPARK-6797
> Project: Spark
> Issue Type: Improvement
> Components: SparkR
> Reporter: Shivaram Venkataraman
> Assignee: Sun Rui
> Priority: Critical
>
> SparkR currently does not work in YARN cluster mode as the R package is not
> shipped along with the assembly jar to the YARN AM. We could try to use the
> support for archives in YARN to send out the R package as a zip file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
