[
https://issues.apache.org/jira/browse/SPARK-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606996#comment-14606996
]
Felix Cheung commented on SPARK-8041:
-------------------------------------
I've done some work to automatically get the RLibDir from R - is that something
that might be useful here?
I think when running sparkR (shell) or YARN client mode we should get that from
R.
YARN cluster mode might be different.
> Consistently pass SparkR library directory to SparkR application
> ----------------------------------------------------------------
>
> Key: SPARK-8041
> URL: https://issues.apache.org/jira/browse/SPARK-8041
> Project: Spark
> Issue Type: Improvement
> Components: SparkR
> Affects Versions: 1.4.0
> Reporter: Sun Rui
>
> The SparkR package library directory path (RLibDir) is needed for SparkR
> applications for loading SparkR package and locating R helper files inside
> the package.
> Currently, there are some places that the RLibDir needs to be specified.
> First of all, when you programs a SparkR application, sparkR.init() allows
> you to pass a RLibDir parameter (by default, it is the same as the SparkR
> package's libname on the driver host). However, it seems not reasonable to
> hard-code RLibDir in a program. Instead, it would be more flexible to pass
> RLibDir via command line or env variable.
> Additionally, for YARN cluster mode, RRunner depends on SPARK_HOME env
> variable to get the RLibDir (assume $SPARK_HOME/R/lib).
> So it would be better to define a consistent way to pass RLibDir to a SparkR
> application in all deployment modes. It could be a command line option for
> bin/sparkR or an env variable. It can be passed to a sparkR application, and
> we can remove the RLibDir parameter of sparkR.init(). When in YARN cluster
> mode, it can be passed to AM using
> spark.yarn.appMasterEnv.[EnvironmentVariableName] configuration option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
