Amit,
You can simply set “MASTER” as “yarn-client” before calling sparkR.init().
Sys.setenv("MASTER"="yarn-client")
I assume that you have set “YARN_CONF_DIR” env variable required for running
Spark on YARN.
If you want to set more YARN specific configurations, you can for example
Sys.setenv ("SPARKR_SUBMIT_ARGS", " --master yarn-client --num-executors 4
sparkr-shell"
Before calling sparkR.init().
From: Amit Behera [mailto:[email protected]]
Sent: Monday, November 9, 2015 2:36 AM
To: [email protected]
Subject: Connecting SparkR through Yarn
Hi All,
Spark Version = 1.5.1
Hadoop Version = 2.6.0
I set up the cluster in Amazon EC2 machines (1+5)
I am able create a SparkContext object using init method from RStudio.
But I do not know how can I create a SparkContext object in yarn mode.
I got the below link to run on yarn. but in this document its given for Spark
version >= 0.9.0 and <= 1.2.
https://github.com/amplab-extras/SparkR-pkg/blob/master/README.md#running-on-yarn
Please help me how can I connect SparkR on Yarn.
Thanks,
Amit.
