I think you can also pass in a zip file using the --files option (http://spark.apache.org/docs/latest/running-on-yarn.html has some examples). The files should then be present in the current working directory of the driver R process.
Thanks Shivaram On Wed, Aug 17, 2016 at 4:16 AM, Felix Cheung <felixcheun...@hotmail.com> wrote: > When you call library(), that is the library loading function in native R. > As of now it does not support HDFS but there are several packages out there > that might help. > > Another approach is to have a prefetch/installation mechanism to call HDFS > command to download the R package from HDFS onto the worker node first. > > > _____________________________ > From: Senthil Kumar <senthilec...@gmail.com> > Sent: Wednesday, August 17, 2016 2:23 AM > Subject: Spark R - Loading Third Party R Library in YARN Executors > To: Senthil kumar <senthilec...@gmail.com>, <du...@ebay.com>, > <jiaj...@ebay.com>, <dev@spark.apache.org> > > > > Hi All , We are using Spark 1.6 Version R library .. Below is our code > which Loads the THIRD Party Library . > > > library("BreakoutDetection", lib.loc = "hdfs://xxxxxx/BreakoutDetection/") : > library("BreakoutDetection", lib.loc = "//xxxxxx/BreakoutDetection/") : > > > When i try to execute the code using LOCAL Mode , Spark R code is Working > fine without any issue . If i submit the Job in Cluster , we will end up > with error. > > error in evaluating the argument 'X' in selecting a method for function > 'lapply': Error in library("BreakoutDetection", lib.loc = > "hdfs://xxxxxxx/BreakoutDetection/") : > no library trees found in 'lib.loc' > Calls: f ... lapply -> FUN -> mainProcess -> angleValid -> library > > > Can't we read libraries in R as below ? > library("BreakoutDetection", lib.loc = "hdfs://xxxxxx/BreakoutDetection/") : > > If not what is the other way to solve this problem ? > > Since our cluster having close to 2500 nodes we cant copy the Third Party > Libs to all nodes .. Copying to all DNs is not good practice too .. > > Can someone help me here How to load R libs from HDFS or any other way ? > > > --Senthil > > > > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org