Re: SparkR in yarn-client mode needs sparkr.zip

Felix, Missed your reply - agree looks like the same issue, resolved mine as Duplicate. Thanks! Ram On Sun, Oct 25, 2015 at 2:47 PM, Felix Cheung wrote: > > > This might be related to https://issues.apache.org/jira/browse/SPARK-10500 > > > > On Sun, Oct 25, 2015 at

Re: SparkR in yarn-client mode needs sparkr.zip

This might be related to https://issues.apache.org/jira/browse/SPARK-10500 On Sun, Oct 25, 2015 at 9:57 AM -0700, "Ted Yu" wrote: In zipRLibraries(): // create a zip file from scratch, do not append to existing file. val zipFile = new File(dir, name) I guess

Re: SparkR in yarn-client mode needs sparkr.zip

Ted Yu, Agree that either picking up sparkr.zip if it already exists, or creating a zip in a local scratch directory will work. This code is called by the client side job submission logic and the resulting zip is already added to the local resources for the YARN job, so I don't think the

Re: SparkR in yarn-client mode needs sparkr.zip

In zipRLibraries(): // create a zip file from scratch, do not append to existing file. val zipFile = new File(dir, name) I guess instead of creating sparkr.zip in the same directory as R lib, the zip file can be created under some directory writable by the user launching the app and

SparkR in yarn-client mode needs sparkr.zip

If you run sparkR in yarn-client mode, it fails with Exception in thread "main" java.io.FileNotFoundException: /usr/hdp/2.3.2.1-12/spark/R/lib/sparkr.zip (Permission denied) at java.io.FileOutputStream.open0(Native Method) at