[ https://issues.apache.org/jira/browse/SPARK-11340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983294#comment-14983294 ]
Shivaram Venkataraman commented on SPARK-11340: ----------------------------------------------- [~felixcheung][~sunrui] It'll be great if we can ask the user who reported this to test the fix ! > Support setting driver properties when starting Spark from R programmatically > or from RStudio > --------------------------------------------------------------------------------------------- > > Key: SPARK-11340 > URL: https://issues.apache.org/jira/browse/SPARK-11340 > Project: Spark > Issue Type: Bug > Components: SparkR > Affects Versions: 1.5.1 > Reporter: Felix Cheung > Assignee: Felix Cheung > Priority: Minor > Fix For: 1.6.0 > > > Currently when sparkR.init() is called in 'client' mode, it launches the JVM > backend but driver properties (like driver-memory) are not passed or settable > by the user calling sparkR.init(). > [~sunrui][~shivaram] and I discussed this offline and think we should support > this. > This is the original thread: > >> From: rui....@intel.com > >> To: dirceu.semigh...@gmail.com > >> CC: u...@spark.apache.org > >> Subject: RE: How to set memory for SparkR with master="local[*]" > >> Date: Mon, 26 Oct 2015 02:24:00 +0000 > >> > >> As documented in > >> http://spark.apache.org/docs/latest/configuration.html#available-prop > >> e > >> rties, > >> > >> Note for “spark.driver.memory”: > >> > >> Note: In client mode, this config must not be set through the > >> SparkConf directly in your application, because the driver JVM has > >> already started at that point. Instead, please set this through the > >> --driver-memory command line option or in your default properties file. > >> > >> > >> > >> If you are to start a SparkR shell using bin/sparkR, then you can use > >> bin/sparkR –driver-memory. You have no chance to set the driver > >> memory size after the R shell has been launched via bin/sparkR. > >> > >> > >> > >> Buf if you are to start a SparkR shell manually without using > >> bin/sparkR (for example, in Rstudio), you can: > >> > >> library(SparkR) > >> > >> Sys.setenv("SPARKR_SUBMIT_ARGS" = "--conf spark.driver.memory=2g > >> sparkr-shell") > >> > >> sc <- sparkR.init() > >> > >> > >> > >> From: Dirceu Semighini Filho [mailto:dirceu.semigh...@gmail.com] > >> Sent: Friday, October 23, 2015 7:53 PM > >> Cc: user > >> Subject: Re: How to set memory for SparkR with master="local[*]" > >> > >> > >> > >> Hi Matej, > >> > >> I'm also using this and I'm having the same behavior here, my driver > >> has only 530mb which is the default value. > >> > >> > >> > >> Maybe this is a bug. > >> > >> > >> > >> 2015-10-23 9:43 GMT-02:00 Matej Holec <hol...@gmail.com>: > >> > >> Hello! > >> > >> How to adjust the memory settings properly for SparkR with > >> master="local[*]" > >> in R? > >> > >> > >> *When running from R -- SparkR doesn't accept memory settings :(* > >> > >> I use the following commands: > >> > >> R> library(SparkR) > >> R> sc <- sparkR.init(master = "local[*]", sparkEnvir = > >> list(spark.driver.memory = "5g")) > >> > >> Despite the variable spark.driver.memory is correctly set (checked in > >> http://node:4040/environment/), the driver has only the default > >> amount of memory allocated (Storage Memory 530.3 MB). > >> > >> *But when running from spark-1.5.1-bin-hadoop2.6/bin/sparkR -- OK* > >> > >> The following command: > >> > >> ]$ spark-1.5.1-bin-hadoop2.6/bin/sparkR --driver-memory 5g > >> > >> creates SparkR session with properly adjustest driver memory (Storage > >> Memory > >> 2.6 GB). > >> > >> > >> Any suggestion? > >> > >> Thanks > >> Matej > >> > >> -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org