Felix Cheung created SPARK-11340: ------------------------------------ Summary: Support setting driver properties when starting Spark from R programmatically or from RStudio Key: SPARK-11340 URL: https://issues.apache.org/jira/browse/SPARK-11340 Project: Spark Issue Type: Bug Components: SparkR Affects Versions: 1.5.1 Reporter: Felix Cheung Priority: Minor
Currently when sparkR.init() is called in 'client' mode, it launches the JVM backend but driver properties (like driver-memory) are not passed or settable by the user calling sparkR.init(). [~sunrui][~shivaram] and I discussed this offline and think we should support this. This is the original thread: >> From: rui....@intel.com >> To: dirceu.semigh...@gmail.com >> CC: u...@spark.apache.org >> Subject: RE: How to set memory for SparkR with master="local[*]" >> Date: Mon, 26 Oct 2015 02:24:00 +0000 >> >> As documented in >> http://spark.apache.org/docs/latest/configuration.html#available-prop >> e >> rties, >> >> Note for “spark.driver.memory”: >> >> Note: In client mode, this config must not be set through the >> SparkConf directly in your application, because the driver JVM has >> already started at that point. Instead, please set this through the >> --driver-memory command line option or in your default properties file. >> >> >> >> If you are to start a SparkR shell using bin/sparkR, then you can use >> bin/sparkR –driver-memory. You have no chance to set the driver >> memory size after the R shell has been launched via bin/sparkR. >> >> >> >> Buf if you are to start a SparkR shell manually without using >> bin/sparkR (for example, in Rstudio), you can: >> >> library(SparkR) >> >> Sys.setenv("SPARKR_SUBMIT_ARGS" = "--conf spark.driver.memory=2g >> sparkr-shell") >> >> sc <- sparkR.init() >> >> >> >> From: Dirceu Semighini Filho [mailto:dirceu.semigh...@gmail.com] >> Sent: Friday, October 23, 2015 7:53 PM >> Cc: user >> Subject: Re: How to set memory for SparkR with master="local[*]" >> >> >> >> Hi Matej, >> >> I'm also using this and I'm having the same behavior here, my driver >> has only 530mb which is the default value. >> >> >> >> Maybe this is a bug. >> >> >> >> 2015-10-23 9:43 GMT-02:00 Matej Holec <hol...@gmail.com>: >> >> Hello! >> >> How to adjust the memory settings properly for SparkR with master="local[*]" >> in R? >> >> >> *When running from R -- SparkR doesn't accept memory settings :(* >> >> I use the following commands: >> >> R> library(SparkR) >> R> sc <- sparkR.init(master = "local[*]", sparkEnvir = >> list(spark.driver.memory = "5g")) >> >> Despite the variable spark.driver.memory is correctly set (checked in >> http://node:4040/environment/), the driver has only the default >> amount of memory allocated (Storage Memory 530.3 MB). >> >> *But when running from spark-1.5.1-bin-hadoop2.6/bin/sparkR -- OK* >> >> The following command: >> >> ]$ spark-1.5.1-bin-hadoop2.6/bin/sparkR --driver-memory 5g >> >> creates SparkR session with properly adjustest driver memory (Storage >> Memory >> 2.6 GB). >> >> >> Any suggestion? >> >> Thanks >> Matej >> >> -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org