Felix Cheung created SPARK-11340:
------------------------------------
Summary: Support setting driver properties when starting Spark
from R programmatically or from RStudio
Key: SPARK-11340
URL: https://issues.apache.org/jira/browse/SPARK-11340
Project: Spark
Issue Type: Bug
Components: SparkR
Affects Versions: 1.5.1
Reporter: Felix Cheung
Priority: Minor
Currently when sparkR.init() is called in 'client' mode, it launches the JVM
backend but driver properties (like driver-memory) are not passed or settable
by the user calling sparkR.init().
[~sunrui][~shivaram] and I discussed this offline and think we should support
this.
This is the original thread:
>> From: [email protected]
>> To: [email protected]
>> CC: [email protected]
>> Subject: RE: How to set memory for SparkR with master="local[*]"
>> Date: Mon, 26 Oct 2015 02:24:00 +0000
>>
>> As documented in
>> http://spark.apache.org/docs/latest/configuration.html#available-prop
>> e
>> rties,
>>
>> Note for “spark.driver.memory”:
>>
>> Note: In client mode, this config must not be set through the
>> SparkConf directly in your application, because the driver JVM has
>> already started at that point. Instead, please set this through the
>> --driver-memory command line option or in your default properties file.
>>
>>
>>
>> If you are to start a SparkR shell using bin/sparkR, then you can use
>> bin/sparkR –driver-memory. You have no chance to set the driver
>> memory size after the R shell has been launched via bin/sparkR.
>>
>>
>>
>> Buf if you are to start a SparkR shell manually without using
>> bin/sparkR (for example, in Rstudio), you can:
>>
>> library(SparkR)
>>
>> Sys.setenv("SPARKR_SUBMIT_ARGS" = "--conf spark.driver.memory=2g
>> sparkr-shell")
>>
>> sc <- sparkR.init()
>>
>>
>>
>> From: Dirceu Semighini Filho [mailto:[email protected]]
>> Sent: Friday, October 23, 2015 7:53 PM
>> Cc: user
>> Subject: Re: How to set memory for SparkR with master="local[*]"
>>
>>
>>
>> Hi Matej,
>>
>> I'm also using this and I'm having the same behavior here, my driver
>> has only 530mb which is the default value.
>>
>>
>>
>> Maybe this is a bug.
>>
>>
>>
>> 2015-10-23 9:43 GMT-02:00 Matej Holec <[email protected]>:
>>
>> Hello!
>>
>> How to adjust the memory settings properly for SparkR with master="local[*]"
>> in R?
>>
>>
>> *When running from R -- SparkR doesn't accept memory settings :(*
>>
>> I use the following commands:
>>
>> R> library(SparkR)
>> R> sc <- sparkR.init(master = "local[*]", sparkEnvir =
>> list(spark.driver.memory = "5g"))
>>
>> Despite the variable spark.driver.memory is correctly set (checked in
>> http://node:4040/environment/), the driver has only the default
>> amount of memory allocated (Storage Memory 530.3 MB).
>>
>> *But when running from spark-1.5.1-bin-hadoop2.6/bin/sparkR -- OK*
>>
>> The following command:
>>
>> ]$ spark-1.5.1-bin-hadoop2.6/bin/sparkR --driver-memory 5g
>>
>> creates SparkR session with properly adjustest driver memory (Storage
>> Memory
>> 2.6 GB).
>>
>>
>> Any suggestion?
>>
>> Thanks
>> Matej
>>
>>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
