[
https://issues.apache.org/jira/browse/SPARK-11340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-11340:
------------------------------------
Assignee: (was: Apache Spark)
> Support setting driver properties when starting Spark from R programmatically
> or from RStudio
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-11340
> URL: https://issues.apache.org/jira/browse/SPARK-11340
> Project: Spark
> Issue Type: Bug
> Components: SparkR
> Affects Versions: 1.5.1
> Reporter: Felix Cheung
> Priority: Minor
>
> Currently when sparkR.init() is called in 'client' mode, it launches the JVM
> backend but driver properties (like driver-memory) are not passed or settable
> by the user calling sparkR.init().
> [~sunrui][~shivaram] and I discussed this offline and think we should support
> this.
> This is the original thread:
> >> From: [email protected]
> >> To: [email protected]
> >> CC: [email protected]
> >> Subject: RE: How to set memory for SparkR with master="local[*]"
> >> Date: Mon, 26 Oct 2015 02:24:00 +0000
> >>
> >> As documented in
> >> http://spark.apache.org/docs/latest/configuration.html#available-prop
> >> e
> >> rties,
> >>
> >> Note for “spark.driver.memory”:
> >>
> >> Note: In client mode, this config must not be set through the
> >> SparkConf directly in your application, because the driver JVM has
> >> already started at that point. Instead, please set this through the
> >> --driver-memory command line option or in your default properties file.
> >>
> >>
> >>
> >> If you are to start a SparkR shell using bin/sparkR, then you can use
> >> bin/sparkR –driver-memory. You have no chance to set the driver
> >> memory size after the R shell has been launched via bin/sparkR.
> >>
> >>
> >>
> >> Buf if you are to start a SparkR shell manually without using
> >> bin/sparkR (for example, in Rstudio), you can:
> >>
> >> library(SparkR)
> >>
> >> Sys.setenv("SPARKR_SUBMIT_ARGS" = "--conf spark.driver.memory=2g
> >> sparkr-shell")
> >>
> >> sc <- sparkR.init()
> >>
> >>
> >>
> >> From: Dirceu Semighini Filho [mailto:[email protected]]
> >> Sent: Friday, October 23, 2015 7:53 PM
> >> Cc: user
> >> Subject: Re: How to set memory for SparkR with master="local[*]"
> >>
> >>
> >>
> >> Hi Matej,
> >>
> >> I'm also using this and I'm having the same behavior here, my driver
> >> has only 530mb which is the default value.
> >>
> >>
> >>
> >> Maybe this is a bug.
> >>
> >>
> >>
> >> 2015-10-23 9:43 GMT-02:00 Matej Holec <[email protected]>:
> >>
> >> Hello!
> >>
> >> How to adjust the memory settings properly for SparkR with
> >> master="local[*]"
> >> in R?
> >>
> >>
> >> *When running from R -- SparkR doesn't accept memory settings :(*
> >>
> >> I use the following commands:
> >>
> >> R> library(SparkR)
> >> R> sc <- sparkR.init(master = "local[*]", sparkEnvir =
> >> list(spark.driver.memory = "5g"))
> >>
> >> Despite the variable spark.driver.memory is correctly set (checked in
> >> http://node:4040/environment/), the driver has only the default
> >> amount of memory allocated (Storage Memory 530.3 MB).
> >>
> >> *But when running from spark-1.5.1-bin-hadoop2.6/bin/sparkR -- OK*
> >>
> >> The following command:
> >>
> >> ]$ spark-1.5.1-bin-hadoop2.6/bin/sparkR --driver-memory 5g
> >>
> >> creates SparkR session with properly adjustest driver memory (Storage
> >> Memory
> >> 2.6 GB).
> >>
> >>
> >> Any suggestion?
> >>
> >> Thanks
> >> Matej
> >>
> >>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
