Hi Ganelin, sorry if it wasn't clear from my previous email, but that is
how I am creating a spark context. I just didn't write out the lines
where I create the new SparkConf and SparkContext. I am also upping the
driver memory when running.
Thanks,
David
On 01/12/2015 11:12 AM, Ganelin, Ilya wrote:
There are two related options:
To solve your problem directly try:
valconf =newSparkConf().set("spark.yarn.driver.memoryOverhead","1024")
valsc =newSparkContext(conf)
And the second, which increases the overall memory available on the driver, as
part of your spark-submit script add:
--driver-memory 2g
Hope this helps!
From: David McWhorter <[email protected] <mailto:[email protected]>>
Date: Monday, January 12, 2015 at 11:01 AM
To: "[email protected] <mailto:[email protected]>"
<[email protected] <mailto:[email protected]>>
Subject: configuring spark.yarn.driver.memoryOverhead on Spark 1.2.0
Hi all,
I'm trying to figure out how to set this option: "
spark.yarn.driver.memoryOverhead" on Spark 1.2.0. I found this
helpful overview
http://apache-spark-user-list.1001560.n3.nabble.com/Stable-spark-streaming-app-td14105.html#a14476,
which suggests to launch with --spark.yarn.driver.memoryOverhead 1024
added to spark-submit. However, when I do that I get this error:
Error: Unrecognized option '--spark.yarn.driver.memoryOverhead'.
Run with --help for usage help or --verbose for debug output
I have also tried calling
sparkConf.set("spark.yarn.driver.memoryOverhead", "1024") on my spark
configuration object but I still get "Will allocate AM container, with
XXXX MB memory including 384 MB overhead" when launching. I'm running
in yarn-cluster mode.
Any help or tips would be appreciated.
Thanks,
David
--
David McWhorter
Software Engineer
Commonwealth Computer Research, Inc.
1422 Sachem Place, Unit #1
Charlottesville, VA 22901
[email protected] | 434.299.0090x204
------------------------------------------------------------------------
The information contained in this e-mail is confidential and/or
proprietary to Capital One and/or its affiliates. The information
transmitted herewith is intended only for use by the individual or
entity to which it is addressed. If the reader of this message is not
the intended recipient, you are hereby notified that any review,
retransmission, dissemination, distribution, copying or other use of,
or taking of any action in reliance upon this information is strictly
prohibited. If you have received this communication in error, please
contact the sender and delete the material from your computer.
--
David McWhorter
Software Engineer
Commonwealth Computer Research, Inc.
1422 Sachem Place, Unit #1
Charlottesville, VA 22901
[email protected] | 434.299.0090x204
