On Thu, Oct 13, 2016 at 1:42 PM, drewrobb <drewr...@gmail.com> wrote:

> When using spark on mesos and deploying a job in cluster mode using
> dispatcher, there appears to be no memory overhead configuration for the
> launched driver processes ("--driver-memory" is the same as Xmx which is
> the
> same as the memory quota). This makes it almost a guarantee that a long
> running driver will be OOM killed by mesos. Yarn cluster mode has an
> equivalent option -- spark.yarn.driver.memoryOverhead. Is there some way
> to
> configure driver memory overhead that I'm missing?
> Bigger picture question-- Is it even best practice to deploy long running
> spark streaming jobs using dispatcher? I could alternatively launch the
> driver by itself using marathon for example, where it would be trivial to
> grant the process additional memory.
> Thanks!
> I wrote a simple wrapper to provide a little head room, I'm currently
using the Chronos framework to run my spark jobs which provides some handy
environment variables like CHRONOS_RESOURCE_MEM which is what's offered
from Mesos.

    execString.append('--driver-memory {:.0f}M'.format(.7 *
    execString.append('--executor-memory {:.0f}M'.format(.98 *
    execString.append('--jars {} {} {}'.format(params.get('classpath'),
params.get('jarfile'), params.get('optargs')))


We basically set the upper limit for the driver to use 70% of what's
allocated in total to Mesos.
This will help get around Mesos killing long running jobs because of OOM.


[image: Orchard Platform] <http://www.orchardplatform.com/>

*Rodrick Brown */ *SRE - DevOPs*

9174456839 / rodr...@orchardplatform.com

Orchard Platform
101 5th Avenue, 4th Floor, New York, NY

*NOTICE TO RECIPIENTS*: This communication is confidential and intended for 
the use of the addressee only. If you are not an intended recipient of this 
communication, please delete it immediately and notify the sender by return 
email. Unauthorized reading, dissemination, distribution or copying of this 
communication is prohibited. This communication does not constitute an 
offer to sell or a solicitation of an indication of interest to purchase 
any loan, security or any other financial product or instrument, nor is it 
an offer to sell or a solicitation of an indication of interest to purchase 
any products or services to any persons who are prohibited from receiving 
such information under applicable law. The contents of this communication 
may not be accurate or complete and are subject to change without notice. 
As such, Orchard App, Inc. (including its subsidiaries and affiliates, 
"Orchard") makes no representation regarding the accuracy or completeness 
of the information contained herein. The intended recipient is advised to 
consult its own professional advisors, including those specializing in 
legal, tax and accounting matters. Orchard does not provide legal, tax or 
accounting advice.

Reply via email to