Ah I see, that's a bit more complicated =). If it's possible, would using `spark.executor.memory` to set the available worker memory used by executors help alleviate the problem of running on a node that already has an executor on it? I would assume that would have a constant worst case overhead per worker and shouldn't matter if two executors on the same node are from one application or two applications. Maybe you have data that states otherwise? I suppose it depends on what resources are causing problems when the executors are imbalanced across the cluster. That could be an indication of possibly not leaving enough free RAM outside the worker and executors heap allocations on worker nodes.
On Tue, Oct 27, 2015 at 4:57 PM, mkhaitman <mark.khait...@chango.com> wrote: > Hi Richard, > > Thanks for the response. > > I should have added that the specific case where this becomes a problem is > when one of the executors for that application is lost/killed prematurely, > and the application attempts to spawn up a new executor without > consideration as to whether an executor already exists on the other node. > > In your example, if one of the executors dies for some reason (memory > exhaustion, or something crashed it), if there are still free cores on the > other nodes, it will spawn an extra executor, which can lead to further > memory problems on the other node that it just spawned on. > > Hopefully that clears up what I mean :) > > Mark. > > > > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-Executor-Cores-question-tp14763p14805.html > Sent from the Apache Spark Developers List mailing list archive at > Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > > -- *Richard Marscher* Software Engineer Localytics Localytics.com <http://localytics.com/> | Our Blog <http://localytics.com/blog> | Twitter <http://twitter.com/localytics> | Facebook <http://facebook.com/localytics> | LinkedIn <http://www.linkedin.com/company/1148792?trk=tyah>
