[ https://issues.apache.org/jira/browse/SPARK-14977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Gummelt closed SPARK-14977. ----------------------------------- Resolution: Not A Problem > Fine grained mode in Mesos is not fair > -------------------------------------- > > Key: SPARK-14977 > URL: https://issues.apache.org/jira/browse/SPARK-14977 > Project: Spark > Issue Type: Bug > Components: Mesos > Affects Versions: 2.1.0 > Environment: Spark commit db75ccb, Debian jessie, Mesos fine grained > Reporter: Luca Bruno > > I've setup a mesos cluster and I'm running spark in fine grained mode. > Spark defaults to 2 executor cores and 2gb of ram. > The total mesos cluster has 8 cores and 8gb of ram. > When I submit two spark jobs simultaneously, spark will always accept full > resources, leading the two frameworks to use 4gb of ram each instead of 2gb. > If I submit another spark job, it will not get offered resources from mesos, > at least using the default HierarchicalDRF allocator module. > Mesos will keep offering 4gb of ram to earlier spark jobs, and spark keeps > accepting full resources for every new task. > Hence new spark jobs have no chance of getting a share. > Is this something to be solved with a custom mesos allocator? Or spark should > be more fair instead? Or maybe provide a configuration option to always > accept with the minimum resources? -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org