Following up my earlier question...Do you have any suggestion on the best way to configure Spark and Hadoop in order to equally share resources?
I see that for Hadoop I have the: mapred.mesos.total.map.slots.minimum and mapred.mesos.total.reduce.slots.minimum but I cannot find a similar property for the maximum slots. Moreover I am not aware of similar configuration properties at Spark . (Is this the most relevant? spark.scheduler.minRegisteredResourcesRatio) Are you aware of a place that I can have documented a full list of available configurations for both frameworks? So far my source is this: https://github.com/mesos/hadoop/blob/master/configuration.md for hadoop and this: http://spark.apache.org/docs/1.2.1/configuration.html#spark-properties for spark but they don't include all the properties. If you are not aware of any documentation maybe suggestion on the parts of code that I could check to find these configuration parameters? thanks, Stratos On Thu, Mar 19, 2015 at 7:32 PM, Stratos Dimopoulos < [email protected]> wrote: > Thank you for the answer Benjamin! > > On Thu, Mar 19, 2015 at 6:05 PM, Benjamin Mahler < > [email protected]> wrote: > >> Guaranteeing fairness in such situations requires pre-emption of running >> tasks/executors, which is not yet provided in mesos. >> >> For now, you can try reserving a minimum amount of resources for each >> framework, note however that this may reduce your efficiency if you >> over-estimate the minimum reservation needed. This is because reserved >> resources cannot be used by other roles. In the future, to avoid the >> efficiency issue, other roles will be able to use reserved resources in a >> pre-emptable way. >> >> On Thu, Mar 19, 2015 at 3:39 PM, Stratos Dimopoulos < >> [email protected]> wrote: >> >>> Hi All, >>> >>> Is there a way to dynamically fair-share resources between the >>> frameworks that running on top of Mesos? What I would like to do is when >>> running for example MapReduce and Spark together, allocate 50% of the >>> resources to each of them (if they need them) and when running MapReduce, >>> Spark, Storm 33% of the resources to each. >>> >>> What I observe is that when I am running Spark in fine grained mode and >>> MapReduce together, MR will gradually take over all the resources as the >>> task trackers of Spark are finished and in the meantime before Spark >>> manages to stage new TaskTrackers Hadoop gets more and more resources until >>> it actually takes over the whole cluster. The opposite happens with I run >>> Spark in coarse grained mode. In this case Spark is faster staging the task >>> trackers and it will manage to get 100% of the cluster before Hadoop gets >>> any. >>> >>> I checked this: >>> http://mesos.apache.org/documentation/latest/framework-rate-limiting/ >>> that might help but what I would really want to is to share the >>> resources equally between registered frameworks. Any ideas? >>> >>> thanks! >>> >>> Stratos >>> >> >> >
