Hi Renjie, You can set number of cores per executor with spark executor cores in fine grain mode.
If you want coarse grain mode to support that it will Be supported in the near term as he coarse grain scheduler is getting revamped now. Tim > On Nov 28, 2015, at 7:31 PM, Renjie Liu <liurenjie2...@gmail.com> wrote: > > Hi, Nagaraj: > Thanks for the response, but this does not solve my problem. > I think executor memory should be proportional to number of cores, or number > of core > in each executor should be the same. >> On Sat, Nov 28, 2015 at 1:48 AM Nagaraj Chandrashekar >> <nchandrashe...@innominds.com> wrote: >> Hi Renjie, >> >> I have not setup Spark Streaming on Mesos but there is something called >> reservations in Mesos. It supports both Static and Dynamic reservations. >> Both types of reservations must have role defined. You may want to explore >> these options. Excerpts from the Apache Mesos documentation. >> >> Cheers >> Nagaraj C >> Reservation >> Mesos provides mechanisms to reserve resources in specific slaves. The >> concept was first introduced with static reservation in 0.14.0 which enabled >> operators to specify the reserved resources on slave startup. This was >> extended with dynamic reservation in 0.23.0 which enabled operators and >> authorized frameworks to dynamically reserve resources in the cluster. >> >> No breaking changes were introduced with dynamic reservation, which means >> the existing static reservation mechanism continues to be fully supported. >> >> In both types of reservations, resources are reserved for a role. >> >> Static Reservation (since 0.14.0) >> An operator can configure a slave with resources reserved for a role. The >> reserved resources are specified via the --resources flag. For example, >> suppose we have 12 CPUs and 6144 MB of RAM available on a slave and that we >> want to reserve 8 CPUs and 4096 MB of RAM for the ads role. We start the >> slave like so: >> >> $ mesos-slave \ >> --master=<ip>:<port> \ >> --resources="cpus:4;mem:2048;cpus(ads):8;mem(ads):4096" >> We now have 8 CPUs and 4096 MB of RAM reserved for ads on this slave. >> >> >> >> From: Renjie Liu <liurenjie2...@gmail.com> >> Date: Friday, November 27, 2015 at 9:57 PM >> To: "user@spark.apache.org" <user@spark.apache.org> >> Subject: Spark Streaming on mesos >> >> Hi, all: >> I'm trying to run spark streaming on mesos and it seems that none of the >> scheduler is suitable for that. Fine grain scheduler will start an executor >> for each task so it will significantly increase the latency. While coarse >> grained mode can only set the max core numbers and executor memory but >> there's no way to set the number of cores for each executor. Has anyone >> deployed spark streaming on mesos? And what's your settings? >> -- >> Liu, Renjie >> Software Engineer, MVAD > > -- > Liu, Renjie > Software Engineer, MVAD
