Hi, Lulian: Are you sure that it'll be a long running process in fine-grained mode? I think you have a misunderstanding about it. An executor will be launched for some tasks, but not a long running process. When a group of tasks finished, it will get shutdown.
On Mon, Nov 30, 2015 at 6:25 PM Iulian Dragoș <iulian.dra...@typesafe.com> wrote: > Hi, > > Latency isn't such a big issue as it sounds. Did you try it out and failed > some performance metrics? > > In short, the *Mesos* executor on a given slave is going to be > long-running (consuming memory, but no CPUs). Each Spark task will be > scheduled using Mesos CPU resources, but they don't suffer much latency. > > iulian > > > On Mon, Nov 30, 2015 at 4:17 AM, Renjie Liu <liurenjie2...@gmail.com> > wrote: > >> Hi, Tim: >> Fine grain mode is not suitable for streaming applications since it need >> to start up an executor each time. When will the revamp get release? In the >> coming 1.6.0? >> >> On Sun, Nov 29, 2015 at 6:16 PM Timothy Chen <t...@mesosphere.io> wrote: >> >>> Hi Renjie, >>> >>> You can set number of cores per executor with spark executor cores in >>> fine grain mode. >>> >>> If you want coarse grain mode to support that it will >>> Be supported in the near term as he coarse grain scheduler is getting >>> revamped now. >>> >>> Tim >>> >>> On Nov 28, 2015, at 7:31 PM, Renjie Liu <liurenjie2...@gmail.com> wrote: >>> >>> Hi, Nagaraj: >>> Thanks for the response, but this does not solve my problem. >>> I think executor memory should be proportional to number of cores, or >>> number of core >>> in each executor should be the same. >>> On Sat, Nov 28, 2015 at 1:48 AM Nagaraj Chandrashekar < >>> nchandrashe...@innominds.com> wrote: >>> >>>> Hi Renjie, >>>> >>>> I have not setup Spark Streaming on Mesos but there is something called >>>> reservations in Mesos. It supports both Static and Dynamic reservations. >>>> Both types of reservations must have role defined. You may want to explore >>>> these options. Excerpts from the Apache Mesos documentation. >>>> >>>> Cheers >>>> Nagaraj C >>>> Reservation >>>> >>>> Mesos provides mechanisms to reserve resources in specific slaves. The >>>> concept was first introduced with static reservation in 0.14.0 which >>>> enabled operators to specify the reserved resources on slave startup. This >>>> was extended with dynamic reservation in 0.23.0 which enabled >>>> operators and authorized frameworks to dynamically reserve resources >>>> in the cluster. >>>> >>>> No breaking changes were introduced with dynamic reservation, which >>>> means the existing static reservation mechanism continues to be fully >>>> supported. >>>> >>>> In both types of reservations, resources are reserved for a role. >>>> Static Reservation (since 0.14.0) >>>> >>>> An operator can configure a slave with resources reserved for a role. >>>> The reserved resources are specified via the --resources flag. For >>>> example, suppose we have 12 CPUs and 6144 MB of RAM available on a slave >>>> and that we want to reserve 8 CPUs and 4096 MB of RAM for the ads role. >>>> We start the slave like so: >>>> >>>> $ mesos-slave \ >>>> --master=<ip>:<port> \ >>>> --resources="cpus:4;mem:2048;cpus(ads):8;mem(ads):4096" >>>> >>>> We now have 8 CPUs and 4096 MB of RAM reserved for ads on this slave. >>>> >>>> >>>> From: Renjie Liu <liurenjie2...@gmail.com> >>>> Date: Friday, November 27, 2015 at 9:57 PM >>>> To: "user@spark.apache.org" <user@spark.apache.org> >>>> Subject: Spark Streaming on mesos >>>> >>>> Hi, all: >>>> I'm trying to run spark streaming on mesos and it seems that none of >>>> the scheduler is suitable for that. Fine grain scheduler will start an >>>> executor for each task so it will significantly increase the latency. While >>>> coarse grained mode can only set the max core numbers and executor memory >>>> but there's no way to set the number of cores for each executor. Has anyone >>>> deployed spark streaming on mesos? And what's your settings? >>>> -- >>>> Liu, Renjie >>>> Software Engineer, MVAD >>>> >>> -- >>> Liu, Renjie >>> Software Engineer, MVAD >>> >>> -- >> Liu, Renjie >> Software Engineer, MVAD >> > > > > -- > > -- > Iulian Dragos > > ------ > Reactive Apps on the JVM > www.typesafe.com > > -- Liu, Renjie Software Engineer, MVAD
