Re: Spark Streaming on mesos

2015-11-30 Thread Renjie Liu
t no CPUs). Each Spark task will be > scheduled using Mesos CPU resources, but they don't suffer much latency. > > iulian > > > On Mon, Nov 30, 2015 at 4:17 AM, Renjie Liu <liurenjie2...@gmail.com> > wrote: > >> Hi, Tim: >> Fine grain mode is not suitable for stre

Re: Spark Streaming on mesos

2015-11-29 Thread Renjie Liu
er of cores per executor with spark executor cores in fine > grain mode. > > If you want coarse grain mode to support that it will > Be supported in the near term as he coarse grain scheduler is getting > revamped now. > > Tim > > On Nov 28, 2015, at 7:31 PM, Renjie L

Re: Spark Streaming on mesos

2015-11-28 Thread Renjie Liu
g. For example, > suppose we have 12 CPUs and 6144 MB of RAM available on a slave and that we > want to reserve 8 CPUs and 4096 MB of RAM for the ads role. We start the > slave like so: > > $ mesos-slave \ > --master=: \ > --resources="cpus:4;mem:2048;cpus(ads):8;mem(

Spark Streaming on mesos

2015-11-27 Thread Renjie Liu
Hi, all: I'm trying to run spark streaming on mesos and it seems that none of the scheduler is suitable for that. Fine grain scheduler will start an executor for each task so it will significantly increase the latency. While coarse grained mode can only set the max core numbers and executor memory