Re: Please reply if you use Mesos fine grained mode

e.org<mailto:dev@spark.apache.org>" mailto:dev@spark.apache.org>> Subject: Re: Please reply if you use Mesos fine grained mode Hi Chris, How does coarse grain mode gives you less starvation in your overloaded cluster? Is it just because it allocates all resources at once (which I th

Re: Please reply if you use Mesos fine grained mode

Probably because only coarse-grained mode respects `spark.cores.max` right now. See (and maybe review ;-)) #9027 (sorry for the shameless plug). iulian On Wed, Nov 4, 2015 at 5:05 PM, Timothy Chen wrote: > Hi Chris, > > How does coarse grain mode give

Re: Please reply if you use Mesos fine grained mode

Hi Chris, How does coarse grain mode gives you less starvation in your overloaded cluster? Is it just because it allocates all resources at once (which I think in a overloaded cluster allows less things to run at once). Tim > On Nov 4, 2015, at 4:21 AM, Heller, Chris wrote: > > We’ve been m

Re: Please reply if you use Mesos fine grained mode

We’ve been making use of both. Fine-grain mode makes sense for more ad-hoc work loads, and coarse-grained for more job like loads on a common data set. My preference is the fine-grain mode in all cases, but the overhead associated with its startup and the possibility that an overloaded cluster w

Re: Please reply if you use Mesos fine grained mode

Fine grain mode does reuse the same JVM but perhaps different placement or different allocated cores comparing to the same total memory allocation. Tim Sent from my iPhone > On Nov 3, 2015, at 6:00 PM, Reynold Xin wrote: > > Soren, > > If I understand how Mesos works correctly, even the fine

Re: Please reply if you use Mesos fine grained mode

Hi, We are using Mesos fine grained mode because we can have multiple instances of spark to share machines and each application get resources dynamically allocated.  Thanks & Regards,  Meethu M On Wednesday, 4 November 2015 5:24 AM, Reynold Xin wrote: If you are using Spark with M

Re: Please reply if you use Mesos fine grained mode

Soren, If I understand how Mesos works correctly, even the fine grained mode keeps the JVMs around? On Tue, Nov 3, 2015 at 4:22 PM, Soren Macbeth wrote: > we use fine-grained mode. coarse-grained mode keeps JVMs around which > often leads to OOMs, which in turn kill the entire executor, causin

Re: Please reply if you use Mesos fine grained mode

We "used" Spark on Mesos to build interactive data analysis platform because the interactive session could be long and might not use Spark for the entire session. It is very wasteful of resources if we used the coarse-grained mode because it keeps resource for the entire session. Therefore, fine-gr

Re: Please reply if you use Mesos fine grained mode

we use fine-grained mode. coarse-grained mode keeps JVMs around which often leads to OOMs, which in turn kill the entire executor, causing entire stages to be retried. In fine-grained mode, only the task fails and subsequently gets retried without taking out an entire stage or worse. On Tue, Nov 3

Please reply if you use Mesos fine grained mode

If you are using Spark with Mesos fine grained mode, can you please respond to this email explaining why you use it over the coarse grained mode? Thanks.