Hi Tim,
Any way I can provide more info on this?
On Thu, Oct 1, 2015 at 4:21 PM, Utkarsh Sengar
wrote:
> Not sure what you mean by that, I shared the data which I see in spark UI.
> Can you point me to a location where I can precisely get the data you need?
>
> When I run the job in fine graine
Not sure what you mean by that, I shared the data which I see in spark UI.
Can you point me to a location where I can precisely get the data you need?
When I run the job in fine grained mode, I see tons are tasks created and
destroyed under a mesos "framework". I have about 80k spark tasks which I
Hi Utkarsh,
I replied earlier asking what is your task assignment like with fine vs
coarse grain mode look like?
Tim
On Thu, Oct 1, 2015 at 4:05 PM, Utkarsh Sengar
wrote:
> Bumping it up, its not really a blocking issue.
> But fine grain mode eats up uncertain number of resources in mesos and
Bumping it up, its not really a blocking issue.
But fine grain mode eats up uncertain number of resources in mesos and
launches tons of tasks, so I would prefer using the coarse grained mode if
only it didn't run out of memory.
Thanks,
-Utkarsh
On Mon, Sep 28, 2015 at 2:24 PM, Utkarsh Sengar
wro
Hi Tim,
1. spark.mesos.coarse:false (fine grain mode)
This is the data dump for config and executors assigned:
https://gist.github.com/utkarsh2012/6401d5526feccab14687
2. spark.mesos.coarse:true (coarse grain mode)
Dump for coarse mode:
https://gist.github.com/utkarsh2012/918cf6f8ed5945627188
As
Hi Utkarsh,
What is your job placement like when you run fine grain mode? You said
coarse grain mode only ran with one node right?
And when the job is running could you open the Spark webui and get stats
about the heap size and other java settings?
Tim
On Thu, Sep 24, 2015 at 10:56 PM, Utkarsh
Bumping this one up, any suggestions on the stacktrace?
spark.mesos.coarse=true is not working and the driver crashed with the
error.
On Wed, Sep 23, 2015 at 3:29 PM, Utkarsh Sengar
wrote:
> Missed to do a reply-all.
>
> Tim,
>
> spark.mesos.coarse = true doesn't work and spark.mesos.coarse = fa
Missed to do a reply-all.
Tim,
spark.mesos.coarse = true doesn't work and spark.mesos.coarse = false works
(sorry there was a typo in my last email, I meant "when I do
"spark.mesos.coarse=false", the job works like a charm. ").
I get this exception with spark.mesos.coarse = true:
15/09/22 20:18
Hi Utkarsh,
Just to be sure you originally set coarse to false but then to true? Or is
it the other way around?
Also what's the exception/stack trace when the driver crashed?
Coarse grain mode per-starts all the Spark executor backends, so has the
least overhead comparing to fine grain. There is
