as lots
>of "Action" supporting it. So parallelism is dynamic, from job to job, or
>even from stage to stage.
>
>
> Yong
>
> --
> Date: Wed, 27 May 2015 15:48:57 +0800
> Subject: Re: How does spark manage the memory of executor
te: Wed, 27 May 2015 15:48:57 +0800
Subject: Re: How does spark manage the memory of executor with multiple tasks
From: ccn...@gmail.com
To: evo.efti...@isecc.com
CC: ar...@sigmoidanalytics.com; user@spark.apache.org
Does anyone can answer my question ? I am curious to know if there's multi
>
>
> Original message
> From: Arush Kharbanda
> Date:2015/05/26 10:55 (GMT+00:00)
> To: canan chen
> Cc: Evo Eftimov ,user@spark.apache.org
> Subject: Re: How does spark manage the memory of executor with multiple
> tasks
>
> Hi Evo,
>
> Worker is the
message From: Arush Kharbanda
Date:2015/05/26 10:55 (GMT+00:00)
To: canan chen Cc: Evo Eftimov
,user@spark.apache.org Subject: Re: How does
spark manage the memory of executor with multiple tasks
Hi Evo,
Worker is the JVM and an executor runs on the JVM. And after Spark 1.4 you
tional RAM e.g. for
>> new Object instances as there is available in the Executor aka JVM Heap
>>
>>
>>
>> *From:* canan chen [mailto:ccn...@gmail.com]
>> *Sent:* Tuesday, May 26, 2015 9:30 AM
>> *To:* Evo Eftimov
>> *Cc:* user@spark.apache.org
>> *Su
ances as there is available in the Executor aka JVM Heap
>
>
>
> *From:* canan chen [mailto:ccn...@gmail.com]
> *Sent:* Tuesday, May 26, 2015 9:30 AM
> *To:* Evo Eftimov
> *Cc:* user@spark.apache.org
> *Subject:* Re: How does spark manage the memory of executor with multipl
9:30 AM
To: Evo Eftimov
Cc: user@spark.apache.org
Subject: Re: How does spark manage the memory of executor with multiple tasks
Yes, I know that one task represent a JVM thread. This is what I confused.
Usually users want to specify the memory on task level, so how can I do it if
task if
Yes, I know that one task represent a JVM thread. This is what I confused.
Usually users want to specify the memory on task level, so how can I do it
if task if thread level and multiple tasks runs in the same executor. And
even I don't know how many threads there will be. Besides that, if one task
An Executor is a JVM instance spawned and running on a Cluster Node (Server
machine). Task is essentially a JVM Thread – you can have as many Threads as
you want per JVM. You will also hear about “Executor Slots” – these are
essentially the CPU Cores available on the machine and granted for use
