Re: Dynamic allocation doesn't work on YARN

2016-03-10 Thread Jy Chen 
Thank you for your help and advice. After I added the log4j conf to expose
more details, I found that Spark had sent the removing request while some
containers did not receive the  SIGTERM signal.

Thanks.

2016-03-10 10:52 GMT+08:00 Saisai Shao <sai.sai.s...@gmail.com>:

> Still I think this information is not enough to explain the reason.
>
> 1. Does your yarn cluster has enough resources to start all 10 executors?
> 2. Would you please try latest version, 1.6.0 or master branch to see if
> this is a bug and already fixed.
> 3. you could add
> "log4j.logger.org.apache.spark.ExecutorAllocationManager=DEBUG" to log4j
> conf to expose more details, then maybe you could dig out some clues.
>
>
> Thanks
> Saisai
>
> On Thu, Mar 10, 2016 at 10:18 AM, Jy Chen <chen.wah...@gmail.com> wrote:
>
>> Sorry,the last configuration is also --conf
>> spark.dynamicAllocation.cachedExecutorIdleTimeout=60s, "--conf" was lost
>> when I copied it to mail.
>>
>> -- Forwarded message --
>> From: Jy Chen <chen.wah...@gmail.com>
>> Date: 2016-03-10 10:09 GMT+08:00
>> Subject: Re: Dynamic allocation doesn't work on YARN
>> To: Saisai Shao <sai.sai.s...@gmail.com>, user@spark.apache.org
>>
>>
>> Hi,
>> My Spark version is 1.5.1 with Hadoop 2.5.0-cdh5.2.0. These are my
>> configurations of dynamic allocation:
>> --master yarn-client --conf spark.dynamicAllocation.enabled=true --conf
>> spark.shuffle.service.enabled=true --conf
>> spark.dynamicAllocation.minExecutors=0 --conf
>> spark.dynamicAllocation.initialExecutors=10
>>  --conf spark.dynamicAllocation.executorIdleTimeout=60s
>> spark.dynamicAllocation.cachedExecutorIdleTimeout=60s
>>
>> At first,it will remove 2 executors and then no more executors will be
>> removed.
>>
>> Thanks
>>
>> 2016-03-09 17:24 GMT+08:00 Saisai Shao <sai.sai.s...@gmail.com>:
>>
>>> Would you please send out the configurations of dynamic allocation so we
>>> could know better.
>>>
>>> On Wed, Mar 9, 2016 at 4:29 PM, Jy Chen <chen.wah...@gmail.com> wrote:
>>>
>>>> Hello everyone:
>>>>
>>>> I'm trying the dynamic allocation in Spark on YARN. I have followed
>>>> configuration steps and started the shuffle service.
>>>>
>>>> Now it can request executors when the workload is heavy but it cannot
>>>> remove executors. I try to open the spark shell and don’t run any command,
>>>> no executor is removed after spark.dynamicAllocation.executorIdleTimeout
>>>> interval.
>>>> Am I missing something?
>>>>
>>>> Thanks.
>>>>
>>>>
>>>
>>
>>
>

Re: Dynamic allocation doesn't work on YARN

2016-03-09 Thread Saisai Shao 
Still I think this information is not enough to explain the reason.

1. Does your yarn cluster has enough resources to start all 10 executors?
2. Would you please try latest version, 1.6.0 or master branch to see if
this is a bug and already fixed.
3. you could add
"log4j.logger.org.apache.spark.ExecutorAllocationManager=DEBUG" to log4j
conf to expose more details, then maybe you could dig out some clues.


Thanks
Saisai

On Thu, Mar 10, 2016 at 10:18 AM, Jy Chen <chen.wah...@gmail.com> wrote:

> Sorry,the last configuration is also --conf
> spark.dynamicAllocation.cachedExecutorIdleTimeout=60s, "--conf" was lost
> when I copied it to mail.
>
> -- Forwarded message --
> From: Jy Chen <chen.wah...@gmail.com>
> Date: 2016-03-10 10:09 GMT+08:00
> Subject: Re: Dynamic allocation doesn't work on YARN
> To: Saisai Shao <sai.sai.s...@gmail.com>, user@spark.apache.org
>
>
> Hi,
> My Spark version is 1.5.1 with Hadoop 2.5.0-cdh5.2.0. These are my
> configurations of dynamic allocation:
> --master yarn-client --conf spark.dynamicAllocation.enabled=true --conf
> spark.shuffle.service.enabled=true --conf
> spark.dynamicAllocation.minExecutors=0 --conf
> spark.dynamicAllocation.initialExecutors=10
>  --conf spark.dynamicAllocation.executorIdleTimeout=60s
> spark.dynamicAllocation.cachedExecutorIdleTimeout=60s
>
> At first,it will remove 2 executors and then no more executors will be
> removed.
>
> Thanks
>
> 2016-03-09 17:24 GMT+08:00 Saisai Shao <sai.sai.s...@gmail.com>:
>
>> Would you please send out the configurations of dynamic allocation so we
>> could know better.
>>
>> On Wed, Mar 9, 2016 at 4:29 PM, Jy Chen <chen.wah...@gmail.com> wrote:
>>
>>> Hello everyone:
>>>
>>> I'm trying the dynamic allocation in Spark on YARN. I have followed
>>> configuration steps and started the shuffle service.
>>>
>>> Now it can request executors when the workload is heavy but it cannot
>>> remove executors. I try to open the spark shell and don’t run any command,
>>> no executor is removed after spark.dynamicAllocation.executorIdleTimeout
>>> interval.
>>> Am I missing something?
>>>
>>> Thanks.
>>>
>>>
>>
>
>

Fwd: Dynamic allocation doesn't work on YARN

2016-03-09 Thread Jy Chen 
Sorry,the last configuration is also --conf
spark.dynamicAllocation.cachedExecutorIdleTimeout=60s, "--conf" was lost
when I copied it to mail.

-- Forwarded message --
From: Jy Chen <chen.wah...@gmail.com>
Date: 2016-03-10 10:09 GMT+08:00
Subject: Re: Dynamic allocation doesn't work on YARN
To: Saisai Shao <sai.sai.s...@gmail.com>, user@spark.apache.org


Hi,
My Spark version is 1.5.1 with Hadoop 2.5.0-cdh5.2.0. These are my
configurations of dynamic allocation:
--master yarn-client --conf spark.dynamicAllocation.enabled=true --conf
spark.shuffle.service.enabled=true --conf
spark.dynamicAllocation.minExecutors=0 --conf
spark.dynamicAllocation.initialExecutors=10
 --conf spark.dynamicAllocation.executorIdleTimeout=60s
spark.dynamicAllocation.cachedExecutorIdleTimeout=60s

At first,it will remove 2 executors and then no more executors will be
removed.

Thanks

2016-03-09 17:24 GMT+08:00 Saisai Shao <sai.sai.s...@gmail.com>:

> Would you please send out the configurations of dynamic allocation so we
> could know better.
>
> On Wed, Mar 9, 2016 at 4:29 PM, Jy Chen <chen.wah...@gmail.com> wrote:
>
>> Hello everyone:
>>
>> I'm trying the dynamic allocation in Spark on YARN. I have followed
>> configuration steps and started the shuffle service.
>>
>> Now it can request executors when the workload is heavy but it cannot
>> remove executors. I try to open the spark shell and don’t run any command,
>> no executor is removed after spark.dynamicAllocation.executorIdleTimeout
>> interval.
>> Am I missing something?
>>
>> Thanks.
>>
>>
>

Re: Dynamic allocation doesn't work on YARN

2016-03-09 Thread Jy Chen 
Hi,
My Spark version is 1.5.1 with Hadoop 2.5.0-cdh5.2.0. These are my
configurations of dynamic allocation:
--master yarn-client --conf spark.dynamicAllocation.enabled=true --conf
spark.shuffle.service.enabled=true --conf
spark.dynamicAllocation.minExecutors=0 --conf
spark.dynamicAllocation.initialExecutors=10
 --conf spark.dynamicAllocation.executorIdleTimeout=60s
spark.dynamicAllocation.cachedExecutorIdleTimeout=60s

At first,it will remove 2 executors and then no more executors will be
removed.

Thanks

2016-03-09 17:24 GMT+08:00 Saisai Shao :

> Would you please send out the configurations of dynamic allocation so we
> could know better.
>
> On Wed, Mar 9, 2016 at 4:29 PM, Jy Chen  wrote:
>
>> Hello everyone:
>>
>> I'm trying the dynamic allocation in Spark on YARN. I have followed
>> configuration steps and started the shuffle service.
>>
>> Now it can request executors when the workload is heavy but it cannot
>> remove executors. I try to open the spark shell and don’t run any command,
>> no executor is removed after spark.dynamicAllocation.executorIdleTimeout
>> interval.
>> Am I missing something?
>>
>> Thanks.
>>
>>
>

Re: Dynamic allocation doesn't work on YARN

2016-03-09 Thread Saisai Shao 
Would you please send out the configurations of dynamic allocation so we
could know better.

On Wed, Mar 9, 2016 at 4:29 PM, Jy Chen  wrote:

> Hello everyone:
>
> I'm trying the dynamic allocation in Spark on YARN. I have followed
> configuration steps and started the shuffle service.
>
> Now it can request executors when the workload is heavy but it cannot
> remove executors. I try to open the spark shell and don’t run any command,
> no executor is removed after spark.dynamicAllocation.executorIdleTimeout
> interval.
> Am I missing something?
>
> Thanks.
>
>

Fwd: Dynamic allocation doesn't work on YARN

2016-03-09 Thread Jy Chen 
Hello everyone:

I'm trying the dynamic allocation in Spark on YARN. I have followed
configuration steps and started the shuffle service.

Now it can request executors when the workload is heavy but it cannot
remove executors. I try to open the spark shell and don’t run any command,
no executor is removed after spark.dynamicAllocation.executorIdleTimeout
interval.
Am I missing something?

Thanks.