Re: Dynamic Allocation in Spark 1.2.0
Thanks Tsuyoshi and Shixiong for the info. Awesome with more documentation
about the feature!
Was afraid that the node manager needed reconfiguration (and restart). Any
idea of how much resources will the shuffle service take on the node
manager? In a multi-tenant Hadoop cluster environment, it would be
undesirable to have a Spark-specific long running service taking up
resources from other types of jobs on the cluster.
Thanks again,
Anders
On Sun, Dec 28, 2014 at 8:08 AM, Shixiong Zhu wrote:
> I encountered the following issue when enabling dynamicAllocation. You may
> want to take a look at it.
>
> https://issues.apache.org/jira/browse/SPARK-4951
>
> Best Regards,
> Shixiong Zhu
>
> 2014-12-28 2:07 GMT+08:00 Tsuyoshi OZAWA :
>
>> Hi Anders,
>>
>> I faced the same issue as you mentioned. Yes, you need to install
>> spark shuffle plugin for YARN. Please check following PRs which add
>> doc to enable dynamicAllocation:
>>
>> https://github.com/apache/spark/pull/3731
>> https://github.com/apache/spark/pull/3757
>>
>> I could run Spark on YARN with dynamicAllocation by following the
>> instructions described in the docs.
>>
>> Thanks,
>> - Tsuyoshi
>>
>> On Sat, Dec 27, 2014 at 11:06 PM, Anders Arpteg
>> wrote:
>> > Hey,
>> >
>> > Tried to get the new spark.dynamicAllocation.enabled feature working on
>> Yarn
>> > (Hadoop 2.2), but am unsuccessful so far. I've tested with the following
>> > settings:
>> >
>> > conf
>> > .set("spark.dynamicAllocation.enabled", "true")
>> > .set("spark.shuffle.service.enabled", "true")
>> > .set("spark.dynamicAllocation.minExecutors", "10")
>> > .set("spark.dynamicAllocation.maxExecutors", "700")
>> >
>> > The app works fine on Spark 1.2 if dynamicAllocation is not enabled, but
>> > with the settings above, it will start the app and the first job is
>> listed
>> > in the web ui. However, no tasks are started and it seems to be stuck
>> > waiting for a container to be allocated forever.
>> >
>> > Any help would be appreciated. Need to do something specific to get the
>> > external yarn shuffle service running in the node manager?
>> >
>> > TIA,
>> > Anders
>>
>>
>>
>> --
>> - Tsuyoshi
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>>
>
Re: Dynamic Allocation in Spark 1.2.0
I encountered the following issue when enabling dynamicAllocation. You may
want to take a look at it.
https://issues.apache.org/jira/browse/SPARK-4951
Best Regards,
Shixiong Zhu
2014-12-28 2:07 GMT+08:00 Tsuyoshi OZAWA :
> Hi Anders,
>
> I faced the same issue as you mentioned. Yes, you need to install
> spark shuffle plugin for YARN. Please check following PRs which add
> doc to enable dynamicAllocation:
>
> https://github.com/apache/spark/pull/3731
> https://github.com/apache/spark/pull/3757
>
> I could run Spark on YARN with dynamicAllocation by following the
> instructions described in the docs.
>
> Thanks,
> - Tsuyoshi
>
> On Sat, Dec 27, 2014 at 11:06 PM, Anders Arpteg
> wrote:
> > Hey,
> >
> > Tried to get the new spark.dynamicAllocation.enabled feature working on
> Yarn
> > (Hadoop 2.2), but am unsuccessful so far. I've tested with the following
> > settings:
> >
> > conf
> > .set("spark.dynamicAllocation.enabled", "true")
> > .set("spark.shuffle.service.enabled", "true")
> > .set("spark.dynamicAllocation.minExecutors", "10")
> > .set("spark.dynamicAllocation.maxExecutors", "700")
> >
> > The app works fine on Spark 1.2 if dynamicAllocation is not enabled, but
> > with the settings above, it will start the app and the first job is
> listed
> > in the web ui. However, no tasks are started and it seems to be stuck
> > waiting for a container to be allocated forever.
> >
> > Any help would be appreciated. Need to do something specific to get the
> > external yarn shuffle service running in the node manager?
> >
> > TIA,
> > Anders
>
>
>
> --
> - Tsuyoshi
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
Re: Dynamic Allocation in Spark 1.2.0
Hi Anders,
I faced the same issue as you mentioned. Yes, you need to install
spark shuffle plugin for YARN. Please check following PRs which add
doc to enable dynamicAllocation:
https://github.com/apache/spark/pull/3731
https://github.com/apache/spark/pull/3757
I could run Spark on YARN with dynamicAllocation by following the
instructions described in the docs.
Thanks,
- Tsuyoshi
On Sat, Dec 27, 2014 at 11:06 PM, Anders Arpteg wrote:
> Hey,
>
> Tried to get the new spark.dynamicAllocation.enabled feature working on Yarn
> (Hadoop 2.2), but am unsuccessful so far. I've tested with the following
> settings:
>
> conf
> .set("spark.dynamicAllocation.enabled", "true")
> .set("spark.shuffle.service.enabled", "true")
> .set("spark.dynamicAllocation.minExecutors", "10")
> .set("spark.dynamicAllocation.maxExecutors", "700")
>
> The app works fine on Spark 1.2 if dynamicAllocation is not enabled, but
> with the settings above, it will start the app and the first job is listed
> in the web ui. However, no tasks are started and it seems to be stuck
> waiting for a container to be allocated forever.
>
> Any help would be appreciated. Need to do something specific to get the
> external yarn shuffle service running in the node manager?
>
> TIA,
> Anders
--
- Tsuyoshi
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Dynamic Allocation in Spark 1.2.0
Hey,
Tried to get the new spark.dynamicAllocation.enabled feature working on
Yarn (Hadoop 2.2), but am unsuccessful so far. I've tested with the
following settings:
conf
.set("spark.dynamicAllocation.enabled", "true")
.set("spark.shuffle.service.enabled", "true")
.set("spark.dynamicAllocation.minExecutors", "10")
.set("spark.dynamicAllocation.maxExecutors", "700")
The app works fine on Spark 1.2 if dynamicAllocation is not enabled, but
with the settings above, it will start the app and the first job is listed
in the web ui. However, no tasks are started and it seems to be stuck
waiting for a container to be allocated forever.
Any help would be appreciated. Need to do something specific to get the
external yarn shuffle service running in the node manager?
TIA,
Anders
Dynamic Allocation in Spark 1.2.0
Hey,
Tried to get the new spark.dynamicAllocation.enabled feature working on
Yarn (Hadoop 2.2), but am unsuccessful so far. I've tested with the
following settings:
conf
.set("spark.dynamicAllocation.enabled", "true")
.set("spark.shuffle.service.enabled", "true")
.set("spark.dynamicAllocation.minExecutors", "10")
.set("spark.dynamicAllocation.maxExecutors", "700")
The app works fine on Spark 1.2 if dynamicAllocation is not enabled, but
with the settings above, it will start the app and the first job is listed
in the web ui. However, no tasks are started and it seems to be stuck
waiting for a container to be allocated forever.
Any help would be appreciated. Need to do something specific to get the
external yarn shuffle service running in the node manager?
TIA,
Anders
