Re: Can not set spark dynamic resource allocation

2016-05-27 Thread Cui, Weifeng
gt; > >Cheers! >David Newberger > >-Original Message- >From: Cui, Weifeng [mailto:weife...@a9.com] >Sent: Friday, May 20, 2016 4:26 PM >To: Marcelo Vanzin >Cc: Ted Yu; Rodrick Brown; user; Zhao, Jun; Aulakh, Sahib; Song, Yiwei >Subject: Re

RE: Can not set spark dynamic resource allocation

2016-05-20 Thread David Newberger
- From: Cui, Weifeng [mailto:weife...@a9.com] Sent: Friday, May 20, 2016 4:26 PM To: Marcelo Vanzin Cc: Ted Yu; Rodrick Brown; user; Zhao, Jun; Aulakh, Sahib; Song, Yiwei Subject: Re: Can not set spark dynamic resource allocation Sorry, here is the node-manager log. application_1463692924309_

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Cui, Weifeng
>> Weifeng >> >> >> >> >> >> From: "Aulakh, Sahib" <aula...@a9.com> >> Date: Friday, May 20, 2016 at 12:43 PM >> To: Ted Yu <yuzhih...@gmail.com> >> Cc: Rodrick Brown <rodr...@orchardplatform.com>, Cui Weifeng &

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Marcelo Vanzin
<weife...@a9.com>, user <user@spark.apache.org>, "Zhao, Jun" > <junz...@a9.com> > Subject: Re: Can not set spark dynamic resource allocation > > > > Yes it is yarn. We have configured spark shuffle service w yarn node manager > but something must

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Cui, Weifeng
t;weife...@a9.com>, user <user@spark.apache.org>, "Zhao, Jun" <junz...@a9.com> Subject: Re: Can not set spark dynamic resource allocation Yes it is yarn. We have configured spark shuffle service w yarn node manager but something must be off. We will send u app log on pa

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Aulakh, Sahib
Yes it is yarn. We have configured spark shuffle service w yarn node manager but something must be off. We will send u app log on paste bin. Sent from my iPhone On May 20, 2016, at 12:35 PM, Ted Yu > wrote: Since yarn-site.xml was cited, I

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Ted Yu
Since yarn-site.xml was cited, I assume the cluster runs YARN. On Fri, May 20, 2016 at 12:30 PM, Rodrick Brown wrote: > Is this Yarn or Mesos? For the later you need to start an external shuffle > service. > > Get Outlook for iOS > > > > >

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Rodrick Brown
Is this Yarn or Mesos? For the later you need to start an external shuffle service.  Get Outlook for iOS On Fri, May 20, 2016 at 11:48 AM -0700, "Cui, Weifeng" wrote: Hi guys,   Our team has a hadoop 2.6.0 cluster with Spark 1.6.1. We want to set

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Ted Yu
Can you retrieve the log for application_1463681113470_0006 and pastebin it ? Thanks On Fri, May 20, 2016 at 11:48 AM, Cui, Weifeng wrote: > Hi guys, > > > > Our team has a hadoop 2.6.0 cluster with Spark 1.6.1. We want to set > dynamic resource allocation for spark and we

Can not set spark dynamic resource allocation

2016-05-20 Thread Cui, Weifeng
Hi guys, Our team has a hadoop 2.6.0 cluster with Spark 1.6.1. We want to set dynamic resource allocation for spark and we followed the following link. After the changes, all spark jobs failed. https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation This test was