yarn memory settings in heterogeneous cluster

2015-08-28 Thread Matt Kowalczyk
Hi, I have deployed a hadoop 2.7.1 cluster with heterogeneous nodes. For the sake of discussion, suppose one node has 100GB of RAM while another has 50 GB. I'm using the Capacity Scheduler and deploy mapred-site.xml and yarn-site.xml configuration files with various memory settings that are

Re: yarn memory settings in heterogeneous cluster

2015-08-28 Thread Vinod Kumar Vavilapalli
Hi Matt, Replies inline. I'm using the Capacity Scheduler and deploy mapred-site.xml and yarn-site.xml configuration files with various memory settings that are tailored to the resources for a particular machine. The master node, and the two slave node classes each get a different

Re: yarn memory settings in heterogeneous cluster

2015-08-28 Thread Matt Kowalczyk
HTH, Thanks for your prompt response. I'll take a look at the per-job memory settings which from your explaining should resolve my issue. -Matt On Fri, Aug 28, 2015 at 12:35 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Hi Matt, Replies inline. I'm using the Capacity