LCE refers to Linux Container Executor Please take a look at yarn-default.xml
Cheers > On Jan 28, 2015, at 12:49 AM, 임정택 <[email protected]> wrote: > > Hi! > > At first, it was my mistake. :( All memory is "in use". > Also I found each Container's information says that "TotalMemoryNeeded 2048 / > TotalVCoreNeeded 1". > I don't understand why container needs 2048m (2G) of memory to run. > > Maybe I have to learn YARN schedulers and relevant configurations. > I'm newbie of YARN, and learning it by reading some docs. :) > > Btw, what's LCE and DRC? > > Thanks again for helping. > > Regards. > Jungtaek Lim (HeartSaVioR) > > > 2015-01-28 17:35 GMT+09:00 Naganarasimha G R (Naga) > <[email protected]>: >> Hi Jungtaek Lim, >> Earlier we faced similar problem of reservation with Capacity scheduler and >> its actually solved with YARN-1769 (part of 2.6 hadoop) >> So hope it might help you if you have configured Capacity scheduler. also >> check whether "yarn.scheduler.capacity.node-locality-delay" is configured >> (might not be direct help but might reduce probability of reservation ). >> I have one doubt with info : In the image it seems to be 20GB and 10 vcores >> reserved but you seem to say all are reserved ? >> Is LCE & DRC also configured ? if so what are the vcores configured for NM >> and the app's containers? >> >> Regards, >> Naga >> From: 임정택 [[email protected]] >> Sent: Wednesday, January 28, 2015 13:23 >> To: [email protected] >> Subject: Re: Question about YARN Memory allocation >> >> Forgot to add one thing, all memory (120G) is reserved now. >> >> Apps Submitted Apps Pending Apps Running Apps Completed >> Containers Running Memory Used Memory Total Memory Reserved >> VCores Used VCores Total VCores Reserved Active Nodes >> Decommissioned Nodes Lost Nodes Unhealthy Nodes Rebooted Nodes >> 2 1 1 0 60 120 GB 120 GB 20 GB 60 80 >> 10 10 0 0 0 0 >> Furthermore, 10 more VCores are reserved. I don't know what is it. >> >> >> 2015-01-28 16:47 GMT+09:00 임정택 <[email protected]>: >>> Hello all! >>> >>> I'm new to YARN, so it could be beginner question. >>> (I've been used MRv1 and changed just now.) >>> >>> I'm using HBase with 3 masters and 10 slaves - CDH 5.2 (Hadoop 2.5.0). >>> In order to migrate MRv1 to YARN, I read several docs, and change >>> configrations. >>> >>> ``` >>> yarn.nodemanager.resource.memory-mb: 12288 >>> yarn.scheduler.minimum-allocation-mb: 512 >>> mapreduce.map.memory.mb: 1536 >>> mapreduce.reduce.memory.mb: 1536 >>> mapreduce.map.java.opts: -Xmx1024m -Dfile.encoding=UTF-8 >>> -Dfile.client.encoding=UTF-8 -Dclient.encoding.override=UTF-8 >>> mapreduce.reduce.java.opts: -Xmx1024m -Dfile.encoding=UTF-8 >>> -Dfile.client.encoding=UTF-8 -Dclient.encoding.override=UTF-8 >>> ``` >>> >>> I'm expecting that it will be 80 containers running concurrently, but in >>> real it's 60 containers. (59 maps ran concurrently, maybe 1 is >>> ApplicationManager.) >>> >>> All YarnChilds' VIRT are higher than 1.5G and lower than 2G now, so I'm >>> suspecting it. >>> But it's better to make clear, to understand YARN clearer. >>> >>> Any helps & explanations are really appreciated. >>> Thanks! >>> >>> Best regards. >>> Jungtaek Lim (HeartSaVioR) >> >> >> >> -- >> Name : 임 정택 >> Blog : http://www.heartsavior.net / http://dev.heartsavior.net >> Twitter : http://twitter.com/heartsavior >> LinkedIn : http://www.linkedin.com/in/heartsavior > > > > -- > Name : 임 정택 > Blog : http://www.heartsavior.net / http://dev.heartsavior.net > Twitter : http://twitter.com/heartsavior > LinkedIn : http://www.linkedin.com/in/heartsavior
