Please note that there are multiple sites making the claim that memory allocation is in multiples of *yarn.scheduler.minimum-allocation-mb*; this may have been true at one time but is no longer true (thanks to Sandesh for fact-checking this).
There is a (?new?) parameter, *yarn.scheduler.increment-allocation-mb*, which serves this purpose as discussed here: http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/ Ram On Tue, Jul 19, 2016 at 11:27 AM, Pradeep A. Dalvi <[email protected]> wrote: > Thanks Chinmay & Ram. > > Troubleshooting page sounds the appropriate location. I shall raise PR with > the given suggestions. > > --prad > > On Tue, Jul 19, 2016 at 5:49 AM, Munagala Ramanath <[email protected]> > wrote: > > > There is already a link to a troubleshooting page at bottom of > > https://apex.apache.org/docs.html > > That page already has some discussion under the section entitled > > "Calculating Container Memory" > > so adding new content there seems like the right thing to do. > > > > Ram > > > > On Mon, Jul 18, 2016 at 11:27 PM, Chinmay Kolhatkar < > > [email protected] > > > wrote: > > > > > Hi Pradeep, > > > > > > This is a great content to add to the documents. These are the common > set > > > of errors which might get googled and hence great to get indexed as > well. > > > > > > You can take a look at: > > > https://github.com/apache/apex-core/tree/master/docs > > > > > > The docs for apex reside there in markdown format. Probably its good a > > > create a troubleshooting page where all such common questions can > reside. > > > > > > After you have the content ready, you can create a pull request to > > > apex-core repo which can get merged to apex-core and later deployed to > > the > > > website by committers. > > > > > > -Chinmay. > > > > > > > > > > > > > > > On Tue, Jul 19, 2016 at 10:46 AM, Pradeep A. Dalvi <[email protected]> > > > wrote: > > > > > >> Container & memory resource allocation has been a common question > around > > >> and so I thought it would be good to explain related configuration > > >> parameters. > > >> > > >> Please feel free to let me know your thoughts. > > >> > > >> Also I'm planning to add following set of information under Apex Docs. > > How > > >> could one add this to Apex Docs? > > >> > > >> =-=-=-= > > >> > > >> "Container is running beyond physical memory limits. Current usage: X > GB > > >> of > > >> Y GB physical memory used; A GB of B GB virtual memory used. Killing > > >> container." > > >> > > >> This is basically for some better understanding on Application > Master's > > >> container requests & Resource Manager's memory resource allocation. > > Please > > >> note that these are individual container request params. All these > > >> parameters are in MB i.e. 1024 => 1GB. > > >> > > >> - AM's container requests to RM shall contain memory in the multiples > of > > >> *yarn.scheduler.minimum-**allocation-mb* & not exceeding > > >> *yarn.scheduler.maximum-**allocation-mb* > > >> - If *yarn.scheduler.minimum-**allocation-mb *is configured as 1024 > > and > > >> container memory requirement is 1025 ( <= 2048 ), container will be > > >> allocated with 2048 memory. > > >> > > >> - With Apex applications, operator memory can be specified by property > > >> *dt.application.<APP_NAME>.operator.<OPERATOR_NAME>.attr.MEMORY_MB* > > >> - Please note this parameter is at Operator level and container > > memory > > >> is calculated based on number of Operators deployed in a container + > > >> additional memory required depending on physical deployment > requirements > > >> e.g. unifier or bufferserver > > >> - Wildcard * can be used at APP_NAME and/or OPERATOR_NAME > > >> > > >> - If container memory is not specified, then AM would request for 1 > unit > > >> of > > >> *yarn.scheduler.minimum-**allocation-mb*, RM would provision container > > >> taking that into consideration. > > >> > > >> Node Manager monitors memory usage of each of these containers and > kills > > >> the ones crossing the configured limit. > > >> > > >> Almost similar stuff is applicable for CPUs. > > >> > > >> --prad > > >> > > > > > > > > >
