I've verified 'multiples of minimum-allocation-mb' on latest Apex. However 'increment-allocation-mb' was not set during that exercise. I shall check that param as well.
Thanks, --prad On Wednesday, July 20, 2016, Munagala Ramanath <r...@datatorrent.com> wrote: > Please note that there are multiple sites making the claim that memory > allocation > is in multiples of *yarn.scheduler.minimum-allocation-mb*; this may have > been true > at one time but is no longer true (thanks to Sandesh for fact-checking > this). > > There is a (?new?) parameter, *yarn.scheduler.increment-allocation-mb*, > which serves > this purpose as discussed here: > > http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/ > > Ram > > On Tue, Jul 19, 2016 at 11:27 AM, Pradeep A. Dalvi <p...@apache.org > <javascript:;>> wrote: > > > Thanks Chinmay & Ram. > > > > Troubleshooting page sounds the appropriate location. I shall raise PR > with > > the given suggestions. > > > > --prad > > > > On Tue, Jul 19, 2016 at 5:49 AM, Munagala Ramanath <r...@datatorrent.com > <javascript:;>> > > wrote: > > > > > There is already a link to a troubleshooting page at bottom of > > > https://apex.apache.org/docs.html > > > That page already has some discussion under the section entitled > > > "Calculating Container Memory" > > > so adding new content there seems like the right thing to do. > > > > > > Ram > > > > > > On Mon, Jul 18, 2016 at 11:27 PM, Chinmay Kolhatkar < > > > chin...@datatorrent.com <javascript:;> > > > > wrote: > > > > > > > Hi Pradeep, > > > > > > > > This is a great content to add to the documents. These are the common > > set > > > > of errors which might get googled and hence great to get indexed as > > well. > > > > > > > > You can take a look at: > > > > https://github.com/apache/apex-core/tree/master/docs > > > > > > > > The docs for apex reside there in markdown format. Probably its good > a > > > > create a troubleshooting page where all such common questions can > > reside. > > > > > > > > After you have the content ready, you can create a pull request to > > > > apex-core repo which can get merged to apex-core and later deployed > to > > > the > > > > website by committers. > > > > > > > > -Chinmay. > > > > > > > > > > > > > > > > > > > > On Tue, Jul 19, 2016 at 10:46 AM, Pradeep A. Dalvi <p...@apache.org > <javascript:;>> > > > > wrote: > > > > > > > >> Container & memory resource allocation has been a common question > > around > > > >> and so I thought it would be good to explain related configuration > > > >> parameters. > > > >> > > > >> Please feel free to let me know your thoughts. > > > >> > > > >> Also I'm planning to add following set of information under Apex > Docs. > > > How > > > >> could one add this to Apex Docs? > > > >> > > > >> =-=-=-= > > > >> > > > >> "Container is running beyond physical memory limits. Current usage: > X > > GB > > > >> of > > > >> Y GB physical memory used; A GB of B GB virtual memory used. Killing > > > >> container." > > > >> > > > >> This is basically for some better understanding on Application > > Master's > > > >> container requests & Resource Manager's memory resource allocation. > > > Please > > > >> note that these are individual container request params. All these > > > >> parameters are in MB i.e. 1024 => 1GB. > > > >> > > > >> - AM's container requests to RM shall contain memory in the > multiples > > of > > > >> *yarn.scheduler.minimum-**allocation-mb* & not exceeding > > > >> *yarn.scheduler.maximum-**allocation-mb* > > > >> - If *yarn.scheduler.minimum-**allocation-mb *is configured as > 1024 > > > and > > > >> container memory requirement is 1025 ( <= 2048 ), container will be > > > >> allocated with 2048 memory. > > > >> > > > >> - With Apex applications, operator memory can be specified by > property > > > >> *dt.application.<APP_NAME>.operator.<OPERATOR_NAME>.attr.MEMORY_MB* > > > >> - Please note this parameter is at Operator level and container > > > memory > > > >> is calculated based on number of Operators deployed in a container + > > > >> additional memory required depending on physical deployment > > requirements > > > >> e.g. unifier or bufferserver > > > >> - Wildcard * can be used at APP_NAME and/or OPERATOR_NAME > > > >> > > > >> - If container memory is not specified, then AM would request for 1 > > unit > > > >> of > > > >> *yarn.scheduler.minimum-**allocation-mb*, RM would provision > container > > > >> taking that into consideration. > > > >> > > > >> Node Manager monitors memory usage of each of these containers and > > kills > > > >> the ones crossing the configured limit. > > > >> > > > >> Almost similar stuff is applicable for CPUs. > > > >> > > > >> --prad > > > >> > > > > > > > > > > > > > >
