Thanks Chinmay & Ram. Troubleshooting page sounds the appropriate location. I shall raise PR with the given suggestions.
--prad On Tue, Jul 19, 2016 at 5:49 AM, Munagala Ramanath <r...@datatorrent.com> wrote: > There is already a link to a troubleshooting page at bottom of > https://apex.apache.org/docs.html > That page already has some discussion under the section entitled > "Calculating Container Memory" > so adding new content there seems like the right thing to do. > > Ram > > On Mon, Jul 18, 2016 at 11:27 PM, Chinmay Kolhatkar < > chin...@datatorrent.com > > wrote: > > > Hi Pradeep, > > > > This is a great content to add to the documents. These are the common set > > of errors which might get googled and hence great to get indexed as well. > > > > You can take a look at: > > https://github.com/apache/apex-core/tree/master/docs > > > > The docs for apex reside there in markdown format. Probably its good a > > create a troubleshooting page where all such common questions can reside. > > > > After you have the content ready, you can create a pull request to > > apex-core repo which can get merged to apex-core and later deployed to > the > > website by committers. > > > > -Chinmay. > > > > > > > > > > On Tue, Jul 19, 2016 at 10:46 AM, Pradeep A. Dalvi <p...@apache.org> > > wrote: > > > >> Container & memory resource allocation has been a common question around > >> and so I thought it would be good to explain related configuration > >> parameters. > >> > >> Please feel free to let me know your thoughts. > >> > >> Also I'm planning to add following set of information under Apex Docs. > How > >> could one add this to Apex Docs? > >> > >> =-=-=-= > >> > >> "Container is running beyond physical memory limits. Current usage: X GB > >> of > >> Y GB physical memory used; A GB of B GB virtual memory used. Killing > >> container." > >> > >> This is basically for some better understanding on Application Master's > >> container requests & Resource Manager's memory resource allocation. > Please > >> note that these are individual container request params. All these > >> parameters are in MB i.e. 1024 => 1GB. > >> > >> - AM's container requests to RM shall contain memory in the multiples of > >> *yarn.scheduler.minimum-**allocation-mb* & not exceeding > >> *yarn.scheduler.maximum-**allocation-mb* > >> - If *yarn.scheduler.minimum-**allocation-mb *is configured as 1024 > and > >> container memory requirement is 1025 ( <= 2048 ), container will be > >> allocated with 2048 memory. > >> > >> - With Apex applications, operator memory can be specified by property > >> *dt.application.<APP_NAME>.operator.<OPERATOR_NAME>.attr.MEMORY_MB* > >> - Please note this parameter is at Operator level and container > memory > >> is calculated based on number of Operators deployed in a container + > >> additional memory required depending on physical deployment requirements > >> e.g. unifier or bufferserver > >> - Wildcard * can be used at APP_NAME and/or OPERATOR_NAME > >> > >> - If container memory is not specified, then AM would request for 1 unit > >> of > >> *yarn.scheduler.minimum-**allocation-mb*, RM would provision container > >> taking that into consideration. > >> > >> Node Manager monitors memory usage of each of these containers and kills > >> the ones crossing the configured limit. > >> > >> Almost similar stuff is applicable for CPUs. > >> > >> --prad > >> > > > > >
