Hi, I guess you can get it from http://<resourcemanager-host>:<rm-port>/jmx or /metrics
Regards Sidharth LinkedIn: www.linkedin.com/in/sidharthkumar2792 On 13-Jun-2017 6:26 PM, "Shmuel Blitz" <shmuel.bl...@similarweb.com> wrote: > (This question has also been published on StackOveflow > <https://stackoverflow.com/q/44484940/416300>) > > I am looking for a way to monitor memory usage of YARN containers over > time. > > Specifically - given a YARN application-id, how can you get a graph, > showing the memory usage of each of its containers over time? > > The main goal is to better fit memory allocation requirements for our YARN > applications (Spark / Map-Reduce), to avoid over allocation and cluster > resource waste. A side goal would be the ability to debug memory issues > when developing our jobs and attempting to pick reasonable resource > allocations. > > We've tried using the Data-Dog integration, But it doesn't break down the > metrics by container. > > Another approach was to parse the hadoop-yarn logs. These logs have > messages like: > > Memory usage of ProcessTree 57251 for container-id > container_e116_1495951495692_35134_01_000001: 1.9 GB of 11 GB physical > memory used; 14.4 GB of 23.1 GB virtual memory used > Parsing the logs correctly can yield data that can be used to plot a graph > of memory usage over time. > > That's exactly what we want, but there are two downsides: > > It involves reading human-readable log lines and parsing them into numeric > data. We'd love to avoid that. > If this data can be consumed otherwise, we're hoping it'll have more > information that we might be interest in in the future. We wouldn't want to > put the time into parsing the logs just to realize we need something else. > Is there any other way to extract these metrics, either by plugging in to > an existing producer or by writing a simple listener? > > Perhaps a whole other approach? > > -- > [image: Logo] > <https://www.similarweb.com/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> > Shmuel Blitz > *Big Data Developer* > www.similarweb.com > <http://www.similarweb.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> > > <https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> > Like > Us > <https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> > > <https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> > Follow > Us > <https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> > > <https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> > Watch > Us > <https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> > > <https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> > Read > Us > <https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> >
