Re: How to monitor YARN application memory per container?

Hello, MAPREDUCE-6829 was about showing the peak memory usage for mapreduce. Here are some of the new counters: [root@42e243b8cf16 hadoop]# bin/yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-jar pi 1 1000 Number of Maps = 1 Samples per Map = 1000 ... Peak Map Physical memory

Re: How to monitor YARN application memory per container?

hi, Please take a look at Timeline Server 2 which supports aggregate nodemenager side info into HBase. These infos include both node level info(e.g., node memory usage, cpu usage) as well as caontainer(e.g., container memory usage and container cpu usage ) level info. I am currently trying to set

Re: How to monitor YARN application memory per container?

Hi, Thanks for your response. We are using CDH, and our version doesn't support the solusions above. Also, ATS is not relevant for us now. We have decided to turn on JMX for all our jobs (spark/hadoop map-reduce) and use jmap to collect the data and send it to datadog. Shmuel On Thu, Jun 15,

Re: How to monitor YARN application memory per container?

Container resource usage has been put into ATS v2 metrics system. But if you do not want heavy ATS v2 subsystem, then i am not sure any of the current interface exposing the actual resource usage of the container which solves your problem. Probably i can think of extending this feature in *Containe

Re: How to monitor YARN application memory per container?

And adding to that, we have aggregated container usage per node. I dont think you ll have a per container real memory usage recorded from YARN. You ll have these 2 entries in ideal cases. Resource Utilization by Node : Resource Utilization by Containers : PMem:0 MB, VMem:0 MB, VCores:0.0 Thanks S

Re: How to monitor YARN application memory per container?

Hi Shmuel This feature is available in Hadoop 2.8 + release lines. Or Hadoop 3 alpha's. Thanks Sunil On Wed, Jun 14, 2017 at 6:31 AM Shmuel Blitz wrote: > Hi Sunil, > > Thanks for your response. > > Here is the response I get when running "yarn node -status {nodeId}" : > > Node Report : >

Re: How to monitor YARN application memory per container?

Hi Sunil, Thanks for your response. Here is the response I get when running "yarn node -status {nodeId}" : Node Report : Node-Id : myNode:4545 Rack : /default Node-State : RUNNING Node-Http-Address : muNode:8042 Last-Health-Update : Wed 14/Jun/17 08:25:43

Re: How to monitor YARN application memory per container?

Hi Shmuel In Hadoop 2.8 release line, you could check "yarn node -status {nodeId}" CLI command or "http:///ws/v1/cluster/nodes/{nodeid}" REST end point to get container's actual resource usage per node. You could also check the same in any of Hadoop 3.0 alpha releases as well. Thanks Sunil On Tu

Re: How to monitor YARN application memory per container?

Hi, Thanks for your response. The /metrics API returns a blank page on our RM. The /jmx API has some metrics, but these are the same metrics we are already loading into data-dog. It's not good enough, because it doesn't break down the memory use by container. I need the by-container breakdown b

Re: How to monitor YARN application memory per container?

Hi, I guess you can get it from http://:/jmx or /metrics Regards Sidharth LinkedIn: www.linkedin.com/in/sidharthkumar2792 On 13-Jun-2017 6:26 PM, "Shmuel Blitz" wrote: > (This question has also been published on StackOveflow > ) > > I am looking for

How to monitor YARN application memory per container?

(This question has also been published on StackOveflow ) I am looking for a way to monitor memory usage of YARN containers over time. Specifically - given a YARN application-id, how can you get a graph, showing the memory usage of each of its container