You would use YARN apis as mentioned my David. Look for “PendingMB” from “RM:8088/jmx” to see allocated/reserved/pending stats on a per queue basis. There is probably a WS that exposes similar data. At the app level, something like "http://RM:8088/ws/v1/cluster/apps/application_1457573549805_0001” will give you details (only for a running app ) on allocated MB, running containers and pending resourceRequests, clusterUsagePercentage, etc.
thanks — Hitesh On Mar 10, 2016, at 9:21 AM, Frank Luo <[email protected]> wrote: > Thanks David/Jeff. > > To avoid further confusions, let me make sure I am clear on what I am trying > to do: I would like to know how many hours in a day my cluster is running at > its full capacity, and when that happens, how long is my waiting queue. I > founded similar information on Ambari as below, but I’d like to dive deeper, > hence asking. > > From what I see, container per job information, especially pending > containers, is only available from an application’s trackingUrl, but that > just applies to M/R jobs. I am not able to get the same information from a > Tez applications’ trackingUrl (Tez’s url doesn’t do anything for hdp2.2). So > how does Ambari find the information out? > > > <image001.png> > > From: David Morel [mailto:[email protected]] > Sent: Thursday, March 10, 2016 1:03 AM > To: Jeff Zhang > Cc: [email protected]; Frank Luo > Subject: Re: how to use Yarn API to find task/attempt status > > The REST API should help. A working implementation (in perl, not java, sorry) > is visible here : http://search.cpan.org/dist/Net-Hadoop-YARN/ > Read the comments, they matter :-) > > Le 10 mars 2016 7:28 AM, "Jeff Zhang" <[email protected]> a écrit : > If it is for M/R, then maybe this is what you want > https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/mapreduce/JobStatus.html > > > > On Thu, Mar 10, 2016 at 1:58 PM, Frank Luo <[email protected]> wrote: > Let’s say there are 10 standard M/R jobs running. How to find how many tasks > are done/running/pending? > > From: Jeff Zhang [mailto:[email protected]] > Sent: Wednesday, March 09, 2016 9:33 PM > To: Frank Luo > Cc: [email protected] > Subject: Re: how to use Yarn API to find task/attempt status > > I don't think it is related with yarn. Yarn don't know about task/task > attempt, it only knows containers. So it should be your application to > provide such function. > > On Thu, Mar 10, 2016 at 11:29 AM, Frank Luo <[email protected]> wrote: > Anyone had a similar issue and knows the answer? > > From: Frank Luo > Sent: Wednesday, March 09, 2016 1:59 PM > To: '[email protected]' > Subject: how to use Yarn API to find task/attempt status > > I have a need to programmatically find out how many tasks are pending in > Yarn. Is there a way to do it through a Java API? > > I looked at YarnClient, but not able to find what I need. > > Thx in advance. > > Frank Luo > This email and any attachments transmitted with it are intended for use by > the intended recipient(s) only. If you have received this email in error, > please notify the sender immediately and then delete it. If you are not the > intended recipient, you must not keep, use, disclose, copy or distribute this > email without the author’s prior permission. We take precautions to minimize > the risk of transmitting software viruses, but we advise you to perform your > own virus checks on any attachment to this message. We cannot accept > liability for any loss or damage caused by software viruses. The information > contained in this communication may be confidential and may be subject to the > attorney-client privilege. > > > > > -- > Best Regards > > Jeff Zhang > This email and any attachments transmitted with it are intended for use by > the intended recipient(s) only. If you have received this email in error, > please notify the sender immediately and then delete it. If you are not the > intended recipient, you must not keep, use, disclose, copy or distribute this > email without the author’s prior permission. We take precautions to minimize > the risk of transmitting software viruses, but we advise you to perform your > own virus checks on any attachment to this message. We cannot accept > liability for any loss or damage caused by software viruses. The information > contained in this communication may be confidential and may be subject to the > attorney-client privilege. > > > > > -- > Best Regards > > Jeff Zhang > This email and any attachments transmitted with it are intended for use by > the intended recipient(s) only. If you have received this email in error, > please notify the sender immediately and then delete it. If you are not the > intended recipient, you must not keep, use, disclose, copy or distribute this > email without the author’s prior permission. We take precautions to minimize > the risk of transmitting software viruses, but we advise you to perform your > own virus checks on any attachment to this message. We cannot accept > liability for any loss or damage caused by software viruses. The information > contained in this communication may be confidential and may be subject to the > attorney-client privilege. > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
