This may sound dumb, but I am not sure how to go about using the REST API. Is there a guide somewhere? Also, I found that the YarnClientImpl has a getRootQueueInfos() method that returns a List<QueueInfo> which you can call getQueueStatistics() on - which returns QueueStatistics, which may give me what I need w/o the REST API (it seemingly includes all the metrics that QueueMetrics does). Can you confirm whether this works/is what I am looking for? Thanks for the prompt response. Charles.
> Date: Sat, 12 Mar 2016 20:22:40 +0000 > From: [email protected] > To: [email protected] > Subject: Re: Question about QueueMetrics/Cluster Metrics > > Would the REST API get you what you need? > https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html > Nathan > > > On Saturday, March 12, 2016 2:11 PM, Charlie Wright > <[email protected]> wrote: > > > Well, what I have been trying to is send a "Cluster Pressure Event" to the > Spark Applications running on a cluster to allow for dynamic re-sizing of > Spark applications running on the cluster. It is a 4th year project that I am > working on for my university program. We have the Spark side implemented > where spark applications are set to have a fair share value for the number of > executors along with the already implemented max and min values. This way, a > spark app will grow until it has it's max and then throttle back to it's fair > share if a Cluster Pressure Event is triggered. The motivation behind this, > is to allow for interactive jobs to come onto the cluster - so that they > don't have to wait for the other jobs to complete. Currently the only way for > a spark application to "shrink down" is if the executors time out for > inactivity. We have defined cluster pressure as (pending resources + > allocated resources / total resources > 1) meaning that there has been more > requests for resources than what is available - so we need to release some. > I had attempted previously to do this inside the resource manager - but I > could not figure out how to get events to the client from the RM. So now I am > trying to do this client side, but I don't have access to this data (all I > need is available memory, pending memory, and total memory available - or > something similar). > If you could provide some insight it would be greatly appreciated. I have > been having trouble with this for months now. > Thanks,Charles. > > > From: [email protected] > > To: [email protected] > > CC: [email protected] > > Subject: Re: Question about QueueMetrics/Cluster Metrics > > Date: Sat, 12 Mar 2016 12:05:52 -0800 > > > > On 3/12/16 12:00 PM, Charlie Wright wrote: > > > Is there a way to get the root QueueMetrics on the client side?-Charles. > > > > > > > The QueueMetrics objects aren't exposed outside of the RM. What are you > > trying to accomplish? If you just want to access the metrics data, you > > can configure a RollingFileSystemSink to write the metrics into HDFS so > > you can pick them up. > > > > Daniel > > >
