Yes, YarnClient should get you access to current capacity information for the
queues.
Nathan
On Saturday, March 12, 2016 2:37 PM, Charlie Wright <[email protected]>
wrote:
This may sound dumb, but I am not sure how to go about using the REST API. Is
there a guide somewhere?
Also, I found that the YarnClientImpl has a getRootQueueInfos() method that
returns a List<QueueInfo> which you can call getQueueStatistics() on - which
returns QueueStatistics, which may give me what I need w/o the REST API (it
seemingly includes all the metrics that QueueMetrics does). Can you confirm
whether this works/is what I am looking for?
Thanks for the prompt response.
Charles.
> Date: Sat, 12 Mar 2016 20:22:40 +0000
> From: [email protected]
> To: [email protected]
> Subject: Re: Question about QueueMetrics/Cluster Metrics
>
> Would the REST API get you what you need?
> https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
> Nathan
>
>
> On Saturday, March 12, 2016 2:11 PM, Charlie Wright
><[email protected]> wrote:
>
>
> Well, what I have been trying to is send a "Cluster Pressure Event" to the
>Spark Applications running on a cluster to allow for dynamic re-sizing of
>Spark applications running on the cluster. It is a 4th year project that I am
>working on for my university program. We have the Spark side implemented where
>spark applications are set to have a fair share value for the number of
>executors along with the already implemented max and min values. This way, a
>spark app will grow until it has it's max and then throttle back to it's fair
>share if a Cluster Pressure Event is triggered. The motivation behind this, is
>to allow for interactive jobs to come onto the cluster - so that they don't
>have to wait for the other jobs to complete. Currently the only way for a
>spark application to "shrink down" is if the executors time out for
>inactivity. We have defined cluster pressure as (pending resources + allocated
>resources / total resources > 1) meaning that there has been more requests for
>resources than what is available - so we need to release some.
> I had attempted previously to do this inside the resource manager - but I
> could not figure out how to get events to the client from the RM. So now I am
> trying to do this client side, but I don't have access to this data (all I
> need is available memory, pending memory, and total memory available - or
> something similar).
> If you could provide some insight it would be greatly appreciated. I have
> been having trouble with this for months now.
> Thanks,Charles.
>
> > From: [email protected]
> > To: [email protected]
> > CC: [email protected]
> > Subject: Re: Question about QueueMetrics/Cluster Metrics
> > Date: Sat, 12 Mar 2016 12:05:52 -0800
> >
> > On 3/12/16 12:00 PM, Charlie Wright wrote:
> > > Is there a way to get the root QueueMetrics on the client side?-Charles.
> > >
> >
> > The QueueMetrics objects aren't exposed outside of the RM. What are you
> > trying to accomplish? If you just want to access the metrics data, you
> > can configure a RollingFileSystemSink to write the metrics into HDFS so
> > you can pick them up.
> >
> > Daniel
>
>
>
