Would the REST API get you what you need?
https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
Nathan
On Saturday, March 12, 2016 2:11 PM, Charlie Wright <[email protected]>
wrote:
Well, what I have been trying to is send a "Cluster Pressure Event" to the
Spark Applications running on a cluster to allow for dynamic re-sizing of Spark
applications running on the cluster. It is a 4th year project that I am working
on for my university program. We have the Spark side implemented where spark
applications are set to have a fair share value for the number of executors
along with the already implemented max and min values. This way, a spark app
will grow until it has it's max and then throttle back to it's fair share if a
Cluster Pressure Event is triggered. The motivation behind this, is to allow
for interactive jobs to come onto the cluster - so that they don't have to wait
for the other jobs to complete. Currently the only way for a spark application
to "shrink down" is if the executors time out for inactivity. We have defined
cluster pressure as (pending resources + allocated resources / total resources
> 1) meaning that there has been more requests for resources than what is
available - so we need to release some.
I had attempted previously to do this inside the resource manager - but I could
not figure out how to get events to the client from the RM. So now I am trying
to do this client side, but I don't have access to this data (all I need is
available memory, pending memory, and total memory available - or something
similar).
If you could provide some insight it would be greatly appreciated. I have been
having trouble with this for months now.
Thanks,Charles.
> From: [email protected]
> To: [email protected]
> CC: [email protected]
> Subject: Re: Question about QueueMetrics/Cluster Metrics
> Date: Sat, 12 Mar 2016 12:05:52 -0800
>
> On 3/12/16 12:00 PM, Charlie Wright wrote:
> > Is there a way to get the root QueueMetrics on the client side?-Charles.
> >
>
> The QueueMetrics objects aren't exposed outside of the RM. What are you
> trying to accomplish? If you just want to access the metrics data, you
> can configure a RollingFileSystemSink to write the metrics into HDFS so
> you can pick them up.
>
> Daniel
