Well, what I have been trying to is send a "Cluster Pressure Event" to the Spark Applications running on a cluster to allow for dynamic re-sizing of Spark applications running on the cluster. It is a 4th year project that I am working on for my university program. We have the Spark side implemented where spark applications are set to have a fair share value for the number of executors along with the already implemented max and min values. This way, a spark app will grow until it has it's max and then throttle back to it's fair share if a Cluster Pressure Event is triggered. The motivation behind this, is to allow for interactive jobs to come onto the cluster - so that they don't have to wait for the other jobs to complete. Currently the only way for a spark application to "shrink down" is if the executors time out for inactivity. We have defined cluster pressure as (pending resources + allocated resources / total resources > 1) meaning that there has been more requests for resources than what is available - so we need to release some. I had attempted previously to do this inside the resource manager - but I could not figure out how to get events to the client from the RM. So now I am trying to do this client side, but I don't have access to this data (all I need is available memory, pending memory, and total memory available - or something similar). If you could provide some insight it would be greatly appreciated. I have been having trouble with this for months now. Thanks,Charles.
> From: [email protected] > To: [email protected] > CC: [email protected] > Subject: Re: Question about QueueMetrics/Cluster Metrics > Date: Sat, 12 Mar 2016 12:05:52 -0800 > > On 3/12/16 12:00 PM, Charlie Wright wrote: > > Is there a way to get the root QueueMetrics on the client side?-Charles. > > > > The QueueMetrics objects aren't exposed outside of the RM. What are you > trying to accomplish? If you just want to access the metrics data, you > can configure a RollingFileSystemSink to write the metrics into HDFS so > you can pick them up. > > Daniel
