Hi Charles, Ok but should it not be the reverse way : polling the other way round to get these stats from the REST (preferable) /RPC and do this calculation in the application or the manager side ? May be if there is any short fall in information from REST for an app then we need to expose them. Already app stats provides allocated Mem & vcores and also the apps asks. through this i think we can do to an extent some calculations
Also i was curios to know what exactly can application like spark do if the app's resource pressure is greater than a particular threshold ? Regards, + Naga ________________________________________ From: Charlie Wright [[email protected]] Sent: Thursday, January 14, 2016 08:55 To: [email protected] Subject: RE: Sending an event from YARN to applications running on the cluster Well im trying to send my own event, a ResourcePressureEvent. I have a thread in the ResourceManager that I have created that periodically checks for "resource pressure" - which is whenever the (pending resources + used resources / total resources >= threshold) - and whenever there is "resource pressure" I want to send an event to applications running on the cluster (specifically Spark applications). Charles. > From: [email protected] > To: [email protected] > Subject: RE: Sending an event from YARN to applications running on the cluster > Date: Thu, 14 Jan 2016 03:15:06 +0000 > > Hi Charles, > > What YARN event needs to be sent ? may be some info on it will be useful to > help you. > > Regards, > + Naga > ________________________________________ > From: Charlie Wright [[email protected]] > Sent: Thursday, January 14, 2016 08:01 > To: [email protected] > Subject: Sending an event from YARN to applications running on the cluster > > Hello, > > I was wondering how I could send an event from YARN to Spark applications > running on the cluster. From what I gather, Spark uses the AMRMClient class > to interface with YARN. How would I go about sending an event from the > ResourceManager to the AMRMClient? > > Any help is appreciated. > > Thanks, > Charles.
