Well, what I have been trying to is send a "Cluster Pressure Event" to the
Spark Applications running on a cluster to allow for dynamic re-sizing of Spark
applications running on the cluster. It is a 4th year project that I am working
on for my university program. We have the Spark side implemented where spark
applications are set to have a fair share value for the number of executors
along with the already implemented max and min values. This way, a spark app
will grow until it has it's max and then throttle back to it's fair share if a
Cluster Pressure Event is triggered. The motivation behind this, is to allow
for interactive jobs to come onto the cluster - so that they don't have to wait
for the other jobs to complete. Currently the only way for a application to
"shrink down" is if the executors time out for inactivity. We have defined
cluster pressure as (pending resources + allocated resources / total resources
> 1) meaning that there has been more requests for resources than what is
available - so we need to release some. Hopefully, that gives you some insight
as to what I am trying to do - I've been trying to figure out how to implement
this for sometime, but have been running into various implementation
roadblocks. If more information is required, I can provide it.
-Charles.
> Subject: Re: Question about sending YARN Scheduler Events to Client
> Applications
> From: [email protected]
> Date: Wed, 9 Mar 2016 14:07:03 -0800
> To: [email protected]
>
> The code path you referred to is completely YARN-internal only and so won’t
> serve any purpose of communication to clients or applications.
>
> What exactly are you trying to do? If you share more details, may be we can
> help you in figuring out how to do it.
>
> Thanks
> +Vinod
>
>
> > On Mar 9, 2016, at 12:09 PM, Charlie Wright <[email protected]> wrote:
> >
> >
> >
> >
> > I am trying to send an event to Spark/Client Applications via the YARN
> > Scheduler and I am not sure how to go about doing so. I have an event type
> > that I have created in SchedulerEventType and have created a corresponding
> > SchedulerEvent. The event is dispatched to the ResourceManager's dispatcher
> > and sent to SchedulerEventDispatcher where it dispatches the event to the
> > Scheduler. After that, I am not sure where to go from there. How can I get
> > events to the client? Is this done through
> > ApplicationReport->YarnApplicationState ?
> > Thanks,Charles.
> >
>
