Hi @rxin, Just to make sure I understand your desired use case, are you suggesting a way (for the driver) to call, at any given time, a general method that can be defined ahead of time on the executors?
On Thu, Aug 30, 2018 at 11:11 PM, Reynold Xin <r...@databricks.com> wrote: > I actually had a similar use case a while ago, but not entirely the same. > In my use case, Spark is already up, but I want to make sure all existing > (and new) executors run some specific code. Can we update the API to > support that? I think that's doable if we split the design into two: one is > the ability to do what I just mentioned, and second is the ability to > register via config class when Spark starts to run the code. > > > On Thu, Aug 30, 2018 at 11:01 PM Felix Cheung <felixcheun...@hotmail.com> > wrote: > >> +1 >> ------------------------------ >> *From:* Mridul Muralidharan <mri...@gmail.com> >> *Sent:* Wednesday, August 29, 2018 1:27:27 PM >> *To:* dev@spark.apache.org >> *Subject:* Re: SPIP: Executor Plugin (SPARK-24918) >> >> +1 >> I left a couple of comments in NiharS's PR, but this is very useful to >> have in spark ! >> >> Regards, >> Mridul >> On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid >> <iras...@cloudera.com.invalid> wrote: >> > >> > I'd like to propose adding a plugin api for Executors, primarily for >> instrumentation and debugging (https://issues.apache.org/ >> jira/browse/SPARK-24918). The changes are small, but as its adding a >> new api, it might be spip-worthy. I mentioned it as well in a recent email >> I sent about memory monitoring >> > >> > The spip proposal is here (and attached to the jira as well): >> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5c >> bHBQtyqIA2hgtc/edit?usp=sharing >> > >> > There are already some comments on the jira and pr, and I hope to get >> more thoughts and opinions on it. >> > >> > thanks, >> > Imran >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
