Re: Executor shutdown hook and initialization

Have a look at this ancient JIRA for a lot more discussion about this: https://issues.apache.org/jira/browse/SPARK-650 You have exactly the same issue described by another user. For your context, your approach is sound. You can set a shutdown hook using the normal Java Runtime API. You may not

Re: Executor shutdown hook and initialization

Hi Sean Could you please elaborate on how can this be done on a per partition basis? Regards Sumit Chawla On Thu, Oct 27, 2016 at 7:44 AM, Walter rakoff wrote: > Thanks for the info Sean. > > I'm initializing them in a singleton but Scala objects are evaluated >

Re: Executor shutdown hook and initialization

Thanks for the info Sean. I'm initializing them in a singleton but Scala objects are evaluated lazily. So it gets initialized only when the first task is run(and makes use of the object). Plan is to start a background thread in the object that does periodic cache refresh too. I'm trying to see if

Re: Executor shutdown hook and initialization

Init is easy -- initialize them in your singleton. Shutdown is harder; a shutdown hook is probably the only reliable way to go. Global state is not ideal in Spark. Consider initializing things like connections per partition, and open/close them with the lifecycle of a computation on a partition

Executor shutdown hook and initialization

Hello, Is there a way I can add an init() call when an executor is created? I'd like to initialize a few connections that are part of my singleton object. Preferably this happens before it runs the first task On the same line, how can I provide an shutdown hook that cleans up these connections on