Re: what's the best way to initialize an executor?

2014-10-23 Thread Sean Owen
It sounds like your code already does its initialization at most once per JVM, and that's about as good as it gets. Each partition asks for init in a thread-safe way and the first request succeeds. On Thu, Oct 23, 2014 at 1:41 PM, Darin McBeath wrote: > I have some code that I only need to be exe

what's the best way to initialize an executor?

2014-10-23 Thread Darin McBeath
I have some code that I only need to be executed once per executor in my spark application.  My current approach is to do something like the following: scala> xmlKeyPair.foreachPartition(i => XPathProcessor.init("ats", "Namespaces/NamespaceContext")) So, If I understand correctly, the XPathProces