Hello,
I was wondering what the best way is to have some form of
singleton per worker...something that'll be instantiated once at the
start of a job for each worker, and shut down when all the work on that
node is completed. For instance, say I have a client library that
initiates a single session to a central server, and you're meant to
reuse that for all little bits of work, and finally close the session
when all the work items are done. Opening and closing sessions for each
data item would be wasteful (and in some cases, cause havoc for the
client).
I've been doing this with foreachPartition (i.e. have
the parameters for creating the singleton outside the loop, do a
foreachPartition, create the instance, loop over entries in the
partition, close the partition), but it's quite cludgy. Is there a
pattern by which I can have an instance of something nonserializable on
each worker?
Regards,
Ashic.