Hi,
On Fri, Dec 5, 2014 at 3:56 AM, Akshat Aranya <[email protected]> wrote:
> Is it possible to have some state across multiple calls to mapPartitions
> on each partition, for instance, if I want to keep a database connection
> open?
>
If you're using Scala, you can use a singleton object, this will exist once
per JVM (i.e., once per executor), like
object DatabaseConnector {
lazy val conn = ...
}
Please be aware that shutting down the connection is much harder than
opening it, because you basically have no idea when processing is done for
an executor, AFAIK.
Tobias
