Olivier Armand commented on SPARK-650:

Sean, a singleton is not the best option in our case. The Spark Streaming 
executors are writing to HBase, we need to initialize the HBase connection. The 
singleton seems (or seemed when we tested it for our customer a few months 
after this issue was raised) to be created when the first RDD is processed by 
the executor, and not when the driver starts. This imposes very high processing 
time for the first events.

> Add a "setup hook" API for running initialization code on each executor
> -----------------------------------------------------------------------
>                 Key: SPARK-650
>                 URL: https://issues.apache.org/jira/browse/SPARK-650
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>            Reporter: Matei Zaharia
>            Priority: Minor
> Would be useful to configure things like reporting libraries

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to