[
https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109179#comment-16109179
]
Louis Bergelson commented on SPARK-650:
---------------------------------------
I can't understand how people are dismissing this as not an issue. There are
many cases where you need to initialize something on an executor, and many of
them need input from the driver. All of the given workarounds are terrible
hacks and at best force bad design, and at worst introduce confusing and
non-deterministic bugs. Any time that the recommended solution to a common
problem that many people are having is to abuse the Serializer in order to
trick it into executing non-serialization code it seems obvious that there's a
missing capability in the system.
The fact that executors can come on and offline at any time during the run
makes it especially essential that we have a robust way of initializing them.
I just really don't understand the opposition to adding an initialization hook,
it would solve so many problems in a clean way and doesn't seem like it would
be particularly problematic on its own.
> Add a "setup hook" API for running initialization code on each executor
> -----------------------------------------------------------------------
>
> Key: SPARK-650
> URL: https://issues.apache.org/jira/browse/SPARK-650
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Reporter: Matei Zaharia
> Priority: Minor
>
> Would be useful to configure things like reporting libraries
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
