[
https://issues.apache.org/jira/browse/SPARK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578894#comment-16578894
]
Imran Rashid commented on SPARK-24918:
--------------------------------------
With dynamic allocation you don't have a good place to run
{code}
df.mapPartitions { it =>
MyResource.initIfNeeded()
it.map(...)
}
{code}
Executors can come and go, you can't ensure that runs everywhere. Even if you
make "too many" tasks, it could be your job starts out with a very small number
of tasks for a while before ramping up. So after you run your initialization
with the added initResource code, many executors get torn down during the first
part of the real job as they sit idle; then when the job ramps up, you get new
executors, which never had your initialization run. You'd have to put
{{MyResource.initIfNeeded()}} inside *every* task. (Note that for the debug
use case, the initializer is totally unnecessary for the task to complete -- if
the task actually depended on it, then of course you should have that logic in
each task.)
I think there are a large class of users who can add "--conf
spark.executor.plugins com.mycompany.WhizzBangDebugPlugin --jars
whizzbangdebug.jar" to the command line arguments, that couldn't add in that
code sample (even with static allocation). They're not the ones that are
*writing* the plugins, they just need to be able to enable it.
{quote}What do you do if init fails? retry or fail?{quote}
good question, Tom asked the same thing on the pr. I suggested the executor
just fails to start. If a plugin wanted to be "safe", it could catch
exceptions in its own initialization.
> Executor Plugin API
> -------------------
>
> Key: SPARK-24918
> URL: https://issues.apache.org/jira/browse/SPARK-24918
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 2.4.0
> Reporter: Imran Rashid
> Priority: Major
> Labels: SPIP, memory-analysis
>
> It would be nice if we could specify an arbitrary class to run within each
> executor for debugging and instrumentation. Its hard to do this currently
> because:
> a) you have no idea when executors will come and go with DynamicAllocation,
> so don't have a chance to run custom code before the first task
> b) even with static allocation, you'd have to change the code of your spark
> app itself to run a special task to "install" the plugin, which is often
> tough in production cases when those maintaining regularly running
> applications might not even know how to make changes to the application.
> For example, https://github.com/squito/spark-memory could be used in a
> debugging context to understand memory use, just by re-running an application
> with extra command line arguments (as opposed to rebuilding spark).
> I think one tricky part here is just deciding the api, and how its versioned.
> Does it just get created when the executor starts, and thats it? Or does it
> get more specific events, like task start, task end, etc? Would we ever add
> more events? It should definitely be a {{DeveloperApi}}, so breaking
> compatibility would be allowed ... but still should be avoided. We could
> create a base class that has no-op implementations, or explicitly version
> everything.
> Note that this is not needed in the driver as we already have SparkListeners
> (even if you don't care about the SparkListenerEvents and just want to
> inspect objects in the JVM, its still good enough).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
