[
https://issues.apache.org/jira/browse/SPARK-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982575#comment-14982575
]
Sean Owen commented on SPARK-11402:
-----------------------------------
The refactoring itself is probably valuable, as are fixing any superfluous
inconsistencies. I am not an expert on this piece but some of the changes look
straightforward and helpful. Reducing complexity is typically a win.
Most of it is introducing an SPI for executor and driver runner classes, and I
think that may be too much to promise to end users. Adding complexity to
decouple the runners from implementation is also debatable, since I don't think
they're intended to be swapped out; that also is a de facto SPI even without
formal configs and so on.
I suggest a way forward is to identify the subset of this change that's only
refactoring, and not adding infrastructure for this SPI. That much may help you
create your own custom version anyway, that you override or modify in your own
version of Spark.
I'm also missing the motivation. Some of this functionality like setting up a
working dir is already possible.
> Allow to define a custom driver runner and executor runner
> ----------------------------------------------------------
>
> Key: SPARK-11402
> URL: https://issues.apache.org/jira/browse/SPARK-11402
> Project: Spark
> Issue Type: Improvement
> Components: Deploy, Spark Core
> Reporter: Jacek Lewandowski
> Priority: Minor
>
> {{DriverRunner}} and {{ExecutorRunner}} are used by Spark Worker in
> standalone mode to spawn driver and executor processes respectively. When
> integrating Spark with some environments, it would be useful to allow
> providing a custom implementation of those components.
> The idea is simple - provide factory class names for driver and executor
> runners in Worker configuration. By default, the current implementations are
> used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
