[
https://issues.apache.org/jira/browse/SPARK-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982030#comment-14982030
]
Jacek Lewandowski commented on SPARK-11402:
-------------------------------------------
That's right, this is not only introduction of pluggable executor and driver
runners. However, at least some of the changes are related to coding practises,
which i believe are good:
- {{WorkerPage}} does use neither {{ExecutorRunner}} nor {{DriverRunner}}, but
traits which expose only the methods to read information
- There is a functionality related to executor and driver runners which is
common to both of them. By introducing interfaces I made this explicit - from
scheduling point of view, either executor or driver are some services which
consume resources so as far as scheduling is concerned, they should be
generalised and handled in uniformly
- the number of arguments which were passed to runner constructors grew to
ridiculous numbers so the related settings were collected in dedicates case
classes - those which represent the worker configuration and those which
represent the process configuration
- fixed some inconsistency - the driver main directory was created inside the
driver runner while executor directory was created in the worker - this was
unified
Do you think the aforementioned changes are wrong? I don't believe they
increase the maintenance and support effort because the refactoring i proposed
is towards better code reuse, separation of concerns and encapsulation. The
ability to plug a custom runner implementation is just a small part of this
whole patch (and afaik using factories is also a good practice).
> Allow to define a custom driver runner and executor runner
> ----------------------------------------------------------
>
> Key: SPARK-11402
> URL: https://issues.apache.org/jira/browse/SPARK-11402
> Project: Spark
> Issue Type: Improvement
> Components: Deploy, Spark Core
> Reporter: Jacek Lewandowski
> Priority: Minor
>
> {{DriverRunner}} and {{ExecutorRunner}} are used by Spark Worker in
> standalone mode to spawn driver and executor processes respectively. When
> integrating Spark with some environments, it would be useful to allow
> providing a custom implementation of those components.
> The idea is simple - provide factory class names for driver and executor
> runners in Worker configuration. By default, the current implementations are
> used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
