Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2798#issuecomment-59540816
Actually my proposal is more like the following.
`CoarseGrainedSchedulerBackend` will define what you propose as the
`Autoscaling trait`, and each of its subclasses will override it. I prefer to
keep the actual autoscaling messages outside of the
`CoarseGrainedSchedulerBackend$DriverActor` because some of them might be
specific to the cluster manager. For instance, for Yarn we'll need to save the
sender `ActorRef` as you have done here, whereas for standalone mode we don't
need to do this because we already have the akka URL of our `Master`.
For Yarn, we can just generalize the `MonitorActor` to become the
equivalent of your `AutoscalingServer`. Currently we only start this on the AM
for client mode, but I don't see a reason why we can't also start it for
cluster mode. Then, on the scheduler side, my plan is to put the equivalent of
your `AutoscalingClient` in a new parent trait `YarnSchedulerBackend` that both
of the existing Yarn scheduler backends implement. There are already messages
in `CoarseGrainedSchedulerBackend` that are pretty Yarn specific (e.g.
`addWebUIFilter`) and this is an opportunity to pull them out into this new
actor.
For standalone mode, the existing communication between the scheduler and
the Master goes through this thing called the `AppClient`, and
`SparkDeploySchedulerBackend` already has a reference to it. I prefer to reuse
this channel and add the new messages there rather than introducing a new
client actor as we are doing in the Yarn case.
Basically my thinking is to add a general communication channel between the
scheduler and the cluster manager, so in the future we can just add messages
specific to the cluster manager there instead of having to create another
actor. This channel already exists for a subset of the cluster managers (e.g.
standalone), so there we can just reuse it. Note that in my proposal I am not
introducing any new classes (only `Actor` subclasses). This feature is
ultimately intended to work with #2746, which already introduces a new class
for the broader feature of SPARK-3174, so I prefer to just implement this in
the existing interfaces.
Does that make sense?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
