Github user PraveenSeluka commented on the pull request:
https://github.com/apache/spark/pull/2798#issuecomment-59502779
Thanks @andrewor14 for taking time to test this out.
- Regarding `AutoscaleServer` is Yarn Specific => Good point. Here is a
proposal to fix this issue so that it works in Mesos mode also.
```
trait Autoscalable {
def addExecutors(count: Int)
def deleteExecutors(execIds: List[String])
}
```
After this, these will be the following changes
```
YarnAllocator extends Autoscalable with Logging
private[spark] class AutoscaleServer(allocator: Autoscalable, sparkConf:
SparkConf, actorSystem: ActorSystem, isClientRemote : Boolean = true)
```
Note that, its not yarn specific after this. If we want to use
AutoscaleSever for Mesos mode, then Mesos equivalent of YarnAllocator must
extend Autoscalable trait. And, Mesos mode can also use AutoscaleSever now
- On reusing `MonitorActor`, lets say we use `MonitorActor` for yarn-client
mode. Now, I am not clear on what you are proposing for yarn-cluster mode -
will it have a different Actor for yarn-cluster mode ? (if so, wont it be
better to keep it consistent)
Let me know if the above sounds good, I have a patch tested and will update
immediately.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
