[
https://issues.apache.org/jira/browse/STORM-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16339495#comment-16339495
]
Robert Joseph Evans commented on STORM-1740:
--------------------------------------------
The following is an initial design for this new feature. I expect to break up
this functionality into smaller pieces and will depend on other pieces of
functionality like STORM-2693 and STORM-2910 to make this ultimately work.
In the past we have run into a number of issues with breaking Storm on Mesos
due to internal changes that we though were not going to break anything, but
did.
The issue as I see it is primarily that the INimbus API is not very clear
about how it should be used, and it does not tie in well with some of the
changes we have made to scheduling, namely the move away from pure slot based
scheduling.
As such I propose that we just drop INimbus entirely, and instead add in a few
new scheduling APIs, and a new Supervisor Launcher/Initialization API.
Requirements:
* We want to be able to have a simple API that allows storm to request
resources from some other place (this could be YARN, or Mesos, or whatever) on
behalf of a topology that does not have the resources it needs.
* We want to be able to to use existing storm schedulers without modification,
but a scheduler may also have the option to improve the requests beyond what a
default implementation would do.
* We want to be able to support security in all of these environments
(although it may not be something that we can fully support initially)
* We want any new APIs to be maintainable, have tests for it and have a very
clearly defined contract with storm so we don't keep breaking our customers.
* We want to be able to give both cluster owners and topology owners some
control over how and when this feature is used.
*First off lets look at scheduling:*
Storm wants to have its own scheduler because it can have a deep view into how
a topology is laid out and, hopefully soon, what it is doing. With this
information we can do a better job of scheduling compared to a generic cluster
level scheduler. It also gives us a lot more control to implement things like
elasticity on our own. But at the same time we need to be able to get
resources from some other place.
So obviously we need a 2 level scheduler to be able to support something like
this. YARN and Mesos are built around this concept and systems like Open Stack,
Kubernetes, and even proprietary cloud service providers can also be used to
get similar functionality.
As such I would like to propose an API for a request that has a very minimal
set of requirements, and at the same time allows for optional hints that can be
used if the higher level scheduler can support them.
{code:java}
//This is just a psudo code example
public enum Priority {
RECOVERY, // Workers have crashed and we really need to get this back up and
running.
NORMAL // It is okay to spend some time on this to get things co-located if
possible.
//Possibly in the future an OPTIMIZATION for cases where the topology is
running, but we have recognized scheduling could be better if we move something
to a better location, or give it more resources.
}
public class SecondLevelTopologyRequest {
public Map<String, Object> getTopoConf();
public String getOwner();
public String getTopologyId();
public Priority getPriority();
public List<SecondLevelSlotRequest> getSlotRequests();
//IP address that we would like to have resources close to.
public String localityRequest();
}
public class SecondLevelSlotRequest {
public Resources getResources();
}
{code}
We then would expose an API through Cluster so a scheduler if it does not have
enough resources to schedule something it can request those needed resources,
or update an existing request. This would likely happen after eviction has
happened, but would be up to the scheduler.
After scheduling was finished we would have a generic piece of code that could
update/create SecondLevelTopologyRequests if the scheduler did not do so on
it's own. This would not be a super simple piece of code because it would have
to take into account both topologies that are resource aware and ones that just
ask for slots. To make this work so the SecondLevelScheduler knows what to work
with we will likely need to use some new settings and exploration of the
topology's command like to guess at least for memory and CPU settings.
After this happens an ISecondLevelScheduler instance would be handed the
requests and it could use them to try and negotiate for resources, and launch
supervisors as needed when those resources are available. This scheduler would
also have the option to tag requests that it cannot or may not be able to meet.
This may be due to config settings (i.e. this particular user is not configured
to request resources on this other cluster), or due to resource constraints on
the other cluster (i.e. the YARN queue for this user is totally full, and there
is no churn on it). In those cases other ISecondLevelScheduler instances may be
given a chance to fulfill those requests.
Ideally an ISecondLevelScheduler should do gang scheduling on a per topology
basis, get all of the resources requested or none of them. There are some
systems where that is not possible. As such it is not a hard requirement, but a
suggestion.
*Security:*
In some cases we may be done at this point. If we are on openstack or AWS we
may have been able to launch a VM that is exactly like our other supervisors so
that they each have access to the needed kerberos credentials and have the
ability to run workers as the user that launched the topology. But for Mesos
and YARN this is not possible.
Their security model is similar to Storm's and I would consider it a security
bug if someone could get sudo like privileges as a regular user on one of those
clusters. So to make this really work we need the concept of a user and or
topology specific supervisor.
This is one of the parts that is going to take some more work.
The issue is that the supervisor currently needs to communicate with Nimbus and
with zookeeper to get what it needs.
The supervisor talks to nimbus to download the dependencies for a topology, and
talks to ZK to heartbeat that it is still alive, and to download the current
scheduling so it knows what it needs to launch.
STORM-2693 makes it so downloading new scheduling from ZK is no longer
required, and we could also make it so heartbeats could go directly to nimbus
instead of going to ZK, similar to what STORM-2693 does for worker heartbeats
to nimbus.
So then we would need a way to authorize a supervisor to communicate with
nimbus. We could create a token similar to a worker token that would allow this
to happen. The lifetime and rotation of this token would be controlled by the
ISecondLevelScheduler that is launching the supervisors as it is application
specific how those tokens could be distributed to the running supervisors in a
secure way.
We would need some kind of plugin in the supervisors (possibly a jaas login
plugin) that would take this token and place it in the current subjects private
credentials so SASL negotiation could work. As it is common for a supervisor to
also need many changes to it's config to work properly on this kind of a new
environment, we may want to make this a formal ISupervisorLauncher or
ISupervisorEnvironment plugin instead. At a minimum we would also need a way
for the supervisor to know what constraints it has (user specific or topology
specific) so it can tell nimbus and enforce those constraints itself. But we
need to get more experience with exactly what/how the different implementations
will want to make this happen. So I expect to get lots of feedback on it.
After STORM-2693 we would also need a way for the supervisor to validate
incoming worker heartbeats. We could just ignore it and have the workers skip
the supervisor and communicate with nimbus directly and to the supervisor only
through the file system like it does now. It would be preferable, though, to
still have the worker to supervisor thrift path work, and we may need to think
about this as a part of STORM-2693. To make this work we could have a separate
supervisor token/secret from the nimbus token/secret. Then the supervisor
secret key for a topology could be distributed to the supervisors through
heartbeat responses and with the updated supervisor assignments. That way the
supervisor would only know about the secrets that are specific to topologies it
is responsible for. One of the big things we would have to be careful of would
be making sure that the keys were distributed before sending the secret to the
workers, or provide an out of band way for the supervisor to fetch the keys.
We would have to have a way for the scheduler to enforce/honor supervisor
constraints:
A supervisor that is specific to either a topology or a user could not have
anything else scheduled on it. The simplest way to work around this is to call
the scheduler multiple times. We would call it once for topologies that do not
have supervisors that are specific to them and with a "cluster" that did not
have any constrained supervisors. We would then call it once for each topology
that does have constraints and a special cluster that includes the resources it
is allowed to use. This is not perfect as the scheduler assumes it sees the
entire cluster and may get really confused about resource guarantees if it
cannot see the entire cluster. But we can update those schedulers so they can
handle all of it at once and work with the constraints, and understand
resources that are enforced by storm, and resources that are enforced by other
schedulers.
*Rolling Upgrades:*
This is something that would likely come later on too, but we would likely need
a good way to ask an ISecondLevelScheduler to do a rolling upgrade.
*UI/Polish:*
Once we have all of that working we need a good way to let users know what in
the world is happening to their topology, and to the cluster. This may be
something that happens later on as well.
Currently we display a lot of resources that are aggregates for the entire
cluster. We likely will want to offer a view that also breaks them up by where
the resources came from.
||Cluster/Application||CPU||MEMORY||
|Total|50,000|200,000|
|Native|40,000|160,000|
|YARN|5,000|20,000|
|YARN: Cluster A|4,000|16,000|
|YARN: Cluster A:Application-29 (topology-2345)|2,000|8,000|
|YARN: Cluster A:Application-45 (topology-2356)|2,000|8,000|
|YARN: Cluster B|1,000|4,000|
|YARN: Cluster B:Application-19 (user bobby)|1,000|4,000|
|Open Stack|5,000|20,000|
It would also really be good for the owners pages to have a breakdown for where
their code is running, and the topology pages to also.
Fragmentation measurements would also need to be updated to take this into
account and give some kind of breakdown by cluster type.
*Future Work:*
At some point soon I would like to look at elasticity too. This would impact
storm on X because we will likely provide ways for the supervisors to increase
or decrease the amount of resources that a worker has based off of their actual
usage. Some clusters support dynamically modifying resources for a container
on the fly, and to make this the most efficient possible it would be great for
the elasticity piece to know which do and which don't so it can request the
changes that would have the least impact possible.
> Storm on X
> ----------
>
> Key: STORM-1740
> URL: https://issues.apache.org/jira/browse/STORM-1740
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-core
> Reporter: Renjie Liu
> Priority: Minor
>
> Storm should have a clean API to be able to get resources from another
> scheduler, like Mesos or Yarn.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)