[ 
https://issues.apache.org/jira/browse/STORM-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16339495#comment-16339495
 ] 

Robert Joseph Evans commented on STORM-1740:
--------------------------------------------

The following is an initial design for this new feature. I expect to break up 
this functionality into smaller pieces and will depend on other pieces of 
functionality like STORM-2693 and STORM-2910 to make this ultimately work.

In the past we have run into a number of issues with breaking Storm on Mesos 
due to internal changes that we though were not going to break anything, but 
did. 
 The issue as I see it is primarily that the INimbus API is not very clear 
about how it should be used, and it does not tie in well with some of the 
changes we have made to scheduling, namely the move away from pure slot based 
scheduling.

As such I propose that we just drop INimbus entirely, and instead add in a few 
new scheduling APIs, and a new Supervisor Launcher/Initialization API.

Requirements:
 * We want to be able to have a simple API that allows storm to request 
resources from some other place (this could be YARN, or Mesos, or whatever) on 
behalf of a topology that does not have the resources it needs.
 * We want to be able to to use existing storm schedulers without modification, 
but a scheduler may also have the option to improve the requests beyond what a 
default implementation would do.
 * We want to be able to support security in all of these environments 
(although it may not be something that we can fully support initially)
 * We want any new APIs to be maintainable, have tests for it and have a very 
clearly defined contract with storm so we don't keep breaking our customers.
 * We want to be able to give both cluster owners and topology owners some 
control over how and when this feature is used.

*First off lets look at scheduling:*
 Storm wants to have its own scheduler because it can have a deep view into how 
a topology is laid out and, hopefully soon, what it is doing. With this 
information we can do a better job of scheduling compared to a generic cluster 
level scheduler. It also gives us a lot more control to implement things like 
elasticity on our own.  But at the same time we need to be able to get 
resources from some other place.

So obviously we need a 2 level scheduler to be able to support something like 
this. YARN and Mesos are built around this concept and systems like Open Stack, 
Kubernetes, and even proprietary cloud service providers can also be used to 
get similar functionality.

As such I would like to propose an API for a request that has a very minimal 
set of requirements, and at the same time allows for optional hints that can be 
used if the higher level scheduler can support them.
{code:java}
//This is just a psudo code example
public enum Priority {
  RECOVERY, // Workers have crashed and we really need to get this back up and 
running.
  NORMAL // It is okay to spend some time on this to get things co-located if 
possible.
  //Possibly in the future an OPTIMIZATION for cases where the topology is 
running, but we have recognized scheduling could be better if we move something 
to a better location, or give it more resources.
}

public class SecondLevelTopologyRequest {
  public Map<String, Object> getTopoConf();
  public String getOwner();
  public String getTopologyId();
  public Priority getPriority();
  public List<SecondLevelSlotRequest> getSlotRequests();
  //IP address that we would like to have resources close to.
  public String localityRequest();
}

public class SecondLevelSlotRequest {
  public Resources getResources();
}
{code}
We then would expose an API through Cluster so a scheduler if it does not have 
enough resources to schedule something it can request those needed resources, 
or update an existing request. This would likely happen after eviction has 
happened, but would be up to the scheduler.

After scheduling was finished we would have a generic piece of code that could 
update/create SecondLevelTopologyRequests if the scheduler did not do so on 
it's own. This would not be a super simple piece of code because it would have 
to take into account both topologies that are resource aware and ones that just 
ask for slots. To make this work so the SecondLevelScheduler knows what to work 
with we will likely need to use some new settings and exploration of the 
topology's command like to guess at least for memory and CPU settings.

After this happens an ISecondLevelScheduler instance would be handed the 
requests and it could use them to try and negotiate for resources, and launch 
supervisors as needed when those resources are available. This scheduler would 
also have the option to tag requests that it cannot or may not be able to meet. 
This may be due to config settings (i.e. this particular user is not configured 
to request resources on this other cluster), or due to resource constraints on 
the other cluster (i.e. the YARN queue for this user is totally full, and there 
is no churn on it). In those cases other ISecondLevelScheduler instances may be 
given a chance to fulfill those requests.

Ideally an ISecondLevelScheduler should do gang scheduling on a per topology 
basis, get all of the resources requested or none of them. There are some 
systems where that is not possible. As such it is not a hard requirement, but a 
suggestion.

*Security:*
 In some cases we may be done at this point. If we are on openstack or AWS we 
may have been able to launch a VM that is exactly like our other supervisors so 
that they each have access to the needed kerberos credentials and have the 
ability to run workers as the user that launched the topology. But for Mesos 
and YARN this is not possible.

Their security model is similar to Storm's and I would consider it a security 
bug if someone could get sudo like privileges as a regular user on one of those 
clusters. So to make this really work we need the concept of a user and or 
topology specific supervisor.

This is one of the parts that is going to take some more work.

The issue is that the supervisor currently needs to communicate with Nimbus and 
with zookeeper to get what it needs.

The supervisor talks to nimbus to download the dependencies for a topology, and 
talks to ZK to heartbeat that it is still alive, and to download the current 
scheduling so it knows what it needs to launch.

STORM-2693 makes it so downloading new scheduling from ZK is no longer 
required, and we could also make it so heartbeats could go directly to nimbus 
instead of going to ZK, similar to what STORM-2693 does for worker heartbeats 
to nimbus.

So then we would need a way to authorize a supervisor to communicate with 
nimbus. We could create a token similar to a worker token that would allow this 
to happen. The lifetime and rotation of this token would be controlled by the 
ISecondLevelScheduler that is launching the supervisors as it is application 
specific how those tokens could be distributed to the running supervisors in a 
secure way.

We would need some kind of plugin in the supervisors (possibly a jaas login 
plugin) that would take this token and place it in the current subjects private 
credentials so SASL negotiation could work. As it is common for a supervisor to 
also need many changes to it's config to work properly on this kind of a new 
environment, we may want to make this a formal ISupervisorLauncher or 
ISupervisorEnvironment plugin instead. At a minimum we would also need a way 
for the supervisor to know what constraints it has (user specific or topology 
specific) so it can tell nimbus and enforce those constraints itself. But we 
need to get more experience with exactly what/how the different implementations 
will want to make this happen. So I expect to get lots of feedback on it.

After STORM-2693 we would also need a way for the supervisor to validate 
incoming worker heartbeats. We could just ignore it and have the workers skip 
the supervisor and communicate with nimbus directly and to the supervisor only 
through the file system like it does now. It would be preferable, though, to 
still have the worker to supervisor thrift path work, and we may need to think 
about this as a part of STORM-2693. To make this work we could have a separate 
supervisor token/secret from the nimbus token/secret. Then the supervisor 
secret key for a topology could be distributed to the supervisors through 
heartbeat responses and with the updated supervisor assignments. That way the 
supervisor would only know about the secrets that are specific to topologies it 
is responsible for. One of the big things we would have to be careful of would 
be making sure that the keys were distributed before sending the secret to the 
workers, or provide an out of band way for the supervisor to fetch the keys.

We would have to have a way for the scheduler to enforce/honor supervisor 
constraints:

A supervisor that is specific to either a topology or a user could not have 
anything else scheduled on it. The simplest way to work around this is to call 
the scheduler multiple times. We would call it once for topologies that do not 
have supervisors that are specific to them and with a "cluster" that did not 
have any constrained supervisors. We would then call it once for each topology 
that does have constraints and a special cluster that includes the resources it 
is allowed to use. This is not perfect as the scheduler assumes it sees the 
entire cluster and may get really confused about resource guarantees if it 
cannot see the entire cluster. But we can update those schedulers so they can 
handle all of it at once and work with the constraints, and understand 
resources that are enforced by storm, and resources that are enforced by other 
schedulers.

*Rolling Upgrades:*

This is something that would likely come later on too, but we would likely need 
a good way to ask an ISecondLevelScheduler to do a rolling upgrade.  

*UI/Polish:*

Once we have all of that working we need a good way to let users know what in 
the world is happening to their topology, and to the cluster.  This may be 
something that happens later on as well.

Currently we display a lot of resources that are aggregates for the entire 
cluster. We likely will want to offer a view that also breaks them up by where 
the resources came from.
||Cluster/Application||CPU||MEMORY||
|Total|50,000|200,000|
|Native|40,000|160,000|
|YARN|5,000|20,000|
|YARN: Cluster A|4,000|16,000|
|YARN: Cluster A:Application-29 (topology-2345)|2,000|8,000|
|YARN: Cluster A:Application-45 (topology-2356)|2,000|8,000|
|YARN: Cluster B|1,000|4,000|
|YARN: Cluster B:Application-19 (user bobby)|1,000|4,000|
|Open Stack|5,000|20,000|

It would also really be good for the owners pages to have a breakdown for where 
their code is running, and the topology pages to also.

Fragmentation measurements would also need to be updated to take this into 
account and give some kind of breakdown by cluster type.

*Future Work:*

At some point soon I would like to look at elasticity too.  This would impact 
storm on X because we will likely provide ways for the supervisors to increase 
or decrease the amount of resources that a worker has based off of their actual 
usage.  Some clusters support dynamically modifying resources for a container 
on the fly, and to make this the most efficient possible it would be great for 
the elasticity piece to know which do and which don't so it can request the 
changes that would have the least impact possible.

> Storm on X
> ----------
>
>                 Key: STORM-1740
>                 URL: https://issues.apache.org/jira/browse/STORM-1740
>             Project: Apache Storm
>          Issue Type: New Feature
>          Components: storm-core
>            Reporter: Renjie Liu
>            Priority: Minor
>
> Storm should have a clean API to be able to get resources from another 
> scheduler, like Mesos or Yarn.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to