Adam to clarify with an example. An offer may have an offer with reserved resources (role hadoop) 2gb Mem, 1 CPU and 31000-31005 ports, and * resources 2 GB Mem 3 CPU and 31500-31999 ports. If one wants to create a task using 4 CPU and 3 GB of memory 31000 and 35001 ports, does one have to add resources of 2gb Mem, 1 CPU, 35001 port of role hadoop and 3 CPU and 2 GB 31000 of role * (or empty)? We had issues with storm not being able to accept both reserved and unreserved resources leading to not being able to use all memory. We worked with Brenden to correct the issue and the solution was to add the two resources to the task. Failure to add both resources and simply using one resource 4 CPU and 3 GB 31000 and 35001 resulted in TaskFailed.
Darin On Nov 2, 2016 5:54 AM, "Adam Bordelon" <a...@mesosphere.io> wrote: > Sorry for the delayed response. If you haven't already, I'd recommend > reading https://mesos.apache.org/documentation/latest/roles/ > Beyond that, let me try to clear up a few things: > > ==FrameworkInfo.role > 1. Every framework registers with Mesos with a role. If you don't specify > one, Mesos defaults the framework to '*'. Reservations and quota cannot be > assigned to "*", but it is in the list (with weight=1) when DRF calculates > which "role" is furthest below its fair share, to decide which role's > framework(s) should get the next offer. > 2. At offer time, every resource offered to a framework is allocated (in > the Mesos allocator) to the framework's role, regardless of reservations. > This is how Mesos determines a role's current usage vs. its "fair share". > If the framework declines the offer, then the resource is "recovered" or > deallocated from the framework's role. If the framework launches a task > with the resource, it is not recovered until the task exits. Mesos > considers offered resources when calculating current "usage" so that > frameworks cannot hoard offers to take over the cluster. > > ==Resource.role > 3. Straight from the protobuf: "The role that this resource is reserved > for. If "*", this indicates that the resource is unreserved. Otherwise, the > resource will only be offered to frameworks that belong to this role." > 4. Resources offered to a framework may be reserved for that framework's > role, so that they are never offered to other frameworks in different > roles. These reserved resources will have role="foo" (for some value of foo > other than "*") set in their offers. > 5. Unreserved resources may be offered to any framework, regardless of what > role the framework registered with. Mesos uses weighted DRF to select the > role that is furthest below its fair share and offer the unreserved > resource to a framework registered with that role (using DRF between > frameworks in the same role). > 6. When unreserved resources are offered, even if Mesos didn't set the > role field, the protobuf parser scheduler-side should set it to the default > "*". Either is equivalent to "unreserved". > > ==Implications for Myriad > 7. If there is code (like Yuliya references) checking if offered resources > are unreserved, we should be checking if role == "*", not role.isEmpty. Or > check for both if you like. > 8. There shouldn't be any reason to select reserved resources over > unreserved resources in the same offer (for the same agent). Mesos > allocates all resources to the framework's role when it makes the offer. > And Mesos ignores the role field in the resources selected for launching a > task. Mesos just verifies that the sum of the resources for the > executor/task do not exceed the available resources on that node. > 9. In fact, you can combine multiple offers (perhaps cached for a second or > two) from the same agent and launch one or more tasks in the same request > with a combination of the offers' resources. For example, with two > different offers of 2GB on the same agent, you can call launchTask with the > list of two offers to launch a 1GB task and a 3GB task. > 10. That said, given an offer of reserved resources on one node and > unreserved resources on another, the Myriad scheduler might want to choose > a) the reserved resources, to be a good citizen and be more likely to > respawn on a reserved node where it already has cached > data/binaries/images; or b) the unreserved resources, so that Myriad can > greedily claim more of the cluster beyond its reserved minimum. > > ==The Future: Multi-role frameworks > 11. In a future Mesos, we plan to allow multi-role frameworks, so that a > framework can register to receive reserved resources from multiple > different roles, and subscribe to their quotas as well. That's currently in > design, but will take months to implement. > > Hope this helps. I've numbered my points in case you have questions about > any of it. > Cheers, > -Adam- > > > On Fri, Oct 28, 2016 at 4:10 PM, yuliya Feldman > <yufeld...@yahoo.com.invalid > > wrote: > > > We clearly need a word from Adam, Mohit, Ken > > My impression is that Myriad will not get any resources that are not > > specific to the role it has (or entitled to), so we may not need much > roles > > manipulation in Myriad code. > > Just my 2c of gut feelings :) > > Thanks,Yuliya > > > > From: Darin Johnson <dbjohnson1...@gmail.com> > > To: Dev <dev@myriad.incubator.apache.org> > > Sent: Friday, October 28, 2016 2:54 PM > > Subject: Re: [DISCUSS] handling roles in Myriad code > > > > Any word from Adam or Mohit? > > > > On Oct 20, 2016 12:36 AM, "Klaus Ma" <klaus1982...@gmail.com> wrote: > > > > > I can help on this discussion; I used to be Mesos contributor for a > year > > > :). > > > > > > Mesos allocate regular resources based on role by DRF; and role is also > > > used for reservation & quotas. So, the framework (like Myriad), may get > > two > > > kind of resources: "*" or "myriad-s role". When Myriad launch tasks, it > > can > > > not overuse any kind of resources: for example, if Myarid got offers: > > > cpu(*):1;cpu(myriad):1, Myriad can not launch tasks by cpu(*):2 which > > will > > > be rejected by Mesos master. > > > > > > Thanks > > > Klaus > > > > > > > > > On Thu, Oct 20, 2016 at 12:10 PM Yuliya <yufeld...@yahoo.com.invalid> > > > wrote: > > > > > > > I really would like Mesosphere guys to comment here. I had a chat > with > > > > Adam today morning and I did not get the same impression > > > > > > > > Thanks, > > > > Yuliya > > > > > > > > > On Oct 19, 2016, at 8:50 PM, Darin Johnson < > dbjohnson1...@gmail.com> > > > > wrote: > > > > > > > > > > We use roles extensively to ensure different frameworks can (or > > can't) > > > > get > > > > > resources via mechanisms such as reserved resorces and quotas. > Also > > if > > > > you > > > > > don't pay attention you can miss a lot of the resources you're > given. > > > I > > > > > wish it was we didn't have to do all the book keeping our selves, > > but I > > > > > suppose there are good reasons for delegating it to the framework, > > for > > > > > instance we can choose when to fave a reserved vs a default > resource. > > > > > > > > > > On Wed, Oct 19, 2016 at 11:30 PM, yuliya Feldman < > > > > > yufeld...@yahoo.com.invalid> wrote: > > > > > > > > > >> I am not sure we should care about role being set or not, what if > in > > > the > > > > >> future we will have multiple rolesNot even sure if > presence/absence > > of > > > > role > > > > >> should play role (no pun intended :) ). > > > > >> > > > > >> From: Darin Johnson <dbjohnson1...@gmail.com> > > > > >> To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman < > > > > >> yufeld...@yahoo.com> > > > > >> Sent: Wednesday, October 19, 2016 7:17 PM > > > > >> Subject: Re: [DISCUSS] handling roles in Myriad code > > > > >> > > > > >> Ah so if I understand correctly, if frameworkRole='*' is present > in > > > the > > > > >> config, it's handled as thought it's the framework role. I > believe > > > > when I > > > > >> was testing I was using frameworkRole="test" or commenting out > > > > >> frameworkRole="test". It looks as though in MyriadConfiguration, > > > > >> getFrameworkRole now returns "*" even if not set. > > > > >> > > > > >> Seems like we should be able to add a check like r.hasRole() && > > > > >> r.getRole().equals(role) > > > > >> && !role.equals("*") in a few places. Though it may be better > > > > >> to pass think about a better approach here. > > > > >> > > > > >> Darin > > > > >> > > > > >> On Wed, Oct 19, 2016 at 9:28 PM, yuliya Feldman > > > > >> <yufeld...@yahoo.com.invalid > > > > >>> wrote: > > > > >> > > > > >>> Hello Darrin, > > > > >>> I kind of see the point regarding JHS ports. May be there is > truth > > to > > > > it. > > > > >>> Regarding my issues with role/no role. > > > > >>> I had this issue for NMs with random ports (not hardcoded), as it > > has > > > > >>> different code path when role is present and when it is not. My > > > > >> impression > > > > >>> those are bugs. > > > > >>> I am happy to point you to the places in the code that caused > > issues > > > on > > > > >>> master (at least for me).[1] does not increment numDefaultValues > if > > > > role > > > > >> is > > > > >>> set (which is always set), subsequently [2] has issues[3] same > > thing > > > - > > > > >>> fills out list only if there is no role, but again it is always > > > there, > > > > >> just > > > > >>> set to "*" > > > > >>> > > > > >>> > > > > >>> Regarding:>>> To handle nodemanager persistence I think we should > > > work > > > > >>> with Klaus's PR's to get thecorrect ports, though we'll need to > use > > > > some > > > > >>> disk persistence as well to > > > > >>> keep the NM state. > > > > >>> Disk persistence won't help here (not even sure NM has much state > > to > > > > >>> persist - even if it does it should be taken care by YARN), as > > > > containers > > > > >>> have to reconnect to NM after it restarts, so they have to know > RPC > > > > port. > > > > >>> Thanks,Yuliya > > > > >>> [1] https://github.com/apache/incubator-myriad/blob/master/ > > > > >>> myriad-scheduler/src/main/java/org/apache/myriad/ > > scheduler/resource/ > > > > >>> RangeResource.java#L85 > > > > >>> [2] https://github.com/apache/incubator-myriad/blob/master/ > > > > >>> myriad-scheduler/src/main/java/org/apache/myriad/ > > scheduler/resource/ > > > > >>> RangeResource.java#L128 > > > > >>> > > > > >>> [3] https://github.com/apache/incubator-myriad/blob/master/ > > > > >>> myriad-scheduler/src/main/java/org/apache/myriad/ > > scheduler/resource/ > > > > >>> RangeResource.java#L140 > > > > >>> > > > > >>> > > > > >>> From: Darin Johnson <dbjohnson1...@gmail.com> > > > > >>> To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman < > > > > >>> yufeld...@yahoo.com> > > > > >>> Sent: Wednesday, October 19, 2016 6:04 PM > > > > >>> Subject: Re: [DISCUSS] handling roles in Myriad code > > > > >>> > > > > >>> Yuyiya, > > > > >>> > > > > >>> Yes on master a lot of refactoring was done, in particular you > > > specify > > > > >>> ports other than 0 in the myriad-default.yaml, it will only > return > > > > those > > > > >>> ports (not random ones). This was done in part because the we > were > > > > >>> attempting the use the JHS on a port like 32001, but it the port > > was > > > > >>> already in use by another app and hence the port wasn't offered > > > myriad > > > > >> was > > > > >>> still launching the JHS only to have it crash. > > > > >>> > > > > >>> If you want to use static ports you can just not put anything in > > the > > > > >>> myriad-default.yaml and configure the yarn-site.xml and > > > mapred-site.xml > > > > >> as > > > > >>> usual (they should be outside the range mesos offers). To handle > > > > >>> nodemanager persistence I think we should work with Klaus's PR's > to > > > get > > > > >> the > > > > >>> correct ports, though we'll need to use some disk persistance as > > well > > > > to > > > > >>> keep the NM state. > > > > >>> > > > > >>> As for a bug in NM's getting zero ports could you send a copy of > > your > > > > >>> configuration and I'll try to recreate the problem? > > > > >>> > > > > >>> On Wed, Oct 19, 2016 at 3:29 PM, yuliya Feldman > > > > >>> <yufeld...@yahoo.com.invalid > > > > >>>> wrote: > > > > >>> > > > > >>>> Hello there, > > > > >>>> I wanted to discuss current handling of roles in Myriad code. > > > > >>> Specifically > > > > >>>> on "master" branch. Most likely due to heavy refactoring. > > > > >>>> As far as I can see we try to handle presence or absence of a > role > > > on > > > > a > > > > >>>> resource(s) based on the fact that framework may or may not > have a > > > > >>> role.On > > > > >>>> the other hand we always set framework role to "*" - which means > > it > > > > >> will > > > > >>>> always have a role, just that role will be "default". > > > > >>>> So far I encountered couple of bugs related to roles in > > > RangeResource > > > > >>>> related to ports and inability to spin up NodeManagers, as no > > ports > > > > >> were > > > > >>>> assigned because of the fact how we handle roles. > > > > >>>> I would like @Adam and other Mesosphere folks to comment on how > > > should > > > > >> we > > > > >>>> handle relationship between frameworkRole and resource role(s) > > > > >>>> Thanks,Yuliya > > > > >> > > > > >> > > > > >> > > > > >> > > > > > > > -- > > > > > > Regards, > > > ---- > > > Da (Klaus), Ma (马达), PMP® | Software Architect > > > IBM Platform Development & Support, STG, IBM GCG > > > +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me > > > > > > > > > >
