Re: [DISCUSS] handling roles in Myriad code

[Darin Johnson]

eason to select reserved resources over
> > > unreserved resources in the same offer (for the same agent). Mesos
> > > allocates all resources to the framework's role when it makes the
> offer.
> > > And Mesos ignores the role field in the resources selected for
> launching
> > a
> > > task. Mesos just verifies that the sum of the resources for the
> > > executor/task do not exceed the available resources on that node.
> > > 9. In fact, you can combine multiple offers (perhaps cached for a
> second
> > or
> > > two) from the same agent and launch one or more tasks in the same
> request
> > > with a combination of the offers' resources. For example, with two
> > > different offers of 2GB on the same agent, you can call launchTask with
> > the
> > > list of two offers to launch a 1GB task and a 3GB task.
> > > 10. That said, given an offer of reserved resources on one node and
> > > unreserved resources on another, the Myriad scheduler might want to
> > choose
> > > a) the reserved resources, to be a good citizen and be more likely to
> > > respawn on a reserved node where it already has cached
> > > data/binaries/images; or b) the unreserved resources, so that Myriad
> can
> > > greedily claim more of the cluster beyond its reserved minimum.
> > >
> > > ==The Future: Multi-role frameworks
> > > 11. In a future Mesos, we plan to allow multi-role frameworks, so that
> a
> > > framework can register to receive reserved resources from multiple
> > > different roles, and subscribe to their quotas as well. That's
> currently
> > in
> > > design, but will take months to implement.
> > >
> > > Hope this helps. I've numbered my points in case you have questions
> about
> > > any of it.
> > > Cheers,
> > > -Adam-
> > >
> > >
> > > On Fri, Oct 28, 2016 at 4:10 PM, yuliya Feldman
> > > <yufeld...@yahoo.com.invalid
> > > > wrote:
> > >
> > > > We clearly need a word from Adam, Mohit, Ken
> > > > My impression is that Myriad will not get any resources that are not
> > > > specific to the role it has (or entitled to), so we may not need much
> > > roles
> > > > manipulation in Myriad code.
> > > > Just my 2c of gut feelings :)
> > > > Thanks,Yuliya
> > > >
> > > >   From: Darin Johnson <dbjohnson1...@gmail.com>
> > > >  To: Dev <dev@myriad.incubator.apache.org>
> > > >  Sent: Friday, October 28, 2016 2:54 PM
> > > >  Subject: Re: [DISCUSS] handling roles in Myriad code
> > > >
> > > > Any word from Adam or Mohit?
> > > >
> > > > On Oct 20, 2016 12:36 AM, "Klaus Ma" <klaus1982...@gmail.com> wrote:
> > > >
> > > > > I can help on this discussion; I used to be Mesos contributor for a
> > > year
> > > > > :).
> > > > >
> > > > > Mesos allocate regular resources based on role by DRF; and role is
> > also
> > > > > used for reservation & quotas. So, the framework (like Myriad), may
> > get
> > > > two
> > > > > kind of resources: "*" or "myriad-s role". When Myriad launch
> tasks,
> > it
> > > > can
> > > > > not overuse any kind of resources: for example, if Myarid got
> offers:
> > > > > cpu(*):1;cpu(myriad):1, Myriad can not launch tasks by cpu(*):2
> which
> > > > will
> > > > > be rejected by Mesos master.
> > > > >
> > > > > Thanks
> > > > > Klaus
> > > > >
> > > > >
> > > > > On Thu, Oct 20, 2016 at 12:10 PM Yuliya
> <yufeld...@yahoo.com.invalid
> > >
> > > > > wrote:
> > > > >
> > > > > > I really would like Mesosphere guys to comment here. I had a chat
> > > with
> > > > > > Adam today morning and I did not get the same impression
> > > > > >
> > > > > > Thanks,
> > > > > > Yuliya
> > > > > >
> > > > > > > On Oct 19, 2016, at 8:50 PM, Darin Johnson <
> > > dbjohnson1...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > We use roles extensively to ensure different frameworks can (or
> > > > can't)
> > > > > > get
> > > > >

Re: [DISCUSS] handling roles in Myriad code

[Adam Bordelon]

es. For example, with two
> > different offers of 2GB on the same agent, you can call launchTask with
> the
> > list of two offers to launch a 1GB task and a 3GB task.
> > 10. That said, given an offer of reserved resources on one node and
> > unreserved resources on another, the Myriad scheduler might want to
> choose
> > a) the reserved resources, to be a good citizen and be more likely to
> > respawn on a reserved node where it already has cached
> > data/binaries/images; or b) the unreserved resources, so that Myriad can
> > greedily claim more of the cluster beyond its reserved minimum.
> >
> > ==The Future: Multi-role frameworks
> > 11. In a future Mesos, we plan to allow multi-role frameworks, so that a
> > framework can register to receive reserved resources from multiple
> > different roles, and subscribe to their quotas as well. That's currently
> in
> > design, but will take months to implement.
> >
> > Hope this helps. I've numbered my points in case you have questions about
> > any of it.
> > Cheers,
> > -Adam-
> >
> >
> > On Fri, Oct 28, 2016 at 4:10 PM, yuliya Feldman
> > <yufeld...@yahoo.com.invalid
> > > wrote:
> >
> > > We clearly need a word from Adam, Mohit, Ken
> > > My impression is that Myriad will not get any resources that are not
> > > specific to the role it has (or entitled to), so we may not need much
> > roles
> > > manipulation in Myriad code.
> > > Just my 2c of gut feelings :)
> > > Thanks,Yuliya
> > >
> > >   From: Darin Johnson <dbjohnson1...@gmail.com>
> > >  To: Dev <dev@myriad.incubator.apache.org>
> > >  Sent: Friday, October 28, 2016 2:54 PM
> > >  Subject: Re: [DISCUSS] handling roles in Myriad code
> > >
> > > Any word from Adam or Mohit?
> > >
> > > On Oct 20, 2016 12:36 AM, "Klaus Ma" <klaus1982...@gmail.com> wrote:
> > >
> > > > I can help on this discussion; I used to be Mesos contributor for a
> > year
> > > > :).
> > > >
> > > > Mesos allocate regular resources based on role by DRF; and role is
> also
> > > > used for reservation & quotas. So, the framework (like Myriad), may
> get
> > > two
> > > > kind of resources: "*" or "myriad-s role". When Myriad launch tasks,
> it
> > > can
> > > > not overuse any kind of resources: for example, if Myarid got offers:
> > > > cpu(*):1;cpu(myriad):1, Myriad can not launch tasks by cpu(*):2 which
> > > will
> > > > be rejected by Mesos master.
> > > >
> > > > Thanks
> > > > Klaus
> > > >
> > > >
> > > > On Thu, Oct 20, 2016 at 12:10 PM Yuliya <yufeld...@yahoo.com.invalid
> >
> > > > wrote:
> > > >
> > > > > I really would like Mesosphere guys to comment here. I had a chat
> > with
> > > > > Adam today morning and I did not get the same impression
> > > > >
> > > > > Thanks,
> > > > > Yuliya
> > > > >
> > > > > > On Oct 19, 2016, at 8:50 PM, Darin Johnson <
> > dbjohnson1...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > We use roles extensively to ensure different frameworks can (or
> > > can't)
> > > > > get
> > > > > > resources via mechanisms such as reserved resorces and quotas.
> > Also
> > > if
> > > > > you
> > > > > > don't pay attention you can miss a lot of the resources you're
> > given.
> > > > I
> > > > > > wish it was we didn't have to do all the book keeping our selves,
> > > but I
> > > > > > suppose there are good reasons for delegating it to the
> framework,
> > > for
> > > > > > instance we can choose when to fave a reserved vs a default
> > resource.
> > > > > >
> > > > > > On Wed, Oct 19, 2016 at 11:30 PM, yuliya Feldman <
> > > > > > yufeld...@yahoo.com.invalid> wrote:
> > > > > >
> > > > > >> I am not sure we should care about role being set or not, what
> if
> > in
> > > > the
> > > > > >> future we will have multiple rolesNot even sure if
> > presence/absence
> > > of
> > > > > role
> > > > > >> should play role (no pun i

Re: [DISCUSS] handling roles in Myriad code

[Darin Johnson]

in
> design, but will take months to implement.
>
> Hope this helps. I've numbered my points in case you have questions about
> any of it.
> Cheers,
> -Adam-
>
>
> On Fri, Oct 28, 2016 at 4:10 PM, yuliya Feldman
> <yufeld...@yahoo.com.invalid
> > wrote:
>
> > We clearly need a word from Adam, Mohit, Ken
> > My impression is that Myriad will not get any resources that are not
> > specific to the role it has (or entitled to), so we may not need much
> roles
> > manipulation in Myriad code.
> > Just my 2c of gut feelings :)
> > Thanks,Yuliya
> >
> >   From: Darin Johnson <dbjohnson1...@gmail.com>
> >  To: Dev <dev@myriad.incubator.apache.org>
> >  Sent: Friday, October 28, 2016 2:54 PM
> >  Subject: Re: [DISCUSS] handling roles in Myriad code
> >
> > Any word from Adam or Mohit?
> >
> > On Oct 20, 2016 12:36 AM, "Klaus Ma" <klaus1982...@gmail.com> wrote:
> >
> > > I can help on this discussion; I used to be Mesos contributor for a
> year
> > > :).
> > >
> > > Mesos allocate regular resources based on role by DRF; and role is also
> > > used for reservation & quotas. So, the framework (like Myriad), may get
> > two
> > > kind of resources: "*" or "myriad-s role". When Myriad launch tasks, it
> > can
> > > not overuse any kind of resources: for example, if Myarid got offers:
> > > cpu(*):1;cpu(myriad):1, Myriad can not launch tasks by cpu(*):2 which
> > will
> > > be rejected by Mesos master.
> > >
> > > Thanks
> > > Klaus
> > >
> > >
> > > On Thu, Oct 20, 2016 at 12:10 PM Yuliya <yufeld...@yahoo.com.invalid>
> > > wrote:
> > >
> > > > I really would like Mesosphere guys to comment here. I had a chat
> with
> > > > Adam today morning and I did not get the same impression
> > > >
> > > > Thanks,
> > > > Yuliya
> > > >
> > > > > On Oct 19, 2016, at 8:50 PM, Darin Johnson <
> dbjohnson1...@gmail.com>
> > > > wrote:
> > > > >
> > > > > We use roles extensively to ensure different frameworks can (or
> > can't)
> > > > get
> > > > > resources via mechanisms such as reserved resorces and quotas.
> Also
> > if
> > > > you
> > > > > don't pay attention you can miss a lot of the resources you're
> given.
> > > I
> > > > > wish it was we didn't have to do all the book keeping our selves,
> > but I
> > > > > suppose there are good reasons for delegating it to the framework,
> > for
> > > > > instance we can choose when to fave a reserved vs a default
> resource.
> > > > >
> > > > > On Wed, Oct 19, 2016 at 11:30 PM, yuliya Feldman <
> > > > > yufeld...@yahoo.com.invalid> wrote:
> > > > >
> > > > >> I am not sure we should care about role being set or not, what if
> in
> > > the
> > > > >> future we will have multiple rolesNot even sure if
> presence/absence
> > of
> > > > role
> > > > >> should play role (no pun intended :) ).
> > > > >>
> > > > >>  From: Darin Johnson <dbjohnson1...@gmail.com>
> > > > >> To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
> > > > >> yufeld...@yahoo.com>
> > > > >> Sent: Wednesday, October 19, 2016 7:17 PM
> > > > >> Subject: Re: [DISCUSS] handling roles in Myriad code
> > > > >>
> > > > >> Ah so if I understand correctly, if frameworkRole='*' is present
> in
> > > the
> > > > >> config, it's handled as thought it's the framework role.  I
> believe
> > > > when I
> > > > >> was testing I was using frameworkRole="test" or commenting out
> > > > >> frameworkRole="test".  It looks as though in MyriadConfiguration,
> > > > >> getFrameworkRole now returns "*" even if not set.
> > > > >>
> > > > >> Seems like we should be able to add a check like r.hasRole() &&
> > > > >> r.getRole().equals(role)
> > > > >> && !role.equals("*") in a few places. Though it may be better
> > > > >> to pass think about a better approach here.
> > > > >>
> > > > >> Darin
> > &

Re: [DISCUSS] handling roles in Myriad code

[Adam Bordelon]

Sorry for the delayed response. If you haven't already, I'd recommend
reading https://mesos.apache.org/documentation/latest/roles/
Beyond that, let me try to clear up a few things:

==FrameworkInfo.role
1. Every framework registers with Mesos with a role. If you don't specify
one, Mesos defaults the framework to '*'. Reservations and quota cannot be
assigned to "*", but it is in the list (with weight=1) when DRF calculates
which "role" is furthest below its fair share, to decide which role's
framework(s) should get the next offer.
2. At offer time, every resource offered to a framework is allocated (in
the Mesos allocator) to the framework's role, regardless of reservations.
This is how Mesos determines a role's current usage vs. its "fair share".
If the framework declines the offer, then the resource is "recovered" or
deallocated from the framework's role. If the framework launches a task
with the resource, it is not recovered until the task exits. Mesos
considers offered resources when calculating current "usage" so that
frameworks cannot hoard offers to take over the cluster.

==Resource.role
3.  Straight from the protobuf: "The role that this resource is reserved
for. If "*", this indicates that the resource is unreserved. Otherwise, the
resource will only be offered to frameworks that belong to this role."
4. Resources offered to a framework may be reserved for that framework's
role, so that they are never offered to other frameworks in different
roles. These reserved resources will have role="foo" (for some value of foo
other than "*") set in their offers.
5. Unreserved resources may be offered to any framework, regardless of what
role the framework registered with. Mesos uses weighted DRF to select the
role that is furthest below its fair share and offer the unreserved
resource to a framework registered with that role (using DRF between
frameworks in the same role).
6.  When unreserved resources are offered, even if Mesos didn't set the
role field, the protobuf parser scheduler-side should set it to the default
"*". Either is equivalent to "unreserved".

==Implications for Myriad
7. If there is code (like Yuliya references) checking if offered resources
are unreserved, we should be checking if role == "*", not role.isEmpty. Or
check for both if you like.
8. There shouldn't be any reason to select reserved resources over
unreserved resources in the same offer (for the same agent). Mesos
allocates all resources to the framework's role when it makes the offer.
And Mesos ignores the role field in the resources selected for launching a
task. Mesos just verifies that the sum of the resources for the
executor/task do not exceed the available resources on that node.
9. In fact, you can combine multiple offers (perhaps cached for a second or
two) from the same agent and launch one or more tasks in the same request
with a combination of the offers' resources. For example, with two
different offers of 2GB on the same agent, you can call launchTask with the
list of two offers to launch a 1GB task and a 3GB task.
10. That said, given an offer of reserved resources on one node and
unreserved resources on another, the Myriad scheduler might want to choose
a) the reserved resources, to be a good citizen and be more likely to
respawn on a reserved node where it already has cached
data/binaries/images; or b) the unreserved resources, so that Myriad can
greedily claim more of the cluster beyond its reserved minimum.

==The Future: Multi-role frameworks
11. In a future Mesos, we plan to allow multi-role frameworks, so that a
framework can register to receive reserved resources from multiple
different roles, and subscribe to their quotas as well. That's currently in
design, but will take months to implement.

Hope this helps. I've numbered my points in case you have questions about
any of it.
Cheers,
-Adam-


On Fri, Oct 28, 2016 at 4:10 PM, yuliya Feldman <yufeld...@yahoo.com.invalid
> wrote:

> We clearly need a word from Adam, Mohit, Ken
> My impression is that Myriad will not get any resources that are not
> specific to the role it has (or entitled to), so we may not need much roles
> manipulation in Myriad code.
> Just my 2c of gut feelings :)
> Thanks,Yuliya
>
>       From: Darin Johnson <dbjohnson1...@gmail.com>
>  To: Dev <dev@myriad.incubator.apache.org>
>  Sent: Friday, October 28, 2016 2:54 PM
>  Subject: Re: [DISCUSS] handling roles in Myriad code
>
> Any word from Adam or Mohit?
>
> On Oct 20, 2016 12:36 AM, "Klaus Ma" <klaus1982...@gmail.com> wrote:
>
> > I can help on this discussion; I used to be Mesos contributor for a year
> > :).
> >
> > Mesos allocate regular resources based on role by DRF; and role is also
> > used for reservation & quotas. So, the framework (lik

Re: [DISCUSS] handling roles in Myriad code

[yuliya Feldman]

We clearly need a word from Adam, Mohit, Ken
My impression is that Myriad will not get any resources that are not specific 
to the role it has (or entitled to), so we may not need much roles manipulation 
in Myriad code.
Just my 2c of gut feelings :)
Thanks,Yuliya

  From: Darin Johnson <dbjohnson1...@gmail.com>
 To: Dev <dev@myriad.incubator.apache.org> 
 Sent: Friday, October 28, 2016 2:54 PM
 Subject: Re: [DISCUSS] handling roles in Myriad code
   
Any word from Adam or Mohit?

On Oct 20, 2016 12:36 AM, "Klaus Ma" <klaus1982...@gmail.com> wrote:

> I can help on this discussion; I used to be Mesos contributor for a year
> :).
>
> Mesos allocate regular resources based on role by DRF; and role is also
> used for reservation & quotas. So, the framework (like Myriad), may get two
> kind of resources: "*" or "myriad-s role". When Myriad launch tasks, it can
> not overuse any kind of resources: for example, if Myarid got offers:
> cpu(*):1;cpu(myriad):1, Myriad can not launch tasks by cpu(*):2 which will
> be rejected by Mesos master.
>
> Thanks
> Klaus
>
>
> On Thu, Oct 20, 2016 at 12:10 PM Yuliya <yufeld...@yahoo.com.invalid>
> wrote:
>
> > I really would like Mesosphere guys to comment here. I had a chat with
> > Adam today morning and I did not get the same impression
> >
> > Thanks,
> > Yuliya
> >
> > > On Oct 19, 2016, at 8:50 PM, Darin Johnson <dbjohnson1...@gmail.com>
> > wrote:
> > >
> > > We use roles extensively to ensure different frameworks can (or can't)
> > get
> > > resources via mechanisms such as reserved resorces and quotas.  Also if
> > you
> > > don't pay attention you can miss a lot of the resources you're given.
> I
> > > wish it was we didn't have to do all the book keeping our selves, but I
> > > suppose there are good reasons for delegating it to the framework, for
> > > instance we can choose when to fave a reserved vs a default resource.
> > >
> > > On Wed, Oct 19, 2016 at 11:30 PM, yuliya Feldman <
> > > yufeld...@yahoo.com.invalid> wrote:
> > >
> > >> I am not sure we should care about role being set or not, what if in
> the
> > >> future we will have multiple rolesNot even sure if presence/absence of
> > role
> > >> should play role (no pun intended :) ).
> > >>
> > >>      From: Darin Johnson <dbjohnson1...@gmail.com>
> > >> To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
> > >> yufeld...@yahoo.com>
> > >> Sent: Wednesday, October 19, 2016 7:17 PM
> > >> Subject: Re: [DISCUSS] handling roles in Myriad code
> > >>
> > >> Ah so if I understand correctly, if frameworkRole='*' is present in
> the
> > >> config, it's handled as thought it's the framework role.  I believe
> > when I
> > >> was testing I was using frameworkRole="test" or commenting out
> > >> frameworkRole="test".  It looks as though in MyriadConfiguration,
> > >> getFrameworkRole now returns "*" even if not set.
> > >>
> > >> Seems like we should be able to add a check like r.hasRole() &&
> > >> r.getRole().equals(role)
> > >> && !role.equals("*") in a few places. Though it may be better
> > >> to pass think about a better approach here.
> > >>
> > >> Darin
> > >>
> > >> On Wed, Oct 19, 2016 at 9:28 PM, yuliya Feldman
> > >> <yufeld...@yahoo.com.invalid
> > >>> wrote:
> > >>
> > >>> Hello Darrin,
> > >>> I kind of see the point regarding JHS ports. May be there is truth to
> > it.
> > >>> Regarding my issues with role/no role.
> > >>> I had this issue for NMs with random ports (not hardcoded), as it has
> > >>> different code path when role is present and when it is not. My
> > >> impression
> > >>> those are bugs.
> > >>> I am happy to point you to the places in the code that caused issues
> on
> > >>> master (at least for me).[1] does not increment numDefaultValues if
> > role
> > >> is
> > >>> set (which is always set), subsequently [2] has issues[3] same thing
> -
> > >>> fills out list only if there is no role, but again it is always
> there,
> > >> just
> > >>> set to "*"
> > >>>
> > >>>
> > >>> Regarding:&

Re: [DISCUSS] handling roles in Myriad code

[Darin Johnson]

Any word from Adam or Mohit?

On Oct 20, 2016 12:36 AM, "Klaus Ma" <klaus1982...@gmail.com> wrote:

> I can help on this discussion; I used to be Mesos contributor for a year
> :).
>
> Mesos allocate regular resources based on role by DRF; and role is also
> used for reservation & quotas. So, the framework (like Myriad), may get two
> kind of resources: "*" or "myriad-s role". When Myriad launch tasks, it can
> not overuse any kind of resources: for example, if Myarid got offers:
> cpu(*):1;cpu(myriad):1, Myriad can not launch tasks by cpu(*):2 which will
> be rejected by Mesos master.
>
> Thanks
> Klaus
>
>
> On Thu, Oct 20, 2016 at 12:10 PM Yuliya <yufeld...@yahoo.com.invalid>
> wrote:
>
> > I really would like Mesosphere guys to comment here. I had a chat with
> > Adam today morning and I did not get the same impression
> >
> > Thanks,
> > Yuliya
> >
> > > On Oct 19, 2016, at 8:50 PM, Darin Johnson <dbjohnson1...@gmail.com>
> > wrote:
> > >
> > > We use roles extensively to ensure different frameworks can (or can't)
> > get
> > > resources via mechanisms such as reserved resorces and quotas.  Also if
> > you
> > > don't pay attention you can miss a lot of the resources you're given.
> I
> > > wish it was we didn't have to do all the book keeping our selves, but I
> > > suppose there are good reasons for delegating it to the framework, for
> > > instance we can choose when to fave a reserved vs a default resource.
> > >
> > > On Wed, Oct 19, 2016 at 11:30 PM, yuliya Feldman <
> > > yufeld...@yahoo.com.invalid> wrote:
> > >
> > >> I am not sure we should care about role being set or not, what if in
> the
> > >> future we will have multiple rolesNot even sure if presence/absence of
> > role
> > >> should play role (no pun intended :) ).
> > >>
> > >>  From: Darin Johnson <dbjohnson1...@gmail.com>
> > >> To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
> > >> yufeld...@yahoo.com>
> > >> Sent: Wednesday, October 19, 2016 7:17 PM
> > >> Subject: Re: [DISCUSS] handling roles in Myriad code
> > >>
> > >> Ah so if I understand correctly, if frameworkRole='*' is present in
> the
> > >> config, it's handled as thought it's the framework role.  I believe
> > when I
> > >> was testing I was using frameworkRole="test" or commenting out
> > >> frameworkRole="test".  It looks as though in MyriadConfiguration,
> > >> getFrameworkRole now returns "*" even if not set.
> > >>
> > >> Seems like we should be able to add a check like r.hasRole() &&
> > >> r.getRole().equals(role)
> > >> && !role.equals("*") in a few places. Though it may be better
> > >> to pass think about a better approach here.
> > >>
> > >> Darin
> > >>
> > >> On Wed, Oct 19, 2016 at 9:28 PM, yuliya Feldman
> > >> <yufeld...@yahoo.com.invalid
> > >>> wrote:
> > >>
> > >>> Hello Darrin,
> > >>> I kind of see the point regarding JHS ports. May be there is truth to
> > it.
> > >>> Regarding my issues with role/no role.
> > >>> I had this issue for NMs with random ports (not hardcoded), as it has
> > >>> different code path when role is present and when it is not. My
> > >> impression
> > >>> those are bugs.
> > >>> I am happy to point you to the places in the code that caused issues
> on
> > >>> master (at least for me).[1] does not increment numDefaultValues if
> > role
> > >> is
> > >>> set (which is always set), subsequently [2] has issues[3] same thing
> -
> > >>> fills out list only if there is no role, but again it is always
> there,
> > >> just
> > >>> set to "*"
> > >>>
> > >>>
> > >>> Regarding:>>> To handle nodemanager persistence I think we should
> work
> > >>> with Klaus's PR's to get thecorrect ports, though we'll need to use
> > some
> > >>> disk persistence as well to
> > >>> keep the NM state.
> > >>> Disk persistence won't help here (not even sure NM has much state to
> > >>> persist - even if it does it should be taken care by YARN), as
> > containers
> >

Re: [DISCUSS] handling roles in Myriad code

[Klaus Ma]

I can help on this discussion; I used to be Mesos contributor for a year :).

Mesos allocate regular resources based on role by DRF; and role is also
used for reservation & quotas. So, the framework (like Myriad), may get two
kind of resources: "*" or "myriad-s role". When Myriad launch tasks, it can
not overuse any kind of resources: for example, if Myarid got offers:
cpu(*):1;cpu(myriad):1, Myriad can not launch tasks by cpu(*):2 which will
be rejected by Mesos master.

Thanks
Klaus


On Thu, Oct 20, 2016 at 12:10 PM Yuliya <yufeld...@yahoo.com.invalid> wrote:

> I really would like Mesosphere guys to comment here. I had a chat with
> Adam today morning and I did not get the same impression
>
> Thanks,
> Yuliya
>
> > On Oct 19, 2016, at 8:50 PM, Darin Johnson <dbjohnson1...@gmail.com>
> wrote:
> >
> > We use roles extensively to ensure different frameworks can (or can't)
> get
> > resources via mechanisms such as reserved resorces and quotas.  Also if
> you
> > don't pay attention you can miss a lot of the resources you're given.  I
> > wish it was we didn't have to do all the book keeping our selves, but I
> > suppose there are good reasons for delegating it to the framework, for
> > instance we can choose when to fave a reserved vs a default resource.
> >
> > On Wed, Oct 19, 2016 at 11:30 PM, yuliya Feldman <
> > yufeld...@yahoo.com.invalid> wrote:
> >
> >> I am not sure we should care about role being set or not, what if in the
> >> future we will have multiple rolesNot even sure if presence/absence of
> role
> >> should play role (no pun intended :) ).
> >>
> >>  From: Darin Johnson <dbjohnson1...@gmail.com>
> >> To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
> >> yufeld...@yahoo.com>
> >> Sent: Wednesday, October 19, 2016 7:17 PM
> >> Subject: Re: [DISCUSS] handling roles in Myriad code
> >>
> >> Ah so if I understand correctly, if frameworkRole='*' is present in the
> >> config, it's handled as thought it's the framework role.  I believe
> when I
> >> was testing I was using frameworkRole="test" or commenting out
> >> frameworkRole="test".  It looks as though in MyriadConfiguration,
> >> getFrameworkRole now returns "*" even if not set.
> >>
> >> Seems like we should be able to add a check like r.hasRole() &&
> >> r.getRole().equals(role)
> >> && !role.equals("*") in a few places. Though it may be better
> >> to pass think about a better approach here.
> >>
> >> Darin
> >>
> >> On Wed, Oct 19, 2016 at 9:28 PM, yuliya Feldman
> >> <yufeld...@yahoo.com.invalid
> >>> wrote:
> >>
> >>> Hello Darrin,
> >>> I kind of see the point regarding JHS ports. May be there is truth to
> it.
> >>> Regarding my issues with role/no role.
> >>> I had this issue for NMs with random ports (not hardcoded), as it has
> >>> different code path when role is present and when it is not. My
> >> impression
> >>> those are bugs.
> >>> I am happy to point you to the places in the code that caused issues on
> >>> master (at least for me).[1] does not increment numDefaultValues if
> role
> >> is
> >>> set (which is always set), subsequently [2] has issues[3] same thing -
> >>> fills out list only if there is no role, but again it is always there,
> >> just
> >>> set to "*"
> >>>
> >>>
> >>> Regarding:>>> To handle nodemanager persistence I think we should work
> >>> with Klaus's PR's to get thecorrect ports, though we'll need to use
> some
> >>> disk persistence as well to
> >>> keep the NM state.
> >>> Disk persistence won't help here (not even sure NM has much state to
> >>> persist - even if it does it should be taken care by YARN), as
> containers
> >>> have to reconnect to NM after it restarts, so they have to know RPC
> port.
> >>> Thanks,Yuliya
> >>> [1] https://github.com/apache/incubator-myriad/blob/master/
> >>> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> >>> RangeResource.java#L85
> >>> [2] https://github.com/apache/incubator-myriad/blob/master/
> >>> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> >>> RangeResource.java#L128
> >>>
> >>> [3] https://github.com/apache/incubator-myriad/bl

Re: [DISCUSS] handling roles in Myriad code

[Yuliya]

I really would like Mesosphere guys to comment here. I had a chat with Adam 
today morning and I did not get the same impression

Thanks,
Yuliya

> On Oct 19, 2016, at 8:50 PM, Darin Johnson <dbjohnson1...@gmail.com> wrote:
> 
> We use roles extensively to ensure different frameworks can (or can't) get
> resources via mechanisms such as reserved resorces and quotas.  Also if you
> don't pay attention you can miss a lot of the resources you're given.  I
> wish it was we didn't have to do all the book keeping our selves, but I
> suppose there are good reasons for delegating it to the framework, for
> instance we can choose when to fave a reserved vs a default resource.
> 
> On Wed, Oct 19, 2016 at 11:30 PM, yuliya Feldman <
> yufeld...@yahoo.com.invalid> wrote:
> 
>> I am not sure we should care about role being set or not, what if in the
>> future we will have multiple rolesNot even sure if presence/absence of role
>> should play role (no pun intended :) ).
>> 
>>  From: Darin Johnson <dbjohnson1...@gmail.com>
>> To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
>> yufeld...@yahoo.com>
>> Sent: Wednesday, October 19, 2016 7:17 PM
>> Subject: Re: [DISCUSS] handling roles in Myriad code
>> 
>> Ah so if I understand correctly, if frameworkRole='*' is present in the
>> config, it's handled as thought it's the framework role.  I believe when I
>> was testing I was using frameworkRole="test" or commenting out
>> frameworkRole="test".  It looks as though in MyriadConfiguration,
>> getFrameworkRole now returns "*" even if not set.
>> 
>> Seems like we should be able to add a check like r.hasRole() &&
>> r.getRole().equals(role)
>> && !role.equals("*") in a few places. Though it may be better
>> to pass think about a better approach here.
>> 
>> Darin
>> 
>> On Wed, Oct 19, 2016 at 9:28 PM, yuliya Feldman
>> <yufeld...@yahoo.com.invalid
>>> wrote:
>> 
>>> Hello Darrin,
>>> I kind of see the point regarding JHS ports. May be there is truth to it.
>>> Regarding my issues with role/no role.
>>> I had this issue for NMs with random ports (not hardcoded), as it has
>>> different code path when role is present and when it is not. My
>> impression
>>> those are bugs.
>>> I am happy to point you to the places in the code that caused issues on
>>> master (at least for me).[1] does not increment numDefaultValues if role
>> is
>>> set (which is always set), subsequently [2] has issues[3] same thing -
>>> fills out list only if there is no role, but again it is always there,
>> just
>>> set to "*"
>>> 
>>> 
>>> Regarding:>>> To handle nodemanager persistence I think we should work
>>> with Klaus's PR's to get thecorrect ports, though we'll need to use some
>>> disk persistence as well to
>>> keep the NM state.
>>> Disk persistence won't help here (not even sure NM has much state to
>>> persist - even if it does it should be taken care by YARN), as containers
>>> have to reconnect to NM after it restarts, so they have to know RPC port.
>>> Thanks,Yuliya
>>> [1] https://github.com/apache/incubator-myriad/blob/master/
>>> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
>>> RangeResource.java#L85
>>> [2] https://github.com/apache/incubator-myriad/blob/master/
>>> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
>>> RangeResource.java#L128
>>> 
>>> [3] https://github.com/apache/incubator-myriad/blob/master/
>>> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
>>> RangeResource.java#L140
>>> 
>>> 
>>> From: Darin Johnson <dbjohnson1...@gmail.com>
>>> To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
>>> yufeld...@yahoo.com>
>>> Sent: Wednesday, October 19, 2016 6:04 PM
>>> Subject: Re: [DISCUSS] handling roles in Myriad code
>>> 
>>> Yuyiya,
>>> 
>>> Yes on master a lot of refactoring was done, in particular you specify
>>> ports other than 0 in the myriad-default.yaml, it will only return those
>>> ports (not random ones).  This was done in part because the we were
>>> attempting the use the JHS on a port like 32001, but it the port was
>>> already in use by another app and hence the port wasn't offered myriad
>> was
>>> still launching the JHS only to have it crash.
>>>

Re: [DISCUSS] handling roles in Myriad code

[Darin Johnson]

We use roles extensively to ensure different frameworks can (or can't) get
resources via mechanisms such as reserved resorces and quotas.  Also if you
don't pay attention you can miss a lot of the resources you're given.  I
wish it was we didn't have to do all the book keeping our selves, but I
suppose there are good reasons for delegating it to the framework, for
instance we can choose when to fave a reserved vs a default resource.

On Wed, Oct 19, 2016 at 11:30 PM, yuliya Feldman <
yufeld...@yahoo.com.invalid> wrote:

> I am not sure we should care about role being set or not, what if in the
> future we will have multiple rolesNot even sure if presence/absence of role
> should play role (no pun intended :) ).
>
>   From: Darin Johnson <dbjohnson1...@gmail.com>
>  To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
> yufeld...@yahoo.com>
>  Sent: Wednesday, October 19, 2016 7:17 PM
>  Subject: Re: [DISCUSS] handling roles in Myriad code
>
> Ah so if I understand correctly, if frameworkRole='*' is present in the
> config, it's handled as thought it's the framework role.  I believe when I
> was testing I was using frameworkRole="test" or commenting out
> frameworkRole="test".  It looks as though in MyriadConfiguration,
> getFrameworkRole now returns "*" even if not set.
>
> Seems like we should be able to add a check like r.hasRole() &&
> r.getRole().equals(role)
> && !role.equals("*") in a few places. Though it may be better
> to pass think about a better approach here.
>
> Darin
>
> On Wed, Oct 19, 2016 at 9:28 PM, yuliya Feldman
> <yufeld...@yahoo.com.invalid
> > wrote:
>
> > Hello Darrin,
> > I kind of see the point regarding JHS ports. May be there is truth to it.
> > Regarding my issues with role/no role.
> > I had this issue for NMs with random ports (not hardcoded), as it has
> > different code path when role is present and when it is not. My
> impression
> > those are bugs.
> > I am happy to point you to the places in the code that caused issues on
> > master (at least for me).[1] does not increment numDefaultValues if role
> is
> > set (which is always set), subsequently [2] has issues[3] same thing -
> > fills out list only if there is no role, but again it is always there,
> just
> > set to "*"
> >
> >
> > Regarding:>>> To handle nodemanager persistence I think we should work
> > with Klaus's PR's to get thecorrect ports, though we'll need to use some
> > disk persistence as well to
> > keep the NM state.
> > Disk persistence won't help here (not even sure NM has much state to
> > persist - even if it does it should be taken care by YARN), as containers
> > have to reconnect to NM after it restarts, so they have to know RPC port.
> > Thanks,Yuliya
> > [1] https://github.com/apache/incubator-myriad/blob/master/
> > myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> > RangeResource.java#L85
> > [2] https://github.com/apache/incubator-myriad/blob/master/
> > myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> > RangeResource.java#L128
> >
> > [3] https://github.com/apache/incubator-myriad/blob/master/
> > myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> > RangeResource.java#L140
> >
> >
> >  From: Darin Johnson <dbjohnson1...@gmail.com>
> >  To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
> > yufeld...@yahoo.com>
> >  Sent: Wednesday, October 19, 2016 6:04 PM
> >  Subject: Re: [DISCUSS] handling roles in Myriad code
> >
> > Yuyiya,
> >
> > Yes on master a lot of refactoring was done, in particular you specify
> > ports other than 0 in the myriad-default.yaml, it will only return those
> > ports (not random ones).  This was done in part because the we were
> > attempting the use the JHS on a port like 32001, but it the port was
> > already in use by another app and hence the port wasn't offered myriad
> was
> > still launching the JHS only to have it crash.
> >
> > If you want to use static ports you can just not put anything in the
> > myriad-default.yaml and configure the yarn-site.xml and mapred-site.xml
> as
> > usual (they should be outside the range mesos offers).  To handle
> > nodemanager persistence I think we should work with Klaus's PR's to get
> the
> > correct ports, though we'll need to use some disk persistance as well to
> > keep the NM state.
> >
> > As for a bug in NM's getting zero ports could you send a copy of your
> > configuration and 

Re: [DISCUSS] handling roles in Myriad code

[yuliya Feldman]

I am not sure we should care about role being set or not, what if in the future 
we will have multiple rolesNot even sure if presence/absence of role should 
play role (no pun intended :) ).

  From: Darin Johnson <dbjohnson1...@gmail.com>
 To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman 
<yufeld...@yahoo.com> 
 Sent: Wednesday, October 19, 2016 7:17 PM
 Subject: Re: [DISCUSS] handling roles in Myriad code
   
Ah so if I understand correctly, if frameworkRole='*' is present in the
config, it's handled as thought it's the framework role.  I believe when I
was testing I was using frameworkRole="test" or commenting out
frameworkRole="test".  It looks as though in MyriadConfiguration,
getFrameworkRole now returns "*" even if not set.

Seems like we should be able to add a check like r.hasRole() &&
r.getRole().equals(role)
&& !role.equals("*") in a few places. Though it may be better
to pass think about a better approach here.

Darin

On Wed, Oct 19, 2016 at 9:28 PM, yuliya Feldman <yufeld...@yahoo.com.invalid
> wrote:

> Hello Darrin,
> I kind of see the point regarding JHS ports. May be there is truth to it.
> Regarding my issues with role/no role.
> I had this issue for NMs with random ports (not hardcoded), as it has
> different code path when role is present and when it is not. My impression
> those are bugs.
> I am happy to point you to the places in the code that caused issues on
> master (at least for me).[1] does not increment numDefaultValues if role is
> set (which is always set), subsequently [2] has issues[3] same thing -
> fills out list only if there is no role, but again it is always there, just
> set to "*"
>
>
> Regarding:>>> To handle nodemanager persistence I think we should work
> with Klaus's PR's to get thecorrect ports, though we'll need to use some
> disk persistence as well to
> keep the NM state.
> Disk persistence won't help here (not even sure NM has much state to
> persist - even if it does it should be taken care by YARN), as containers
> have to reconnect to NM after it restarts, so they have to know RPC port.
> Thanks,Yuliya
> [1] https://github.com/apache/incubator-myriad/blob/master/
> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> RangeResource.java#L85
> [2] https://github.com/apache/incubator-myriad/blob/master/
> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> RangeResource.java#L128
>
> [3] https://github.com/apache/incubator-myriad/blob/master/
> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> RangeResource.java#L140
>
>
>      From: Darin Johnson <dbjohnson1...@gmail.com>
>  To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
> yufeld...@yahoo.com>
>  Sent: Wednesday, October 19, 2016 6:04 PM
>  Subject: Re: [DISCUSS] handling roles in Myriad code
>
> Yuyiya,
>
> Yes on master a lot of refactoring was done, in particular you specify
> ports other than 0 in the myriad-default.yaml, it will only return those
> ports (not random ones).  This was done in part because the we were
> attempting the use the JHS on a port like 32001, but it the port was
> already in use by another app and hence the port wasn't offered myriad was
> still launching the JHS only to have it crash.
>
> If you want to use static ports you can just not put anything in the
> myriad-default.yaml and configure the yarn-site.xml and mapred-site.xml as
> usual (they should be outside the range mesos offers).  To handle
> nodemanager persistence I think we should work with Klaus's PR's to get the
> correct ports, though we'll need to use some disk persistance as well to
> keep the NM state.
>
> As for a bug in NM's getting zero ports could you send a copy of your
> configuration and I'll try to recreate the problem?
>
> On Wed, Oct 19, 2016 at 3:29 PM, yuliya Feldman
> <yufeld...@yahoo.com.invalid
> > wrote:
>
> > Hello there,
> > I wanted to discuss current handling of roles in Myriad code.
> Specifically
> > on "master" branch. Most likely due to heavy refactoring.
> > As far as I can see we try to handle presence or absence of a role on a
> > resource(s) based on the fact that framework may or may not have a
> role.On
> > the other hand we always set framework role to "*" - which means it will
> > always have a role, just that role will be "default".
> > So far I encountered couple of bugs related to roles in RangeResource
> > related to ports and inability to spin up NodeManagers, as no ports were
> > assigned because of the fact how we handle roles.
> > I would like @Adam and other Mesosphere folks to comment on how should we
> > handle relationship between frameworkRole and resource role(s)
> > Thanks,Yuliya
>
>
>
>


   

Re: [DISCUSS] handling roles in Myriad code

[Darin Johnson]

I'd really like to see if Mohit could answer how the dcos-commons library
might be able to help here.  Also can dcos commons work with vanilla mesos
or just dcos?

On Wed, Oct 19, 2016 at 9:51 PM, Klaus Ma <klaus1982...@gmail.com> wrote:

> And for the role, we also need to handle the principal with it. For
> example, it need principal to use reserved resources if necessary.
>
> 
> Da (Klaus), Ma (马达) | PMP® | Software Architect
> Platform OpenSource Technology, STG, IBM GCG
> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>
> On Thu, Oct 20, 2016 at 9:28 AM, yuliya Feldman
> <yufeld...@yahoo.com.invalid
> > wrote:
>
> > Hello Darrin,
> > I kind of see the point regarding JHS ports. May be there is truth to it.
> > Regarding my issues with role/no role.
> > I had this issue for NMs with random ports (not hardcoded), as it has
> > different code path when role is present and when it is not. My
> impression
> > those are bugs.
> > I am happy to point you to the places in the code that caused issues on
> > master (at least for me).[1] does not increment numDefaultValues if role
> is
> > set (which is always set), subsequently [2] has issues[3] same thing -
> > fills out list only if there is no role, but again it is always there,
> just
> > set to "*"
> >
> >
> > Regarding:>>> To handle nodemanager persistence I think we should work
> > with Klaus's PR's to get thecorrect ports, though we'll need to use some
> > disk persistence as well to
> > keep the NM state.
> > Disk persistence won't help here (not even sure NM has much state to
> > persist - even if it does it should be taken care by YARN), as containers
> > have to reconnect to NM after it restarts, so they have to know RPC port.
> > Thanks,Yuliya
> > [1] https://github.com/apache/incubator-myriad/blob/master/
> > myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> > RangeResource.java#L85
> > [2] https://github.com/apache/incubator-myriad/blob/master/
> > myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> > RangeResource.java#L128
> >
> > [3] https://github.com/apache/incubator-myriad/blob/master/
> > myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> > RangeResource.java#L140
> >
> >
> >   From: Darin Johnson <dbjohnson1...@gmail.com>
> >  To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
> > yufeld...@yahoo.com>
> >  Sent: Wednesday, October 19, 2016 6:04 PM
> >  Subject: Re: [DISCUSS] handling roles in Myriad code
> >
> > Yuyiya,
> >
> > Yes on master a lot of refactoring was done, in particular you specify
> > ports other than 0 in the myriad-default.yaml, it will only return those
> > ports (not random ones).  This was done in part because the we were
> > attempting the use the JHS on a port like 32001, but it the port was
> > already in use by another app and hence the port wasn't offered myriad
> was
> > still launching the JHS only to have it crash.
> >
> > If you want to use static ports you can just not put anything in the
> > myriad-default.yaml and configure the yarn-site.xml and mapred-site.xml
> as
> > usual (they should be outside the range mesos offers).  To handle
> > nodemanager persistence I think we should work with Klaus's PR's to get
> the
> > correct ports, though we'll need to use some disk persistance as well to
> > keep the NM state.
> >
> > As for a bug in NM's getting zero ports could you send a copy of your
> > configuration and I'll try to recreate the problem?
> >
> > On Wed, Oct 19, 2016 at 3:29 PM, yuliya Feldman
> > <yufeld...@yahoo.com.invalid
> > > wrote:
> >
> > > Hello there,
> > > I wanted to discuss current handling of roles in Myriad code.
> > Specifically
> > > on "master" branch. Most likely due to heavy refactoring.
> > > As far as I can see we try to handle presence or absence of a role on a
> > > resource(s) based on the fact that framework may or may not have a
> > role.On
> > > the other hand we always set framework role to "*" - which means it
> will
> > > always have a role, just that role will be "default".
> > > So far I encountered couple of bugs related to roles in RangeResource
> > > related to ports and inability to spin up NodeManagers, as no ports
> were
> > > assigned because of the fact how we handle roles.
> > > I would like @Adam and other Mesosphere folks to comment on how should
> we
> > > handle relationship between frameworkRole and resource role(s)
> > > Thanks,Yuliya
> >
> >
> >
> >
>

Re: [DISCUSS] handling roles in Myriad code

[Darin Johnson]

Yuyiya,

Yes on master a lot of refactoring was done, in particular you specify
ports other than 0 in the myriad-default.yaml, it will only return those
ports (not random ones).  This was done in part because the we were
attempting the use the JHS on a port like 32001, but it the port was
already in use by another app and hence the port wasn't offered myriad was
still launching the JHS only to have it crash.

If you want to use static ports you can just not put anything in the
myriad-default.yaml and configure the yarn-site.xml and mapred-site.xml as
usual (they should be outside the range mesos offers).  To handle
nodemanager persistence I think we should work with Klaus's PR's to get the
correct ports, though we'll need to use some disk persistance as well to
keep the NM state.

As for a bug in NM's getting zero ports could you send a copy of your
configuration and I'll try to recreate the problem?

On Wed, Oct 19, 2016 at 3:29 PM, yuliya Feldman  wrote:

> Hello there,
> I wanted to discuss current handling of roles in Myriad code. Specifically
> on "master" branch. Most likely due to heavy refactoring.
> As far as I can see we try to handle presence or absence of a role on a
> resource(s) based on the fact that framework may or may not have a role.On
> the other hand we always set framework role to "*" - which means it will
> always have a role, just that role will be "default".
> So far I encountered couple of bugs related to roles in RangeResource
> related to ports and inability to spin up NodeManagers, as no ports were
> assigned because of the fact how we handle roles.
> I would like @Adam and other Mesosphere folks to comment on how should we
> handle relationship between frameworkRole and resource role(s)
> Thanks,Yuliya