We use roles extensively to ensure different frameworks can (or can't) get
resources via mechanisms such as reserved resorces and quotas. Also if you
don't pay attention you can miss a lot of the resources you're given. I
wish it was we didn't have to do all the book keeping our selves, but I
suppose there are good reasons for delegating it to the framework, for
instance we can choose when to fave a reserved vs a default resource.
On Wed, Oct 19, 2016 at 11:30 PM, yuliya Feldman <
yufeld...@yahoo.com.invalid> wrote:
> I am not sure we should care about role being set or not, what if in the
> future we will have multiple rolesNot even sure if presence/absence of role
> should play role (no pun intended :) ).
>
> From: Darin Johnson <dbjohnson1...@gmail.com>
> To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
> yufeld...@yahoo.com>
> Sent: Wednesday, October 19, 2016 7:17 PM
> Subject: Re: [DISCUSS] handling roles in Myriad code
>
> Ah so if I understand correctly, if frameworkRole='*' is present in the
> config, it's handled as thought it's the framework role. I believe when I
> was testing I was using frameworkRole="test" or commenting out
> frameworkRole="test". It looks as though in MyriadConfiguration,
> getFrameworkRole now returns "*" even if not set.
>
> Seems like we should be able to add a check like r.hasRole() &&
> r.getRole().equals(role)
> && !role.equals("*") in a few places. Though it may be better
> to pass think about a better approach here.
>
> Darin
>
> On Wed, Oct 19, 2016 at 9:28 PM, yuliya Feldman
> <yufeld...@yahoo.com.invalid
> > wrote:
>
> > Hello Darrin,
> > I kind of see the point regarding JHS ports. May be there is truth to it.
> > Regarding my issues with role/no role.
> > I had this issue for NMs with random ports (not hardcoded), as it has
> > different code path when role is present and when it is not. My
> impression
> > those are bugs.
> > I am happy to point you to the places in the code that caused issues on
> > master (at least for me).[1] does not increment numDefaultValues if role
> is
> > set (which is always set), subsequently [2] has issues[3] same thing -
> > fills out list only if there is no role, but again it is always there,
> just
> > set to "*"
> >
> >
> > Regarding:>>> To handle nodemanager persistence I think we should work
> > with Klaus's PR's to get thecorrect ports, though we'll need to use some
> > disk persistence as well to
> > keep the NM state.
> > Disk persistence won't help here (not even sure NM has much state to
> > persist - even if it does it should be taken care by YARN), as containers
> > have to reconnect to NM after it restarts, so they have to know RPC port.
> > Thanks,Yuliya
> > [1] https://github.com/apache/incubator-myriad/blob/master/
> > myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> > RangeResource.java#L85
> > [2] https://github.com/apache/incubator-myriad/blob/master/
> > myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> > RangeResource.java#L128
> >
> > [3] https://github.com/apache/incubator-myriad/blob/master/
> > myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> > RangeResource.java#L140
> >
> >
> > From: Darin Johnson <dbjohnson1...@gmail.com>
> > To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
> > yufeld...@yahoo.com>
> > Sent: Wednesday, October 19, 2016 6:04 PM
> > Subject: Re: [DISCUSS] handling roles in Myriad code
> >
> > Yuyiya,
> >
> > Yes on master a lot of refactoring was done, in particular you specify
> > ports other than 0 in the myriad-default.yaml, it will only return those
> > ports (not random ones). This was done in part because the we were
> > attempting the use the JHS on a port like 32001, but it the port was
> > already in use by another app and hence the port wasn't offered myriad
> was
> > still launching the JHS only to have it crash.
> >
> > If you want to use static ports you can just not put anything in the
> > myriad-default.yaml and configure the yarn-site.xml and mapred-site.xml
> as
> > usual (they should be outside the range mesos offers). To handle
> > nodemanager persistence I think we should work with Klaus's PR's to get
> the
> > correct ports, though we'll need to use some disk persistance as well to
> > keep the NM state.
> >
> > As for a bug in NM's getting zero ports could you send a copy of your
> > configuration and I'll try to recreate the problem?
> >
> > On Wed, Oct 19, 2016 at 3:29 PM, yuliya Feldman
> > <yufeld...@yahoo.com.invalid
> > > wrote:
> >
> > > Hello there,
> > > I wanted to discuss current handling of roles in Myriad code.
> > Specifically
> > > on "master" branch. Most likely due to heavy refactoring.
> > > As far as I can see we try to handle presence or absence of a role on a
> > > resource(s) based on the fact that framework may or may not have a
> > role.On
> > > the other hand we always set framework role to "*" - which means it
> will
> > > always have a role, just that role will be "default".
> > > So far I encountered couple of bugs related to roles in RangeResource
> > > related to ports and inability to spin up NodeManagers, as no ports
> were
> > > assigned because of the fact how we handle roles.
> > > I would like @Adam and other Mesosphere folks to comment on how should
> we
> > > handle relationship between frameworkRole and resource role(s)
> > > Thanks,Yuliya
> >
> >
> >
> >
>
>
>
>
