I really would like Mesosphere guys to comment here. I had a chat with Adam today morning and I did not get the same impression
Thanks, Yuliya > On Oct 19, 2016, at 8:50 PM, Darin Johnson <dbjohnson1...@gmail.com> wrote: > > We use roles extensively to ensure different frameworks can (or can't) get > resources via mechanisms such as reserved resorces and quotas. Also if you > don't pay attention you can miss a lot of the resources you're given. I > wish it was we didn't have to do all the book keeping our selves, but I > suppose there are good reasons for delegating it to the framework, for > instance we can choose when to fave a reserved vs a default resource. > > On Wed, Oct 19, 2016 at 11:30 PM, yuliya Feldman < > yufeld...@yahoo.com.invalid> wrote: > >> I am not sure we should care about role being set or not, what if in the >> future we will have multiple rolesNot even sure if presence/absence of role >> should play role (no pun intended :) ). >> >> From: Darin Johnson <dbjohnson1...@gmail.com> >> To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman < >> yufeld...@yahoo.com> >> Sent: Wednesday, October 19, 2016 7:17 PM >> Subject: Re: [DISCUSS] handling roles in Myriad code >> >> Ah so if I understand correctly, if frameworkRole='*' is present in the >> config, it's handled as thought it's the framework role. I believe when I >> was testing I was using frameworkRole="test" or commenting out >> frameworkRole="test". It looks as though in MyriadConfiguration, >> getFrameworkRole now returns "*" even if not set. >> >> Seems like we should be able to add a check like r.hasRole() && >> r.getRole().equals(role) >> && !role.equals("*") in a few places. Though it may be better >> to pass think about a better approach here. >> >> Darin >> >> On Wed, Oct 19, 2016 at 9:28 PM, yuliya Feldman >> <yufeld...@yahoo.com.invalid >>> wrote: >> >>> Hello Darrin, >>> I kind of see the point regarding JHS ports. May be there is truth to it. >>> Regarding my issues with role/no role. >>> I had this issue for NMs with random ports (not hardcoded), as it has >>> different code path when role is present and when it is not. My >> impression >>> those are bugs. >>> I am happy to point you to the places in the code that caused issues on >>> master (at least for me).[1] does not increment numDefaultValues if role >> is >>> set (which is always set), subsequently [2] has issues[3] same thing - >>> fills out list only if there is no role, but again it is always there, >> just >>> set to "*" >>> >>> >>> Regarding:>>> To handle nodemanager persistence I think we should work >>> with Klaus's PR's to get thecorrect ports, though we'll need to use some >>> disk persistence as well to >>> keep the NM state. >>> Disk persistence won't help here (not even sure NM has much state to >>> persist - even if it does it should be taken care by YARN), as containers >>> have to reconnect to NM after it restarts, so they have to know RPC port. >>> Thanks,Yuliya >>> [1] https://github.com/apache/incubator-myriad/blob/master/ >>> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/ >>> RangeResource.java#L85 >>> [2] https://github.com/apache/incubator-myriad/blob/master/ >>> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/ >>> RangeResource.java#L128 >>> >>> [3] https://github.com/apache/incubator-myriad/blob/master/ >>> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/ >>> RangeResource.java#L140 >>> >>> >>> From: Darin Johnson <dbjohnson1...@gmail.com> >>> To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman < >>> yufeld...@yahoo.com> >>> Sent: Wednesday, October 19, 2016 6:04 PM >>> Subject: Re: [DISCUSS] handling roles in Myriad code >>> >>> Yuyiya, >>> >>> Yes on master a lot of refactoring was done, in particular you specify >>> ports other than 0 in the myriad-default.yaml, it will only return those >>> ports (not random ones). This was done in part because the we were >>> attempting the use the JHS on a port like 32001, but it the port was >>> already in use by another app and hence the port wasn't offered myriad >> was >>> still launching the JHS only to have it crash. >>> >>> If you want to use static ports you can just not put anything in the >>> myriad-default.yaml and configure the yarn-site.xml and mapred-site.xml >> as >>> usual (they should be outside the range mesos offers). To handle >>> nodemanager persistence I think we should work with Klaus's PR's to get >> the >>> correct ports, though we'll need to use some disk persistance as well to >>> keep the NM state. >>> >>> As for a bug in NM's getting zero ports could you send a copy of your >>> configuration and I'll try to recreate the problem? >>> >>> On Wed, Oct 19, 2016 at 3:29 PM, yuliya Feldman >>> <yufeld...@yahoo.com.invalid >>>> wrote: >>> >>>> Hello there, >>>> I wanted to discuss current handling of roles in Myriad code. >>> Specifically >>>> on "master" branch. Most likely due to heavy refactoring. >>>> As far as I can see we try to handle presence or absence of a role on a >>>> resource(s) based on the fact that framework may or may not have a >>> role.On >>>> the other hand we always set framework role to "*" - which means it >> will >>>> always have a role, just that role will be "default". >>>> So far I encountered couple of bugs related to roles in RangeResource >>>> related to ports and inability to spin up NodeManagers, as no ports >> were >>>> assigned because of the fact how we handle roles. >>>> I would like @Adam and other Mesosphere folks to comment on how should >> we >>>> handle relationship between frameworkRole and resource role(s) >>>> Thanks,Yuliya >> >> >> >>