Hello Darrin, I kind of see the point regarding JHS ports. May be there is truth to it. Regarding my issues with role/no role. I had this issue for NMs with random ports (not hardcoded), as it has different code path when role is present and when it is not. My impression those are bugs. I am happy to point you to the places in the code that caused issues on master (at least for me).[1] does not increment numDefaultValues if role is set (which is always set), subsequently [2] has issues[3] same thing - fills out list only if there is no role, but again it is always there, just set to "*"
Regarding:>>> To handle nodemanager persistence I think we should work with Klaus's PR's to get thecorrect ports, though we'll need to use some disk persistence as well to keep the NM state. Disk persistence won't help here (not even sure NM has much state to persist - even if it does it should be taken care by YARN), as containers have to reconnect to NM after it restarts, so they have to know RPC port. Thanks,Yuliya [1] https://github.com/apache/incubator-myriad/blob/master/myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/RangeResource.java#L85 [2] https://github.com/apache/incubator-myriad/blob/master/myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/RangeResource.java#L128 [3] https://github.com/apache/incubator-myriad/blob/master/myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/RangeResource.java#L140 From: Darin Johnson <dbjohnson1...@gmail.com> To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <yufeld...@yahoo.com> Sent: Wednesday, October 19, 2016 6:04 PM Subject: Re: [DISCUSS] handling roles in Myriad code Yuyiya, Yes on master a lot of refactoring was done, in particular you specify ports other than 0 in the myriad-default.yaml, it will only return those ports (not random ones). This was done in part because the we were attempting the use the JHS on a port like 32001, but it the port was already in use by another app and hence the port wasn't offered myriad was still launching the JHS only to have it crash. If you want to use static ports you can just not put anything in the myriad-default.yaml and configure the yarn-site.xml and mapred-site.xml as usual (they should be outside the range mesos offers). To handle nodemanager persistence I think we should work with Klaus's PR's to get the correct ports, though we'll need to use some disk persistance as well to keep the NM state. As for a bug in NM's getting zero ports could you send a copy of your configuration and I'll try to recreate the problem? On Wed, Oct 19, 2016 at 3:29 PM, yuliya Feldman <yufeld...@yahoo.com.invalid > wrote: > Hello there, > I wanted to discuss current handling of roles in Myriad code. Specifically > on "master" branch. Most likely due to heavy refactoring. > As far as I can see we try to handle presence or absence of a role on a > resource(s) based on the fact that framework may or may not have a role.On > the other hand we always set framework role to "*" - which means it will > always have a role, just that role will be "default". > So far I encountered couple of bugs related to roles in RangeResource > related to ports and inability to spin up NodeManagers, as no ports were > assigned because of the fact how we handle roles. > I would like @Adam and other Mesosphere folks to comment on how should we > handle relationship between frameworkRole and resource role(s) > Thanks,Yuliya
