I really would like Mesosphere guys to comment here. I had a chat with Adam
today morning and I did not get the same impression
Thanks,
Yuliya
> On Oct 19, 2016, at 8:50 PM, Darin Johnson <dbjohnson1...@gmail.com> wrote:
>
> We use roles extensively to ensure different frameworks can (or can't) get
> resources via mechanisms such as reserved resorces and quotas. Also if you
> don't pay attention you can miss a lot of the resources you're given. I
> wish it was we didn't have to do all the book keeping our selves, but I
> suppose there are good reasons for delegating it to the framework, for
> instance we can choose when to fave a reserved vs a default resource.
>
> On Wed, Oct 19, 2016 at 11:30 PM, yuliya Feldman <
> yufeld...@yahoo.com.invalid> wrote:
>
>> I am not sure we should care about role being set or not, what if in the
>> future we will have multiple rolesNot even sure if presence/absence of role
>> should play role (no pun intended :) ).
>>
>> From: Darin Johnson <dbjohnson1...@gmail.com>
>> To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
>> yufeld...@yahoo.com>
>> Sent: Wednesday, October 19, 2016 7:17 PM
>> Subject: Re: [DISCUSS] handling roles in Myriad code
>>
>> Ah so if I understand correctly, if frameworkRole='*' is present in the
>> config, it's handled as thought it's the framework role. I believe when I
>> was testing I was using frameworkRole="test" or commenting out
>> frameworkRole="test". It looks as though in MyriadConfiguration,
>> getFrameworkRole now returns "*" even if not set.
>>
>> Seems like we should be able to add a check like r.hasRole() &&
>> r.getRole().equals(role)
>> && !role.equals("*") in a few places. Though it may be better
>> to pass think about a better approach here.
>>
>> Darin
>>
>> On Wed, Oct 19, 2016 at 9:28 PM, yuliya Feldman
>> <yufeld...@yahoo.com.invalid
>>> wrote:
>>
>>> Hello Darrin,
>>> I kind of see the point regarding JHS ports. May be there is truth to it.
>>> Regarding my issues with role/no role.
>>> I had this issue for NMs with random ports (not hardcoded), as it has
>>> different code path when role is present and when it is not. My
>> impression
>>> those are bugs.
>>> I am happy to point you to the places in the code that caused issues on
>>> master (at least for me).[1] does not increment numDefaultValues if role
>> is
>>> set (which is always set), subsequently [2] has issues[3] same thing -
>>> fills out list only if there is no role, but again it is always there,
>> just
>>> set to "*"
>>>
>>>
>>> Regarding:>>> To handle nodemanager persistence I think we should work
>>> with Klaus's PR's to get thecorrect ports, though we'll need to use some
>>> disk persistence as well to
>>> keep the NM state.
>>> Disk persistence won't help here (not even sure NM has much state to
>>> persist - even if it does it should be taken care by YARN), as containers
>>> have to reconnect to NM after it restarts, so they have to know RPC port.
>>> Thanks,Yuliya
>>> [1] https://github.com/apache/incubator-myriad/blob/master/
>>> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
>>> RangeResource.java#L85
>>> [2] https://github.com/apache/incubator-myriad/blob/master/
>>> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
>>> RangeResource.java#L128
>>>
>>> [3] https://github.com/apache/incubator-myriad/blob/master/
>>> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
>>> RangeResource.java#L140
>>>
>>>
>>> From: Darin Johnson <dbjohnson1...@gmail.com>
>>> To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
>>> yufeld...@yahoo.com>
>>> Sent: Wednesday, October 19, 2016 6:04 PM
>>> Subject: Re: [DISCUSS] handling roles in Myriad code
>>>
>>> Yuyiya,
>>>
>>> Yes on master a lot of refactoring was done, in particular you specify
>>> ports other than 0 in the myriad-default.yaml, it will only return those
>>> ports (not random ones). This was done in part because the we were
>>> attempting the use the JHS on a port like 32001, but it the port was
>>> already in use by another app and hence the port wasn't offered myriad
>> was
>>> still launching the JHS only to have it crash.
>>>
>>> If you want to use static ports you can just not put anything in the
>>> myriad-default.yaml and configure the yarn-site.xml and mapred-site.xml
>> as
>>> usual (they should be outside the range mesos offers). To handle
>>> nodemanager persistence I think we should work with Klaus's PR's to get
>> the
>>> correct ports, though we'll need to use some disk persistance as well to
>>> keep the NM state.
>>>
>>> As for a bug in NM's getting zero ports could you send a copy of your
>>> configuration and I'll try to recreate the problem?
>>>
>>> On Wed, Oct 19, 2016 at 3:29 PM, yuliya Feldman
>>> <yufeld...@yahoo.com.invalid
>>>> wrote:
>>>
>>>> Hello there,
>>>> I wanted to discuss current handling of roles in Myriad code.
>>> Specifically
>>>> on "master" branch. Most likely due to heavy refactoring.
>>>> As far as I can see we try to handle presence or absence of a role on a
>>>> resource(s) based on the fact that framework may or may not have a
>>> role.On
>>>> the other hand we always set framework role to "*" - which means it
>> will
>>>> always have a role, just that role will be "default".
>>>> So far I encountered couple of bugs related to roles in RangeResource
>>>> related to ports and inability to spin up NodeManagers, as no ports
>> were
>>>> assigned because of the fact how we handle roles.
>>>> I would like @Adam and other Mesosphere folks to comment on how should
>> we
>>>> handle relationship between frameworkRole and resource role(s)
>>>> Thanks,Yuliya
>>
>>
>>
>>
