Re: Add hostname or agentid in rescind offers callback

[Varun Gupta]

Implementation is only needed for V1 API.
On Wed, May 2, 2018 at 7:31 PM Varun Gupta  wrote:

> We aggregate all the offers for a host, such that placement engine can
> pack multiple tasks that can be launched on this host using aggregated
> resources. If offers are unused for that host then they will be implicitly
> declined.
>
> Placement cycle has two components
> - determine tasks that can be launched on this host and enque those tasks
> to be launched.
> - second is to deque and launch them.
>
> During this placement cycle, offers can be rescinded and accordingly
> placement has to be adjusted. For that we maintain second map of offer id
> -> host name.
>
> Benefits of adding host name and agent id in rescind offer callback.
> - Mutex locks to synchronize both maps, leads to some performance hit.
> - Managing second map, is more code and prone to bugs.
> - Little overhead on heap memory and GC.
> On Wed, May 2, 2018 at 5:08 PM Benjamin Mahler  wrote:
>
>> I'm a -1 on adding redundant information in the message.
>>
>> The scheduler can maintain an index of offers by offer id to address this
>> issue:
>>
>> hostname -> offers
>> offer_id -> offer
>>
>> On Wed, May 2, 2018 at 11:39 AM, Vinod Kone  wrote:
>>
>> > Can I ask why you are indexing the offers by hostname? Is it to better
>> > handle agent removal / unreachable signal?
>> >
>> > Looking at the code
>> > <
>> https://github.com/apache/mesos/blob/master/src/master/master.cpp#L11036>
>> > ,
>> > I think master has the requested information (agent id, hostname) so we
>> can
>> > include it in the rescind message!
>> >
>> > But there are couple things to discuss.
>> >
>> > The extra information to be included in rescind message is technically
>> > redundant. So we need to figure out a guideline on what information
>> should
>> > be included / not included (e.g., should we include agent IP too) in
>> such
>> > calls.
>> >
>> > Second, adding this extra information in v1 scheduler API would be
>> > relatively easy. But adding this to v0 API would be hard. Which API do
>> you
>> > need to be updated?
>> >
>> >
>> > On Wed, May 2, 2018 at 10:31 AM, Varun Gupta  wrote:
>> >
>> > > Hi,
>> > >
>> > > Currently in our implementation we maintain two maps.
>> > >
>> > > Hostname -> []Offers
>> > >
>> > > offerID -> Hostname
>> > >
>> > > Second map is needed because rescind offers callback only provides
>> > offerid
>> > > and we need hostname to do performant lookup in first map.
>> > >
>> > > Is it feasible to add hostname or agentid in rescind offers?
>> > >
>> > > Thanks,
>> > > Varun
>> > >
>> >
>>
>

Re: Add hostname or agentid in rescind offers callback

[Varun Gupta]

We aggregate all the offers for a host, such that placement engine can pack
multiple tasks that can be launched on this host using aggregated
resources. If offers are unused for that host then they will be implicitly
declined.

Placement cycle has two components
- determine tasks that can be launched on this host and enque those tasks
to be launched.
- second is to deque and launch them.

During this placement cycle, offers can be rescinded and accordingly
placement has to be adjusted. For that we maintain second map of offer id
-> host name.

Benefits of adding host name and agent id in rescind offer callback.
- Mutex locks to synchronize both maps, leads to some performance hit.
- Managing second map, is more code and prone to bugs.
- Little overhead on heap memory and GC.
On Wed, May 2, 2018 at 5:08 PM Benjamin Mahler  wrote:

> I'm a -1 on adding redundant information in the message.
>
> The scheduler can maintain an index of offers by offer id to address this
> issue:
>
> hostname -> offers
> offer_id -> offer
>
> On Wed, May 2, 2018 at 11:39 AM, Vinod Kone  wrote:
>
> > Can I ask why you are indexing the offers by hostname? Is it to better
> > handle agent removal / unreachable signal?
> >
> > Looking at the code
> > <
> https://github.com/apache/mesos/blob/master/src/master/master.cpp#L11036>
> > ,
> > I think master has the requested information (agent id, hostname) so we
> can
> > include it in the rescind message!
> >
> > But there are couple things to discuss.
> >
> > The extra information to be included in rescind message is technically
> > redundant. So we need to figure out a guideline on what information
> should
> > be included / not included (e.g., should we include agent IP too) in such
> > calls.
> >
> > Second, adding this extra information in v1 scheduler API would be
> > relatively easy. But adding this to v0 API would be hard. Which API do
> you
> > need to be updated?
> >
> >
> > On Wed, May 2, 2018 at 10:31 AM, Varun Gupta  wrote:
> >
> > > Hi,
> > >
> > > Currently in our implementation we maintain two maps.
> > >
> > > Hostname -> []Offers
> > >
> > > offerID -> Hostname
> > >
> > > Second map is needed because rescind offers callback only provides
> > offerid
> > > and we need hostname to do performant lookup in first map.
> > >
> > > Is it feasible to add hostname or agentid in rescind offers?
> > >
> > > Thanks,
> > > Varun
> > >
> >
>

Re: Add hostname or agentid in rescind offers callback

[Benjamin Mahler]

I'm a -1 on adding redundant information in the message.

The scheduler can maintain an index of offers by offer id to address this
issue:

hostname -> offers
offer_id -> offer

On Wed, May 2, 2018 at 11:39 AM, Vinod Kone  wrote:

> Can I ask why you are indexing the offers by hostname? Is it to better
> handle agent removal / unreachable signal?
>
> Looking at the code
> 
> ,
> I think master has the requested information (agent id, hostname) so we can
> include it in the rescind message!
>
> But there are couple things to discuss.
>
> The extra information to be included in rescind message is technically
> redundant. So we need to figure out a guideline on what information should
> be included / not included (e.g., should we include agent IP too) in such
> calls.
>
> Second, adding this extra information in v1 scheduler API would be
> relatively easy. But adding this to v0 API would be hard. Which API do you
> need to be updated?
>
>
> On Wed, May 2, 2018 at 10:31 AM, Varun Gupta  wrote:
>
> > Hi,
> >
> > Currently in our implementation we maintain two maps.
> >
> > Hostname -> []Offers
> >
> > offerID -> Hostname
> >
> > Second map is needed because rescind offers callback only provides
> offerid
> > and we need hostname to do performant lookup in first map.
> >
> > Is it feasible to add hostname or agentid in rescind offers?
> >
> > Thanks,
> > Varun
> >
>

Re: Add hostname or agentid in rescind offers callback

[Vinod Kone]

Can I ask why you are indexing the offers by hostname? Is it to better
handle agent removal / unreachable signal?

Looking at the code
 ,
I think master has the requested information (agent id, hostname) so we can
include it in the rescind message!

But there are couple things to discuss.

The extra information to be included in rescind message is technically
redundant. So we need to figure out a guideline on what information should
be included / not included (e.g., should we include agent IP too) in such
calls.

Second, adding this extra information in v1 scheduler API would be
relatively easy. But adding this to v0 API would be hard. Which API do you
need to be updated?


On Wed, May 2, 2018 at 10:31 AM, Varun Gupta  wrote:

> Hi,
>
> Currently in our implementation we maintain two maps.
>
> Hostname -> []Offers
>
> offerID -> Hostname
>
> Second map is needed because rescind offers callback only provides offerid
> and we need hostname to do performant lookup in first map.
>
> Is it feasible to add hostname or agentid in rescind offers?
>
> Thanks,
> Varun
>

Add hostname or agentid in rescind offers callback

[Varun Gupta]

Hi,

Currently in our implementation we maintain two maps.

Hostname -> []Offers

offerID -> Hostname

Second map is needed because rescind offers callback only provides offerid
and we need hostname to do performant lookup in first map.

Is it feasible to add hostname or agentid in rescind offers?

Thanks,
Varun

[GitHub] mesos pull request #287: Add fiu to the list of contributor

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/mesos/pull/287


---