Re: [openstack-dev] [oslo] New and next-gen libraries (a BCN followup)

[John Dickinson]

On 4 Nov 2016, at 7:50, Jim Rollenhagen wrote:

> On Thu, Nov 3, 2016 at 3:04 PM, Joshua Harlow  wrote:
>> Jay Faulkner wrote:

 On Nov 3, 2016, at 11:27 AM, Joshua Harlow  wrote:

 Just as a followup from the summit,

 One of the sessions (the new lib one) had a few proposals:

 https://etherpad.openstack.org/p/ocata-oslo-bring-ideas

 And I wanted to try to get clear owners for each part (there was some
 followup work for each); so just wanted to start this email to get the
 thoughts going on what to do for next steps.

 *A hash ring library*

 So this one it feels like we need at least a tiny oslo-spec for and for
 someone to write down the various implementations, what they share, what
 they do not share (talking to swift, nova, ironic and others? to figure 
 this
 out). I think alexis was thinking he might want to work through some of 
 that
 but I'll leave it for him to chime in on that (or others feel free to 
 also).

 This one doesn't seem very controversial and the majority of the work is
 probably on doing some analysis of what exists and then picking a library
 name and coding that up, testing it, and then integrating (pretty 
 standard).

>>>
>>> Ironic and Nova both share a hash ring implementation currently
>>> (ironic-conductor and nova-compute driver for ironic). It would be sensible
>>> to reuse this implementation, oslo-ify it, and have that code shared.
>>>
>>> I question the value of re-implementing something like this from scratch
>>> though.
>>>
>>> Thanks,
>>> Jay Faulkner
>>> OSIC
>>>
>>
>> Right I don't think the intention would be to implement it from scratch, but
>> to do some basic analysis of what exists (and think about and document the
>> patterns), and try to find the common parts (which likely involves renaming
>> some specific nova/ironic methods from what I see); especially if we can get
>> swift to perhaps (TBD) also use and contribute to this library.
>
> As the person who copied that code into Nova, the Nova code is a strict subset
> of the Ironic code.
>
> Some of us talked to John Dickinson off-list, and it seems the Swift hash ring
> has very different use cases and very different implementation. I
> think we should
> focus on pulling the Nova/Ironic code out first, and then talking to
> Swift if we can
> also make it work for them (sounds like it's not helpful today).
>
> // jim


We had some great conversations last week face to face about this. The summary 
is that the "ring" in Ironic/Nova and the placement "ring" in Swift are vastly 
different in scope, requirements, and capabilities. I don't think it makes 
sense to try to unify them at this time.

As always, I'm available to talk further about this, if you want.


--John







signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] New and next-gen libraries (a BCN followup)

[Doug Hellmann]

Excerpts from Jim Rollenhagen's message of 2016-11-04 10:50:34 -0400:
> On Thu, Nov 3, 2016 at 3:04 PM, Joshua Harlow  wrote:
> > Jay Faulkner wrote:
> >>>
> >>> On Nov 3, 2016, at 11:27 AM, Joshua Harlow  wrote:
> >>>
> >>> Just as a followup from the summit,
> >>>
> >>> One of the sessions (the new lib one) had a few proposals:
> >>>
> >>> https://etherpad.openstack.org/p/ocata-oslo-bring-ideas
> >>>
> >>> And I wanted to try to get clear owners for each part (there was some
> >>> followup work for each); so just wanted to start this email to get the
> >>> thoughts going on what to do for next steps.
> >>>
> >>> *A hash ring library*
> >>>
> >>> So this one it feels like we need at least a tiny oslo-spec for and for
> >>> someone to write down the various implementations, what they share, what
> >>> they do not share (talking to swift, nova, ironic and others? to figure 
> >>> this
> >>> out). I think alexis was thinking he might want to work through some of 
> >>> that
> >>> but I'll leave it for him to chime in on that (or others feel free to 
> >>> also).
> >>>
> >>> This one doesn't seem very controversial and the majority of the work is
> >>> probably on doing some analysis of what exists and then picking a library
> >>> name and coding that up, testing it, and then integrating (pretty 
> >>> standard).
> >>>
> >>
> >> Ironic and Nova both share a hash ring implementation currently
> >> (ironic-conductor and nova-compute driver for ironic). It would be sensible
> >> to reuse this implementation, oslo-ify it, and have that code shared.
> >>
> >> I question the value of re-implementing something like this from scratch
> >> though.
> >>
> >> Thanks,
> >> Jay Faulkner
> >> OSIC
> >>
> >
> > Right I don't think the intention would be to implement it from scratch, but
> > to do some basic analysis of what exists (and think about and document the
> > patterns), and try to find the common parts (which likely involves renaming
> > some specific nova/ironic methods from what I see); especially if we can get
> > swift to perhaps (TBD) also use and contribute to this library.
> 
> As the person who copied that code into Nova, the Nova code is a strict subset
> of the Ironic code.
> 
> Some of us talked to John Dickinson off-list, and it seems the Swift hash ring
> has very different use cases and very different implementation. I
> think we should
> focus on pulling the Nova/Ironic code out first, and then talking to
> Swift if we can
> also make it work for them (sounds like it's not helpful today).
> 
> // jim
> 

Thanks for the background, Jim. I thought the code shared more with
Swift's implementation than it sounds like it does, so I agree with your
proposed plan.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] New and next-gen libraries (a BCN followup)

[Jim Rollenhagen]

On Thu, Nov 3, 2016 at 3:04 PM, Joshua Harlow  wrote:
> Jay Faulkner wrote:
>>>
>>> On Nov 3, 2016, at 11:27 AM, Joshua Harlow  wrote:
>>>
>>> Just as a followup from the summit,
>>>
>>> One of the sessions (the new lib one) had a few proposals:
>>>
>>> https://etherpad.openstack.org/p/ocata-oslo-bring-ideas
>>>
>>> And I wanted to try to get clear owners for each part (there was some
>>> followup work for each); so just wanted to start this email to get the
>>> thoughts going on what to do for next steps.
>>>
>>> *A hash ring library*
>>>
>>> So this one it feels like we need at least a tiny oslo-spec for and for
>>> someone to write down the various implementations, what they share, what
>>> they do not share (talking to swift, nova, ironic and others? to figure this
>>> out). I think alexis was thinking he might want to work through some of that
>>> but I'll leave it for him to chime in on that (or others feel free to also).
>>>
>>> This one doesn't seem very controversial and the majority of the work is
>>> probably on doing some analysis of what exists and then picking a library
>>> name and coding that up, testing it, and then integrating (pretty standard).
>>>
>>
>> Ironic and Nova both share a hash ring implementation currently
>> (ironic-conductor and nova-compute driver for ironic). It would be sensible
>> to reuse this implementation, oslo-ify it, and have that code shared.
>>
>> I question the value of re-implementing something like this from scratch
>> though.
>>
>> Thanks,
>> Jay Faulkner
>> OSIC
>>
>
> Right I don't think the intention would be to implement it from scratch, but
> to do some basic analysis of what exists (and think about and document the
> patterns), and try to find the common parts (which likely involves renaming
> some specific nova/ironic methods from what I see); especially if we can get
> swift to perhaps (TBD) also use and contribute to this library.

As the person who copied that code into Nova, the Nova code is a strict subset
of the Ironic code.

Some of us talked to John Dickinson off-list, and it seems the Swift hash ring
has very different use cases and very different implementation. I
think we should
focus on pulling the Nova/Ironic code out first, and then talking to
Swift if we can
also make it work for them (sounds like it's not helpful today).

// jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] New and next-gen libraries (a BCN followup)

[Mehdi Abaakouk]

Le 2016-11-03 19:27, Joshua Harlow a écrit :

*Next-gen oslo.service replacement*

This one may require a little more of a plan on how to make it work,
but the gist is that medhi (and others) has created
https://github.com/sileht/cotyledon which is a nice replacement for
oslo.service that ceilometer is using (and others?) and the idea was
to start to figure out how to move away from (or replace with?)
olso.service with that library.


If people are interested, the doc is here [0]


I'd like to see a spec with some of the details/thoughts on how that
could be possible, what changes would still be needed. I think from
that session that the following questions were raised:

- Can multiprocessing (or subprocess?) be used (instead of os.fork)
- What to do about windows?


I have already solved those two by using multiprocessing, disable 
SIGHUP and write down the limitation [1]


- Is it possible to create a oslo.service compat layer that preserves 
the oslo.service API but uses cotyledon under the covers to smooth the 
transition/adoption of other projects to cotyledon


Not sure it's easy to do, cotyledon API encourages user to not create 
any python objects before the process is forked to ensure we didn't have 
any rpc/mysql/... connections open and unused, FDs open, lock acquired, 
(put any thing here that can result in race when using os.fork()). While 
oslo.service forces the user to create objects before the fork occurs.


- Perhaps in general we should write how an adoption could happen for a 
consuming project (maybe just writing down how ceilometer made the 
switch would be a good start, what issues were encountered, how they 
were resolved...)


This is avialable here [2]

[0] http://cotyledon.readthedocs.io/en/latest/
[1] http://cotyledon.readthedocs.io/en/latest/non-posix-support.html
[2] http://cotyledon.readthedocs.io/en/latest/oslo-service-migration.html
--
Mehdi Abaakouk
mail: sil...@sileht.net
:w
irc: sileht

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] New and next-gen libraries (a BCN followup)

[Joshua Harlow]

Jay Faulkner wrote:

On Nov 3, 2016, at 11:27 AM, Joshua Harlow  wrote:

Just as a followup from the summit,

One of the sessions (the new lib one) had a few proposals:

https://etherpad.openstack.org/p/ocata-oslo-bring-ideas

And I wanted to try to get clear owners for each part (there was some followup 
work for each); so just wanted to start this email to get the thoughts going on 
what to do for next steps.

*A hash ring library*

So this one it feels like we need at least a tiny oslo-spec for and for someone 
to write down the various implementations, what they share, what they do not 
share (talking to swift, nova, ironic and others? to figure this out). I think 
alexis was thinking he might want to work through some of that but I'll leave 
it for him to chime in on that (or others feel free to also).

This one doesn't seem very controversial and the majority of the work is 
probably on doing some analysis of what exists and then picking a library name 
and coding that up, testing it, and then integrating (pretty standard).



Ironic and Nova both share a hash ring implementation currently 
(ironic-conductor and nova-compute driver for ironic). It would be sensible to 
reuse this implementation, oslo-ify it, and have that code shared.

I question the value of re-implementing something like this from scratch though.

Thanks,
Jay Faulkner
OSIC



Right I don't think the intention would be to implement it from scratch, 
but to do some basic analysis of what exists (and think about and 
document the patterns), and try to find the common parts (which likely 
involves renaming some specific nova/ironic methods from what I see); 
especially if we can get swift to perhaps (TBD) also use and contribute 
to this library.


-Josh

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] New and next-gen libraries (a BCN followup)

[Jay Faulkner]

> On Nov 3, 2016, at 11:27 AM, Joshua Harlow  wrote:
> 
> Just as a followup from the summit,
> 
> One of the sessions (the new lib one) had a few proposals:
> 
> https://etherpad.openstack.org/p/ocata-oslo-bring-ideas
> 
> And I wanted to try to get clear owners for each part (there was some 
> followup work for each); so just wanted to start this email to get the 
> thoughts going on what to do for next steps.
> 
> *A hash ring library*
> 
> So this one it feels like we need at least a tiny oslo-spec for and for 
> someone to write down the various implementations, what they share, what they 
> do not share (talking to swift, nova, ironic and others? to figure this out). 
> I think alexis was thinking he might want to work through some of that but 
> I'll leave it for him to chime in on that (or others feel free to also).
> 
> This one doesn't seem very controversial and the majority of the work is 
> probably on doing some analysis of what exists and then picking a library 
> name and coding that up, testing it, and then integrating (pretty standard).
> 

Ironic and Nova both share a hash ring implementation currently 
(ironic-conductor and nova-compute driver for ironic). It would be sensible to 
reuse this implementation, oslo-ify it, and have that code shared. 

I question the value of re-implementing something like this from scratch though.

Thanks,
Jay Faulkner
OSIC

> *Failure capturing/formation/(de)serialization library*
> 
> This one I've just decided to push through, though more comments on the spec 
> are always welcome @ https://review.openstack.org/#/c/229194/ the repo where 
> I started just doing this work (while in the airports traveling back) is @ 
> https://github.com/harlowja/failure
> 
> Ideally once that is in a slightly better state we should be able to start to 
> converge the various (at least 3 similar kinds of implementations) to that 
> one and ideally get less duplicated (or slightly same code) out of the 
> various libraries and projects that have copied/recreated it.
> 
> Anyone desiring to help in that is more than welcome to jump in :)
> 
> *Next-gen oslo.service replacement*
> 
> This one may require a little more of a plan on how to make it work, but the 
> gist is that medhi (and others) has created 
> https://github.com/sileht/cotyledon which is a nice replacement for 
> oslo.service that ceilometer is using (and others?) and the idea was to start 
> to figure out how to move away from (or replace with?) olso.service with that 
> library.
> 
> I'd like to see a spec with some of the details/thoughts on how that could be 
> possible, what changes would still be needed. I think from that session that 
> the following questions were raised:
> 
> - Can multiprocessing (or subprocess?) be used (instead of os.fork)
> - What to do about windows?
> - Is it possible to create a oslo.service compat layer that preserves the 
> oslo.service API but uses cotyledon under the covers to smooth the 
> transition/adoption of other projects to cotyledon
>   - Perhaps in general we should write how an adoption could happen for a 
> consuming project (maybe just writing down how ceilometer made the switch 
> would be a good start, what issues were encountered, how they were 
> resolved...)
> - Something else that people forgot to write down in the etherpad here :-P
> 
> I think that was the majority of thoughts coming out of that session (there 
> were a few others, but those were not especially loud and may have just been 
> me rambling, ha). Anything I forgot feel free to add in :)
> 
> Whose in to make the above happen?!?!
> 
> -Josh
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev