Re: [openstack-dev] [oslo] zeromq work for kilo

[Doug Hellmann]

On Oct 12, 2014, at 12:32 AM, Li Ma  wrote:

> 
> On 2014/9/17 22:34, Doug Hellmann wrote:
>> 
>> The documentation in the oslo.messaging repository [2] would be a good place 
>> to start for that. If we decide deployers/operators need the information we 
>> can either refer to it from the guides managed by the documentation team or 
>> we can move/copy the information. How about if you start a new drivers 
>> subdirectory there, and add information about zmq. We can have other driver 
>> authors provide similar detail about their drivers in the same directory.
>> 
>> [2] http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source
> Hi all, I wrote a deployment guide of ZeroMQ for oslo.messaging, which is 
> located at
> 
> https://github.com/li-ma/zmq-for-oslo
> 
> Do I need to issue a bug or propose a blueprint to trace and merge it to 
> oslo.messaging doc tree?

A wishlist bug would be a good idea for tracking and to ensure that reviews are 
prioritized.

> 
>>> 3) an analysis of what it would take to be able to run functional
>>> tests for zeromq on our CI infrastructure, not necessarily the full
>>> tempest run or devstack-gate job, probably functional tests we place
>>> in the tree with the driver (we will be doing this for all of the
>>> drivers) + besides writing new functional tests, we need to bring the
>>> unit tests for zeromq into the oslo.messaging repository
>>> 
>>> Kapil Thangavelu started work on both functional tests for the ZMQ
>>> driver last week; the output from the sprint is here:
>>> 
>>>https://github.com/ostack-musketeers/oslo.messaging
>>> 
>>> it covers the ZMQ driver (including messaging through the zmq-receiver
>>> proxy) and the associated MatchMakers (local, ring, redis) at a
>>> varying levels of coverage, but I feel it moves things in the right
>>> direction - Kapil's going to raise a review for this in the next
>>> couple of days.
>>> 
>>> Doug - has any structure been agreed within the oslo.messaging tree
>>> for unit/functional test splits? Right now we have them all in one place.
>> I think we will want them split up, but we don’t have an agreed existing 
>> structure for that. I would like to see a test framework of some sort that 
>> defines the tests in a way that can be used to run the same functional for 
>> all of the drivers as separate jobs (with appropriate hooks for ensuring the 
>> needed services are running, etc.). Setting that up warrants its own spec, 
>> because there are going to be quite a few details to work out. We will also 
>> need to participate in the larger conversation about how to set up those 
>> functional test jobs to be consistent with the other projects.
> That's good to hear someone working on the test stuff. I suggest to deal with 
> the unit tests first for ZeroMQ.
> 
> Anyway, are there any sessions related to this topic in the summit?

We are still working on summit planning. Based on our meeting last week, it 
looks like we will have a session combining several topics related to 
oslo.messaging, including what to do with zeromq. The full list of topics and 
notes is in the etherpad we’re using for planning: 
https://etherpad.openstack.org/p/kilo-oslo-summit-topics

Doug

> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] zeromq work for kilo

[Li Ma]

On 2014/9/17 22:34, Doug Hellmann wrote:


The documentation in the oslo.messaging repository [2] would be a good place to 
start for that. If we decide deployers/operators need the information we can 
either refer to it from the guides managed by the documentation team or we can 
move/copy the information. How about if you start a new drivers subdirectory 
there, and add information about zmq. We can have other driver authors provide 
similar detail about their drivers in the same directory.

[2] http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source
Hi all, I wrote a deployment guide of ZeroMQ for oslo.messaging, which 
is located at


https://github.com/li-ma/zmq-for-oslo

Do I need to issue a bug or propose a blueprint to trace and merge it to 
oslo.messaging doc tree?



3) an analysis of what it would take to be able to run functional
tests for zeromq on our CI infrastructure, not necessarily the full
tempest run or devstack-gate job, probably functional tests we place
in the tree with the driver (we will be doing this for all of the
drivers) + besides writing new functional tests, we need to bring the
unit tests for zeromq into the oslo.messaging repository

Kapil Thangavelu started work on both functional tests for the ZMQ
driver last week; the output from the sprint is here:

https://github.com/ostack-musketeers/oslo.messaging

it covers the ZMQ driver (including messaging through the zmq-receiver
proxy) and the associated MatchMakers (local, ring, redis) at a
varying levels of coverage, but I feel it moves things in the right
direction - Kapil's going to raise a review for this in the next
couple of days.

Doug - has any structure been agreed within the oslo.messaging tree
for unit/functional test splits? Right now we have them all in one place.

I think we will want them split up, but we don’t have an agreed existing 
structure for that. I would like to see a test framework of some sort that 
defines the tests in a way that can be used to run the same functional for all 
of the drivers as separate jobs (with appropriate hooks for ensuring the needed 
services are running, etc.). Setting that up warrants its own spec, because 
there are going to be quite a few details to work out. We will also need to 
participate in the larger conversation about how to set up those functional 
test jobs to be consistent with the other projects.
That's good to hear someone working on the test stuff. I suggest to deal 
with the unit tests first for ZeroMQ.


Anyway, are there any sessions related to this topic in the summit?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] zeromq work for kilo

[Li Ma]

Hi all, I almost lost this new thread for discussing with ZeroMQ issue.

On 2014/9/19 5:29, Eric Windisch wrote:


I believe it makes good sense for all drivers, in the long term. 
However, the most immediate benefits would be in offloading any 
drivers that need substantial work or improvements, aka velocity. That 
would mean the AMQP and ZeroMQ drivers.


It's very interesting. If we separate all the drivers out of the main 
framework, we can have different review groups working on sub-projects 
and people can work with the sub-group according to their specialty and 
preference. It can greatly improve the quality of reviewing and speed up 
the process as well.


Another thing I'll note is that before pulling Ironic in, Nova had an 
API contract test. This can be useful for making sure that changes in 
the upstream project doesn't break drivers, or that breakages could at 
least invoke action by the driver team: 
https://github.com/openstack/nova/blob/4ce3f55d169290015063131134f93fca236807ed/nova/tests/virt/test_ironic_api_contracts.py



Yes, it is necessary to make API consistent.

Cheers,
Li Ma


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] zeromq work for kilo

[Doug Hellmann]

On Sep 19, 2014, at 10:34 AM, Ken Giusti  wrote:

> On Mon Sep 8 15:18:35 UTC 2014, Doug Hellmann wrote:
>> On Sep 8, 2014, at 10:35 AM, Antonio Messina > gmail.com> wrote:
>> 
>>> Hi All,
>>> 
>>> We tested briefly ZeroMQ with Havana last year, but we couldn't find
>>> any good documentation on how to implement it, and we were not able to
>>> get it working. We also got the impression that the support was not at
>>> all mature, so we decided to use RabbitMQ instead.
>>> 
>>> However, I must say that the broker-less design of ZeroMQ is very
>>> appealing, and we would like to give it a try, assuming
>>> 1) the documentation is improved
>>> 2) there is some assurance that support for ZeroMQ is not going to be 
>>> dropped.
>>> 
>>> I can help with 1) if there is someone that knows a bit of the
>>> internals and can bootstrap me, because I have no first hand
>>> experience on how message queues are used in OpenStack, and little
>>> experience with ZeroMQ.
>> 
>> Unfortunately, the existing Oslo team doesn’t have a lot of
>> experience with ZeroMQ either (hence this thread). It sounds like Li
>> Ma’s team has made it work, though, so maybe you could work
>> together. We should prioritize documentation and then functional
>> testing, I think.
>> 
>> About 2), well this is a decision for the developers, but IMHO there
>> *should* be support for ZeroMQ in OpenStack: its broker-less
>> architecture would eliminate a SPoF (the message broker), could ease
>> the deployment (especially in HA setup) and grant very high
>> performance.
>> 
>> I agree, it would be useful to support it. This is purely a resource
>> allocation problem for me. I don't have anyone willing to do the work
>> needed to ensure the driver is functional and can be deployed sanely
>> (although maybe I’ve found a couple of volunteers now :-).
>> 
>> There is another effort going on to support AMQP 1.0, which (as I
>> understand it) includes similar broker-less deployment options. Before
>> we decide whether to invest in ZeroMQ for that reason alone, it would
>> be useful to know if AMQP 1.0 support makes potential ZeroMQ support
>> less interesting.
>> 
> 
> While the AMQP 1.0 protocol permits it, the current implementation of
> the new driver does not support broker-less point-to-point - yet.
> 
> I'm planning on adding that support to the AMQP 1.0 driver in the
> future.  I have yet to spend any time ramping up on the existing
> brokerless support implemented by the zmq driver, so forgive my
> ignorance, but I'm hoping to leverage what is there if it makes sense.
> 
> If it doesn't make sense, and the existing code is zmq specific, then
> I'd be interested in working with the zmq folks to help develop a more
> generic implementation that functions across both drivers.

It would be great to have some common broker-less base class or something like 
that to share implementations, if that is possible.

Doug

> 
>> Doug
> 
> 
> -- 
> Ken Giusti  (kgiu...@gmail.com)
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] zeromq work for kilo

[Mehdi Abaakouk]

Hi,

For me, the lack of tests is why reviews are not done, even some of them 
are critical.


An other issue, less problematic, of this driver is it relies on 
eventlet directly,
but all the eventlet code should be only localized into 
oslo.messaging._executor.impl_eventlet.


Otherwise, I have real opinion for pushing the zmq driver in an other 
repo or not, if we really have people to maintain this driver,

I'm sure the code will be merged.

Having the testing issue fixed for K-1 sounds a good milestone, to see 
if we are able to maintain it even with split or not the repo.


Regards,
---
Mehdi Abaakouk
mail: sil...@sileht.net
irc: sileht

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] zeromq work for kilo

[Ben Nemec]

On 09/19/2014 08:38 AM, Ken Giusti wrote:
> On Thu, 18 Sep 2014 17:29:27 Eric Windisch wrote:
>>>
>>>
>>> That?s great feedback, Eric, thank you. I know some of the other projects
> 
> +1 - yes, excellent feedback - having just worked on the AMQP 1.0
> driver, I think you've nicely described some of my own experiences.
> 
>>> are moving drivers out of the main core tree, and we could certainly
>>> consider doing that here as well, if we have teams willing to sign up for
>>> the work it means doing.
>>>
>>> In addition to the zmq driver, we have a fairly stable rabbit driver, a
>>> qpid driver whose quality I don?t know , and a new experimental AMQP 1.0
>>> driver. Are we talking about moving those out, too, or just zmq because we
>>> were already considering removing it entirely?
>>>
>>
>> I believe it makes good sense for all drivers, in the long term. However,
>> the most immediate benefits would be in offloading any drivers that need
>> substantial work or improvements, aka velocity. That would mean the AMQP
>> and ZeroMQ drivers.
>>
> 
> I'm tentatively in favor of this - 'tentative' because, noob that I am,
> I'm not sure I understand the trade-offs, if any, that moving these
> drivers outside of oslo.messaging would bring.
> 
> To be clear: I'm 100% for any change that makes it easier to have
> non-core developers that have the proper domain knowledge contribute
> to these drivers.  However, there's a few things I need to understand:
> 
> Does this move make it harder for users to deploy these
> drivers?  How would we insure that the proper, tested version of a
> driver is delivered along with oslo.messaging proper?

On the point of non-core developers being better able to contribute, in
oslo-incubator we have the concept of Maintainers, who have sort of a
super +1 that counts as +2 on the code they maintain (note that this is
a policy thing, not something enforced by Gerrit in any way).  What this
means is that in theory you could have two +1's from maintainers and
then all you need an oslo.messaging core for is the approval.  In
practice I think it's more common that you get a maintainer +1 and a
core +2A, but for oslo.messaging drivers where there is likely to be
more than one person interested in maintaining it the two +1 case might
be more common.

In any case, that might be an option besides splitting out the drivers
completely.

> 
>> With the Nova drivers, what's useful is that we have tempest and we can use
>> that as an integration gate. I suppose that's technically possible with
>> oslo.messaging and its drivers as well, although I prefer to see a
>> separation of concerns were I presume there are messaging patterns you want
>> to validate that aren't exercised by Tempest.
> 
> This is critical IMHO - any proposed changes to oslo.messaging
> proper, or any particular driver for that matter, needs to be vetted
> against the other out-of-tree drivers automagically.  E.g. If a
> proposed change to oslo.messaging breaks the out of tree AMQP 1.0
> driver, that needs to be flagged by jenkins during the gerrit review
> of the proposed oslo.messaging patch.
> 
>>
>> Another thing I'll note is that before pulling Ironic in, Nova had an API
>> contract test. This can be useful for making sure that changes in the
>> upstream project doesn't break drivers, or that breakages could at least
>> invoke action by the driver team:
>> https://github.com/openstack/nova/blob/4ce3f55d169290015063131134f93fca236807ed/nova/tests/virt/test_ironic_api_contracts.py
>>
>> --
>> Regards,
>> Eric Windisch
> 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] zeromq work for kilo

[Ken Giusti]

On Mon Sep 8 15:18:35 UTC 2014, Doug Hellmann wrote:
>On Sep 8, 2014, at 10:35 AM, Antonio Messina  
>wrote:
>
>> Hi All,
>>
>> We tested briefly ZeroMQ with Havana last year, but we couldn't find
>> any good documentation on how to implement it, and we were not able to
>> get it working. We also got the impression that the support was not at
>> all mature, so we decided to use RabbitMQ instead.
>>
>> However, I must say that the broker-less design of ZeroMQ is very
>> appealing, and we would like to give it a try, assuming
>> 1) the documentation is improved
>> 2) there is some assurance that support for ZeroMQ is not going to be 
>> dropped.
>>
>> I can help with 1) if there is someone that knows a bit of the
>> internals and can bootstrap me, because I have no first hand
>> experience on how message queues are used in OpenStack, and little
>> experience with ZeroMQ.
>
>Unfortunately, the existing Oslo team doesn’t have a lot of
>experience with ZeroMQ either (hence this thread). It sounds like Li
>Ma’s team has made it work, though, so maybe you could work
>together. We should prioritize documentation and then functional
>testing, I think.
>
> About 2), well this is a decision for the developers, but IMHO there
> *should* be support for ZeroMQ in OpenStack: its broker-less
> architecture would eliminate a SPoF (the message broker), could ease
> the deployment (especially in HA setup) and grant very high
> performance.
>
>I agree, it would be useful to support it. This is purely a resource
>allocation problem for me. I don't have anyone willing to do the work
>needed to ensure the driver is functional and can be deployed sanely
>(although maybe I’ve found a couple of volunteers now :-).
>
>There is another effort going on to support AMQP 1.0, which (as I
>understand it) includes similar broker-less deployment options. Before
>we decide whether to invest in ZeroMQ for that reason alone, it would
>be useful to know if AMQP 1.0 support makes potential ZeroMQ support
>less interesting.
>

While the AMQP 1.0 protocol permits it, the current implementation of
the new driver does not support broker-less point-to-point - yet.

I'm planning on adding that support to the AMQP 1.0 driver in the
future.  I have yet to spend any time ramping up on the existing
brokerless support implemented by the zmq driver, so forgive my
ignorance, but I'm hoping to leverage what is there if it makes sense.

If it doesn't make sense, and the existing code is zmq specific, then
I'd be interested in working with the zmq folks to help develop a more
generic implementation that functions across both drivers.

>Doug


-- 
Ken Giusti  (kgiu...@gmail.com)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] zeromq work for kilo

[Ken Giusti]

On Thu, 18 Sep 2014 17:29:27 Eric Windisch wrote:
>>
>>
>> That?s great feedback, Eric, thank you. I know some of the other projects

+1 - yes, excellent feedback - having just worked on the AMQP 1.0
driver, I think you've nicely described some of my own experiences.

>> are moving drivers out of the main core tree, and we could certainly
>> consider doing that here as well, if we have teams willing to sign up for
>> the work it means doing.
>>
>> In addition to the zmq driver, we have a fairly stable rabbit driver, a
>> qpid driver whose quality I don?t know , and a new experimental AMQP 1.0
>> driver. Are we talking about moving those out, too, or just zmq because we
>> were already considering removing it entirely?
>>
>
>I believe it makes good sense for all drivers, in the long term. However,
>the most immediate benefits would be in offloading any drivers that need
>substantial work or improvements, aka velocity. That would mean the AMQP
>and ZeroMQ drivers.
>

I'm tentatively in favor of this - 'tentative' because, noob that I am,
I'm not sure I understand the trade-offs, if any, that moving these
drivers outside of oslo.messaging would bring.

To be clear: I'm 100% for any change that makes it easier to have
non-core developers that have the proper domain knowledge contribute
to these drivers.  However, there's a few things I need to understand:

Does this move make it harder for users to deploy these
drivers?  How would we insure that the proper, tested version of a
driver is delivered along with oslo.messaging proper?

>With the Nova drivers, what's useful is that we have tempest and we can use
>that as an integration gate. I suppose that's technically possible with
>oslo.messaging and its drivers as well, although I prefer to see a
>separation of concerns were I presume there are messaging patterns you want
>to validate that aren't exercised by Tempest.

This is critical IMHO - any proposed changes to oslo.messaging
proper, or any particular driver for that matter, needs to be vetted
against the other out-of-tree drivers automagically.  E.g. If a
proposed change to oslo.messaging breaks the out of tree AMQP 1.0
driver, that needs to be flagged by jenkins during the gerrit review
of the proposed oslo.messaging patch.

>
>Another thing I'll note is that before pulling Ironic in, Nova had an API
>contract test. This can be useful for making sure that changes in the
>upstream project doesn't break drivers, or that breakages could at least
>invoke action by the driver team:
>https://github.com/openstack/nova/blob/4ce3f55d169290015063131134f93fca236807ed/nova/tests/virt/test_ironic_api_contracts.py
>
>--
>Regards,
>Eric Windisch

-- 
Ken Giusti  (kgiu...@gmail.com)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] zeromq work for kilo

[Eric Windisch]

>
>
> That’s great feedback, Eric, thank you. I know some of the other projects
> are moving drivers out of the main core tree, and we could certainly
> consider doing that here as well, if we have teams willing to sign up for
> the work it means doing.
>
> In addition to the zmq driver, we have a fairly stable rabbit driver, a
> qpid driver whose quality I don’t know , and a new experimental AMQP 1.0
> driver. Are we talking about moving those out, too, or just zmq because we
> were already considering removing it entirely?
>

I believe it makes good sense for all drivers, in the long term. However,
the most immediate benefits would be in offloading any drivers that need
substantial work or improvements, aka velocity. That would mean the AMQP
and ZeroMQ drivers.

With the Nova drivers, what's useful is that we have tempest and we can use
that as an integration gate. I suppose that's technically possible with
oslo.messaging and its drivers as well, although I prefer to see a
separation of concerns were I presume there are messaging patterns you want
to validate that aren't exercised by Tempest.

Another thing I'll note is that before pulling Ironic in, Nova had an API
contract test. This can be useful for making sure that changes in the
upstream project doesn't break drivers, or that breakages could at least
invoke action by the driver team:
https://github.com/openstack/nova/blob/4ce3f55d169290015063131134f93fca236807ed/nova/tests/virt/test_ironic_api_contracts.py

-- 
Regards,
Eric Windisch
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] zeromq work for kilo

[Doug Hellmann]

On Sep 18, 2014, at 4:16 PM, Eric Windisch  wrote:

> 
> 
> On Thu, Sep 18, 2014 at 3:55 PM, Doug Hellmann  wrote:
> 
> On Sep 18, 2014, at 10:16 AM, Kapil Thangavelu 
>  wrote:
> 
> >
> >
> > On Thu, Sep 18, 2014 at 4:18 AM, Flavio Percoco  wrote:
> > On 09/17/2014 04:34 PM, Doug Hellmann wrote:
> > > This thread [1] has turned more “future focused", so I’m moving the 
> > > conversation to the -dev list where we usually have those sorts of 
> > > discussions.
> > >
> > > [1] 
> > > http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
> > >
> >
> >
> > I saw this mentioned in the `openstack` thread but I'd like us to
> > reconsider it.
> >
> > Since we've gone through the "hey please don't deprecate it, I'll help"
> > process a couple of times with the zmq driver, I'm thinking we should
> > pull it out of the code base anyway.
> >
> > I think the primary issue has been two fold, one is the lack of 
> > reviews/merges on extant patches that fix critical items that have been 
> > outstanding for months. I think that is compounded by the other issue which 
> > was the lack of tests. We've sprinted last week on adding in both unit 
> > tests, the extant patches and functionally verifying them by automating 
> > cloud deployments with zmq for the messaging driver. We're also committing 
> > as a company to supporting it on an ongoing basis. If we get the gates 
> > going for kilo, i don't see any reason for the churn below, if the gates 
> > don't get going we can yank to external in kilo anyway.
> 
> I imagine that, as with drivers in other projects, part of the issue here is 
> that there are not enough oslo.messaging core reviewers comfortable with zmq 
> to feel confident reviewing the driver. Flavio’s proposal also has the 
> benefit of addressing this, since some of the currently interested parties 
> could be given core reviewer privileges on an oslo.messaging.zmq repository.
> 
> For now, I’m in favor of keeping an eye on the current interest and seeing 
> how things progress before making a final decision on whether to delete the 
> driver, move it to its own repository, or keep it where it is. I’d like to 
> hear from some of the rest of oslo-core and oslo-messaging-core about their 
> opinions, too, though — it isn’t my call alone.
> 
> While I'm no longer involved and quite unlikely to again become involved, I 
> will say that I'm +1 on maintaining it separately.
> 
> At the time we merged the ZeroMQ driver, it was suggested that we keep it out 
> and maintain it outside the tree. I lobbied hard to get it in, and it was... 
> but it was at great cost. The review process is highly detrimental to 
> fast-moving code bases and imposes a severe handicap to gaining maturity and 
> full API compliance, etc. Once the ZeroMQ code went into Nova and 
> subsequently Oslo, it was incredibly difficult to balance the need to 
> increment and improve the code and improve testing while also managing 
> reviews. 
> 
> The reason I lobbied hard to get the code in was because it couldn't really 
> be tested much otherwise. We couldn't keep track of changes in Nova and the 
> needs of OpenStack. Ultimately, as a the project evolved and introduced 
> changes to messaging for the needs of projects such as Ceilometer, what was 
> once an advantage became a disadvantage, when coupled with the long review 
> times.
> 
> However, the barriers to maintaining code out of the tree is lower than ever. 
>  I currently maintain the Docker driver out of Nova's tree and it's doing 
> fine. Well enough that I no longer want it to go back into Nova's tree.  
> While I'd like it to become part of Nova, I'd like to maintain a separate or 
> superset of core reviewers as not to impede development and maintenance, 
> without changes to Gerrit, this means using a separate git repository -- and 
> that's okay.
> 
> The only reason I see moving the Docker code back into Nova would be 
> political, not based on the merit of a technical approach or ease and cost of 
> maintenance. In the last year, I've become very aware of the financial 
> requirements that the OpenStack community has unwittingly imposed on its 
> contributing members and I really wish, as much as possible, to roll this 
> back and reduce the cost of contributing. Breaking code out, while accepting 
> it may still be "valid" and "included" (if not core), is a big step for 
> OpenStack in reducing that cost. Obviously, all I've just said could be 
> applied to the ZeroMQ driver as well as it applies to Docker.
> 
> The OpenStack CI system is now advanced and mature enough that breaking 
> ZeroMQ out into a stackforge repo and creating dependencies between the 
> projects and setting up testing will be, in my opinion, better for any new 
> maintainers and users of this driver.

That’s great feedback, Eric, thank you. I know some of the other projects are 
moving drivers out of the main core tree, and we could certainly consider doing 
that here as well, 

Re: [openstack-dev] [oslo] zeromq work for kilo

[Kapil Thangavelu]

Hi Davanum

The two primary fixes to oslo.messaging outstanding to get things running
are

https://review.openstack.org/#/c/84938/
https://review.openstack.org/#/c/120745/

cheers,

Kapil


On Thu, Sep 18, 2014 at 11:32 AM, Davanum Srinivas 
wrote:

> Kapil,
>
> I see just 2 relevant reviews for zmq in the oslo.messaging queue:
>
> https://review.openstack.org/#/q/project:openstack/oslo.messaging+status:open+file:%255E.*zmq.*,n,z
>
> Are there others i am missing? ("fix critical items", "tests" from your
> email)
>
> thanks,
> dims
>
> On Thu, Sep 18, 2014 at 10:16 AM, Kapil Thangavelu
>  wrote:
> >
> >
> > On Thu, Sep 18, 2014 at 4:18 AM, Flavio Percoco 
> wrote:
> >>
> >> On 09/17/2014 04:34 PM, Doug Hellmann wrote:
> >> > This thread [1] has turned more “future focused", so I’m moving the
> >> > conversation to the -dev list where we usually have those sorts of
> >> > discussions.
> >> >
> >> > [1]
> >> >
> http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
> >> >
> >>
> >>
> >> I saw this mentioned in the `openstack` thread but I'd like us to
> >> reconsider it.
> >>
> >> Since we've gone through the "hey please don't deprecate it, I'll help"
> >> process a couple of times with the zmq driver, I'm thinking we should
> >> pull it out of the code base anyway.
> >
> >
> > I think the primary issue has been two fold, one is the lack of
> > reviews/merges on extant patches that fix critical items that have been
> > outstanding for months. I think that is compounded by the other issue
> which
> > was the lack of tests. We've sprinted last week on adding in both unit
> > tests, the extant patches and functionally verifying them by automating
> > cloud deployments with zmq for the messaging driver. We're also
> committing
> > as a company to supporting it on an ongoing basis. If we get the gates
> going
> > for kilo, i don't see any reason for the churn below, if the gates don't
> get
> > going we can yank to external in kilo anyway.
> >
> > cheers,
> >
> > Kapil
> >
> >>
> >> Here's a rough plan of what I think we should do until the zmq is
> >> updated and has a proper gate working:
> >>
> >> 1. Pull it out the code base into its own repo (stackforge?)
> >> 2. Setup the basic `oslo.messaging` gates for it
> >> 3. Release the driver on pypi as: `oslo.messaging.zmq`
> >> 4. Make lots of noise and make sure people using zmq knows that they
> >> have to install a separate package now. (I know this means creating a
> >> new package for many distros).
> >>
> >> If there are folks interested in maintaining this driver, they can still
> >> do it with the driver outside the code base. If it ever gets enough
> >> attention and maturity, it could be re-proposed for inclusion into the
> >> code base.
> >>
> >> I know there are folks interested in a broker-less solution and we now
> >> have the not-tested-in-production-yet amqp1.0 driver which I think is a
> >> very good solution and also shares most of the semantics we rely on
> >> protocol-wise.
> >>
> >> I didn't go through the whole thread so, I'm sorry if I'm repeating
> >> something that has already been said/proposed/discussed.
> >>
> >> Flavio
> >>
> >> > On Sep 17, 2014, at 7:54 AM, James Page 
> wrote:
> >> >
> >> >> Signed PGP part
> >> >> Hi Li
> >> >>
> >> >> On 17/09/14 11:58, Li Ma wrote:
> >>  The scale potential is very appealing and is something I want to
> >>  test - - hopefully in the next month or so.
> >> 
> >>  Canonical are interested in helping to maintain this driver and
> >>  hopefully we help any critical issues prior to Juno release.
> >> 
> >> 
> >> >>> That sounds good. I just went through all the bugs reported in the
> >> >>> community.
> >> >>>
> >> >>> The only critical bug which makes ZeroMQ malfunction is
> >> >>> https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
> >> >>> corresponding review is under way:
> >> >>> https://review.openstack.org/#/c/84938/
> >> >>
> >> >> Agreed
> >> >>
> >> >>> Others are tagged to 'zmq' in
> >> >>> https://bugs.launchpad.net/oslo.messaging
> >> >>
> >> >> Looking through Doug's suggested list of information and collating
> >> >> what I know of from our work last week:
> >> >>
> >> >> 1) documentation for how to configure and use zeromq with
> >> >> oslo.messaging (note, not the version in oslo-incubator, the version
> >> >> in the messaging library repository)
> >> >>
> >> >> As part of our sprint, I worked on automating deployment of OpenStack
> >> >> + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
> >> >> that work into some general documentation on how best to configure
> >> >> ZeroMQ with OpenStack - the current documentation is a bit raw and
> >> >> does not talk about how to configure the oslo-messaging-zmq-receiver
> >> >> at all.
> >> >>
> >> >> I also plan some packaging updates for Debian/Ubuntu in our next dev
> >> >> cycle to make this a little easier to configure and digest - for
> >> >> example, right now

Re: [openstack-dev] [oslo] zeromq work for kilo

[Eric Windisch]

On Thu, Sep 18, 2014 at 3:55 PM, Doug Hellmann 
wrote:

>
> On Sep 18, 2014, at 10:16 AM, Kapil Thangavelu <
> kapil.thangav...@canonical.com> wrote:
>
> >
> >
> > On Thu, Sep 18, 2014 at 4:18 AM, Flavio Percoco 
> wrote:
> > On 09/17/2014 04:34 PM, Doug Hellmann wrote:
> > > This thread [1] has turned more “future focused", so I’m moving the
> conversation to the -dev list where we usually have those sorts of
> discussions.
> > >
> > > [1]
> http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
> > >
> >
> >
> > I saw this mentioned in the `openstack` thread but I'd like us to
> > reconsider it.
> >
> > Since we've gone through the "hey please don't deprecate it, I'll help"
> > process a couple of times with the zmq driver, I'm thinking we should
> > pull it out of the code base anyway.
> >
> > I think the primary issue has been two fold, one is the lack of
> reviews/merges on extant patches that fix critical items that have been
> outstanding for months. I think that is compounded by the other issue which
> was the lack of tests. We've sprinted last week on adding in both unit
> tests, the extant patches and functionally verifying them by automating
> cloud deployments with zmq for the messaging driver. We're also committing
> as a company to supporting it on an ongoing basis. If we get the gates
> going for kilo, i don't see any reason for the churn below, if the gates
> don't get going we can yank to external in kilo anyway.
>
> I imagine that, as with drivers in other projects, part of the issue here
> is that there are not enough oslo.messaging core reviewers comfortable with
> zmq to feel confident reviewing the driver. Flavio’s proposal also has the
> benefit of addressing this, since some of the currently interested parties
> could be given core reviewer privileges on an oslo.messaging.zmq repository.
>
> For now, I’m in favor of keeping an eye on the current interest and seeing
> how things progress before making a final decision on whether to delete the
> driver, move it to its own repository, or keep it where it is. I’d like to
> hear from some of the rest of oslo-core and oslo-messaging-core about their
> opinions, too, though — it isn’t my call alone.
>

While I'm no longer involved and quite unlikely to again become involved, I
will say that I'm +1 on maintaining it separately.

At the time we merged the ZeroMQ driver, it was suggested that we keep it
out and maintain it outside the tree. I lobbied hard to get it in, and it
was... but it was at great cost. The review process is highly detrimental
to fast-moving code bases and imposes a severe handicap to gaining maturity
and full API compliance, etc. Once the ZeroMQ code went into Nova and
subsequently Oslo, it was incredibly difficult to balance the need to
increment and improve the code and improve testing while also managing
reviews.

The reason I lobbied hard to get the code in was because it couldn't really
be tested much otherwise. We couldn't keep track of changes in Nova and the
needs of OpenStack. Ultimately, as a the project evolved and introduced
changes to messaging for the needs of projects such as Ceilometer, what was
once an advantage became a disadvantage, when coupled with the long review
times.

However, the barriers to maintaining code out of the tree is lower than
ever.  I currently maintain the Docker driver out of Nova's tree and it's
doing fine. Well enough that I no longer want it to go back into Nova's
tree.  While I'd like it to become part of Nova, I'd like to maintain a
separate or superset of core reviewers as not to impede development and
maintenance, without changes to Gerrit, this means using a separate git
repository -- and that's okay.

The only reason I see moving the Docker code back into Nova would be
political, not based on the merit of a technical approach or ease and cost
of maintenance. In the last year, I've become very aware of the financial
requirements that the OpenStack community has unwittingly imposed on its
contributing members and I really wish, as much as possible, to roll this
back and reduce the cost of contributing. Breaking code out, while
accepting it may still be "valid" and "included" (if not core), is a big
step for OpenStack in reducing that cost. Obviously, all I've just said
could be applied to the ZeroMQ driver as well as it applies to Docker.

The OpenStack CI system is now advanced and mature enough that breaking
ZeroMQ out into a stackforge repo and creating dependencies between the
projects and setting up testing will be, in my opinion, better for any new
maintainers and users of this driver.

Regards,
Eric Windisch
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] zeromq work for kilo

[Doug Hellmann]

On Sep 18, 2014, at 10:16 AM, Kapil Thangavelu  
wrote:

> 
> 
> On Thu, Sep 18, 2014 at 4:18 AM, Flavio Percoco  wrote:
> On 09/17/2014 04:34 PM, Doug Hellmann wrote:
> > This thread [1] has turned more “future focused", so I’m moving the 
> > conversation to the -dev list where we usually have those sorts of 
> > discussions.
> >
> > [1] 
> > http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
> >
> 
> 
> I saw this mentioned in the `openstack` thread but I'd like us to
> reconsider it.
> 
> Since we've gone through the "hey please don't deprecate it, I'll help"
> process a couple of times with the zmq driver, I'm thinking we should
> pull it out of the code base anyway.
> 
> I think the primary issue has been two fold, one is the lack of 
> reviews/merges on extant patches that fix critical items that have been 
> outstanding for months. I think that is compounded by the other issue which 
> was the lack of tests. We've sprinted last week on adding in both unit tests, 
> the extant patches and functionally verifying them by automating cloud 
> deployments with zmq for the messaging driver. We're also committing as a 
> company to supporting it on an ongoing basis. If we get the gates going for 
> kilo, i don't see any reason for the churn below, if the gates don't get 
> going we can yank to external in kilo anyway.

I imagine that, as with drivers in other projects, part of the issue here is 
that there are not enough oslo.messaging core reviewers comfortable with zmq to 
feel confident reviewing the driver. Flavio’s proposal also has the benefit of 
addressing this, since some of the currently interested parties could be given 
core reviewer privileges on an oslo.messaging.zmq repository.

For now, I’m in favor of keeping an eye on the current interest and seeing how 
things progress before making a final decision on whether to delete the driver, 
move it to its own repository, or keep it where it is. I’d like to hear from 
some of the rest of oslo-core and oslo-messaging-core about their opinions, 
too, though — it isn’t my call alone.

Doug

>  
> cheers,
> 
> Kapil
> 
> 
> Here's a rough plan of what I think we should do until the zmq is
> updated and has a proper gate working:
> 
> 1. Pull it out the code base into its own repo (stackforge?)
> 2. Setup the basic `oslo.messaging` gates for it
> 3. Release the driver on pypi as: `oslo.messaging.zmq`
> 4. Make lots of noise and make sure people using zmq knows that they
> have to install a separate package now. (I know this means creating a
> new package for many distros).
> 
> If there are folks interested in maintaining this driver, they can still
> do it with the driver outside the code base. If it ever gets enough
> attention and maturity, it could be re-proposed for inclusion into the
> code base.
> 
> I know there are folks interested in a broker-less solution and we now
> have the not-tested-in-production-yet amqp1.0 driver which I think is a
> very good solution and also shares most of the semantics we rely on
> protocol-wise.
> 
> I didn't go through the whole thread so, I'm sorry if I'm repeating
> something that has already been said/proposed/discussed.
> 
> Flavio
> 
> > On Sep 17, 2014, at 7:54 AM, James Page  wrote:
> >
> >> Signed PGP part
> >> Hi Li
> >>
> >> On 17/09/14 11:58, Li Ma wrote:
>  The scale potential is very appealing and is something I want to
>  test - - hopefully in the next month or so.
> 
>  Canonical are interested in helping to maintain this driver and
>  hopefully we help any critical issues prior to Juno release.
> 
> 
> >>> That sounds good. I just went through all the bugs reported in the
> >>> community.
> >>>
> >>> The only critical bug which makes ZeroMQ malfunction is
> >>> https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
> >>> corresponding review is under way:
> >>> https://review.openstack.org/#/c/84938/
> >>
> >> Agreed
> >>
> >>> Others are tagged to 'zmq' in
> >>> https://bugs.launchpad.net/oslo.messaging
> >>
> >> Looking through Doug's suggested list of information and collating
> >> what I know of from our work last week:
> >>
> >> 1) documentation for how to configure and use zeromq with
> >> oslo.messaging (note, not the version in oslo-incubator, the version
> >> in the messaging library repository)
> >>
> >> As part of our sprint, I worked on automating deployment of OpenStack
> >> + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
> >> that work into some general documentation on how best to configure
> >> ZeroMQ with OpenStack - the current documentation is a bit raw and
> >> does not talk about how to configure the oslo-messaging-zmq-receiver
> >> at all.
> >>
> >> I also plan some packaging updates for Debian/Ubuntu in our next dev
> >> cycle to make this a little easier to configure and digest - for
> >> example, right now no systemd unit/upstart configuration/sysv init
> >> script is provided to manage the 

Re: [openstack-dev] [oslo] zeromq work for kilo

[Davanum Srinivas]

Kapil,

I see just 2 relevant reviews for zmq in the oslo.messaging queue:
https://review.openstack.org/#/q/project:openstack/oslo.messaging+status:open+file:%255E.*zmq.*,n,z

Are there others i am missing? ("fix critical items", "tests" from your email)

thanks,
dims

On Thu, Sep 18, 2014 at 10:16 AM, Kapil Thangavelu
 wrote:
>
>
> On Thu, Sep 18, 2014 at 4:18 AM, Flavio Percoco  wrote:
>>
>> On 09/17/2014 04:34 PM, Doug Hellmann wrote:
>> > This thread [1] has turned more “future focused", so I’m moving the
>> > conversation to the -dev list where we usually have those sorts of
>> > discussions.
>> >
>> > [1]
>> > http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
>> >
>>
>>
>> I saw this mentioned in the `openstack` thread but I'd like us to
>> reconsider it.
>>
>> Since we've gone through the "hey please don't deprecate it, I'll help"
>> process a couple of times with the zmq driver, I'm thinking we should
>> pull it out of the code base anyway.
>
>
> I think the primary issue has been two fold, one is the lack of
> reviews/merges on extant patches that fix critical items that have been
> outstanding for months. I think that is compounded by the other issue which
> was the lack of tests. We've sprinted last week on adding in both unit
> tests, the extant patches and functionally verifying them by automating
> cloud deployments with zmq for the messaging driver. We're also committing
> as a company to supporting it on an ongoing basis. If we get the gates going
> for kilo, i don't see any reason for the churn below, if the gates don't get
> going we can yank to external in kilo anyway.
>
> cheers,
>
> Kapil
>
>>
>> Here's a rough plan of what I think we should do until the zmq is
>> updated and has a proper gate working:
>>
>> 1. Pull it out the code base into its own repo (stackforge?)
>> 2. Setup the basic `oslo.messaging` gates for it
>> 3. Release the driver on pypi as: `oslo.messaging.zmq`
>> 4. Make lots of noise and make sure people using zmq knows that they
>> have to install a separate package now. (I know this means creating a
>> new package for many distros).
>>
>> If there are folks interested in maintaining this driver, they can still
>> do it with the driver outside the code base. If it ever gets enough
>> attention and maturity, it could be re-proposed for inclusion into the
>> code base.
>>
>> I know there are folks interested in a broker-less solution and we now
>> have the not-tested-in-production-yet amqp1.0 driver which I think is a
>> very good solution and also shares most of the semantics we rely on
>> protocol-wise.
>>
>> I didn't go through the whole thread so, I'm sorry if I'm repeating
>> something that has already been said/proposed/discussed.
>>
>> Flavio
>>
>> > On Sep 17, 2014, at 7:54 AM, James Page  wrote:
>> >
>> >> Signed PGP part
>> >> Hi Li
>> >>
>> >> On 17/09/14 11:58, Li Ma wrote:
>>  The scale potential is very appealing and is something I want to
>>  test - - hopefully in the next month or so.
>> 
>>  Canonical are interested in helping to maintain this driver and
>>  hopefully we help any critical issues prior to Juno release.
>> 
>> 
>> >>> That sounds good. I just went through all the bugs reported in the
>> >>> community.
>> >>>
>> >>> The only critical bug which makes ZeroMQ malfunction is
>> >>> https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
>> >>> corresponding review is under way:
>> >>> https://review.openstack.org/#/c/84938/
>> >>
>> >> Agreed
>> >>
>> >>> Others are tagged to 'zmq' in
>> >>> https://bugs.launchpad.net/oslo.messaging
>> >>
>> >> Looking through Doug's suggested list of information and collating
>> >> what I know of from our work last week:
>> >>
>> >> 1) documentation for how to configure and use zeromq with
>> >> oslo.messaging (note, not the version in oslo-incubator, the version
>> >> in the messaging library repository)
>> >>
>> >> As part of our sprint, I worked on automating deployment of OpenStack
>> >> + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
>> >> that work into some general documentation on how best to configure
>> >> ZeroMQ with OpenStack - the current documentation is a bit raw and
>> >> does not talk about how to configure the oslo-messaging-zmq-receiver
>> >> at all.
>> >>
>> >> I also plan some packaging updates for Debian/Ubuntu in our next dev
>> >> cycle to make this a little easier to configure and digest - for
>> >> example, right now no systemd unit/upstart configuration/sysv init
>> >> script is provided to manage the zmq-receiver.
>> >>
>> >> I'd also like to document the current design of the ZMQ driver - Doug
>> >> - where is the best place todo this? I thought in the source tree
>> >> somewhere.
>> >
>> > The documentation in the oslo.messaging repository [2] would be a good
>> > place to start for that. If we decide deployers/operators need the
>> > information we can either refer to it from the guides managed by th

Re: [openstack-dev] [oslo] zeromq work for kilo

[Flavio Percoco]

On 09/18/2014 04:16 PM, Kapil Thangavelu wrote:
> 
> 
> On Thu, Sep 18, 2014 at 4:18 AM, Flavio Percoco  > wrote:
> 
> On 09/17/2014 04:34 PM, Doug Hellmann wrote:
> > This thread [1] has turned more “future focused", so I’m moving the 
> conversation to the -dev list where we usually have those sorts of 
> discussions.
> >
> > [1] 
> http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
> >
> 
> 
> I saw this mentioned in the `openstack` thread but I'd like us to
> reconsider it.
> 
> Since we've gone through the "hey please don't deprecate it, I'll help"
> process a couple of times with the zmq driver, I'm thinking we should
> pull it out of the code base anyway.
> 
> 
> I think the primary issue has been two fold, one is the lack of
> reviews/merges on extant patches that fix critical items that have been
> outstanding for months. I think that is compounded by the other issue
> which was the lack of tests. We've sprinted last week on adding in both
> unit tests, the extant patches and functionally verifying them by
> automating cloud deployments with zmq for the messaging driver. We're
> also committing as a company to supporting it on an ongoing basis. If we
> get the gates going for kilo, i don't see any reason for the churn
> below, if the gates don't get going we can yank to external in kilo anyway.

The above sounds good to me. Thanks for sharing.

Can we have a deadline for all this? Last time we discussed zmq driver's
faith, we said that k-1 was a good deadline for it to get back in shape.
If the above plan sounds good to other folks, then I'd prefer us to
stick to k-1 for all the above to happen.

Thoughts?

Thanks again, Kapil
Flavio

>  
> cheers,
> 
> Kapil
> 
> 
> Here's a rough plan of what I think we should do until the zmq is
> updated and has a proper gate working:
> 
> 1. Pull it out the code base into its own repo (stackforge?)
> 2. Setup the basic `oslo.messaging` gates for it
> 3. Release the driver on pypi as: `oslo.messaging.zmq`
> 4. Make lots of noise and make sure people using zmq knows that they
> have to install a separate package now. (I know this means creating a
> new package for many distros).
> 
> If there are folks interested in maintaining this driver, they can still
> do it with the driver outside the code base. If it ever gets enough
> attention and maturity, it could be re-proposed for inclusion into the
> code base.
> 
> I know there are folks interested in a broker-less solution and we now
> have the not-tested-in-production-yet amqp1.0 driver which I think is a
> very good solution and also shares most of the semantics we rely on
> protocol-wise.
> 
> I didn't go through the whole thread so, I'm sorry if I'm repeating
> something that has already been said/proposed/discussed.
> 
> Flavio
> 
> > On Sep 17, 2014, at 7:54 AM, James Page  > wrote:
> >
> >> Signed PGP part
> >> Hi Li
> >>
> >> On 17/09/14 11:58, Li Ma wrote:
>  The scale potential is very appealing and is something I want to
>  test - - hopefully in the next month or so.
> 
>  Canonical are interested in helping to maintain this driver and
>  hopefully we help any critical issues prior to Juno release.
> 
> 
> >>> That sounds good. I just went through all the bugs reported in the
> >>> community.
> >>>
> >>> The only critical bug which makes ZeroMQ malfunction is
> >>> https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
> >>> corresponding review is under way:
> >>> https://review.openstack.org/#/c/84938/
> >>
> >> Agreed
> >>
> >>> Others are tagged to 'zmq' in
> >>> https://bugs.launchpad.net/oslo.messaging
> >>
> >> Looking through Doug's suggested list of information and collating
> >> what I know of from our work last week:
> >>
> >> 1) documentation for how to configure and use zeromq with
> >> oslo.messaging (note, not the version in oslo-incubator, the version
> >> in the messaging library repository)
> >>
> >> As part of our sprint, I worked on automating deployment of OpenStack
> >> + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
> >> that work into some general documentation on how best to configure
> >> ZeroMQ with OpenStack - the current documentation is a bit raw and
> >> does not talk about how to configure the oslo-messaging-zmq-receiver
> >> at all.
> >>
> >> I also plan some packaging updates for Debian/Ubuntu in our next dev
> >> cycle to make this a little easier to configure and digest - for
> >> example, right now no systemd unit/upstart configuration/sysv init
> >> script is provided to manage the zmq-receiver.
> >>
> >> I'd also like to document t

Re: [openstack-dev] [oslo] zeromq work for kilo

[Kapil Thangavelu]

On Thu, Sep 18, 2014 at 4:18 AM, Flavio Percoco  wrote:

> On 09/17/2014 04:34 PM, Doug Hellmann wrote:
> > This thread [1] has turned more “future focused", so I’m moving the
> conversation to the -dev list where we usually have those sorts of
> discussions.
> >
> > [1]
> http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
> >
>
>
> I saw this mentioned in the `openstack` thread but I'd like us to
> reconsider it.
>
> Since we've gone through the "hey please don't deprecate it, I'll help"
> process a couple of times with the zmq driver, I'm thinking we should
> pull it out of the code base anyway.
>

I think the primary issue has been two fold, one is the lack of
reviews/merges on extant patches that fix critical items that have been
outstanding for months. I think that is compounded by the other issue which
was the lack of tests. We've sprinted last week on adding in both unit
tests, the extant patches and functionally verifying them by automating
cloud deployments with zmq for the messaging driver. We're also committing
as a company to supporting it on an ongoing basis. If we get the gates
going for kilo, i don't see any reason for the churn below, if the gates
don't get going we can yank to external in kilo anyway.

cheers,

Kapil


> Here's a rough plan of what I think we should do until the zmq is
> updated and has a proper gate working:
>
> 1. Pull it out the code base into its own repo (stackforge?)
> 2. Setup the basic `oslo.messaging` gates for it
> 3. Release the driver on pypi as: `oslo.messaging.zmq`
> 4. Make lots of noise and make sure people using zmq knows that they
> have to install a separate package now. (I know this means creating a
> new package for many distros).
>
> If there are folks interested in maintaining this driver, they can still
> do it with the driver outside the code base. If it ever gets enough
> attention and maturity, it could be re-proposed for inclusion into the
> code base.
>
> I know there are folks interested in a broker-less solution and we now
> have the not-tested-in-production-yet amqp1.0 driver which I think is a
> very good solution and also shares most of the semantics we rely on
> protocol-wise.
>
> I didn't go through the whole thread so, I'm sorry if I'm repeating
> something that has already been said/proposed/discussed.
>
> Flavio
>
> > On Sep 17, 2014, at 7:54 AM, James Page  wrote:
> >
> >> Signed PGP part
> >> Hi Li
> >>
> >> On 17/09/14 11:58, Li Ma wrote:
>  The scale potential is very appealing and is something I want to
>  test - - hopefully in the next month or so.
> 
>  Canonical are interested in helping to maintain this driver and
>  hopefully we help any critical issues prior to Juno release.
> 
> 
> >>> That sounds good. I just went through all the bugs reported in the
> >>> community.
> >>>
> >>> The only critical bug which makes ZeroMQ malfunction is
> >>> https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
> >>> corresponding review is under way:
> >>> https://review.openstack.org/#/c/84938/
> >>
> >> Agreed
> >>
> >>> Others are tagged to 'zmq' in
> >>> https://bugs.launchpad.net/oslo.messaging
> >>
> >> Looking through Doug's suggested list of information and collating
> >> what I know of from our work last week:
> >>
> >> 1) documentation for how to configure and use zeromq with
> >> oslo.messaging (note, not the version in oslo-incubator, the version
> >> in the messaging library repository)
> >>
> >> As part of our sprint, I worked on automating deployment of OpenStack
> >> + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
> >> that work into some general documentation on how best to configure
> >> ZeroMQ with OpenStack - the current documentation is a bit raw and
> >> does not talk about how to configure the oslo-messaging-zmq-receiver
> >> at all.
> >>
> >> I also plan some packaging updates for Debian/Ubuntu in our next dev
> >> cycle to make this a little easier to configure and digest - for
> >> example, right now no systemd unit/upstart configuration/sysv init
> >> script is provided to manage the zmq-receiver.
> >>
> >> I'd also like to document the current design of the ZMQ driver - Doug
> >> - where is the best place todo this? I thought in the source tree
> >> somewhere.
> >
> > The documentation in the oslo.messaging repository [2] would be a good
> place to start for that. If we decide deployers/operators need the
> information we can either refer to it from the guides managed by the
> documentation team or we can move/copy the information. How about if you
> start a new drivers subdirectory there, and add information about zmq. We
> can have other driver authors provide similar detail about their drivers in
> the same directory.
> >
> > [2]
> http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source
> >
> >>
> >> 2) a list of the critical bugs that need to be fixed + any existing
> >> patches associated with those bugs, so they

Re: [openstack-dev] [oslo] zeromq work for kilo

[Flavio Percoco]

On 09/17/2014 04:34 PM, Doug Hellmann wrote:
> This thread [1] has turned more “future focused", so I’m moving the 
> conversation to the -dev list where we usually have those sorts of 
> discussions.
> 
> [1] http://lists.openstack.org/pipermail/openstack/2014-September/009253.html
> 


I saw this mentioned in the `openstack` thread but I'd like us to
reconsider it.

Since we've gone through the "hey please don't deprecate it, I'll help"
process a couple of times with the zmq driver, I'm thinking we should
pull it out of the code base anyway.

Here's a rough plan of what I think we should do until the zmq is
updated and has a proper gate working:

1. Pull it out the code base into its own repo (stackforge?)
2. Setup the basic `oslo.messaging` gates for it
3. Release the driver on pypi as: `oslo.messaging.zmq`
4. Make lots of noise and make sure people using zmq knows that they
have to install a separate package now. (I know this means creating a
new package for many distros).

If there are folks interested in maintaining this driver, they can still
do it with the driver outside the code base. If it ever gets enough
attention and maturity, it could be re-proposed for inclusion into the
code base.

I know there are folks interested in a broker-less solution and we now
have the not-tested-in-production-yet amqp1.0 driver which I think is a
very good solution and also shares most of the semantics we rely on
protocol-wise.

I didn't go through the whole thread so, I'm sorry if I'm repeating
something that has already been said/proposed/discussed.

Flavio

> On Sep 17, 2014, at 7:54 AM, James Page  wrote:
> 
>> Signed PGP part
>> Hi Li
>>
>> On 17/09/14 11:58, Li Ma wrote:
 The scale potential is very appealing and is something I want to
 test - - hopefully in the next month or so.

 Canonical are interested in helping to maintain this driver and
 hopefully we help any critical issues prior to Juno release.


>>> That sounds good. I just went through all the bugs reported in the
>>> community.
>>>
>>> The only critical bug which makes ZeroMQ malfunction is
>>> https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
>>> corresponding review is under way:
>>> https://review.openstack.org/#/c/84938/
>>
>> Agreed
>>
>>> Others are tagged to 'zmq' in
>>> https://bugs.launchpad.net/oslo.messaging
>>
>> Looking through Doug's suggested list of information and collating
>> what I know of from our work last week:
>>
>> 1) documentation for how to configure and use zeromq with
>> oslo.messaging (note, not the version in oslo-incubator, the version
>> in the messaging library repository)
>>
>> As part of our sprint, I worked on automating deployment of OpenStack
>> + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
>> that work into some general documentation on how best to configure
>> ZeroMQ with OpenStack - the current documentation is a bit raw and
>> does not talk about how to configure the oslo-messaging-zmq-receiver
>> at all.
>>
>> I also plan some packaging updates for Debian/Ubuntu in our next dev
>> cycle to make this a little easier to configure and digest - for
>> example, right now no systemd unit/upstart configuration/sysv init
>> script is provided to manage the zmq-receiver.
>>
>> I'd also like to document the current design of the ZMQ driver - Doug
>> - where is the best place todo this? I thought in the source tree
>> somewhere.
> 
> The documentation in the oslo.messaging repository [2] would be a good place 
> to start for that. If we decide deployers/operators need the information we 
> can either refer to it from the guides managed by the documentation team or 
> we can move/copy the information. How about if you start a new drivers 
> subdirectory there, and add information about zmq. We can have other driver 
> authors provide similar detail about their drivers in the same directory.
> 
> [2] http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source
> 
>>
>> 2) a list of the critical bugs that need to be fixed + any existing
>> patches associated with those bugs, so they can be reviewed early in kilo
>>
>> This blocks operation of nova+neutron environements:
>>
>> https://bugs.launchpad.net/oslo.messaging/+bug/1301723
>>  Summary: Message was sent to wrong node with zmq as rpc_backend
>>  Patch: https://review.openstack.org/84938
>>
>> Also notifcations are effectively unimplemented which prevents use
>> with Ceilometer so I'd also add:
>>
>> https://bugs.launchpad.net/oslo.messaging/+bug/1368154
>>  Summary: https://bugs.launchpad.net/oslo.messaging/+bug/
>>  Patch: https://review.openstack.org/120745
> 
> That’s a good list, and shorter than I expected. I have added these bugs to 
> the next-kilo milestone.
> 
>>
>> 3) an analysis of what it would take to be able to run functional
>> tests for zeromq on our CI infrastructure, not necessarily the full
>> tempest run or devstack-gate job, probably functional tests we

Re: [openstack-dev] [oslo] zeromq work for kilo

[James Page]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 17/09/14 15:34, Doug Hellmann wrote:
> This thread [1] has turned more “future focused", so I’m moving the
> conversation to the -dev list where we usually have those sorts of
> discussions.
[...]
>> I'd also like to document the current design of the ZMQ driver -
>> Doug - where is the best place todo this? I thought in the source
>> tree somewhere.
> 
> The documentation in the oslo.messaging repository [2] would be a
> good place to start for that. If we decide deployers/operators need
> the information we can either refer to it from the guides managed
> by the documentation team or we can move/copy the information. How
> about if you start a new drivers subdirectory there, and add
> information about zmq. We can have other driver authors provide
> similar detail about their drivers in the same directory.
> 
> [2]
> http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source

OK
> 
- - I'll raise a review with some appropriate design documentation
sometime in the next few weeks.

>> 2) a list of the critical bugs that need to be fixed + any
>> existing patches associated with those bugs, so they can be
>> reviewed early in kilo
[...]
> That’s a good list, and shorter than I expected. I have added these
> bugs to the next-kilo milestone.

Thanks

>> 3) an analysis of what it would take to be able to run
>> functional tests for zeromq on our CI infrastructure, not
>> necessarily the full tempest run or devstack-gate job, probably
>> functional tests we place in the tree with the driver (we will be
>> doing this for all of the drivers) + besides writing new
>> functional tests, we need to bring the unit tests for zeromq into
>> the oslo.messaging repository
>> 
>> Kapil Thangavelu started work on both functional tests for the
>> ZMQ driver last week; the output from the sprint is here:
>> 
>> https://github.com/ostack-musketeers/oslo.messaging
>> 
>> it covers the ZMQ driver (including messaging through the
>> zmq-receiver proxy) and the associated MatchMakers (local, ring,
>> redis) at a varying levels of coverage, but I feel it moves
>> things in the right direction - Kapil's going to raise a review
>> for this in the next couple of days.
>> 
>> Doug - has any structure been agreed within the oslo.messaging
>> tree for unit/functional test splits? Right now we have them all
>> in one place.
> 
> I think we will want them split up, but we don’t have an agreed
> existing structure for that. I would like to see a test framework
> of some sort that defines the tests in a way that can be used to
> run the same functional for all of the drivers as separate jobs
> (with appropriate hooks for ensuring the needed services are
> running, etc.). Setting that up warrants its own spec, because
> there are going to be quite a few details to work out. We will also
> need to participate in the larger conversation about how to set up
> those functional test jobs to be consistent with the other
> projects.

I guess we can put then up for review as is and then refactor to
support any future framework changes that come along.

>> 4) and some improvements that we would like to make longer term
>> 
>> a) Connection re-use on outbound messaging avoiding the current
>> tcp setup overhead for every sent message.  This may also bring
>> further performance benefits due to underlying messaging batching
>> in ZMQ.
> 
> This sounds like it would be a good thing to do, but making what we
> have work correctly and testing it feels more important for now.

Agreed - these are all futures.

>> b) Moving from tcp PUSH/PULL sockets between servers to
>> DEALER/DEALER (or something similar) to allow for heartbeating
>> and more immediate failure detection
> 
> I would need to understand how much of a rewrite that represents
> before commenting further.

Sorry - that item was a little hand-wavey - it could be minimal in
terms of impact but I would probably see it tied to a) so it might
involve the zmq-receiver moving towards zmq-proxy for both inbound and
outbound tcp connections.

>> 
>> c) Crypto support
> 
> There are some other discussions about adding crypto to messaging,
> and I hope we can do that without having to touch each driver, if
> possible.

That would be even better IMHO.

- -- 
James Page
Ubuntu and Debian Developer
james.p...@ubuntu.com
jamesp...@debian.org
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBCAAGBQJUGo8jAAoJEL/srsug59jDAMMP/0RCc3ElFC/E/UqZJrY+X2oE
YMB87TM06AX26wS2uKe7F7xEAsv+vMTcSK/FC13xGfd+xBTT13pHquBQjeHwUx2C
+2Fbp6QoLwbvZF69IWL0E/8M/KueGth+JrHRpPqQqK5OcuwmDbDOeq2eychDMg1w
pkpXF0PPsa0tBRo3VQUHc43po9fAqT9Vpf8nG1M5vNTuOcCnrrmGg3DOFCzjVlG1
u2Mfo3HNIL48ZdkrSLHMk+7V1j4qHdU8ADxKaE3aPbxAaSieTlsKtf4pxJSM9Ikg
6WBsCChIVZA62Ox9xRZntidWdHTKcc4Lsv/K9ZhnFzST6OoWvEkpNxol1g/YX9lV
30TUdVBhNGm+ZU+2J8vxJ5suyVD0oW/J2hZGwhomi3tpKk9euKb5HMe5Ln0Fy1k9
63//XE8IzGIkNzwnBd4qnEubo1hg7Jfuw30uij9jJ85IdWRrpXiiei4XKZNJsAgD
/RsKzQt+BwlcztcWoQ0RX3OuOonQXb+5EgfqBce8DDHs+xi

[openstack-dev] [oslo] zeromq work for kilo

[Doug Hellmann]

This thread [1] has turned more “future focused", so I’m moving the 
conversation to the -dev list where we usually have those sorts of discussions.

[1] http://lists.openstack.org/pipermail/openstack/2014-September/009253.html

On Sep 17, 2014, at 7:54 AM, James Page  wrote:

> Signed PGP part
> Hi Li
> 
> On 17/09/14 11:58, Li Ma wrote:
> >> The scale potential is very appealing and is something I want to
> >> test - - hopefully in the next month or so.
> >>
> >> Canonical are interested in helping to maintain this driver and
> >> hopefully we help any critical issues prior to Juno release.
> >>
> >>
> > That sounds good. I just went through all the bugs reported in the
> > community.
> >
> > The only critical bug which makes ZeroMQ malfunction is
> > https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the
> > corresponding review is under way:
> > https://review.openstack.org/#/c/84938/
> 
> Agreed
> 
> > Others are tagged to 'zmq' in
> > https://bugs.launchpad.net/oslo.messaging
> 
> Looking through Doug's suggested list of information and collating
> what I know of from our work last week:
> 
> 1) documentation for how to configure and use zeromq with
> oslo.messaging (note, not the version in oslo-incubator, the version
> in the messaging library repository)
> 
> As part of our sprint, I worked on automating deployment of OpenStack
> + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig
> that work into some general documentation on how best to configure
> ZeroMQ with OpenStack - the current documentation is a bit raw and
> does not talk about how to configure the oslo-messaging-zmq-receiver
> at all.
> 
> I also plan some packaging updates for Debian/Ubuntu in our next dev
> cycle to make this a little easier to configure and digest - for
> example, right now no systemd unit/upstart configuration/sysv init
> script is provided to manage the zmq-receiver.
> 
> I'd also like to document the current design of the ZMQ driver - Doug
> - where is the best place todo this? I thought in the source tree
> somewhere.

The documentation in the oslo.messaging repository [2] would be a good place to 
start for that. If we decide deployers/operators need the information we can 
either refer to it from the guides managed by the documentation team or we can 
move/copy the information. How about if you start a new drivers subdirectory 
there, and add information about zmq. We can have other driver authors provide 
similar detail about their drivers in the same directory.

[2] http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source

> 
> 2) a list of the critical bugs that need to be fixed + any existing
> patches associated with those bugs, so they can be reviewed early in kilo
> 
> This blocks operation of nova+neutron environements:
> 
> https://bugs.launchpad.net/oslo.messaging/+bug/1301723
>   Summary: Message was sent to wrong node with zmq as rpc_backend
>   Patch: https://review.openstack.org/84938
> 
> Also notifcations are effectively unimplemented which prevents use
> with Ceilometer so I'd also add:
> 
> https://bugs.launchpad.net/oslo.messaging/+bug/1368154
>   Summary: https://bugs.launchpad.net/oslo.messaging/+bug/
>   Patch: https://review.openstack.org/120745

That’s a good list, and shorter than I expected. I have added these bugs to the 
next-kilo milestone.

> 
> 3) an analysis of what it would take to be able to run functional
> tests for zeromq on our CI infrastructure, not necessarily the full
> tempest run or devstack-gate job, probably functional tests we place
> in the tree with the driver (we will be doing this for all of the
> drivers) + besides writing new functional tests, we need to bring the
> unit tests for zeromq into the oslo.messaging repository
> 
> Kapil Thangavelu started work on both functional tests for the ZMQ
> driver last week; the output from the sprint is here:
> 
>https://github.com/ostack-musketeers/oslo.messaging
> 
> it covers the ZMQ driver (including messaging through the zmq-receiver
> proxy) and the associated MatchMakers (local, ring, redis) at a
> varying levels of coverage, but I feel it moves things in the right
> direction - Kapil's going to raise a review for this in the next
> couple of days.
> 
> Doug - has any structure been agreed within the oslo.messaging tree
> for unit/functional test splits? Right now we have them all in one place.

I think we will want them split up, but we don’t have an agreed existing 
structure for that. I would like to see a test framework of some sort that 
defines the tests in a way that can be used to run the same functional for all 
of the drivers as separate jobs (with appropriate hooks for ensuring the needed 
services are running, etc.). Setting that up warrants its own spec, because 
there are going to be quite a few details to work out. We will also need to 
participate in the larger conversation about how to set up those functional 
test jobs to be consistent wit