Re: [openstack-dev] [oslo] No complains about rabbitmq SSL problems: could we have this in the logs?

[Ken Giusti]

Hi Mohammed,

What release of openstack are you using?  (ocata, pike, etc)

Also just to confirm my understanding:  you do see the SSL connections come
up, but after some time they 'hang' - what do you mean by 'hang'?  Do the
connections drop?  Or do the connections remain up but you start seeing
messages (RPC calls) time out?

thanks,

On Wed, Oct 31, 2018 at 9:40 AM Mohammed Naser  wrote:

> For what it’s worth: I ran into the same issue.  I think the problem lies
> a bit deeper because it’s a problem with kombu as when debugging I saw that
> Oslo messaging tried to connect and hung after.
>
> Sent from my iPhone
>
> > On Oct 31, 2018, at 2:29 PM, Thomas Goirand  wrote:
> >
> > Hi,
> >
> > It took me a long long time to figure out that my SSL setup was wrong
> > when trying to connect Heat to rabbitmq over SSL. Unfortunately, Oslo
> > (or heat itself) never warn me that something was wrong, I just got
> > nothing working, and no log at all.
> >
> > I'm sure I wouldn't be the only one happy about having this type of
> > problems being yelled out loud in the logs. Right now, it does work if I
> > turn off SSL, though I'm still not sure what's wrong in my setup, and
> > I'm given no clue if the issue is on rabbitmq-server or on the client
> > side (ie: heat, in my current case).
> >
> > Just a wishlist... :)
> > Cheers,
> >
> > Thomas Goirand (zigo)
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] No complains about rabbitmq SSL problems: could we have this in the logs?

[Ken Giusti]

Hi,

There does seem to be something currently wonky with SSL &
oslo.messaging.   I'm looking into it now.

And there's this recently reported issue:

https://bugs.launchpad.net/oslo.messaging/+bug/1800957

In the above bug something seems to have broken SSL between ocata and
pike.  The current suspected change is a patch that fixed a threading issue.

Stay tuned...


On Thu, Nov 1, 2018 at 3:53 AM Thomas Goirand  wrote:

> On 10/31/18 2:40 PM, Mohammed Naser wrote:
> > For what it’s worth: I ran into the same issue.  I think the problem
> lies a bit deeper because it’s a problem with kombu as when debugging I saw
> that Oslo messaging tried to connect and hung after.
> >
> > Sent from my iPhone
> >
> >> On Oct 31, 2018, at 2:29 PM, Thomas Goirand  wrote:
> >>
> >> Hi,
> >>
> >> It took me a long long time to figure out that my SSL setup was wrong
> >> when trying to connect Heat to rabbitmq over SSL. Unfortunately, Oslo
> >> (or heat itself) never warn me that something was wrong, I just got
> >> nothing working, and no log at all.
> >>
> >> I'm sure I wouldn't be the only one happy about having this type of
> >> problems being yelled out loud in the logs. Right now, it does work if I
> >> turn off SSL, though I'm still not sure what's wrong in my setup, and
> >> I'm given no clue if the issue is on rabbitmq-server or on the client
> >> side (ie: heat, in my current case).
> >>
> >> Just a wishlist... :)
> >> Cheers,
> >>
> >> Thomas Goirand (zigo)
>
> I've opened a bug here:
>
> https://bugs.launchpad.net/oslo.messaging/+bug/1801011
>
> Cheers,
>
> Thomas Goirand (zigo)
>
> __________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Oslo][nova][openstack-ansible] About Rabbitmq warning problems on nova-compute side

[Ken Giusti]

Hi Gokhan,

There's been a flurry of folks reporting issues recently related to pike
and SSL.   See:

https://bugs.launchpad.net/oslo.messaging/+bug/1800957
and
https://bugs.launchpad.net/oslo.messaging/+bug/1801011

I'm currently working on this - no status yet.

As a test would it be possible to try disabling SSL in your configuration
to see if the problem persists?


On Thu, Nov 1, 2018 at 7:53 AM Gökhan IŞIK (BİLGEM BTE) <
gokhan.i...@tubitak.gov.tr> wrote:

> Hi folks,
>
> I have problems about rabbitmq on nova-compute side. I see lots of
> warnings in log file like that “client unexpectedly closed TCP
> connection”.[1]
>
> I have a HA OpenStack environment on ubuntu 16.04.5 which is installed
> with Openstack Ansible Project. My OpenStack environment version is Pike.
> My environment consists of 3 controller nodes ,23 compute nodes and 1 log
> node. Cinder volume service is installed on compute nodes and I am using
> NetApp Storage.
>
> I tried lots of configs on nova about oslo messaging and rabbitmq side,
> but I didn’t resolve this problem. My latest configs are below:
>
> rabbitmq.config is : http://paste.openstack.org/show/733767/
>
> nova.conf is: http://paste.openstack.org/show/733768/
>
> Services versions are : http://paste.openstack.org/show/733769/
>
>
> Can you share your experiences on rabbitmq side and How can I solve these
> warnings on nova-compute side ? What will you advice ?
>
>
> [1] http://paste.openstack.org/show/733766/
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral][oslo][messaging] Removing “blocking” executor from oslo.messaging

[Ken Giusti]

Hi Renat,
After discussing this a bit with Ben on IRC we're going to push the removal
off to T milestone 1.

I really like Ben's idea re: adding a blocking entry to your project's
setup.cfg file.  We can remove the explicit check for blocking in
oslo.messaging so you won't get an annoying warning if you want to load
blocking on your own.

Let me know what you think, thanks.

On Fri, Oct 19, 2018 at 12:02 AM Renat Akhmerov 
wrote:

> Hi,
>
>
> @Ken, I understand your considerations. I get that. I’m only asking not to
> remove it *for now*. And yes, if you think it should be discouraged from
> using it’s totally fine. But practically, it’s been the only reliable
> option for Mistral so far that may be our fault, I have to admit, because
> we weren’t able to make it work well with other executor types but we’ll
> try to fix that.
>
> By the way, I was playing with different options yesterday and it seems
> like that setting the executor to “threading” and the
> “executor_thread_pool_size” property to 1 behaves the same way as
> “blocking”. So may be that’s an option for us too, even if “blocking” is
> completely removed. But I would still be in favour of having some extra
> time to prove that with thorough testing.
>
> @Ben, including the executor via setup.cfg also looks OK to me. I see no
> issues with this approach.
>
>
> Thanks
>
> Renat Akhmerov
> @Nokia
> On 18 Oct 2018, 23:35 +0700, Ben Nemec , wrote:
>
>
>
> On 10/18/18 9:59 AM, Ken Giusti wrote:
>
> Hi Renat,
>
> The biggest issue with the blocking executor (IMHO) is that it blocks
> the protocol I/O while  RPC processing is in progress.  This increases
> the likelihood that protocol processing will not get done in a timely
> manner and things start to fail in weird ways.  These failures are
> timing related and are typically hard to reproduce or root-cause.   This
> isn't something we can fix as blocking is the nature of the executor.
>
> If we are to leave it in we'd really want to discourage its use.
>
>
> Since it appears the actual executor code lives in futurist, would it be
> possible to remove the entrypoint for blocking from oslo.messaging and
> have mistral just pull it in with their setup.cfg? Seems like they
> should be able to add something like:
>
> oslo.messaging.executors =
> blocking = futurist:SynchronousExecutor
>
> to their setup.cfg to keep it available to them even if we drop it from
> oslo.messaging itself. That seems like a good way to strongly discourage
> use of it while still making it available to projects that are really
> sure they want it.
>
>
> However I'm ok with leaving it available if the policy for using
> blocking is 'use at your own risk', meaning that bug reports may have to
> be marked 'won't fix' if we have reason to believe that blocking is at
> fault.  That implies removing 'blocking' as the default executor value
> in the API and having applications explicitly choose it.  And we keep
> the deprecation warning.
>
> We could perhaps implement time duration checks around the executor
> callout and log a warning if the executor blocked for an extended amount
> of time (extended=TBD).
>
> Other opinions so we can come to a consensus?
>
>
> On Thu, Oct 18, 2018 at 3:24 AM Renat Akhmerov  <mailto:renat.akhme...@gmail.com>> wrote:
>
> Hi Oslo Team,
>
> Can we retain “blocking” executor for now in Oslo Messaging?
>
>
> Some background..
>
> For a while we had to use Oslo Messaging with “blocking” executor in
> Mistral because of incompatibility of MySQL driver with green
> threads when choosing “eventlet” executor. Under certain conditions
> we would get deadlocks between green threads. Some time ago we
> switched to using PyMysql driver which is eventlet friendly and did
> a number of tests that showed that we could safely switch to
> “eventlet” executor (with that driver) so we introduced a new option
> in Mistral where we could choose an executor in Oslo Messaging. The
> corresponding bug is [1].
>
> The issue is that we recently found that not everything actually
> works as expected when using combination PyMysql + “eventlet”
> executor. We also tried “threading” executor and the system *seems*
> to work with it but surprisingly performance is much worse.
>
> Given all of that we’d like to ask Oslo Team not to remove
> “blocking” executor for now completely, if that’s possible. We have
> a strong motivation to switch to “eventlet” for other reasons
> (parallelism => better performance etc.) but seems like we need some
> time to make it smoothly.
>
>
> [1] https://bugs.launchpad.net/mistral/+bug/1696469
>
>
> Thanks
>
> Renat Akhmerov
> @Nokia
> ___

Re: [openstack-dev] [mistral][oslo][messaging] Removing “blocking” executor from oslo.messaging

[Ken Giusti]

Hi Renat,

The biggest issue with the blocking executor (IMHO) is that it blocks the
protocol I/O while  RPC processing is in progress.  This increases the
likelihood that protocol processing will not get done in a timely manner
and things start to fail in weird ways.  These failures are timing related
and are typically hard to reproduce or root-cause.   This isn't something
we can fix as blocking is the nature of the executor.

If we are to leave it in we'd really want to discourage its use.

However I'm ok with leaving it available if the policy for using blocking
is 'use at your own risk', meaning that bug reports may have to be marked
'won't fix' if we have reason to believe that blocking is at fault.  That
implies removing 'blocking' as the default executor value in the API and
having applications explicitly choose it.  And we keep the deprecation
warning.

We could perhaps implement time duration checks around the executor callout
and log a warning if the executor blocked for an extended amount of time
(extended=TBD).

Other opinions so we can come to a consensus?


On Thu, Oct 18, 2018 at 3:24 AM Renat Akhmerov 
wrote:

> Hi Oslo Team,
>
> Can we retain “blocking” executor for now in Oslo Messaging?
>
>
> Some background..
>
> For a while we had to use Oslo Messaging with “blocking” executor in
> Mistral because of incompatibility of MySQL driver with green threads when
> choosing “eventlet” executor. Under certain conditions we would get
> deadlocks between green threads. Some time ago we switched to using PyMysql
> driver which is eventlet friendly and did a number of tests that showed
> that we could safely switch to “eventlet” executor (with that driver) so we
> introduced a new option in Mistral where we could choose an executor in
> Oslo Messaging. The corresponding bug is [1].
>
> The issue is that we recently found that not everything actually works as
> expected when using combination PyMysql + “eventlet” executor. We also
> tried “threading” executor and the system *seems* to work with it but
> surprisingly performance is much worse.
>
> Given all of that we’d like to ask Oslo Team not to remove “blocking”
> executor for now completely, if that’s possible. We have a strong
> motivation to switch to “eventlet” for other reasons (parallelism => better
> performance etc.) but seems like we need some time to make it smoothly.
>
>
> [1] https://bugs.launchpad.net/mistral/+bug/1696469
>
>
> Thanks
>
> Renat Akhmerov
> @Nokia
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][glance][cinder][keystone][requirements] blocking oslo.messaging 9.0.0

[Ken Giusti]

On Tue, Oct 9, 2018 at 12:30 PM Doug Hellmann  wrote:

> Ken Giusti  writes:
>
> > On Tue, Oct 9, 2018 at 11:56 AM Doug Hellmann 
> wrote:
> >
> >> Matthew Thode  writes:
> >>
> >> > On 18-10-09 11:12:30, Doug Hellmann wrote:
> >> >> Matthew Thode  writes:
> >> >>
> >> >> > several projects have had problems with the new release, some have
> >> ways
> >> >> > of working around it, and some do not.  I'm sending this just to
> raise
> >> >> > the issue and allow a place to discuss solutions.
> >> >> >
> >> >> > Currently there is a review proposed to blacklist 9.0.0, but if
> this
> >> is
> >> >> > going to still be an issue somehow in further releases we may need
> >> >> > another solution.
> >> >> >
> >> >> > https://review.openstack.org/#/c/608835/
> >> >> >
> >> >> > --
> >> >> > Matthew Thode (prometheanfire)
> >> >> >
> >>
> __
> >> >> > OpenStack Development Mailing List (not for usage questions)
> >> >> > Unsubscribe:
> >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >>
> >> >> Do you have links to the failure logs or bug reports or something?
> If I
> >> >> wanted to help I wouldn't even know where to start.
> >> >>
> >> >
> >> >
> >>
> http://logs.openstack.org/21/607521/2/check/cross-cinder-py35/e15722e/testr_results.html.gz
> >>
> >> These failures look like we should add a proper API to oslo.messaging to
> >> set the notification and rpc backends for testing. The configuration
> >> options are *not* part of the API of the library.
> >>
> >> There is already an oslo_messaging.conffixture module with a fixture
> >> class, but it looks like it defaults to rabbit. Maybe someone wants to
> >> propose a patch to make that a parameter to the constructor?
> >>
> >
> > oslo.messaging's conffixture uses whatever the config default for
> > transport_url is unless the test
> > specifically overrides it by setting the transport_url attribute.
> > The o.m. unit tests's base test class sets conffixture.transport_url to
> > "fake:/" to use the fake in memory driver.
> > That's the existing practice (I believe it's used like that outside of
> o.m.)
>
> OK, so it sounds like the fixture is relying on the configuration to be
> set up in advance, and that's the thing we need to change. We don't want
> users outside of the library to set up tests by using the configuration
> options, right?
>

That's the intent of ConfFixture it seems - provide a wrapper API so tests
don't have to monkey directly with the config.

How about this:

  https://review.openstack.org/609063


>
> Doug
>


-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][glance][cinder][keystone][requirements] blocking oslo.messaging 9.0.0

[Ken Giusti]

On Tue, Oct 9, 2018 at 11:56 AM Doug Hellmann  wrote:

> Matthew Thode  writes:
>
> > On 18-10-09 11:12:30, Doug Hellmann wrote:
> >> Matthew Thode  writes:
> >>
> >> > several projects have had problems with the new release, some have
> ways
> >> > of working around it, and some do not.  I'm sending this just to raise
> >> > the issue and allow a place to discuss solutions.
> >> >
> >> > Currently there is a review proposed to blacklist 9.0.0, but if this
> is
> >> > going to still be an issue somehow in further releases we may need
> >> > another solution.
> >> >
> >> > https://review.openstack.org/#/c/608835/
> >> >
> >> > --
> >> > Matthew Thode (prometheanfire)
> >> >
> __
> >> > OpenStack Development Mailing List (not for usage questions)
> >> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >> Do you have links to the failure logs or bug reports or something? If I
> >> wanted to help I wouldn't even know where to start.
> >>
> >
> >
> http://logs.openstack.org/21/607521/2/check/cross-cinder-py35/e15722e/testr_results.html.gz
>
> These failures look like we should add a proper API to oslo.messaging to
> set the notification and rpc backends for testing. The configuration
> options are *not* part of the API of the library.
>
> There is already an oslo_messaging.conffixture module with a fixture
> class, but it looks like it defaults to rabbit. Maybe someone wants to
> propose a patch to make that a parameter to the constructor?
>

oslo.messaging's conffixture uses whatever the config default for
transport_url is unless the test
specifically overrides it by setting the transport_url attribute.
The o.m. unit tests's base test class sets conffixture.transport_url to
"fake:/" to use the fake in memory driver.
That's the existing practice (I believe it's used like that outside of o.m.)


>
> >
> http://logs.openstack.org/21/607521/2/check/cross-glance-py35/e2161d7/testr_results.html.gz
>
> These failures should be fixed by releasing the patch that Mehdi
> provided that ensures there is a valid default transport configured.
>
> >
> http://logs.openstack.org/21/607521/2/check/cross-keystone-py35/908a1c2/testr_results.html.gz
>
> Lance has already described these as mocking implementation details of
> the library. I expect we'll need someone with keystone experience to
> work out what the best solution is to do there.
>
> >
> > --
> > Matthew Thode (prometheanfire)
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][tacker][daisycloud-core][meteos] Removal of rpc_backend config opt from oslo.messaging

[Ken Giusti]

Thanks to work done by Steve Kowalik we're ready to remove the old
rpc_backend transport configuration option that has been deprecated since
mid 2016.  This removal involves changes to the oslo.messaging.ConfFixture
as well.

Steve has provided patches to those projects affected by these changes
Almost all projects have merged these patches.

There are a few projects - included in the subject line - where the
necessary patches have not yet landed.  If you're a committer on one of
these projects please make an effort to review the patches proposed for
your project:

https://review.openstack.org/#/q/topic:bug/1712399+status:open

Our goal is to land the removal next week.

thanks

-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [goals][python3] mixed versions?

[Ken Giusti]

On Thu, Sep 13, 2018 at 7:39 PM Doug Hellmann  wrote:

> Excerpts from Jim Rollenhagen's message of 2018-09-13 12:08:08 -0600:
> > On Wed, Sep 12, 2018 at 2:28 PM, Doug Hellmann 
> > wrote:
> >
> > > Excerpts from Doug Hellmann's message of 2018-09-12 12:04:02 -0600:
> > > > Excerpts from Clark Boylan's message of 2018-09-12 10:44:55 -0700:
> > > > > On Wed, Sep 12, 2018, at 10:23 AM, Jim Rollenhagen wrote:
> > > > > > The process of operators upgrading Python versions across their
> > > fleet came
> > > > > > up this morning. It's fairly obvious that operators will want to
> do
> > > this in
> > > > > > a rolling fashion.
> > > > > >
> > > > > > Has anyone considered doing this in CI? For example, running
> > > multinode
> > > > > > grenade with python 2 on one node and python 3 on the other node.
> > > > > >
> > > > > > Should we (openstack) test this situation, or even care?
> > > > > >
> > > > >
> > > > > This came up in a Vancouver summit session (the python3 one I
> think).
> > > General consensus there seemed to be that we should have grenade jobs
> that
> > > run python2 on the old side and python3 on the new side and test the
> update
> > > from one to another through a release that way. Additionally there was
> > > thought that the nova partial job (and similar grenade jobs) could
> hold the
> > > non upgraded node on python2 and that would talk to a python3 control
> plane.
> > > > >
> > > > > I haven't seen or heard of anyone working on this yet though.
> > > > >
> > > > > Clark
> > > > >
> > > >
> > > > IIRC, we also talked about not supporting multiple versions of
> > > > python on a given node, so all of the services on a node would need
> > > > to be upgraded together.
> > > >
> > > > Doug
> > >
> > > I spent a little time talking with the QA team about setting up
> > > this job, and Attila pointed out that we should think about what
> > > exactly we think would break during a 2-to-3 in-place upgrade like
> > > this.
> > >
> > > Keeping in mind that we are still testing initial installation under
> > > both versions and upgrades under python 2, do we have any specific
> > > concerns about the python *version* causing upgrade issues?
> > >
> >
> > A specific example brought up in the ironic room was the way we encode
> > exceptions in oslo.messaging for transmitting over RPC. I know that we've
> > found encoding bugs in that in the past, and one can imagine that RPC
> > between a service running on py2 and a service running on py3 could have
> > similar issues.
>
> Mixing python 2 and 3 components of the same service across nodes
> does seem like an interesting case. I wonder if it's something we
> could build a functional test job in oslo.messaging for, though,
> without having to test every service separately. I'd be happy if
> someone did that.
>
>
Currently that's a hole in the oslo.messaging tests.  I've opened a work
item to address this in launchpad:
https://bugs.launchpad.net/oslo.messaging/+bug/1792977


> > It's definitely edge cases that we'd be catching here (if any), so I'm
> > personally fine with assuming it will just work. But I wanted to pose the
> > question to the list, as we agreed this isn't only an ironic problem.
>
> Yes, definitely. I think it's likely to be a bit of work to set up the
> jobs and run them for all services, which is why I'm trying to
> understand if it's really needed. Thinking through the cases on the list
> is a good way to get folks to poke holes in any assertions, so I
> appreciate that you started the thread and that everyone is
> participating.
>
> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] proposing Moisés Guimarães for oslo.config core

[Ken Giusti]

On Wed, Aug 8, 2018 at 9:19 AM Doug Hellmann  wrote:
>
> Excerpts from Doug Hellmann's message of 2018-08-01 09:27:09 -0400:
> > Moisés Guimarães (moguimar) did quite a bit of work on oslo.config
> > during the Rocky cycle to add driver support. Based on that work,
> > and a discussion we have had since then about general cleanup needed
> > in oslo.config, I think he would make a good addition to the
> > oslo.config review team.
> >
> > Please indicate your approval or concerns with +1/-1.
> >
> > Doug
>
> Normally I would have added moguimar to the oslo-config-core team
> today, after a week's wait. Funny story, though. There is no
> oslo-config-core team.
>
> oslo.config is one of a few of our libraries that we never set up with a
> separate review team. It is managed by oslo-core. We could set up a new
> review team for that library, but after giving it some thought I
> realized that *most* of the libraries are fairly stable, our team is
> pretty small, and Moisés is a good guy so maybe we don't need to worry
> about that.
>
> I spoke with Moisés, and he agreed to be part of the larger core team.
> He pointed out that the next phase of the driver work is going to happen
> in castellan, so it would be useful to have another reviewer there. And
> I'm sure we can trust him to be careful with reviews in other repos
> until he learns his way around.
>
> So, I would like to amend my original proposal and suggest that we add
> Moisés to the oslo-core team.
>
> Please indicate support with +1 or present any concerns you have. I
> apologize for the confusion on my part.
>
> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

+1

-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Proposing Zane Bitter as oslo.service core

[Ken Giusti]

+1!

On Fri, Aug 3, 2018 at 12:58 PM, Ben Nemec  wrote:
> Hi,
>
> Zane has been doing some good work in oslo.service recently and I would like
> to add him to the core team.  I know he's got a lot on his plate already,
> but he has taken the time to propose and review patches in oslo.service and
> has demonstrated an understanding of the code.
>
> Please respond with +1 or any concerns you may have.  Thanks.
>
> -Ben
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][ci][infra][aodh][telemetry] telemetry test broken on oslo.messaging stable/queens

[Ken Giusti]

Updated subject to include [aodh] and [telemetry]

On Tue, Jun 5, 2018 at 11:41 AM, Doug Hellmann  wrote:
> Excerpts from Ken Giusti's message of 2018-06-05 10:47:17 -0400:
>> Hi,
>>
>> The telemetry integration test for oslo.messaging has started failing
>> on the stable/queens branch [0].
>>
>> A quick review of the logs points to a change in heat-tempest-plugin
>> that is incompatible with the version of gabbi from queens upper
>> constraints (1.40.0) [1][2].
>>
>> The job definition [3] includes required-projects that do not have
>> stable/queens branches - including heat-tempest-plugin.
>>
>> My question - how do I prevent this job from breaking when these
>> unbranched projects introduce changes that are incompatible with
>> upper-constrants for a particular branch?
>
> Aren't those projects co-gating on the oslo.messaging test job?
>
> How are the tests working for heat's stable/queens branch? Or telemetry?
> (whichever project is pulling in that tempest repo)
>

I've run the stable/queens branches of both Aodh[1] and Heat[2] - both failed.

Though the heat failure is different from what we're seeing on
oslo.messaging [3],
the same warning about gabbi versions is there [4].

However the Aodh failure is exactly the same as the oslo.messaging one
[5] - this makes sense since the oslo.messaging test is basically
running the same telemetry-tempest-plugin test.

So this isn't something unique to oslo.messaging - the telemetry
integration test is busted in stable/queens.

I'm going to mark these tests as non-voting on oslo.messaging's queens
branch for now so we can land some pending patches.


[1] https://review.openstack.org/#/c/574306/
[2] https://review.openstack.org/#/c/574311/
[3] 
http://logs.openstack.org/11/574311/1/check/heat-functional-orig-mysql-lbaasv2/21cce1d/job-output.txt.gz#_2018-06-11_17_30_51_106223
[4] 
http://logs.openstack.org/11/574311/1/check/heat-functional-orig-mysql-lbaasv2/21cce1d/logs/devstacklog.txt.gz#_2018-06-11_17_09_39_691
[5] 
http://logs.openstack.org/06/574306/1/check/telemetry-dsvm-integration/0a9620a/job-output.txt.gz#_2018-06-11_16_53_33_982143



>>
>> I've tried to use override-checkout in the job definition, but that
>> seems a bit hacky in this case since the tagged versions don't appear
>> to work and I've resorted to a hardcoded ref [4].
>>
>> Advice appreciated, thanks!
>>
>> [0] https://review.openstack.org/#/c/567124/
>> [1] 
>> http://logs.openstack.org/24/567124/1/check/oslo.messaging-telemetry-dsvm-integration-rabbit/e7fdc7d/logs/devstack-gate-post_test_hook.txt.gz#_2018-05-16_05_20_05_624
>> [2] 
>> http://logs.openstack.org/24/567124/1/check/oslo.messaging-telemetry-dsvm-integration-rabbit/e7fdc7d/logs/devstacklog.txt.gz#_2018-05-16_05_19_06_332
>> [3] 
>> https://git.openstack.org/cgit/openstack/oslo.messaging/tree/.zuul.yaml?h=stable/queens#n250
>> [4] https://review.openstack.org/#/c/572193/2/.zuul.yaml
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [heat][ci][infra] telemetry test broken on oslo.messaging stable/queens

[Ken Giusti]

Hi,

The telemetry integration test for oslo.messaging has started failing
on the stable/queens branch [0].

A quick review of the logs points to a change in heat-tempest-plugin
that is incompatible with the version of gabbi from queens upper
constraints (1.40.0) [1][2].

The job definition [3] includes required-projects that do not have
stable/queens branches - including heat-tempest-plugin.

My question - how do I prevent this job from breaking when these
unbranched projects introduce changes that are incompatible with
upper-constrants for a particular branch?

I've tried to use override-checkout in the job definition, but that
seems a bit hacky in this case since the tagged versions don't appear
to work and I've resorted to a hardcoded ref [4].

Advice appreciated, thanks!

[0] https://review.openstack.org/#/c/567124/
[1] 
http://logs.openstack.org/24/567124/1/check/oslo.messaging-telemetry-dsvm-integration-rabbit/e7fdc7d/logs/devstack-gate-post_test_hook.txt.gz#_2018-05-16_05_20_05_624
[2] 
http://logs.openstack.org/24/567124/1/check/oslo.messaging-telemetry-dsvm-integration-rabbit/e7fdc7d/logs/devstacklog.txt.gz#_2018-05-16_05_19_06_332
[3] 
https://git.openstack.org/cgit/openstack/oslo.messaging/tree/.zuul.yaml?h=stable/queens#n250
[4] https://review.openstack.org/#/c/572193/2/.zuul.yaml
-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Oslo][Nova][Sahara][Tempest][Cinder][Magnum] Removing broken tox missing requirements tests

[Ken Giusti]

Folks,

Here in Oslo land a number of projects define a tox test for missing
dependencies. These tests are based on a tool - pip-check-reqs - that
no longer functions under the latest release of pip.  The project's
upstream github repo hasn't had any commit activity in a year and
appears to no longer be maintained.

See my previous email about this tool:
http://lists.openstack.org/pipermail/openstack-dev/2018-April/129697.html

In lieu of a suitable replacement, I've started removing the broken
tox tests from the oslo project to prevent anyone else having that
"Hmm - why doesn't this test pass?" moment I hit last week.

I've created a epad that lists the projects that define tox tests
based on this tool:

https://etherpad.openstack.org/p/pip_(missing|check)_reqs

There are other non-Oslo projects - Nova, Cinder, etc - that may want
to also remove that test. See the epad for details.

I've started patches for a couple of projects, but if anyone is
willing to help out please use the epad so we don't step on each
other's toes.

thanks,

-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Catching missing or stale package requirements in requirements.txt

[Ken Giusti]

Hi Folks,

Some of the Oslo libraries have a tox test that does the above [0].
This ensures that our requirements.txt file is kept current with the
code.

This test uses a tool called pip_check_reqs [1].  Unfortunately this
tool is not compatible with pip version 10, and it appears as if the
github project hasn't seen any development activity in the last 2
years.  Seems unlikely that pip 10 support will be added anytime soon.

Can anyone recommend a suitable alternative to the pip_check_reqs tool?

Thanks in advance,

[0] https://git.openstack.org/cgit/openstack/oslo.messaging/tree/tox.ini#n116
[1] https://github.com/r1chardj0n3s/pip-check-reqs
-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all][oslo] Notice to users of the ZeroMQ transport in oslo.messaging

[Ken Giusti]

Folks,

It's been over a year since the last commit was made to the ZeroMQ
driver in oslo.messaging.  It is at the point where some of the
related unit tests are beginning to fail due to bit rot.  None of the
current oslo.messaging contributors have a good enough understanding
of the codebase to effectively fix it.

Personally I'm not sure the driver will work in production at all.

Given this it was decided in Dublin that the ZeroMQ driver no longer
meets the official policy for in tree driver support [0] and will be
deprecated in Rocky.  However it would be insincere for the team to
give the impression that the driver is maintained for the normal 2
cycle deprecation process.  Therefore the driver code will be removed
in 'S'.

The ZeroMQ driver is the largest body of code of any driver in the
oslo.messaging repo, weighing in at over 5k lines of code.  For
comparison, the rabbitmq kombu driver consists of only about 2K lines
of code.  If any individuals are willing to commit to ownership of
this codebase and keep the driver compliant with policy (see [0]),
please follow up with bnemec or myself (kgiusti) on #openstack-oslo.

Thanks,


[0] 
https://docs.openstack.org/oslo.messaging/latest/contributor/supported-messaging-drivers.html


-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][all] Deprecation Notice: Pika driver for oslo.messaging

[Ken Giusti]

Folks,

Last year at the Boston summit the Oslo team decided to deprecate
support for the Pika transport in oslo.messaging with removal planned
for Rocky [0].

This was announced on the operators list last May [1].

No objections have been raised to date. We're not aware of any
deployments using this transport and its
removal is not anticipated to affect anyone.

This is notice that the removal is currently underway [2].

Thanks,

[0] 
https://etherpad.openstack.org/p/BOS_Forum_Oslo.Messaging_driver_recommendations
[1] 
http://lists.openstack.org/pipermail/openstack-operators/2017-May/013579.html
[2] https://review.openstack.org/#/c/536960/

-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Oslo PTG Summary

[Ken Giusti]

Hi Doug,

Andy updated the etherpad [0] with a new link [1].
Holler if it's still broken...


[0] https://etherpad.openstack.org/p/oslo-ptg-rocky
[1] 
https://docs.google.com/presentation/d/1PWJAGQohAvlwod4gMTp6u1jtZT1cuaE-whRmnV8uiMM/edit?usp=sharing

On Mon, Mar 12, 2018 at 11:54 AM, Doug Hellmann <d...@doughellmann.com> wrote:
> I can’t see
>
> https://docs.google.com/presentation/d/e/2PACX-1vQpaSSm7Amk9q4sBEAUi_IpyJ4l07qd3t5T_BPZkdLWfYbtSpSmF7obSB1qRGA65wjiiq2Sb7H2ylJo/pub?start=false=false=3000=id.p
>
>
>
> On Mar 12, 2018, at 11:39 AM, Ken Giusti <kgiu...@gmail.com> wrote:
>
> Hi Josh - I'm able to view all of them, but I probably have special
> google powers ;)
>
> Which links are broken for you?
>
> thanks,
>
> On Thu, Mar 8, 2018 at 3:53 PM, Joshua Harlow <harlo...@fastmail.com> wrote:
>
>
> Can we get some of those doc links opened.
>
> 'You need permission to access this published document.' I am getting for a
> few of them :(
>
>
> Ben Nemec wrote:
>
>
> Hi,
>
> Here's my summary of the discussions we had in the Oslo room at the PTG.
> Please feel free to reply with any additions if I missed something or
> correct anything I've misrepresented.
>
> oslo.config drivers for secret management
> -
>
> The oslo.config implementation is in progress, while the Castellan
> driver still needs to be written. We want to land this early in Rocky as
> it is a significant change in architecture for oslo.config and we want
> it to be well-exercised before release.
>
> There are discussions with the TripleO team around adding support for
> this feature to its deployment tooling and there will be a functional
> test job for the Castellan driver with Custodia.
>
> There is a weekly meeting in #openstack-meeting-3 on Tuesdays at 1600
> UTC for discussion of this feature.
>
> oslo.config driver implementation: https://review.openstack.org/#/c/513844
> spec:
>
> https://specs.openstack.org/openstack/oslo-specs/specs/queens/oslo-config-drivers.html
>
> Custodia key management support for Castellan:
> https://review.openstack.org/#/c/515190/
>
> "stable" libraries
> --
>
> Some of the Oslo libraries are in a mature state where there are very
> few, if any, meaningful changes to them. With the removal of the
> requirements sync process in Rocky, we may need to change the release
> process for these libraries. My understanding was that there were no
> immediate action items for this, but it was something we need to be
> aware of.
>
> dropping support for mox3
> -
>
> There was some concern that no one from the Oslo team is actually in a
> position to support mox3 if something were to break (such as happened in
> some libraries with Python 3.6). Since there is a community goal to
> remove mox from all OpenStack projects in Rocky this will hopefully not
> be a long-term problem, but there was some discussion that if projects
> needed to keep mox for some reason that they would be asked to provide a
> maintainer for mox3. This topic is kind of on hold pending the outcome
> of the community goal this cycle.
>
> automatic configuration migration on upgrade
> 
>
> There is a desire for oslo.config to provide a mechanism to
> automatically migrate deprecated options to their new location on
> version upgrades. This is a fairly complex topic that I can't cover
> adequately in a summary email, but there is a spec proposed at
> https://review.openstack.org/#/c/520043/ and POC changes at
> https://review.openstack.org/#/c/526314/ and
> https://review.openstack.org/#/c/526261/
>
> One outcome of the discussion was that in the initial version we would
> not try to handle complex migrations, such as the one that happened when
> we combined all of the separate rabbit connection opts into a single
> connection string. To start with we will just raise a warning to the
> user that they need to handle those manually, but a templated or
> hook-based method of automating those migrations could be added as a
> follow-up if there is sufficient demand.
>
> oslo.messaging plans
> 
>
> There was quite a bit discussed under this topic. I'm going to break it
> down into sub-topics for clarity.
>
> oslo.messaging heartbeats
> =
>
> Everyone seemed to be in favor of this feature, so we anticipate
> development moving forward in Rocky. There is an initial patch proposed
> at https://review.openstack.org/546763
>
> We felt that it should be possible to opt in and out of the feature, and
> that the configuration shou

Re: [openstack-dev] [oslo] Oslo PTG Summary

[Ken Giusti]

/ Turns out it's
>> really simple!
>>
>> Nova is also using this functionality for more complex options related
>> to upgrades, so that would be a good place to look for more advanced use
>> cases.
>>
>> Full documentation for the mutable config options is at
>> https://docs.openstack.org/oslo.config/latest/reference/mutable.html
>>
>> The goal status is being tracked in
>> https://storyboard.openstack.org/#!/story/2001545
>>
>> Chang Bo was also going to talk to Lance about possibly coming up with a
>> burndown chart like the one he had for the policy in code work.
>>
>> oslo healthcheck middleware
>> ---
>>
>> As this ended up being the only major topic for the afternoon, the
>> session was unfortunately lightly attended. However, the self-healing
>> SIG was talking about related topics at the same time so we ended up
>> moving to that room and had a good discussion.
>>
>> Overall the feature seemed to be well-received. There is some security
>> concern with exposing service information over an un-authenticated
>> endpoint, but because there is no authentication supported by the health
>> checking functionality in things like Kubernetes or HAProxy this is
>> unavoidable. The feature won't be mandatory, so if this exposure is
>> unacceptable it can be turned off (with a corresponding loss of
>> functionality, of course).
>>
>> There was also some discussion of dropping the asynchronous nature of
>> the checks in the initial version in order to keep the complexity to a
>> minimum. Asynchronous testing can always be added later if it proves
>> necessary.
>>
>> The full spec is at https://review.openstack.org/#/c/531456
>>
>> oslo.config strict validation
>> -
>>
>> I actually had discussions with multiple people about this during the
>> week. In both cases, they were just looking for a minimal amount of
>> validation that would catch an error such at "devug=True". Such a
>> validation might be fairly simple to write now that we have the
>> YAML-based sample config with (ideally) information about all the
>> options available to set in a project. It should be possible to compare
>> the options set in the config file with the ones listed in the sample
>> config and raise warnings for any that don't exist.
>>
>> There is also a more complete validation spec at
>>
>> http://specs.openstack.org/openstack/oslo-specs/specs/ocata/oslo-validator.html
>> and a patch proposed at https://review.openstack.org/#/c/384559/
>>
>> Unfortunately there has been little movement on that as of late, so it
>> might be worthwhile to implement something more minimalist initially and
>> then build from there. The existing patch is quite significant and
>> difficult to review.
>>
>> Conclusion
>> --
>>
>> I feel like there were a lot of good discussions at the PTG and we have
>> plenty of work to keep the small Oslo team busy for the Rocky cycle. :-)
>>
>> Thanks to everyone who participated and I look forward to seeing how
>> much progress we've made at the next Summit and PTG.
>>
>> -Ben
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] proposing Stephen Finucan for oslo-core

[Ken Giusti]

+1 for Stephen!

On Mon, Jan 8, 2018 at 9:55 AM, Doug Hellmann <d...@doughellmann.com> wrote:
> Stephen (sfinucan) has been working on pbr, oslo.config, and
> oslo.policy and reviewing several of the other Oslo libraries for
> a while now. His reviews are always helpful and I think he would
> make a good addition to the oslo-core team.
>
> As per our usual practice, please reply here with a +1 or -1 and
> any reservations.
>
> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Oslo team updates

[Ken Giusti]

+1, and a big thank you for all your contributions

On Tue, Jan 2, 2018 at 9:39 AM, Davanum Srinivas <dava...@gmail.com> wrote:
> +1 from me as well. Thanks everyone!
>
> On Tue, Jan 2, 2018 at 9:31 AM, Doug Hellmann <d...@doughellmann.com> wrote:
>> Excerpts from ChangBo Guo's message of 2018-01-02 11:53:02 +0800:
>>> In last two cycles some people's situation changed, can't focus on Oslo
>>> code review, so I propose  some changes in Oslo team.  Remove following
>>> people, thanks their past hard wok to make Oslo well, and welcome them back
>>> if they want to join the team again.  please +1/-1 for the change
>>>
>>> Generalist Code Reviewers:
>>>  Brant Knudson
>>>
>>> Specialist API Maintainers:
>>> oslo-cache-core:  Brant Kundson  David Stanek
>>> oslo-db-core: Viktor Serhieiev
>>> oslo-messaging-core:Dmitriy Ukhlov Oleksii Zamiatin Viktor Serhieiev
>>> oslo-policy-core: Brant Kundson  David Stanek guang-yee
>>> oslo-service-core: Marian Horban
>>>
>>> We welcome anyone join the team or contribution in Oslo. The Oslo program
>>> brings together generalist code reviewers and specialist API maintainers
>>> They share a common interest in tackling copy-and-paste technical debt
>>> across the OpenStack project. For more information please refer to wiki
>>> [1].
>>>
>>> [1] https://wiki.openstack.org/wiki/Oslo
>>
>> +1 -- it's sad to see the team shrink a bit, but it's good to keep the
>> list accurate based on when people can contribute.
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Davanum Srinivas :: https://twitter.com/dims
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Oslo][oslo.messaging][all] Notice: upcoming change to oslo.messaging RPC server

[Ken Giusti]

Hi Folks,

Just a head's up:

In Queens the default access policy for RPC Endpoints will change from
LegacyRPCAccessPolicy to DefaultRPCAccessPolicy.  RPC calls to private
('_' prefix) methods will no longer be possible.  If you want to allow
RPC Clients to invoke private methods, you must explicitly set the
access_policy to LegacyRPCAccessPolicy when you call get_rpc_server()
or instantiate an RPCDispatcher.  This change [0] has been merged to
oslo.messaging master and will appear in the next release of
oslo.messaging.

"Umm What?"

Good question!  Here's the TL;DR details:

Since forever it's been possible for a client to make an RPC call
against _any_ method defined in the RPC Endpoint object.  And by "any"
we mean "all methods including private ones (method names prefixed by
'_' )"

Naturally this ability came as a surprise many folk [1], including
yours truly and others on the oslo team [2].  It was agreed that
having this be the default behavior was indeed A Bad Thing.

So starting in Ocata oslo.messaging has provided a means for
controlling access to Endpoint methods [3].  Oslo.messaging now
defines three different "access control policies" that can be applied
to an RPC Server:

LegacyRPCAccessPolicy: original behavior - any method can be invoked
by an RPC client
DefaultRPCAccessPolicy: prevent RPC access to private '_' methods, all
others may be invoked
ExplicitRPCAccessPolicy: only allow access to those methods that have
been decorated with @expose decorator

See [4] for more details.

In order not to break anything at the time the default access policy
was set to 'LegacyRPCAccessPolicy'.  This has been the default for
Ocata and Pike.

Starting in Queens this will no longer be the case.
DefaultRPCAccessPolicy will become the default if no access policy is
specified when calling get_rpc_server() or directly instantiating an
RPCDispatcher.  To keep the old behavior you must explicitly set the
access policy to LegacyRPCAccessPolicy:

from oslo_messaging.rpc import LegacyRPCAccessPolicy
...
server = get_rpc_server(transport, target, endpoints,
 access_policy=LegacyRPCAccessPolicy)



Reply here if you have any questions or hit any issues, thanks!

-K

[0] https://review.openstack.org/#/c/500456/
[1] https://bugs.launchpad.net/oslo.messaging/+bug/1194279
[2] https://bugs.launchpad.net/oslo.messaging/+bug/1555845
[3] https://review.openstack.org/#/c/358359/
[4] https://docs.openstack.org/oslo.messaging/latest/reference/server.html
-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][barbican][sahara] start RPC service before launcher wait?

[Ken Giusti]

On Thu, Sep 14, 2017 at 7:33 PM, Adam Spiers <aspi...@suse.com> wrote:
>
> Hi Ken,
>
> Thanks a lot for the analysis, and sorry for the slow reply!
> Comments inline...
>
> Ken Giusti <kgiu...@gmail.com> wrote:
> > Hi Adam,
> >
> > I think there's a couple of problems here.
> >
> > Regardless of worker count, the service.wait() is called before
> > service.start().  And from looking at the oslo.service code, the 'wait()'
> > method is call after start(), then again after stop().  This doesn't match
> > up with the intended use of oslo.messaging.server.wait(), which should only
> > be called after .stop().
>
> Hmm, so are you saying that there might be a bug in oslo.service's
> usage of oslo.messaging, and that this Sahara bugfix was the wrong
> approach too?
>
> https://review.openstack.org/#/c/280741/1/sahara/cli/sahara_engine.py
>

Well, I don't think the explicit call to start() is going to help,
esp. if the number of workers is > 1 since the workers are forked and
need to call start() from their own process space..
In fact, if # of workers > 1 then you not only get an RPC server in
each worker process, you'll end up with an extra RPC
server in the calling thread.

Take a look at a test service I've created for oslo.messaging:

https://pastebin.com/rSA6AD82

If you change the main code to call the new sequence, you'll end up
with 3 rpc servers (2 in the workers, one in the main process).

In that code I've made the wait() call a no op if the server hasn't
been started first.   And the stop method will call stop and wait on
the rpc server, which is the expected sequence as far as
oslo.messaging is concerned.

To me it seems that the bug is in oslo.service - calling wait() before
start() doesn't make sense to me.

> > Perhaps a bigger issue is that in the multi threaded case all threads
> > appear to be calling start, wait, and stop on the same instance of the
> > service (oslo.messaging rpc server).  At least that's what I'm seeing in my
> > muchly reduced test code:

I was wrong about this - I failed to notice that each service had
forked and was dealing with its own copy of the server.

> >
> > https://paste.fedoraproject.org/paste/-73zskccaQvpSVwRJD11cA
> >
> > The log trace shows multiple calls to start, wait, stop via different
> > threads to the same TaskServer instance:
> >
> > https://paste.fedoraproject.org/paste/dyPq~lr26sQZtMzHn5w~Vg
> >
> > Is that expected?
>
> Unfortunately in the interim, your pastes seem to have vanished - any
> chance you could repaste them?
>

Ugh - didn't keep a copy.  If you pull down that test code you can use
it to generate those traces.


> Thanks,
> Adam
>
> > On Mon, Jul 31, 2017 at 9:32 PM, Adam Spiers <aspi...@suse.com> wrote:
> > > Ken Giusti <kgiu...@gmail.com> wrote:
> > >> On Mon, Jul 31, 2017 at 10:01 AM, Adam Spiers <aspi...@suse.com> wrote:
> > >>> I recently discovered a bug where barbican-worker would hang on
> > >>> shutdown if queue.asynchronous_workers was changed from 1 to 2:
> > >>>
> > >>>https://bugs.launchpad.net/barbican/+bug/1705543
> > >>>
> > >>> resulting in a warning like this:
> > >>>
> > >>>WARNING oslo_messaging.server [-] Possible hang: stop is waiting for
> > >>> start to complete
> > >>>
> > >>> I found a similar bug in Sahara:
> > >>>
> > >>>https://bugs.launchpad.net/sahara/+bug/1546119
> > >>>
> > >>> where the fix was to call start() on the RPC service before making the
> > >>> launcher wait() on it, so I ported the fix to Barbican, and it seems
> > >>> to work fine:
> > >>>
> > >>>https://review.openstack.org/#/c/485755
> > >>>
> > >>> I noticed that both projects use ProcessLauncher; barbican uses
> > >>> oslo_service.service.launch() which has:
> > >>>
> > >>>if workers is None or workers == 1:
> > >>>launcher = ServiceLauncher(conf, restart_method=restart_method)
> > >>>else:
> > >>>launcher = ProcessLauncher(conf, restart_method=restart_method)
> > >>>
> > >>> However, I'm not an expert in oslo.service or oslo.messaging, and one
> > >>> of Barbican's core reviewers (thanks Kaitlin!) noted that not many
> > >>> other projects start the task before calling wait() on the launcher,
> > >>> so I thought I'd check here whether that is the correct fix, or
> > 

Re: [openstack-dev] [openstack][oslo][all] Clean up oslo deprecated stuff !

[Ken Giusti]

On Tue, Aug 22, 2017 at 3:24 AM, ChangBo Guo <glongw...@gmail.com> wrote:

> Hi ALL,
>
> We discussed a little about how to avoid breaking consuming projects' gate
> in oslo weekly meeting last week.  The easy improvement is that clean up
> deprecated stuff in oslo at the beginning of Queens.  I collected
> deprecated stuff  in [1].  This need Oslo team and other team work
> toghether. I think we can start the work when cycle Queens is open. I
> also reserved a room in PTG
> for 2 hours to do hacking.[2]
>
>
> [1] https://etherpad.openstack.org/p/oslo-queens-tasks
> [2] https://ethercalc.openstack.org/Queens-PTG-Discussion-Rooms
>
> --
> ChangBo Guo(gcb)
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
I've gone through oslo.messaging and have updated the etherpad [1] with a
few more entries.  I've opened launchpad bugs where appropriate, adding the
oslo-debt-cleanup tag to each for good measure.

The original list included a few items that as best I can tell were not
tagged as deprecated via debtcollector until Pike. IIUC we can't remove
those in Queens, correct?

-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][oslo.messaging] Proposing Andy Smith as an oslo.messaging core reviewers

[Ken Giusti]

+1

On Mon, Aug 14, 2017 at 6:59 AM, ChangBo Guo <glongw...@gmail.com> wrote:

> I propose that we add Andy Smith to the oslo.messaging team.
>
> Andy Smith has been actively contributing to oslo.messaging for a while
> now, both
> in helping make oslo.messaging better via code contribution(s) and by
> helping with
> the review load when he can. He's been involved on the AMQP 1.0 side for
> awhile. He's really interested in taking ownership of the experimental
> Kafka driver, which would be great to have someone able to drive that.
>
> Please respond with +1/-1
>
> Voting will last 2 weeks and will end at 28th of August.
>
> Cheers,
> --
> ChangBo Guo(gcb)
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [FEMDC][MassivelyDistributed] Strawman proposal for message bus analysis

[Ken Giusti]

v] [FEMDC][MassivelyDistributed] Strawman
> > > > proposal for message bus analysis
> > > >
> > > > Hi Matthieu,
> > > >
> > > > You mentioned 15000 connections with 1000 compute nodes.
> > > > Was that mostly Nova? Was ceilometer involved?
> > > > I would be curious to know how much AMQP traffic is
> Control
> > > > related
> > > > (e.g. spinning up VMs) vs how much is telemetry related
> in a
> > > > typical
> > > > openstack deployment.
> > > > Do we know that?
> > > >
> > > > I have also left some comments in the doc.
> > > >
> > > > Paul-Andre
> > > >
> > > >
> > > > -Original Message-
> > > > From: Matthieu Simonin <matthieu.simo...@inria.fr>
> > > > Reply-To: "OpenStack Development Mailing List (not for usage
> > > > questions)"
> > > > <openstack-dev@lists.openstack.org>
> > > > Date: Wednesday, June 21, 2017 at 6:54 PM
> > > > To: "OpenStack Development Mailing List (not for usage
> > > > questions)"
> > > > <openstack-dev@lists.openstack.org>
> > > > Subject: Re: [openstack-dev] [FEMDC][MassivelyDistributed]
> > > > Strawman
> > > > proposal for  message bus analysis
> > > >
> > > > Hi Ken,
> > > >
> > > > Thanks for starting this !
> > > > I've made a first pass on the epad and left some notes
> and
> > > > questions
> > > > there.
> > > >
> > > > Best,
> > > >
> > > > Matthieu
> > > > - Mail original -
> > > > > De: "Ken Giusti" <kgiu...@gmail.com>
> > > > > À: "OpenStack Development Mailing List (not for usage
> > > > > questions)"
> > > > > <openstack-dev@lists.openstack.org>
> > > > > Envoyé: Mercredi 21 Juin 2017 15:23:26
> > > > > Objet: [openstack-dev] [FEMDC][MassivelyDistributed]
> > > > > Strawman
> > > > > proposal formessage bus analysis
> > > > >
> > > > > Hi All,
> > > > >
> > > > > Andy and I have taken a stab at defining some test
> > > > > scenarios
> > > > > for
> > > > > anal the
> > > > > different message bus technologies:
> > > > >
> > > > > https://etherpad.openstack.org/p/1BGhFHDIoi
> > > > >
> > > > > We've started with tests for just the oslo.messaging
> layer
> > > > > to
> > > > > analyze
> > > > > throughput and latency as the number of message bus
> clients
> > > > > -
> > > > > and
> > > > > the bus
> > > > > itself - scale out.
> > > > >
> > > > > The next step will be to define messaging oriented test
> > > > > scenarios
> > > > > for an
> > > > > openstack deployment.  We've started by enumerating a
> few
> > > > > of
> > > > > the
> > > > > tools,
> > > > > topologies, and fault conditions that need to be
> covered.
> > > > >
> > > > > Let's use this epad as a starting point for analyzing
> > > > > messaging -
> > > > > please
> > > > > feel free to contribute, question, and criticize :)
> > > > >
> > > > > thanks,
> > > > >
> > > > >
> > > > >
> > &g

[openstack-dev] [oslo][barbican][sahara] start RPC service before launcher wait?

[Ken Giusti]

Oop - didn't reply all
-- Forwarded message --
From: Ken Giusti <kgiu...@gmail.com>
Date: Tue, Aug 1, 2017 at 12:51 PM
Subject: Re: [openstack-dev] [oslo][barbican][sahara] start RPC service
before launcher wait?
To: Adam Spiers <aspi...@suse.com>


Hi Adam,

I think there's a couple of problems here.

Regardless of worker count, the service.wait() is called before
service.start().  And from looking at the oslo.service code, the 'wait()'
method is call after start(), then again after stop().  This doesn't match
up with the intended use of oslo.messaging.server.wait(), which should only
be called after .stop().

Perhaps a bigger issue is that in the multi threaded case all threads
appear to be calling start, wait, and stop on the same instance of the
service (oslo.messaging rpc server).  At least that's what I'm seeing in my
muchly reduced test code:

https://paste.fedoraproject.org/paste/-73zskccaQvpSVwRJD11cA

The log trace shows multiple calls to start, wait, stop via different
threads to the same TaskServer instance:

https://paste.fedoraproject.org/paste/dyPq~lr26sQZtMzHn5w~Vg

Is that expected?

On Mon, Jul 31, 2017 at 9:32 PM, Adam Spiers <aspi...@suse.com> wrote:

> Ken Giusti <kgiu...@gmail.com> wrote:
>
>> On Mon, Jul 31, 2017 at 10:01 AM, Adam Spiers <aspi...@suse.com> wrote:
>>
>>> I recently discovered a bug where barbican-worker would hang on
>>> shutdown if queue.asynchronous_workers was changed from 1 to 2:
>>>
>>>https://bugs.launchpad.net/barbican/+bug/1705543
>>>
>>> resulting in a warning like this:
>>>
>>>WARNING oslo_messaging.server [-] Possible hang: stop is waiting for
>>> start to complete
>>>
>>> I found a similar bug in Sahara:
>>>
>>>https://bugs.launchpad.net/sahara/+bug/1546119
>>>
>>> where the fix was to call start() on the RPC service before making the
>>> launcher wait() on it, so I ported the fix to Barbican, and it seems
>>> to work fine:
>>>
>>>https://review.openstack.org/#/c/485755
>>>
>>> I noticed that both projects use ProcessLauncher; barbican uses
>>> oslo_service.service.launch() which has:
>>>
>>>if workers is None or workers == 1:
>>>launcher = ServiceLauncher(conf, restart_method=restart_method)
>>>else:
>>>launcher = ProcessLauncher(conf, restart_method=restart_method)
>>>
>>> However, I'm not an expert in oslo.service or oslo.messaging, and one
>>> of Barbican's core reviewers (thanks Kaitlin!) noted that not many
>>> other projects start the task before calling wait() on the launcher,
>>> so I thought I'd check here whether that is the correct fix, or
>>> whether there's something else odd going on.
>>>
>>> Any oslo gurus able to shed light on this?
>>>
>>
>> As far as an oslo.messaging server is concerned, the order of operations
>> is:
>>
>> server.start()
>> # do stuff until ready to stop the server...
>> server.stop()
>> server.wait()
>>
>> The final wait blocks until all requests that are in progress when stop()
>> is called finish and cleanup.
>>
>
> Thanks - that makes sense.  So the question is, why would
> barbican-worker only hang on shutdown when there are multiple workers?
> Maybe the real bug is somewhere in oslo_service.service.ProcessLauncher
> and it's not calling start() correctly?
>



-- 
Ken Giusti  (kgiu...@gmail.com)



-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][barbican][sahara] start RPC service before launcher wait?

[Ken Giusti]

On Mon, Jul 31, 2017 at 10:01 AM, Adam Spiers <aspi...@suse.com> wrote:

> Hi all,
>
> I recently discovered a bug where barbican-worker would hang on
> shutdown if queue.asynchronous_workers was changed from 1 to 2:
>
>https://bugs.launchpad.net/barbican/+bug/1705543
>
> resulting in a warning like this:
>
>WARNING oslo_messaging.server [-] Possible hang: stop is waiting for
> start to complete
>
> I found a similar bug in Sahara:
>
>https://bugs.launchpad.net/sahara/+bug/1546119
>
> where the fix was to call start() on the RPC service before making the
> launcher wait() on it, so I ported the fix to Barbican, and it seems
> to work fine:
>
>https://review.openstack.org/#/c/485755
>
> I noticed that both projects use ProcessLauncher; barbican uses
> oslo_service.service.launch() which has:
>
>if workers is None or workers == 1:
>launcher = ServiceLauncher(conf, restart_method=restart_method)
>else:
>launcher = ProcessLauncher(conf, restart_method=restart_method)
>
> However, I'm not an expert in oslo.service or oslo.messaging, and one
> of Barbican's core reviewers (thanks Kaitlin!) noted that not many
> other projects start the task before calling wait() on the launcher,
> so I thought I'd check here whether that is the correct fix, or
> whether there's something else odd going on.
>
> Any oslo gurus able to shed light on this?
>
> Thanks!
> Adam
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


As far as an oslo.messaging server is concerned, the order of operations is:

server.start()
# do stuff until ready to stop the server...
server.stop()
server.wait()

The final wait blocks until all requests that are in progress when stop()
is called finish and cleanup.

-K


-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [FEMDC][MassivelyDistributed] Strawman proposal for message bus analysis

[Ken Giusti]

On Wed, Jun 21, 2017 at 10:13 AM, Ilya Shakhat <shak...@gmail.com> wrote:

> Hi Ken,
>
> Please check scenarios and reports that exist in Performance Docs. In
> particular you may be interested in:
>  * O.M.Simulator - https://github.com/openstack/o
> slo.messaging/blob/master/tools/simulator.py
>  * MQ  performance scenario - https://docs.openstack.org/dev
> eloper/performance-docs/test_plans/mq/plan.html#message-queue-performance
>  * One of RabbitMQ reports - https://docs.openstack.org/dev
> eloper/performance-docs/test_results/mq/rabbitmq/cmsm/index.html
>  * MQ HA scenario - https://docs.openstack.org/dev
> eloper/performance-docs/test_plans/mq_ha/plan.html
>  * One of RabbitMQ HA reports - https://docs.openstack.org/dev
> eloper/performance-docs/test_results/mq_ha/rabbitmq-ha-
> queues/cs1ss2-ks2-ha/omsimulator-ha-call-cs1ss2-ks2-ha/index.html
>
>
Thank you Ilya - these tests you reference are indeed valuable.

But, IIUC, those tests benchmark queue throughput, using a single (highly
threaded) client->single server traffic flow.   If that is the case, I
think the tests we're trying to define might be a bit more specific to the
FEMDC [0] use cases:  multiple servers consuming from different topics
while many clients distributed across the message bus are connecting,
generating traffic, failing over, etc.

The goal of these tests would be to quantify the behavior of the message
bus as a whole under different messaging loads, failure conditions, etc.

[0] https://wiki.openstack.org/wiki/Fog_Edge_Massively_Distributed_Clouds




>
> Thanks,
> Ilya
>
> 2017-06-21 15:23 GMT+02:00 Ken Giusti <kgiu...@gmail.com>:
>
>> Hi All,
>>
>> Andy and I have taken a stab at defining some test scenarios for anal the
>> different message bus technologies:
>>
>> https://etherpad.openstack.org/p/1BGhFHDIoi
>>
>> We've started with tests for just the oslo.messaging layer to analyze
>> throughput and latency as the number of message bus clients - and the bus
>> itself - scale out.
>>
>> The next step will be to define messaging oriented test scenarios for an
>> openstack deployment.  We've started by enumerating a few of the tools,
>> topologies, and fault conditions that need to be covered.
>>
>> Let's use this epad as a starting point for analyzing messaging - please
>> feel free to contribute, question, and criticize :)
>>
>> thanks,
>>
>>
>>
>> --
>> Ken Giusti  (kgiu...@gmail.com)
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [FEMDC][MassivelyDistributed] Strawman proposal for message bus analysis

[Ken Giusti]

On Wed, Jun 21, 2017 at 11:24 AM, Jay Pipes <jaypi...@gmail.com> wrote:

> On 06/21/2017 09:23 AM, Ken Giusti wrote:
>
>> Andy and I have taken a stab at defining some test scenarios for anal the
>> different message bus...
>>
>
> That was a particularly unfortunatey choice of words.
>
>
Ugh. Sorry - most unfortunate fat-finger...or Freudian slip...


>
> Best,
> -jay
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [FEMDC][MassivelyDistributed] Strawman proposal for message bus analysis

[Ken Giusti]

Hi All,

Andy and I have taken a stab at defining some test scenarios for anal the
different message bus technologies:

https://etherpad.openstack.org/p/1BGhFHDIoi

We've started with tests for just the oslo.messaging layer to analyze
throughput and latency as the number of message bus clients - and the bus
itself - scale out.

The next step will be to define messaging oriented test scenarios for an
openstack deployment.  We've started by enumerating a few of the tools,
topologies, and fault conditions that need to be covered.

Let's use this epad as a starting point for analyzing messaging - please
feel free to contribute, question, and criticize :)

thanks,



-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging]Optimize RPC performance by reusing callback queue

[Ken Giusti]

Hi,

Keep in mind the rabbit driver creates a single reply queue per *transport*
- that is per call to oslo.messaging's
get_transport/get_rpc_transport/get_notification_transport.

If you have multiple RPCClients sharing the same transport, then all
clients issuing RPC calls over that transport will use the same reply queue
(and multiplex incoming replies using a unique id in the reply itself).
See
https://git.openstack.org/cgit/openstack/oslo.messaging/tree/oslo_messaging/_drivers/amqpdriver.py?h=stable/newton#n452
for all the details.

But it cannot share the reply queue across transports - and certainly not
across processes :)

-K



On Wed, Jun 7, 2017 at 10:29 PM, int32bit <kryst...@gmail.com> wrote:

> Hi,
>
> Currently, I find our RPC client always need create a new callback queue
> for every call requests to track the reply belongs, at least in Newton.
> That's pretty inefficient and lead to poor performance. I also  find some
> RPC implementations no need to create a new queue, they track the request
> and response by correlation id in message header(rabbitmq well supports,
> not sure is it AMQP standard?). The rabbitmq official document provide a
> simple demo, see [1].
>
> So I am confused that why our oslo.messaging doesn't use this way
> to optimize RPC performance. Is it for any consideration or do I miss
> some potential cases?
>
> Thanks for any reply and discussion!
>
>
> [1] https://www.rabbitmq.com/tutorials/tutorial-six-python.html.
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Ken Giusti  (kgiu...@gmail.com)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][oslo.messaging] Call to deprecate the 'pika' driver in the oslo.messaging project

[Ken Giusti]

So it's been over a week with no objections.

I will start the deprecation process, including an announcement on the
operator's email list.

Thanks for the feedback.

On Mon, May 22, 2017 at 8:06 PM, ChangBo Guo <glongw...@gmail.com> wrote:
> +1 , let's focus on key drivers.
>
> 2017-05-17 2:02 GMT+08:00 Joshua Harlow <harlo...@fastmail.com>:
>>
>> Fine with me,
>>
>> I'd personally rather get down to say 2 'great' drivers for RPC,
>>
>> And say 1 (or 2?) for notifications.
>>
>> So ya, wfm.
>>
>> -Josh
>>
>>
>> Mehdi Abaakouk wrote:
>>>
>>> +1 too, I haven't seen its contributors since a while.
>>>
>>> On Mon, May 15, 2017 at 09:42:00PM -0400, Flavio Percoco wrote:
>>>>
>>>> On 15/05/17 15:29 -0500, Ben Nemec wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 05/15/2017 01:55 PM, Doug Hellmann wrote:
>>>>>>
>>>>>> Excerpts from Davanum Srinivas (dims)'s message of 2017-05-15
>>>>>> 14:27:36 -0400:
>>>>>>>
>>>>>>> On Mon, May 15, 2017 at 2:08 PM, Ken Giusti <kgiu...@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Folks,
>>>>>>>>
>>>>>>>> It was decided at the oslo.messaging forum at summit that the pika
>>>>>>>> driver will be marked as deprecated [1] for removal.
>>>>>>>
>>>>>>>
>>>>>>> [dims} +1 from me.
>>>>>>
>>>>>>
>>>>>> +1
>>>>>
>>>>>
>>>>> Also +1
>>>>
>>>>
>>>> +1
>>>>
>>>> Flavio
>>>>
>>>> --
>>>> @flaper87
>>>> Flavio Percoco
>>>
>>>
>>>
>>>
>>>>
>>>> __
>>>>
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>
>>> __________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> --
> ChangBo Guo(gcb)
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][oslo.messaging] Call to deprecate the 'pika' driver in the oslo.messaging project

[Ken Giusti]

Folks,

It was decided at the oslo.messaging forum at summit that the pika
driver will be marked as deprecated [1] for removal.

The pika driver is another rabbitmq-based driver.  It was developed as
a replacement for the current rabbit driver (rabbit://).  The pika
driver is based on the 'pika' rabbitmq client library [2], rather than
the kombu library [3] of the current rabbitmq driver.  The pika
library was recommended by the rabbitmq community a couple of summits
ago as a better client than the kombu client.

However, testing done against this driver did not show "appreciable
difference in performance or reliability" over the existing rabbitmq
driver.

Given this, and the recent departure of some very talented
contributors, the consensus is to deprecate pika and recommend users
stay with the original rabbitmq driver.

The plan is to mark the driver as deprecated in Pike, removal in Rocky.

thanks,


[1] 
https://etherpad.openstack.org/p/BOS_Forum_Oslo.Messaging_driver_recommendations
  (~ line 80)
[2] https://github.com/pika/pika
[3] https://github.com/celery/kombu

-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Oslo][oslo.messaging] FYI: forum for "Oslo.Messaging Non-rabbitmq Backend Recommendations" scheduled 5/11 1:30 MR104

[Ken Giusti]

Just a head's up: this is an additional session to continue the
discussion from Weds

etherpad: 
https://etherpad.openstack.org/p/BOS_Forum_Oslo.Messaging_driver_recommendations

thanks,

-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo]Introduction of new driver for oslo.messaging

[Ken Giusti]

This is gonna be _huge_.

As a matter of fact, I like this so much I'm abandoning my proposal to
support for RPC over rfc2549.

-K

On Sat, Apr 1, 2017 at 10:49 AM, Amrith Kumar <amrith.ku...@gmail.com> wrote:
> Great idea, happy to try it out for Trove. We love o.m.rpc :) But it needs to 
> be secure; other comment has been posed in review, I'm doing a talk about o.m 
> use by trove in Boston anyway, maybe we can get Melissa to join me for that?
>
> -amrith
>
>
> -Original Message-
> From: Deja, Dawid [mailto:dawid.d...@intel.com]
> Sent: Friday, March 31, 2017 10:41 AM
> To: openstack-dev@lists.openstack.org
> Subject: [openstack-dev] [oslo]Introduction of new driver for oslo.messaging
>
> Hi all,
>
> To work around issues with rabbitMQ scalability we'd like to introduce new 
> driver in oslo messaging that have nearly no scaling limits[1].
> We'd like to have as much eyes on this as possible since we believe that this 
> is the technology of the future. Thanks for all reviews.
>
> Dawid Deja
>
> [1] https://review.openstack.org/#/c/452219/
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo_messaging] Limiting the number of retries for kafka driver

[Ken Giusti]

On Tue, Feb 14, 2017 at 2:52 PM, Elancheran Subramanian
<esubraman...@godaddy.com> wrote:
> Hello All,
> This is reg limiting the number of retries for Kafka driver support on Nova
> and Neutron.
>
> While trying out the oslo messaging notifications support for Kafka on Nova
> and Neutron, the Kafka driver doesn’t support limiting the number of retries
> for failed messages. When I checked the code, currently there is no
> configuration which support that, though the send_notification has retry
> https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_kafka.py#L336
> but it’s not set or passed from component’s (nova or neutron’s)
> configuration. Is there any configuration which I’m missing? Please let me
> know.
>

You haven't missed anything - the kafka driver doesn't provide a means
to set a default retry via its configuration.
The expectation is that the caller (nova/neutron) would provide a
retry value when constructing a Notifier instance.

There was such a config option for the rabbitmq driver
(rabbit_max_retries) but that was removed because it broke
*something* - can't remember exactly the reason tho, sorry.

>
> Thanks in advance,
> Cheran
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Oslo] Oslo logo files (was 'log files')

[Ken Giusti]

These are great!

I've tweaked the subject lest folks think this only applies to logging.

"Watch me pull an RPC call outta my hat... nothing up my sleeve... Presto!"

(google 'bullwinkle' - pop culture reference... clearly showing my age
(and mentality) :)

-K

On Mon, Feb 13, 2017 at 11:35 PM, ChangBo Guo <glongw...@gmail.com> wrote:
> We got  cool logos in different format, just use them :-)
> I’m excited to finally be able to share final project logo files with you.
> Inside this folder, you’ll find full-color and one-color versions of the
> logo, plus a mascot-only version, in EPS, JPG and PNG formats. You can use
> them on presentations and wherever else you’d like to add some team flair.
>
> https://www.dropbox.com/sh/kj0e3sdu47pqr3e/AABllB31vJZDlw4OkZRK_AZia?dl=0
>
> --
> ChangBo Guo(gcb)
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Not running for Oslo PTL for Pike

[Ken Giusti]

On Tue, Jan 3, 2017 at 3:03 PM, Joshua Harlow <harlo...@fastmail.com> wrote:
> Hi Oslo folks (and others),
>
> Happy new year!
>
> After serving for about a year I think it's a good opportunity for myself to
> let another qualified individual run for Oslo PTL (seems common to only go
> for two terms and hand-off to another).
>
> So I just wanted to let folks know that I will be doing this, so that we can
> grow others in the community that wish to try out being a PTL.
>
> I don't plan on leaving the Oslo community btw, just want to make sure we
> spread the knowledge (and the fun!) of being a PTL.
>
> Hopefully I've been a decent PTL (with  room to improve) during this
> time :-)
>

Dude - you've been a most excellent PTL!

Thanks for all the help (and laughs :) you've provided in the past year.

> -Josh
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Splitting notifications from rpc (and questions + work around this)

[Ken Giusti]

On Wed, Nov 2, 2016 at 8:11 PM, Joshua Harlow <harlo...@fastmail.com> wrote:
> Hi folks,
>
> There was a bunch of chatter at the summit about how there are really two
> different types of (oslo) messaging usage that exist in openstack and how
> they need not be backed by the same solution type (rabbitmq, qpid,
> kafka...).
>
> For those that were not at the oslo sessions:
>
> https://wiki.openstack.org/wiki/Design_Summit/Ocata/Etherpads#Oslo
>
> The general gist was though that we need to make sure people really do know
> that there are two very different types of messaging usage in openstack and
> to ensure that operators (and developers) are picking the right backing
> technology for each type.
>
> So some questions naturally arise out of this.
>
> * Where are the best practices with regard to selection of the best backend
> type for rpc (and one for notifications); is this something oslo.messaging
> should work through (or can the docs team and operator group also help in
> making this)?
>
> * What are the tradeoffs in using the same (or different) technology for rpc
> and notifications?
>

I think the olso.messaging team should take the lead here and educate
users as to what the options are, and  how the two supported messaging
services (RPC and Notifications) differ with respect to backend
requirements.   These topics really should be part of the
oslo.messaging 'Theory of Operations' documentation that was discussed
during the Arch WG summit meeting.

Currently the biggest functional difference between the backends is
the support of store-and-forward (e.g. queueing) verses point-to-point
message transfer.  Oslo could at least explain the pros and cons of
each approach with respect to the RPC and Notification services so
that folks understand what the tradeoffs and advantages are in the
first place.

Furthermore the team should also document the functional differences
between the various choices of backends.  For instance it would be
useful to understand how the two supported point-to-point backends
(zmq and dispatch router) differ in both behavior and recommended
deployment.


> * Is it even possible for all oslo.messaging consuming projects to be able
> to choose 2 different backends, are consuming projects consuming the library
> correctly so that they can use 2 different backends?
>
> * Is devstack able to run with say kafka for notifications and rabbitmq for
> rpc (if not, is there any work item the oslo group can help with to make
> this possible) so that we can ensure and test that all projects can work
> correctly with appropriate (and possibly different) backends?
>
> * Any other messaging, arch-wg work that we (oslo or others) can help out
> with to make sure that projects (and operators) are using the right
> technology for the right use (and not just defaulting to RPC over rabbitmq
> because it exists, when in reality something else might be a better choice)?
>

Ultimately there should be recommendations for which backends are
optimal for a range of different deployment scenarios, but at this
point we really don't have enough data and experience with these
backends to create such recommendations.

> * More(?)
>
> Just wanted to get this conversation started, because afaik it's one that
> has not been widely circulated (and operators have been setting up rabbitmq
> in various HA and clustered and ... modes, when in reality thinking through
> what and how it is used may be more appropriate); this also applies to
> developers since some technical solutions in openstack seem to be created
> due to (in-part) rabbitmq shortcomings (cells v1 afaik was *in* part created
> due to scaling issues).
>
> -Josh
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Adding gdavoian to oslo-core

[Ken Giusti]

+1 for Gevorg!

On Mon, Oct 3, 2016 at 2:02 PM, Davanum Srinivas <dava...@gmail.com> wrote:
> +1 from me!
>
> On Mon, Oct 3, 2016 at 1:40 PM, Joshua Harlow <harlo...@fastmail.com> wrote:
>> Greetings all stackers,
>>
>> I propose that we add Gevorg Davoian[1] to the oslo-core[2] team.
>>
>> Gevorg has been actively contributing to oslo for a while now, both in
>> helping make oslo better via code contribution(s) and by helping with the
>> review load when he can. He has provided quality reviews and is doing an
>> awesome job with the various oslo concepts and helping make oslo the best it
>> can be!
>>
>> Overall I think he would make a great addition to the core review team.
>>
>> Please respond with +1/-1.
>>
>> Thanks much!
>>
>> - Joshua Harlow
>>
>> [1] https://launchpad.net/~gdavoian
>> [2] https://launchpad.net/oslo
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Davanum Srinivas :: https://twitter.com/dims
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Adding ozamiatin to oslo-core

[Ken Giusti]

+1 for sure.

On Mon, Oct 3, 2016 at 1:42 PM, Joshua Harlow <harlo...@fastmail.com> wrote:
> Greetings all stackers,
>
> I propose that we add Oleksii Zamiatin[1] to the oslo-core[2] team.
>
> Oleksii has been actively contributing to oslo for a while now, both in
> helping make oslo better via code contribution(s) and by helping with the
> review load when he can. He has provided quality reviews and is doing an
> awesome job with the various oslo concepts and helping make oslo the best it
> can be!
>
> Overall I think he would make a great addition to the core review team.
>
> Please respond with +1/-1.
>
> Thanks much!
>
> - Joshua Harlow
>
> [1] https://launchpad.net/~ozamiatin
> [2] https://launchpad.net/oslo
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][massively distributed][architecture]Coordination between actions/WGs

[Ken Giusti]

On Thu, Sep 1, 2016 at 4:53 PM, Ian Wells <ijw.ubu...@cack.org.uk> wrote:
> On 1 September 2016 at 06:52, Ken Giusti <kgiu...@gmail.com> wrote:
>>
>> On Wed, Aug 31, 2016 at 3:30 PM, Ian Wells <ijw.ubu...@cack.org.uk> wrote:
>
>> > I have opinions about other patterns we could use, but I don't want to
>> > push
>>
>> > my solutions here, I want to see if this is really as much of a problem
>> > as
>> > it looks and if people concur with my summary above.  However, the right
>> > approach is most definitely to create a new and more fitting set of oslo
>> > interfaces for communication patterns, and then to encourage people to
>> > move
>> > to the new ones from the old.  (Whether RabbitMQ is involved is neither
>> > here
>> > nor there, as this is really a question of Oslo APIs, not their
>> > implementation.)
>> >
>>
>> Hmm... maybe.   Message bus technology is varied, and so is it's
>> behavior.  There are brokerless, point-to-point backends supported by
>> oslo.messaging [1],[2] which will exhibit different
>> capabilities/behaviors from the traditional broker-based
>> store-and-forward backend (e.g. message acking end-to-end vs to the
>> intermediary).
>
>
> The important thing is that you shouldn't have to look behind the curtain.
> We can offer APIs that are driven by the implementation (designed for test,
> and trivial to implement correctly given handy open source projects we know
> and trust) and the choice of design will therefore be dependent on the
> backend mechanisms we consider for use to implement the API.  APIs are
> always a point of negotiation between what the caller needs and what can be
> implemented in a practical amount of time.  But *I do not care* whether
> you're using rabbits or carrier pigeons just so long as what you have
> documented that the API promises me is actually true.  I *do not expect* to
> have to read RabbitMQ ior AMQP documentation to work out what behaviour I
> should expect for my messaging.  And its behaviour should be consistent if I
> have a choice of messaging backends.
>

And I agree totally - this is the way it _should_ be.  And to get
there we do have to address the ambiguities in the existing API, as
well as extend it so applications can explicitly state their service
needs.

My point is that the API also has to be _backend_ agnostic.  That
really hasn't been the priority it should be IMHO.  The current API as
it stands leaks too much of the backend behavior past the API.

For example here's where we are with the current API: a majority of
deployments are broker based - applications using oslo.messaging  have
come to rely _indirectly_ on the behavioral side effects of using a
broker backend.  In fact RabbitMQ's operational characteristics have
become the de-facto "correct" behavior.  Any other backend that
doesn't exhibit exactly the same behavior as RabbitMQ is considered
buggy.   Consider qpidd for example - simple differences in default
queue lifecycle and default flow control settings resulted in
messaging behavior different from RabbitMQ.  These were largely
considered bugs in qpidd.  I think this played a large part in the
lack of adoption of qpidd.

And qpidd is the same type of messaging backend as rabbitmq - a
broker.  Imagine what deployer's are going to hit when they attempt to
use a completely different technology - non-brokered backends like
Zeromq or message routing.

Improving the API as you describe will go a long way to solving this
situation.  And believe me I agree 100% that this API work needs to be
done.

But the API redesign should be done in a backend-agnostic manner.  We
(the oslo.messaging devs) have to ensure that _required_ API features
cannot be tied to any one backend implementation.  For example things
like notification pools are trivial to support for broker backends,
but hard/impossible for point to point distributed technologies.  It
must be clear to the application devs that using those optional
features that cannot be effectively implemented for a given backend
basically forces the deployer's hand.

My point is yes we need to improve that API but it should be done in a
backend agnostic way. There are currently features/behaviors that
essentially require a broker back end.  We should avoid making such
features mandatory elements of the API and ensure that the API users
are well aware of the consequences for deployers when using such
features.


>> All the more reason to have explicit delivery guarantees and well
>> understood failure scenarios defined by the API.
>
> And on this point we totally agree.
>
> I think the point of an API is to subdivide who carries which
> responsibilities - the caller for handling exceptional cases and th

Re: [openstack-dev] [all][massively distributed][architecture]Coordination between actions/WGs

[Ken Giusti]

On Wed, Aug 31, 2016 at 3:30 PM, Ian Wells <ijw.ubu...@cack.org.uk> wrote:
> On 31 August 2016 at 10:12, Clint Byrum <cl...@fewbar.com> wrote:
>>
>> Excerpts from Duncan Thomas's message of 2016-08-31 12:42:23 +0300:
>> > On 31 August 2016 at 11:57, Bogdan Dobrelya <bdobre...@mirantis.com>
>> > wrote:
>> >
>> > > I agree that RPC design pattern, as it is implemented now, is a major
>> > > blocker for OpenStack in general. It requires a major redesign,
>> > > including handling of corner cases, on both sides, *especially* RPC
>> > > call
>> > > clients. Or may be it just have to be abandoned to be replaced by a
>> > > more
>> > > cloud friendly pattern.
>> >
>> >
>> > Is there a writeup anywhere on what these issues are? I've heard this
>> > sentiment expressed multiple times now, but without a writeup of the
>> > issues
>> > and the design goals of the replacement, we're unlikely to make progress
>> > on
>> > a replacement - even if somebody takes the heroic approach and writes a
>> > full replacement themselves, the odds of getting community by-in are
>> > very
>> > low.
>>
>> Right, this is exactly the sort of thing I'd like to gather a group of
>> design-minded folks around in an Architecture WG. Oslo is busy with the
>> implementations we have now, but I'm sure many oslo contributors would
>> like to come up for air and talk about the design issues, and come up
>> with a current design, and some revisions to it, or a whole new one,
>> that can be used to put these summit hallway rumors to rest.
>
>
> I'd say the issue is comparatively easy to describe.  In a call sequence:
>
> 1. A sends a message to B
> 2. B receives messages
> 3. B acts upon message
> 4. B responds to message
> 5. A receives response
> 6. A acts upon response
>
> ... you can have a fault at any point in that message flow (consider crashes
> or program restarts).  If you ask for something to happen, you wait for a
> reply, and you don't get one, what does it mean?  The operation may have
> happened, with or without success, or it may not have gotten to the far end.
> If you send the message, does that mean you'd like it to cause an action
> tomorrow?  A year from now?  Or perhaps you'd like it to just not happen?
> Do you understand what Oslo promises you here, and do you think every person
> who ever wrote an RPC call in the whole OpenStack solution also understood
> it?
>

Precisely - IMHO it's a shortcoming of the current o.m. RPC (and
Notification) API in that it does not let the API user explicitly set
the desired delivery guarantee when publishing.  Right now it's
implied that the delivery guarantee is "At Most Once" but that's
mostly not precisely defined in any meaningful way.

Any messaging API should be explicit regarding what delivery
guarantee(s) are possible.  In addition, an API should allow the user
to designate the importance of a message on a per-send basis:  can
this message be dropped?  can this message be duplicated?  At what
point in time does the message become invalid (already offered for RPC
via timeout, but not Notifications IIRC), etc

And well-understood failure modes... things always fail...


> I have opinions about other patterns we could use, but I don't want to push
> my solutions here, I want to see if this is really as much of a problem as
> it looks and if people concur with my summary above.  However, the right
> approach is most definitely to create a new and more fitting set of oslo
> interfaces for communication patterns, and then to encourage people to move
> to the new ones from the old.  (Whether RabbitMQ is involved is neither here
> nor there, as this is really a question of Oslo APIs, not their
> implementation.)
>

Hmm... maybe.   Message bus technology is varied, and so is it's
behavior.  There are brokerless, point-to-point backends supported by
oslo.messaging [1],[2] which will exhibit different
capabilities/behaviors from the traditional broker-based
store-and-forward backend (e.g. message acking end-to-end vs to the
intermediary).

All the more reason to have explicit delivery guarantees and well
understood failure scenarios defined by the API.

[1] http://docs.openstack.org/developer/oslo.messaging/zmq_driver.html
[2] http://docs.openstack.org/developer/oslo.messaging/AMQP1.0.html


> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][massively distributed][architecture]Coordination between actions/WGs

[Ken Giusti]

On Wed, Aug 31, 2016 at 6:02 PM, Clint Byrum <cl...@fewbar.com> wrote:
> Excerpts from Ian Wells's message of 2016-08-31 12:30:45 -0700:
>> On 31 August 2016 at 10:12, Clint Byrum <cl...@fewbar.com> wrote:
>>
>> > Excerpts from Duncan Thomas's message of 2016-08-31 12:42:23 +0300:
>> > > On 31 August 2016 at 11:57, Bogdan Dobrelya <bdobre...@mirantis.com>
>> > wrote:
>> > >
>> > > > I agree that RPC design pattern, as it is implemented now, is a major
>> > > > blocker for OpenStack in general. It requires a major redesign,
>> > > > including handling of corner cases, on both sides, *especially* RPC
>> > call
>> > > > clients. Or may be it just have to be abandoned to be replaced by a
>> > more
>> > > > cloud friendly pattern.
>> > >
>> > >
>> > > Is there a writeup anywhere on what these issues are? I've heard this
>> > > sentiment expressed multiple times now, but without a writeup of the
>> > issues
>> > > and the design goals of the replacement, we're unlikely to make progress
>> > on
>> > > a replacement - even if somebody takes the heroic approach and writes a
>> > > full replacement themselves, the odds of getting community by-in are very
>> > > low.
>> >
>> > Right, this is exactly the sort of thing I'd like to gather a group of
>> > design-minded folks around in an Architecture WG. Oslo is busy with the
>> > implementations we have now, but I'm sure many oslo contributors would
>> > like to come up for air and talk about the design issues, and come up
>> > with a current design, and some revisions to it, or a whole new one,
>> > that can be used to put these summit hallway rumors to rest.
>> >
>>
>> I'd say the issue is comparatively easy to describe.  In a call sequence:
>>
>> 1. A sends a message to B
>> 2. B receives messages
>> 3. B acts upon message
>> 4. B responds to message
>> 5. A receives response
>> 6. A acts upon response
>>
>> ... you can have a fault at any point in that message flow (consider
>> crashes or program restarts).  If you ask for something to happen, you wait
>> for a reply, and you don't get one, what does it mean?  The operation may
>> have happened, with or without success, or it may not have gotten to the
>> far end.  If you send the message, does that mean you'd like it to cause an
>> action tomorrow?  A year from now?  Or perhaps you'd like it to just not
>> happen?  Do you understand what Oslo promises you here, and do you think
>> every person who ever wrote an RPC call in the whole OpenStack solution
>> also understood it?
>>
>> I have opinions about other patterns we could use, but I don't want to push
>> my solutions here, I want to see if this is really as much of a problem as
>> it looks and if people concur with my summary above.  However, the right
>> approach is most definitely to create a new and more fitting set of oslo
>> interfaces for communication patterns, and then to encourage people to move
>> to the new ones from the old.  (Whether RabbitMQ is involved is neither
>> here nor there, as this is really a question of Oslo APIs, not their
>> implementation.)
>
> I think it's about time we get some Architecture WG meetings started,
> and put "Document RPC design" on the agenda.
>

+1 I'm certainly interested in helping out here.



> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] what do you work on upstream of OpenStack?

[Ken Giusti]

Hi Doug,

The pyngus library - it was originally designed as part of the
oslo.messaging amqp1 driver but proved to be useful as a stand alone
messaging API built for proton.

https://pypi.python.org/pypi/pyngus

On Mon, Jun 13, 2016 at 3:11 PM, Doug Hellmann <d...@doughellmann.com> wrote:
> I'm trying to pull together some information about contributions
> that OpenStack community members have made *upstream* of OpenStack,
> via code, docs, bug reports, or anything else to dependencies that
> we have.
>
> If you've made a contribution of that sort, I would appreciate a
> quick note.  Please reply off-list, there's no need to spam everyone,
> and I'll post the summary if folks want to see it.
>
> Thanks,
> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][mistral] Saga of process than ack and where can we go from here...

[Ken Giusti]

een targeting at
> that point, but is more like a work-queue) be in another library with a
> clear API that explicitly is targeted at this kind of model. Thus
> instead of having a multi-personality codebase with hidden features like
> this (say in oslo.messaging) instead it gets its own codebase and API
> that is 'just right' (or more close to being 'right') for it's concept
> (vs trying to stuff it into oslo.messaging).
>
>
> What happened to the idea of adding new functions at the level of the
> call & cast functions we have now, that work with at-least-once instead
> of at-most-once semantics? Yes this is a different sort of use case, but
> it's still "messaging".
>
>
> The idea I think is dead. Joshua essentially told the reasons in the
> previous message.
>
> Renat Akhmerov
> @Nokia
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Common RPC Message Trace Mechanism

[Ken Giusti]

Hi,

The 'trace' boolean offered by the AMQP 1.0 driver exposes a debug feature
that is provided by the Proton library.  This is specific to the Proton
library - I'm not sure kombu/zmq/etc offer a similar feature.

As Xuanzhou points out, this debug tool merely prints to stdout a summary
of each AMQP 1.0 protocol frame before it is written/after it is read from
the socket.  It prints the entire protocol exchange (control frames, etc)
and is not limited to just the message traffic.  Given that, I don't think
the transport drivers can implement such a low level debug feature unless
it is offered by the protocol libraries themselves.

-K


On Sun, Mar 6, 2016 at 11:55 PM Xuanzhou Perry Dong 
wrote:

> Hi, Boris,
>
> Thanks for your response.
>
> I refer to the very simple type of "trace": just print out the RPC
> messages to stdout/stderr/syslog. I have checked the osprofiler project and
> think that it is very good and could solve my problem if it is used by the
> Openstack projects to trace their RPC calls.
>
>
> Best Regards,
> Xuanzhou Perry Dong
>
> At 2016-03-07 12:27:12, "Boris Pavlovic"  wrote:
>
> Xuanzhou,
>
> I am not sure what do you mean by "trace". But if you need something that
> allows to do cross service/project tracing then you should take a look at
> osprofiler:
> https://github.com/openstack/osprofiler
>
> Best regards,
> Boris Pavlovic
>
> On Sun, Mar 6, 2016 at 8:15 PM, Xuanzhou Perry Dong 
> wrote:
>
>> Hi,
>>
>> I am looking for a common RPC message trace mechanism in oslo_messaging.
>> This message trace mechanism needs to be common to all drivers. Currently
>> some documentation mentions that oslo_messaging_amqp.trace can activate the
>> message trace (say,
>> http://docs.openstack.org/liberty/config-reference/content/networking-configuring-rpc.html).
>> But it seems that this oslo_messaging_amqp.trace is only available to the
>> Proton driver.
>>
>> Do I miss any existing common RPC message trace mechanism in oslo? If
>> there is no such mechanism, I would propose to create such a mechanism for
>> oslo.
>>
>> Any response is appreciated.
>> Thanks.
>> Best Regards,
>> Xuanzhou Perry Dong
>>
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Adding Dmitry Ukhlov to Oslo-Messaging-Core

[Ken Giusti]

+1

Yay Dmitry!!


On Thu, Jan 28, 2016 at 11:00 AM Oleksii Zamiatin 
wrote:

> My big +1 for Dmitry!
>
> On Thu, Jan 28, 2016 at 5:40 PM, Doug Hellmann 
> wrote:
>
>> Excerpts from Davanum Srinivas (dims)'s message of 2016-01-28 09:25:56
>> -0600:
>> > Team,
>> >
>> > Dmitry has been focused solely on the Pika Driver this cycle:
>> > http://stackalytics.com/?user_id=dukhlov=commits
>> >
>> > Now that we have Pika driver in master, i'd like to invite Dmitry to
>> > continue his work on all of Oslo.Messaging in addition to Pika.
>> > Clearly over time he will expand to other Oslo stuff (hopefully!).
>> > Let's please make add him to the Oslo-Messaging-Core in the meantime.
>> >
>> > Thanks,
>> > Dims
>> >
>>
>> +1 -- Thanks for your contributions, Dmitry!
>>
>> Doug
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] nominating Brant Knudson for Oslo core

[Ken Giusti]

+1

On Mon, Sep 28, 2015 at 4:37 AM Victor Stinner  wrote:

> +1 for Brant
>
> Victor
>
> Le 24/09/2015 19:12, Doug Hellmann a écrit :
> > Oslo team,
> >
> > I am nominating Brant Knudson for Oslo core.
> >
> > As liaison from the Keystone team Brant has participated in meetings,
> > summit sessions, and other discussions at a level higher than some
> > of our own core team members.  He is already core on oslo.policy
> > and oslo.cache, and given his track record I am confident that he would
> > make a good addition to the team.
> >
> > Please indicate your opinion by responding with +1/-1 as usual.
> >
> > Doug
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [qpid] [zmq] [RabbitMQ] [oslo] Pending deprecation of driver(s).

[Ken Giusti]

On Mon, Jun 22, 2015 at 2:27 PM Adam Young ayo...@redhat.com wrote:

 On 06/20/2015 10:28 AM, Flavio Percoco wrote:
 
 
  As promissed: https://review.openstack.org/#/c/193804/
 
  Cheers,
 You can't deprecate a driver without providing a viable alternative.

 Right now, QPID is the only driver that supports  Kerberos.

 TO support Kerberos, tyou need support for the GSSAPI library, which is
 usually done via support for SASL.  Why is it so convoluted...historical...

 We've talked with both teams (I work with Ken) and I think Proton is
 likely going to be the first to have support.  The folks working on
 Rabbit have the double hurdle of getting SASL support into Erlang first,
 and then support for SASL into Rabbit. They've indicated a preference
 for getting it in to the AMQP 1.0 driver, and not bothering with the
 exisiting, but, check me on this, the Oso.Messaging  code only support
 the pre 1.0 Rabbit.


 So..until we have a viable alternative, please leave QPID alone. I've
 not been bothering people about it, as there seems to be work to get
 ahead, but until either Rabbit or  Proton support Kerberos, I need QPID
 as is.


Re: proton - Kerberos support is in progress upstream [0],[1], but as you
point out it's not yet available.  That blocks kerberos support for the
amqp1.0 driver.

Once proton does release that, and the amqp1.0 driver adopts it, you'll be
able to migrate to the amqp1.0 driver and continue to work with the QPID
broker (as long as the version of the QPID broker supports AMQP 1.0).

That doesn't help you now, but it is something to plan for.

[0]https://issues.apache.org/jira/browse/PROTON-334
[1]https://issues.apache.org/jira/browse/PROTON-911


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [qpid] [zmq] [RabbitMQ] [oslo] Pending deprecation of driver(s).

[Ken Giusti]

On Fri, Jun 19, 2015 at 4:47 PM Clint Byrum cl...@fewbar.com wrote:

 Excerpts from Ken Giusti's message of 2015-06-19 08:01:46 -0700:
  On Fri, Jun 19, 2015 at 2:15 AM Flavio Percoco fla...@redhat.com
 wrote:
 
   On 18/06/15 16:37 -0400, Doug Hellmann wrote:
   Excerpts from Clint Byrum's message of 2015-06-18 12:47:21 -0700:
Hello! I know there's been a lot of churn and misunderstanding over
 the
recent devstack changes, so I wanted to make it clear where we're
 going
with messaging drivers now that the policy [1] was approved.
   
According to the policy, drivers need to have at least 60% unit test
coverage, and an integration test suite with at least 3 popular
OpenStack projects, with preference for Nova, Cinder, and Glance,
 and
individuals who will support said test suite.
   
So, with that, the following is the status of each driver in tree
 right
now:
   
rabbit - 89% Unit test coverage. Being the default in devstack, and
the default in nearly every project's config files, this one is
 heavily
integration tested. There are multiple individuals who have proven
 to
be able to debug failures related to RabbitMQ and are well known in
the community.
   
   +1
   
   
qpid - Unit test coverage is at 77%, so it passes that bar. I cannot
find any integration testing being done, so it fails that. I also
 have
not found anyone who will step up and support it. So this should be
deprecated immediately.
   
   +1 - I believe Ken and the other folks interested in this indicated
 that
   the AMQP 1.0 driver should replace this one.
  
   The qpid driver should be deprecated, I'll be doing so in the next
   couple of days. Look forward to the patch.
  
   +1
 
   
   Speaking of AMQP 1.0, you don't mention that one (it uses qpid, but is
   separate from the driver named qpid).
  
   I'd like to clarify something about the AMQP 1.0 driver. It's not a
   direct replacement for the qpidd one because it uses an entirely
   different protocol that recently became a standard.
  
   The reason I mention this is because it doesn't really require qpidd -
   not the double d - which is the broker daemon in the qpid family. I
   guess the confusion comes because the library it sits on top off is
   called qpid-proton.
  
   The qpid family is a set of tools that provide messaging capabilities.
   Among those you find qpidd (broker daemon), qpid-proton (amqp1.0
   library), qpid-dispatch (message router). It's confusing indeed.
  
   The importance of this distinction is that the amqp1.0 driver in
   oslo.messaging is intended as a protocol based driver and not
   targetting any technology. That is to say, that it could be written
   using a library that is not qpid-proton and it can talk to any message
   router or message broker that has support for amqp 1.0.
  
  
  +1 - yeah, we really shouldn't be considering the amqp1 driver as simply
  the replacement qpid driver - as Flavio points out it has the potential
  to provide compatibility with other messaging back ends.
 
  Clint - can you also include separate metrics for the amqp1 driver?
 

 It's far less clear to me how to measure the unit test coverage of the
 amqp driver. I'll wait for Flavio's answer to my question about where
 the code lives, because it is definitely not organized like the others.


No, it isn't.  At the time we proposed this structure for the amqp1 driver,
there really wasn't much formal structure within the driver directory.  We
had impl_rabbit, which was copied to impl_qpid in the hopes that all the
rest of the code could be shared (it didn't work out very well).  And the
zmq code wasn't working at the time.  And all of it was crammed into one
flat directory.

Rather than heap yet more stuff into that directory, we proposed to put the
amqp1 driver into its own sub-directory.  We chose a 'protocols'
subdirectory, into which the amqp1 driver lives in the 'amqp'
subdirectory.  The hope was that other protocols would put their
implementation bits into that protocols directory rather than lump them in
directly under _drivers.

It looks like the zmq driver will be structured a bit different from that -
there will be a top level impl_zmq file along side a zmq_driver directory.

I like the way zmq has laid out the sources, and maybe that should be the
'official' structure to olso.messaging drivers.  If that's the case I'll be
more that happy to arrange the amqp1 driver in a similar manner.

I've gone off into the weeds, sorry.

The amqp1 driver lives in _drivers/protocols/amqp directory (for now).  And
yes, those unit tests should be in the tests/drivers directory like
everything else - when the tests/drivers directory was created that file
wasn't moved.  Oversight - I'll submit a patch to fix it.

As far as coverage testing is concerned - I'm not sure how best to do this
accurately.  The amqp1 driver doesn't use any of the code in amqpdriver.py,
amqp.py, common.py (I think), 

Re: [openstack-dev] [all] [qpid] [zmq] [RabbitMQ] [oslo] Pending deprecation of driver(s).

[Ken Giusti]

On Fri, Jun 19, 2015 at 2:15 AM Flavio Percoco fla...@redhat.com wrote:

 On 18/06/15 16:37 -0400, Doug Hellmann wrote:
 Excerpts from Clint Byrum's message of 2015-06-18 12:47:21 -0700:
  Hello! I know there's been a lot of churn and misunderstanding over the
  recent devstack changes, so I wanted to make it clear where we're going
  with messaging drivers now that the policy [1] was approved.
 
  According to the policy, drivers need to have at least 60% unit test
  coverage, and an integration test suite with at least 3 popular
  OpenStack projects, with preference for Nova, Cinder, and Glance, and
  individuals who will support said test suite.
 
  So, with that, the following is the status of each driver in tree right
  now:
 
  rabbit - 89% Unit test coverage. Being the default in devstack, and
  the default in nearly every project's config files, this one is heavily
  integration tested. There are multiple individuals who have proven to
  be able to debug failures related to RabbitMQ and are well known in
  the community.
 
 +1
 
 
  qpid - Unit test coverage is at 77%, so it passes that bar. I cannot
  find any integration testing being done, so it fails that. I also have
  not found anyone who will step up and support it. So this should be
  deprecated immediately.
 
 +1 - I believe Ken and the other folks interested in this indicated that
 the AMQP 1.0 driver should replace this one.

 The qpid driver should be deprecated, I'll be doing so in the next
 couple of days. Look forward to the patch.

 +1


 
 Speaking of AMQP 1.0, you don't mention that one (it uses qpid, but is
 separate from the driver named qpid).

 I'd like to clarify something about the AMQP 1.0 driver. It's not a
 direct replacement for the qpidd one because it uses an entirely
 different protocol that recently became a standard.

 The reason I mention this is because it doesn't really require qpidd -
 not the double d - which is the broker daemon in the qpid family. I
 guess the confusion comes because the library it sits on top off is
 called qpid-proton.

 The qpid family is a set of tools that provide messaging capabilities.
 Among those you find qpidd (broker daemon), qpid-proton (amqp1.0
 library), qpid-dispatch (message router). It's confusing indeed.

 The importance of this distinction is that the amqp1.0 driver in
 oslo.messaging is intended as a protocol based driver and not
 targetting any technology. That is to say, that it could be written
 using a library that is not qpid-proton and it can talk to any message
 router or message broker that has support for amqp 1.0.


+1 - yeah, we really shouldn't be considering the amqp1 driver as simply
the replacement qpid driver - as Flavio points out it has the potential
to provide compatibility with other messaging back ends.

Clint - can you also include separate metrics for the amqp1 driver?




 The ones we're targetting for the gate are rabbitmq (with the amqp 1.0
 plugin enabled) and qpidd.

 Since we're at it, let me share some updates:

 The driver unittests now run on every python2 job and the work on
 python3 is almost done. There's also a functional tests gate like we
 have for other drivers.

 The missing bit is an integration gate, which we'll be start working
 on in the next couple of days.

 Hope the above helps clarifying confusions around this driver.

 
 
  zmq - Unit test coverage is 74%. There are no currently active
 integration
  tests in OpenStack's infrastructure. Several individuals have self
  identified as being willing to work on creating them. We have not had
  the conversations yet about ongoing support. I recommend we continue to
  support their effort to become policy compliant. If that does not
 solidify
  by M3 of Liberty, or if the new zmq driver appears with integration
  tests and support manpower, then we can deprecate at that time.
 
 +1 - I know interest has been growing in this, so let's keep it going
 and see where we end up.
 
 
  There's also the question of _how_ to deprecate them. I figure we should
  deprecate when the driver is loaded. Is there an existing mechanism
  that someone can point me to, or should I look at adding that to
  oslo.messaging/stevedore?
 
 Normally we would recommend using versionutils from oslo.log, but we've
 been trying to avoid making oslo.log a dependency of the oslo libs
 because it uses some of them and that introduces cycles. Dims had a
 patch recently that just used a DeprecationWarning, and relied on
 oslo.log to redirect the warning to the log file. That seems like a good
 pattern to repeat.

 Can we use debtcollector to decorate the main driver class? A warning
 will be printted every time an instance of such class is created
 (rather than at import time).

 If we don't want to add a dependency on that, we could just use a
 DeprecationWarning as Doug mentioned.

 
 Doug
 
 
  [1]
 http://specs.openstack.org/openstack/openstack-specs/specs/supported-messaging-drivers.html
 
 
 

Re: [openstack-dev] [all] FYI - dropping non RabbitMQ support in devstack

[Ken Giusti]

I've already +1'd this patch - timing issues aside - I think this is a Good
Thing from a developer's point of view.  Particularly, my own selfish point
of view :)  I envision the ability to support multiple different messaging
services via the amqp1 driver.  Having to keep devstack updated with an
array of supported configurations is gonna be a nightmare for all
involved.  I'd much rather have a small independent plugin to work on
rather than having to get every change included into devstack proper.

And thanks to Sean's excellent example I've started a plugin for the
amqp1.0 driver (totally untested at this point), so we'll have that covered
[0].   Thanks Sean!

That said, the only concern I have with this patch is whether it will
result in a less well-tested oslo.messaging.

O.M. is supposed to be an abstraction of the messaging bus - it's not just
RPC and Notifications over RabbitMQ, is it?   How do we validate that
abstraction if we don't thoroughly integration test O.M. over multiple
different drivers/backends?  Other folks have already raised the issue of
rabbit-specific behaviors likely leaking through the API, especially with
respect to failure scenarios.   If we make it harder to run integration
tests over anything but the rabbit driver, then we risk breaking that
abstraction in such a way that using anything _other_ than rabbit will be
practically impossible.

[0] https://github.com/kgiusti/amqp1-devstack

On Thu, Jun 18, 2015 at 12:28 PM Clint Byrum cl...@fewbar.com wrote:

 Excerpts from Sean Dague's message of 2015-06-18 07:09:56 -0700:
  On 06/18/2015 09:54 AM, ozamiatin wrote:
   Hi Sean,
  
   Thanks a lot for the plugin!
   I was a little bit confused with a commit message and dropping of
   drivers support.
   It turns really not so hard to test zeromq driver with plugin.
 
  Yes, that was the design goal with the whole plugin mechanism. To
  provide an experience that was so minimally different from vanilla
  devstack, that most people would never even notice. It's honestly often
  easier to explain to people how to enable a service via a plugin than
  the config variables in tree.
 

 +1

   So I have no objections any more and removing my -1.
 
  Cool, great. It would be great if you or someone else could post a
  review to pull this code into gerrit somewhere. You'll need the code in
  gerrit to use it in devstack-gate jobs, as we don't allow projects
  outside of gerrit to be cloned during tests.
 
   But I also agree with Doug Hellmann and other speakers that we should
   not make such changes
   so fast.
 
  The reason I didn't think the time table was unreasonable was how quick
  this transition could be made, and how little code is needed to make one
  of these plugins. And the fact that from there on out you get to be in
  control of landing fixes or enhancements for your backend on the
  timetable that works for you.
 
  It will make getting the devstack-gate jobs working reliably a lot
  simpler and quicker for your team.
 

 Agreed on all points. I believe that the mistake was simply that
 there wasn't any need to hold hands for those who we are enabling to
 move faster and more independently. We do, in fact, need to transfer
 ownership gracefully. Thanks so much for writing the plugin for zmq,
 that is a huge win for zmq developers. I can't speak for oslo directly,
 but it seems like that plugin should land under oslo's direct stewardship
 and then we can move forward with this.

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?

[Ken Giusti]

On Wed, Apr 15, 2015 at 8:18 PM, Joshua Harlow harlo...@outlook.com wrote:
 Ken Giusti wrote:

 On Wed, Apr 15, 2015 at 1:33 PM, Doug Hellmannd...@doughellmann.com
 wrote:

 Excerpts from Ken Giusti's message of 2015-04-15 09:31:18 -0400:

 On Tue, Apr 14, 2015 at 6:23 PM, Joshua Harlowharlo...@outlook.com
 wrote:

 Ken Giusti wrote:

 Just to be clear: you're asking specifically about the 0-10 based
 impl_qpid.py driver, correct?   This is the driver that is used for
 the qpid:// transport (aka rpc_backend).

 I ask because I'm maintaining the AMQP 1.0 driver (transport
 amqp://) that can also be used with qpidd.

 However, the AMQP 1.0 driver isn't yet Python 3 compatible due to its
 dependency on Proton, which has yet to be ported to python 3 - though
 that's currently being worked on [1].

 I'm planning on porting the AMQP 1.0 driver once the dependent
 libraries are available.

 [1]: https://issues.apache.org/jira/browse/PROTON-490


 What's the expected date on this as it appears this also blocks python
 3
 work as well... Seems like that hasn't been updated since nov 2014
 which
 doesn't inspire that much confidence (especially for what appears to be
 mostly small patches).

 Good point.  I reached out to the bug owner.  He got it 'mostly
 working' but got hung up on porting the proton unit tests.   I've
 offered to help this along and he's good with that.  I'll make this a
 priority to move this along.

 In terms of availability - proton tends to do releases about every 4-6
 months.  They just released 0.9, so the earliest availability would be
 in that 4-6 month window (assuming that should be enough time to
 complete the work).   Then there's the time it will take for the
 various distros to pick it up...

 so, definitely not 'real soon now'. :(

 This seems like a case where if we can get the libs we need to a point
 where they install via pip, we can let the distros catch up instead of
 waiting for them.


 Sadly just the python wrappers are available via pip.  Its C extension
 requires that the native proton shared library (libqpid-proton) is
 available.   To date we've relied on the distro to provide that
 library.


 How does that (c extension) work with eventlet? Does it?


I haven't experienced any issues in my testing.

To be clear - the libqpid-proton library is non-blocking and
non-threading.  It's simply an protocol processing engine - the driver
hands it raw network data and messages magically pop out (and vise
versa).

 All I/O, blocking, threading etc is done in the python driver itself.
I suspect there's nothing eventlet needs to do that requires
overloading functionality provided by the binary proton library, but
my knowledge of eventlet is pretty slim.




 Similarly, if we have *an* approach for Python 3 on oslo.messaging, that
 means the library isn't blocking us from testing applications with
 Python 3. If some of the drivers lag, their test jobs may need to be
 removed or disabled if the apps start testing under Python 3.

 Doug


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?

[Ken Giusti]

On Tue, Apr 14, 2015 at 6:23 PM, Joshua Harlow harlo...@outlook.com wrote:
 Ken Giusti wrote:

 Just to be clear: you're asking specifically about the 0-10 based
 impl_qpid.py driver, correct?   This is the driver that is used for
 the qpid:// transport (aka rpc_backend).

 I ask because I'm maintaining the AMQP 1.0 driver (transport
 amqp://) that can also be used with qpidd.

 However, the AMQP 1.0 driver isn't yet Python 3 compatible due to its
 dependency on Proton, which has yet to be ported to python 3 - though
 that's currently being worked on [1].

 I'm planning on porting the AMQP 1.0 driver once the dependent
 libraries are available.

 [1]: https://issues.apache.org/jira/browse/PROTON-490


 What's the expected date on this as it appears this also blocks python 3
 work as well... Seems like that hasn't been updated since nov 2014 which
 doesn't inspire that much confidence (especially for what appears to be
 mostly small patches).


Good point.  I reached out to the bug owner.  He got it 'mostly
working' but got hung up on porting the proton unit tests.   I've
offered to help this along and he's good with that.  I'll make this a
priority to move this along.

In terms of availability - proton tends to do releases about every 4-6
months.  They just released 0.9, so the earliest availability would be
in that 4-6 month window (assuming that should be enough time to
complete the work).   Then there's the time it will take for the
various distros to pick it up...

so, definitely not 'real soon now'. :(



 On Tue, Apr 14, 2015 at 1:22 PM, Clint Byrumcl...@fewbar.com  wrote:

 Hello! There's been some recent progress on python3 compatibility for
 core libraries that OpenStack depends on[1], and this is likely to open
 the flood gates for even more python3 problems to be found and fixed.

 Recently a proposal was made to make oslo.messaging start to run python3
 tests[2], and it was found that qpid-python is not python3 compatible
 yet.

 This presents us with questions: Is anyone using QPID, and if so, should
 we add gate testing for it? If not, can we deprecate the driver? In the
 most recent survey results I could find [3] I don't even see message
 broker mentioned, whereas Databases in use do vary somewhat.

 Currently it would appear that only oslo.messaging runs functional tests
 against QPID. I was unable to locate integration testing for it, but I
 may not know all of the places to dig around to find that.

 So, please let us know if QPID is important to you. Otherwise it may be
 time to unburden ourselves of its maintenance.

 [1] https://pypi.python.org/pypi/eventlet/0.17.3
 [2] https://review.openstack.org/#/c/172135/
 [3]
 http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?

[Ken Giusti]

On Wed, Apr 15, 2015 at 1:33 PM, Doug Hellmann d...@doughellmann.com wrote:
 Excerpts from Ken Giusti's message of 2015-04-15 09:31:18 -0400:
 On Tue, Apr 14, 2015 at 6:23 PM, Joshua Harlow harlo...@outlook.com wrote:
  Ken Giusti wrote:
 
  Just to be clear: you're asking specifically about the 0-10 based
  impl_qpid.py driver, correct?   This is the driver that is used for
  the qpid:// transport (aka rpc_backend).
 
  I ask because I'm maintaining the AMQP 1.0 driver (transport
  amqp://) that can also be used with qpidd.
 
  However, the AMQP 1.0 driver isn't yet Python 3 compatible due to its
  dependency on Proton, which has yet to be ported to python 3 - though
  that's currently being worked on [1].
 
  I'm planning on porting the AMQP 1.0 driver once the dependent
  libraries are available.
 
  [1]: https://issues.apache.org/jira/browse/PROTON-490
 
 
  What's the expected date on this as it appears this also blocks python 3
  work as well... Seems like that hasn't been updated since nov 2014 which
  doesn't inspire that much confidence (especially for what appears to be
  mostly small patches).
 

 Good point.  I reached out to the bug owner.  He got it 'mostly
 working' but got hung up on porting the proton unit tests.   I've
 offered to help this along and he's good with that.  I'll make this a
 priority to move this along.

 In terms of availability - proton tends to do releases about every 4-6
 months.  They just released 0.9, so the earliest availability would be
 in that 4-6 month window (assuming that should be enough time to
 complete the work).   Then there's the time it will take for the
 various distros to pick it up...

 so, definitely not 'real soon now'. :(

 This seems like a case where if we can get the libs we need to a point
 where they install via pip, we can let the distros catch up instead of
 waiting for them.


Sadly just the python wrappers are available via pip.  Its C extension
requires that the native proton shared library (libqpid-proton) is
available.   To date we've relied on the distro to provide that
library.

 Similarly, if we have *an* approach for Python 3 on oslo.messaging, that
 means the library isn't blocking us from testing applications with
 Python 3. If some of the drivers lag, their test jobs may need to be
 removed or disabled if the apps start testing under Python 3.

 Doug

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?

[Ken Giusti]

Just to be clear: you're asking specifically about the 0-10 based
impl_qpid.py driver, correct?   This is the driver that is used for
the qpid:// transport (aka rpc_backend).

I ask because I'm maintaining the AMQP 1.0 driver (transport
amqp://) that can also be used with qpidd.

However, the AMQP 1.0 driver isn't yet Python 3 compatible due to its
dependency on Proton, which has yet to be ported to python 3 - though
that's currently being worked on [1].

I'm planning on porting the AMQP 1.0 driver once the dependent
libraries are available.

[1]: https://issues.apache.org/jira/browse/PROTON-490

On Tue, Apr 14, 2015 at 1:22 PM, Clint Byrum cl...@fewbar.com wrote:
 Hello! There's been some recent progress on python3 compatibility for
 core libraries that OpenStack depends on[1], and this is likely to open
 the flood gates for even more python3 problems to be found and fixed.

 Recently a proposal was made to make oslo.messaging start to run python3
 tests[2], and it was found that qpid-python is not python3 compatible yet.

 This presents us with questions: Is anyone using QPID, and if so, should
 we add gate testing for it? If not, can we deprecate the driver? In the
 most recent survey results I could find [3] I don't even see message
 broker mentioned, whereas Databases in use do vary somewhat.

 Currently it would appear that only oslo.messaging runs functional tests
 against QPID. I was unable to locate integration testing for it, but I
 may not know all of the places to dig around to find that.

 So, please let us know if QPID is important to you. Otherwise it may be
 time to unburden ourselves of its maintenance.

 [1] https://pypi.python.org/pypi/eventlet/0.17.3
 [2] https://review.openstack.org/#/c/172135/
 [3] 
 http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Oslo][Neutron] Fork() safety and oslo.messaging

[Ken Giusti]

I've come up with a patch to the amqp 1.0 driver that resets the
connection state if the pid of the current process is different from
the process that has created the connection:

https://review.openstack.org/#/c/134684/

I've managed to get this to work using rpc_workers=4 in neutron.conf,
which failed consistently pre-patch.



On Tue, Nov 25, 2014 at 11:16 AM, Mehdi Abaakouk sil...@sileht.net wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256



 Mmmm... I don't think it's that clear (re: an application issue).  I
 mean, yes - the application is doing the os.fork() at the 'wrong'
 time, but where is this made clear in the oslo.messaging API
 documentation?
 I think this is the real issue here:  what is the official guidance
 for using os.fork() and its interaction with oslo libraries?

 In the case of oslo.messaging, I can't find any mention of os.fork()
 in the API docs (I may have missed it - please correct me if so).
 That would imply - at least to me - that there is _no_ restrictions of
 using os.fork() together with oslo.messaging.


 Yes, I agree we should add a note on that in oslo.messaging (and perhaps in
 oslo.db too).

 And also the os.fork() is done by the service.ProcessLauncher of
 oslo-incubator,
 and it's not (yet) documented. But once oslo.service library will be
 released, it will.

 But in the case of qpid, that is definitely _not_ the case.

 The legacy qpid driver - impl_qpid - imports a 3rd party library, the
 qpid.messaging API.   This library uses threading.Thread internally,
 we (consumers of this library) have no control over how that thread is
 managed.  So for impl_qpid, os.fork()'ing after the driver is loaded
 can't be guaranteed to work.   In fact, I'd say os.fork() and
 impl_qpid will not work - full stop.


 Yes, I have tried it, and I have catch what happen and I can confirm that
 too now, unfortunately :( And this can occurs with any driver if the 3rd
 party library
 doesn't work when we use os.fork()

 For the amqp1 driver case, I think this is the same things, it seems to
 do lazy creation of the connection too.


 We have more flexibility here, since the driver directly controls when
 the thread is spawned.  But the very fact that the thread is used
 places a restriction on how oslo.messaging and os.fork() can be used
 together, which isn't made clear in the documentation for the library.

 I'm not familiar with the rabbit driver - I've seen some patch for
 heatbeating in rabbit introduce threading, so there may also be an
 implication there as well.


 Yes, we need to check that.

 Personally, I don't like this API, because the behavior difference
 between
 '__init__' and 'start' is too implicit.


 That's true, but I'd say that the problem of implicitness re:
 os.fork() needs to be clarified at the library level as well.


 I agree.

 I will write the documentation patch for oslo.messaging.

 Cheers,

 - ---
 Mehdi Abaakouk
 mail: sil...@sileht.net
 irc: sileht
 -BEGIN PGP SIGNATURE-
 Version: OpenPGP.js v.1.20131017
 Comment: http://openpgpjs.org

 wsFcBAEBCAAQBQJUdKtPCRAYkrQvzqrryAAAqCIP/RvONngQ1MOKXEcktcku
 Ok1Lr9O12O3j/zESkXaKInJbqak0EjM0FXnoXeah4qPbSoJYZIfztmCOtX4u
 CSdhkAWhFqXdpHUtngXImHeB3P/eRH0Vo7R3PAmUbv5VWkn+lwcO+Ym1g79Z
 vCJbcwVpldGiTDRJRDAPPb14UakJbZJGnkRDgYscNlBG+alzLw0MsaqnJ7LS
 8Yj4YkXSgthpHLF2N8Yem9r7Lh7CbYLKzlhOylgAJTd8gpGGtncwWMwYJvKc
 lsMJNY34PMiNkPk1a+VSlrWcPJpafBl3aOBbrIpmMSpMe9pXC/yHW2nrtGez
 VXxliFpqQ7kA5827AuhPAM8EzeMUDetLhZvLxzqY7f/SlaoQ/s/9VhfemmHv
 d4wT8uiayrWSMdXVUJZcMUdM2XlJDdObokMI0ZQKQYX8OhKQL8LdaHR2xr6B
 OjS4Mp4+/W4Y9wMUFqlRyGnW1LLwCFYWHpyKlhXKmYSSdKTn5L7Pcvmmfw8d
 JzDcMxfKCBnM4mNRzlBqYV4/ysb6FNMUZwu+D1YxCVHmAH2O1/oODujNJFkZ
 gSWAmh9kYawJKbbS0Lh7nkOJs1iOnYxG0IQmz61sffg8T2FrpbH4FNWh1/+a
 mQhmYWH2L5noJIwncVQSloSRuoSWLj9rfeiTIHjq2ZnTUD5tbXK6S5dTvv4m
 4bij
 =G9oX
 -END PGP SIGNATURE-




-- 
Ken Giusti  (kgiu...@gmail.com)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Oslo][Neutron] Fork() safety and oslo.messaging

[Ken Giusti]

Hi Mehdi

On Tue, Nov 25, 2014 at 5:38 AM, Mehdi Abaakouk sil...@sileht.net wrote:

 Hi,

 I think the main issue is the behavior of the API
 of oslo-incubator/openstack/common/service.py, specially:

  * ProcessLauncher.launch_service(MyService())

 And then the MyService have this behavior:

 class MyService:
def __init__(self):
# CODE DONE BEFORE os.fork()

def start(self):
# CODE DONE AFTER os.fork()

 So if an application created a FD inside MyService.__init__ or before 
 ProcessLauncher.launch_service, it will be shared between
 processes and we got this kind of issues...

 For the rabbitmq/qpid driver, the first connection is created when the rpc 
 server is started or when the first rpc call/cast/... is done.

 So if the application doesn't do that inside MyService.__init__ or before 
 ProcessLauncher.launch_service everything works as expected.

 But if the issue is raised I think this is an application issue (rpc stuff 
 done before the os.fork())


Mmmm... I don't think it's that clear (re: an application issue).  I
mean, yes - the application is doing the os.fork() at the 'wrong'
time, but where is this made clear in the oslo.messaging API
documentation?

I think this is the real issue here:  what is the official guidance
for using os.fork() and its interaction with oslo libraries?

In the case of oslo.messaging, I can't find any mention of os.fork()
in the API docs (I may have missed it - please correct me if so).
That would imply - at least to me - that there is _no_ restrictions of
using os.fork() together with oslo.messaging.

But in the case of qpid, that is definitely _not_ the case.

The legacy qpid driver - impl_qpid - imports a 3rd party library, the
qpid.messaging API.   This library uses threading.Thread internally,
we (consumers of this library) have no control over how that thread is
managed.  So for impl_qpid, os.fork()'ing after the driver is loaded
can't be guaranteed to work.   In fact, I'd say os.fork() and
impl_qpid will not work - full stop.

 For the amqp1 driver case, I think this is the same things, it seems to do 
 lazy creation of the connection too.


We have more flexibility here, since the driver directly controls when
the thread is spawned.  But the very fact that the thread is used
places a restriction on how oslo.messaging and os.fork() can be used
together, which isn't made clear in the documentation for the library.

I'm not familiar with the rabbit driver - I've seen some patch for
heatbeating in rabbit introduce threading, so there may also be an
implication there as well.


 I will take a look to the neutron code, if I found a rpc usage
 before the os.fork().


I've done some tracing of neutron-server's behavior in this case - you
may want to take a look at

 https://bugs.launchpad.net/neutron/+bug/1330199/comments/8


 Personally, I don't like this API, because the behavior difference between
 '__init__' and 'start' is too implicit.


That's true, but I'd say that the problem of implicitness re:
os.fork() needs to be clarified at the library level as well.

thanks,

-K

 Cheers,

 ---
 Mehdi Abaakouk
 mail: sil...@sileht.net
 irc: sileht


 Le 2014-11-24 20:27, Ken Giusti a écrit :

 Hi all,

 As far as oslo.messaging is concerned, should it be possible for the
 main application to safely os.fork() when there is already an active
 connection to a messaging broker?

 I ask because I'm hitting what appears to be fork-related issues with
 the new AMQP 1.0 driver.  I think the same problems have been seen
 with the older impl_qpid driver as well [0]

 Both drivers utilize a background threading.Thread that handles all
 async socket I/O and protocol timers.

 In the particular case I'm trying to debug, rpc_workers is set to 4 in
 neutron.conf.  As far as I can tell, this causes neutron.service to
 os.fork() four workers, but does so after it has created a listener
 (and therefore a connection to the broker).

 This results in multiple processes all select()'ing the same set of
 networks sockets, and stuff breaks :(

 Even without the background process, wouldn't this use still result in
 sockets being shared across the parent/child processes?   Seems
 dangerous.

 Thoughts?

 [0] https://bugs.launchpad.net/oslo.messaging/+bug/1330199




-- 
Ken Giusti  (kgiu...@gmail.com)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Oslo][Neutron] Fork() safety and oslo.messaging

[Ken Giusti]

Hi all,

As far as oslo.messaging is concerned, should it be possible for the
main application to safely os.fork() when there is already an active
connection to a messaging broker?

I ask because I'm hitting what appears to be fork-related issues with
the new AMQP 1.0 driver.  I think the same problems have been seen
with the older impl_qpid driver as well [0]

Both drivers utilize a background threading.Thread that handles all
async socket I/O and protocol timers.

In the particular case I'm trying to debug, rpc_workers is set to 4 in
neutron.conf.  As far as I can tell, this causes neutron.service to
os.fork() four workers, but does so after it has created a listener
(and therefore a connection to the broker).

This results in multiple processes all select()'ing the same set of
networks sockets, and stuff breaks :(

Even without the background process, wouldn't this use still result in
sockets being shared across the parent/child processes?   Seems
dangerous.

Thoughts?

[0] https://bugs.launchpad.net/oslo.messaging/+bug/1330199

-- 
Ken Giusti  (kgiu...@gmail.com)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo.messaging][feature freeze exception] Can I get an exception for AMQP 1.0?

[Ken Giusti]

Hello,

I'm proposing a freeze exception for the oslo.messaging AMQP 1.0
driver:

   https://review.openstack.org/#/c/75815/

Blueprint:

   
https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation

I presented this work at the Juno summit [1]. The associated spec has
been approved and merged [2].

The proposed patch has been in review since before icehouse, with a
couple of non-binding +1's.  A little more time is necessary to get
core reviews.

The patch includes a number of functional tests, and I've proposed a
CI check that will run those tests [3].  This patch is currently
pending support for bare fedora 20 nodes in CI.  I'm planning to add
additional test cases and devstack support in the future.

I'm in the process of adding documentation to the RPC section of the
Openstack manual.

Justification:

I think there's a benefit to have this driver available as an
_experimental_ feature in Juno, and the risk of inclusion is minimal
as the driver is optional, disabled by default, and will not have
impact on any system that does not explicitly enable it.

Unlike previous versions of the protocol, AMQP 1.0 is the official
standard for AMQP messaging (ISO/IEC 19464).  Support for it is
arriving from multiple different messaging system vendors [4].

Having access to AMQP 1.0 functionality in openstack sooner rather
than later gives the developers of AMQP 1.0 messaging systems the
opportunity to validate their AMQP 1.0 support in the openstack
environment.  Likewise, easier access to this driver by the openstack
developer community will help us find and fix any issues in a timely
manner as adoption of the standard grows.

Please consider this feature to be a part of Juno-3 release.

Thanks,

Ken


-- 
Ken Giusti  (kgiu...@gmail.com)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo.messaging][feature freeze exception] Can I get an exception for AMQP 1.0?

[Ken Giusti]

On Wed Sep 3 19:23:52 UTC 2014, Doug Hellmann wrote:
On Sep 3, 2014, at 2:03 PM, Ken Giusti kgiusti at gmail.com wrote:
 Hello,

 I'm proposing a freeze exception for the oslo.messaging AMQP 1.0
 driver:

   https://review.openstack.org/#/c/75815/
SNIP
 Thanks,

 Ken

Ken,

I think we’re generally in favor of including the new driver, but
before I say so officially can you fill us in on the state of the
additional external libraries it needs? I see pyngus on pypi, and you
mention in the “Request to include AMQP 1.0 support in Juno-3” thread
that proton is being packaged in EPEL and work is ongoing for
Debian. Is that done (it has only been a few days, I know)?


Hi Doug,

Yes, AMQP 1.0 tech is pretty new, so the availability of the packages
is the sticking point.

That said, there are two different dependencies to consider:
oslo.messaging dependencies, and AMQP 1.0 support in the brokers.

For oslo.messaging, the dependency is on the Proton C developer
libraries.  This library is currently available on EPEL for centos6+
and fedora. It is not available in the Ubuntu repos yet - though they
have recently been accepted to Debian sid.  For developers, the QPID
project maintains a PPA that can be used to get the packages on
Debian/Ubuntu (though this is not acceptable for openstack CI
support).  The python bindings
that interface with this library are available on pypi (see the
amqp1-requirements.txt in the driver patch).

Qpid upstream has been shipping with 1.0 support for awhile now, but
unfortunately the popular distros don't have the latest Qpid brokers
available.  Qpid with AMQP 1.0 support is available via EPEL for
centos7 and fedora. RedHat deprecated Qpid on rhel6, so now centos6 is
stuck with an old version of qpid since we can't override base
packages in EPEL.  Same deal with Debian/Ubuntu, though the QPID PPA
should have the latest packages (I'll have to follow up on that).


I would like to avoid having that packaging work be a blocker, so if
we do include the driver, what do you think is the best way to convey
the instructions for installing those packages? I know you’ve done
some work recently on documenting the experimental status, did that
include installation tips?

Not at present, but that's what I'm working on at the moment.  I'll
definitely call out the installation dependencies and configuration
settings necessary for folks to get the driver up and running with a
minimum of pain.

I assume I'll have to update at least:

The wiki: https://wiki.openstack.org/wiki/Oslo/Messaging
The user manuals:
http://docs.openstack.org/icehouse/config-reference/content/configuring-rpc.html

I can also add a README to the protocols/amqp directory, if that makes sense.


Thanks,
Doug

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging] Request to include AMQP 1.0 support in Juno-3

[Ken Giusti]

On Thu, 28 Aug 2014 13:36:46 +0100, Mark McLoughlin wrote:
 On Thu, 2014-08-28 at 13:24 +0200, Flavio Percoco wrote:
  On 08/27/2014 03:35 PM, Ken Giusti wrote:
   Hi All,
  
   I believe Juno-3 is our last chance to get this feature [1] included
   into olso.messaging.
  
SNIP!
 
 
  Hi Ken,
 
  Thanks a lot for your hard work here. As I stated in my last comment on
  the driver's review, I think we should let this driver land and let
  future patches improve it where/when needed.
 
  I agreed on letting the driver land as-is based on the fact that there
  are patches already submitted ready to enable the gates for this driver.

 I feel bad that the driver has been in a pretty complete state for quite
 a while but hasn't received a whole lot of reviews. There's a lot of
 promise to this idea, so it would be ideal if we could unblock it.

 One thing I've been meaning to do this cycle is add concrete advice for
 operators on the state of each driver. I think we'd be a lot more
 comfortable merging this in Juno if we could somehow make it clear to
 operators that it's experimental right now. My idea was:

   - Write up some notes which discusses the state of each driver e.g.

   - RabbitMQ - the default, used by the majority of OpenStack
 deployments, perhaps list some of the known bugs, particularly
 around HA.

   - Qpid - suitable for production, but used in a limited number of
 deployments. Again, list known issues. Mention that it will
 probably be removed with the amqp10 driver matures.

   - Proton/AMQP 1.0 - experimental, in active development, will
 support  multiple brokers and topologies, perhaps a pointer to a
 wiki page with the current TODO list

   - ZeroMQ - unmaintained and deprecated, planned for removal in
 Kilo

Sounds like a plan - I'll take on the Qpid and Proton notes.  I've
been (trying) to keep the status of the Proton stuff up to date on the
blueprint page:

https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation

Is there a more appropriate home for these notes?  Etherpad?


   - Propose this addition to the API docs and ask the operators list
 for feedback

   - Propose a patch which adds a load-time deprecation warning to the
 ZeroMQ driver

   - Include a load-time experimental warning in the proton driver

Done!


 Thoughts on that?

 (I understand the ZeroMQ situation needs further discussion - I don't
 think that's on-topic for the thread, I was just using it as example of
 what kind of advice we'd be giving in these docs)

 Mark.

 -
Ken Giusti  (kgiu...@gmail.com)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo.messaging] Request to include AMQP 1.0 support in Juno-3

[Ken Giusti]

Hi All,

I believe Juno-3 is our last chance to get this feature [1] included
into olso.messaging.

I honestly believe this patch is about as low risk as possible for a
change that introduces a whole new transport into oslo.messaging.  The
patch shouldn't affect the existing transports at all, and doesn't
come into play unless the application specifically turns on the new
'amqp' transport, which won't be the case for existing applications.

The patch includes a set of functional tests which exercise all the
messaging patterns, timeouts, and even broker failover. These tests do
not mock out any part of the driver - a simple test broker is included
which allows the full driver codepath to be executed and verified.

IFAIK, the only remaining technical block to adding this feature,
aside from core reviews [2], is sufficient infrastructure test coverage.
We discussed this a bit at the last design summit.  The root of the
issue is that this feature is dependent on a platform-specific library
(proton) that isn't in the base repos for most of the CI platforms.
But it is available via EPEL, and the Apache QPID team is actively
working towards getting the packages into Debian (a PPA is available
in the meantime).

In the interim I've proposed a non-voting CI check job that will
sanity check the new driver on EPEL based systems [3].  I'm also
working towards adding devstack support [4], which won't be done in
time for Juno but nevertheless I'm making it happen.

I fear that this feature's inclusion is stuck in a chicken/egg
deadlock: the driver won't get merged until there is CI support, but
the CI support won't run correctly (and probably won't get merged)
until the driver is available.  The driver really has to be merged
first, before I can continue with CI/devstack development.

[1] 
https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation
[2] https://review.openstack.org/#/c/75815/
[3] https://review.openstack.org/#/c/115752/
[4] https://review.openstack.org/#/c/109118/

-- 
Ken Giusti  (kgiu...@gmail.com)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack-Infra] How to get new package requirements into the CI system using a PPA and EPEL?

[Ken Giusti]

On Tue, Jul 15, 2014 at 7:28 PM, Ian Wienand iwien...@redhat.com wrote:
 On 07/15/2014 11:55 PM, Ken Giusti wrote:
 Good to hear about epel's availability.  But on the Ubuntu/Debian
 side - is it possible to add the Qpid project's PPA to the config
 project?  From a quick 'grep' of the sources, it appears as if Pypy
 requires a PPA.  It's configured in
 modules/openstack_project/manifests/slave_common.pp).  Can I use
 this as an example for adding Qpid's PPA?

 This is probably a good example of the puppet classes to use, but...

 As discussed, it's questionable what you want here.  Probably for your
 unit tests, you could mock-out calls to the library?  So you may not
 need it installed at all?

 If you want to test it for real; i.e. in a running environment with
 real RPC happening between components, etc, then that would be in a
 devstack environment.  It sounds like you'd probably be wanting to
 define a new rpc-backend [1] that could be optionally enabled.

 Once you had that in devstack, you'd have to start looking at the
 jenkins-job-builder configs [2] and add a specific test that enabled
 the flag for this back-end and add it as probably a non-voting job to
 some component.



Thanks,  I've spent some time hacking this and have the following:

1) a patch to devstack that adds a configuration option to enable AMQP
1.0 as the RPC messaging protocol:

https://review.openstack.org/#/c/109118/

2) a patch to openstack-infra/config that adds a new non-voting job
for oslo.messaging that runs on a devstack node with AMQP 1.0 enabled.
The job runs the AMQP 1.0 functional tests via tox:

https://review.openstack.org/#/c/115752/

#2 is fairly straightforward - I've copypasted the code from existing
neutron-functional tests.  Still needs testing, however.

#1 is the change I'm most concerned about as it adds the Apache qpid
PPA for ubuntu systems.  I've tested this on my Trusty and Centos6
vm's and it works well for me.

Can anyone on the devstack or infra teams give me some feedback on
these changes?  I'm hoping these infrastructure changes will unblock
the AMQP 1.0 blueprint in time for Juno-3 (fingers, toes, eyes
crossed).

thanks!

 -i

 [1] http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/rpc_backend
 [2] 
 https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/



-- 
Ken Giusti  (kgiu...@gmail.com)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging][infra] Adding support for AMQP 1.0 Messaging to Oslo.Messaging and infra/config

[Ken Giusti]

On Wed, 30 Jul 2014 15:04:41 -0700, Matt Riedemann wrote:
On 7/30/2014 11:59 AM, Ken Giusti wrote:
 On Wed, 30 Jul 2014 14:25:46 +0100, Daniel P. Berrange wrote:
 On Wed, Jul 30, 2014 at 08:54:01AM -0400, Ken Giusti wrote:
 Greetings,
snip
 At this point, there are no integration tests that exercise the
 driver.  However, the new unit tests include a test 'broker', which
 allow the unit tests to fully exercise the new driver, right down to
 the network.  That's a bonus of AMQP 1.0 - it can support peer-2-peer
 messaging.

 So its the new unit tests that have the 'hard' requirement of the
 proton libraries.And mocking-out the proton libraries really
 doesn't allow us to do any meaningful tests of the driver.

snip

If your unit tests are dependent on a specific dependent library aren't
they no longer unit tests but integration tests anyway?


Good point - yes, they are certainly more than just unit tests.  I'd
consider them more functional tests than integration tests, tho:
they only test from the new driver API down to the wire (and back up
again via the fake loopback broker).  For integration testing, I'd
want to put a real broker in there, and run real subprojects over
oslo.messaging using the new driver (neutron, etc).

I'd really like to avoid the classic unit test approach of mocking out
the underlying messaging client api if possible.  Even though that
would avoid the dependency, I think it could result in the same issues
we've had with the existing impl_qpid tests passing in mock, but
failing when run against qpidd.

Just wondering, not trying to put up road-blocks because I'd like to see
how this code performs but haven't had time yet to play with it.


np, a good question, thanks!  When you do get a chance to kick the tires,
feel free to ping me with any questions/issues you have.  Thanks!

--

Thanks,

Matt Riedemann
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging][infra] Adding support for AMQP 1.0 Messaging to Oslo.Messaging and infra/config

[Ken Giusti]

On Wed, 30 Jul 2014 22:14:51 +, Jeremy Stanley wrote:
On 2014-07-30 14:59:09 -0400 (-0400), Ken Giusti wrote:
 Thanks Daniel.  It was my understanding - which may be wrong - that
 having devstack install the 'out of band' packages would only help in
 the case of the devstack-based integration tests, not in the case of
 CI running the unit tests.  Is that indeed the case?
[...]
 I'm open to any thoughts on how best to solve this, thanks.

Since they're in EPEL and we run Python 2.6 unit tests today on
CentOS 6 servers, if the proton libraries install successfully there
perhaps we could opportunistically exercise it only under Python 2.6
for now? Not ideal, but it does get it enforced upstream with
minimal fuss. I'd really rather not start accumulating arbitrary PPA
sources on our job workers... I know we've done it for a handful of
multi-project efforts where we needed select backports from non-LTS
releases, but we've so far limited that to only PPAs maintained by
the same package teams as the mainline distro packages themselves.


Yeah, it's becoming pretty clear that adding this PPA to infra is not
The Right Thing To Do.  How does this sound as an alternative:

1) _for_ _now_, make the dependent unit tests optional for
oslo.messaging.  Specifically, by default tox will not run them, but
I'll add a new testenv that adds a requirement for the dependent
packages and runs all the unit tests (default tests + new amqp1.0
tests).  Eg, do 'tox -e amqp1' to pull in the python packages that
require the libraries, and run all unit tests.  This allows those
developers that have installed the proton libraries to run the tests,
and avoid making life hard for those devs who don't have the libraries
installed.

2) Propose a new optional configuration flag in devstack that enables
the AMQP 1.0 messaging protocol (default off).  Something like
$RPC_MESSAGING_PROTOCOL == AMQP1.  When this is set in the devstack
config, rpc_backend will install the AMQP 1.0 libraries, adding the
Qpid PPA in the case of ubuntu for now.

3) Create a non-voting oslo.messaging gate test [0] that merely
runs the 'tox -e amqp1' tests.  Of course, additional integration
tests are a Good Thing, but at the very least we should start with
this. This would give us a heads up should new patches break the amqp
1.0 driver.  This test could eventually become gating once the driver
matures and the packages find their way into all the proper repos.

4) Profit (couldn't resist :)

Opinions?

[0] I honestly have no idea how to do this, or if it's even feasible
btw - I've never written a gating test before.  I'd appreciate any
pointers to get me started, thanks!


Longer term, I'd suggest getting it sponsored into Debian
unstable/testing ASAP, interesting the Ubuntu OpenStack team in
importing it into the development tree for the next Ubuntu release,
and then incorporating it into the Trusty Ubuntu Cloud Archive.
We're not using UCA yet, but on Trusty we probably should consider
adding it sooner rather than later since when we tried to tack on
the Precise UCA in the last couple cycles we had too many headaches
from trying to jump ahead substantially on fundamental bits like
libvirt. Breaking sooner and more often means those incremental
issues are easier to identify and address, usually.

Ah - I didn't know that, thanks!  I know one of the Qpid devs is
currently engaged in getting these packages into Debian.  I'll reach
out to him and see if he can work on getting it into UCA next.

Thanks again - very valuable info!

--
Jeremy Stanley


-- 
Ken Giusti  (kgiu...@gmail.com)
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo.messaging][infra] Adding support for AMQP 1.0 Messaging to Oslo.Messaging and infra/config

[Ken Giusti]

Greetings,

Apologies for the cross-post: this should be of interest to both infra
and olso.messaging developers.

The blueprint [0] that adds support for version 1.0 of the AMQP messaging
protocol is blocked due to CI test failures [1]. These failures are due
to a new package dependency this blueprint adds to oslo.messaging.

The AMQP 1.0 functionality is provided by the Apache Qpid's Proton
AMQP 1.0 toolkit.  The blueprint uses the Python bindings for this
toolkit, which are available on Pypi.  These bindings, however, include
a C extension that depends on the Proton toolkit development libraries
in order to build and install.  The lack of this toolkit is the cause
of the blueprint's current CI failures.

This toolkit is written in C, and thus requires platform-specific
libraries.

Now here's the problem: packages for Proton are not included by
default in most distro's base repositories (yet).  The Apache Qpid
team has provided packages for EPEL, and has a PPA available for
Ubuntu.  Packages for Debian are also being proposed.

I'm proposing this patch to openstack-infra/config to address the
dependency problem [2].  It adds the proton toolkit packages to the
common slave configuration.  Does this make sense?  Are there any
better alternatives?

[0] 
https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation
[1] https://review.openstack.org/#/c/75815/
[2] https://review.openstack.org/#/c/110431/



-- 
Ken Giusti  (kgiu...@gmail.com)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging][infra] Adding support for AMQP 1.0 Messaging to Oslo.Messaging and infra/config

[Ken Giusti]

On Wed, 30 Jul 2014 14:25:46 +0100, Daniel P. Berrange wrote:
 On Wed, Jul 30, 2014 at 08:54:01AM -0400, Ken Giusti wrote:
  Greetings,
 
  Apologies for the cross-post: this should be of interest to both infra
  and olso.messaging developers.
 
  The blueprint [0] that adds support for version 1.0 of the AMQP messaging
  protocol is blocked due to CI test failures [1]. These failures are due
  to a new package dependency this blueprint adds to oslo.messaging.
 
  The AMQP 1.0 functionality is provided by the Apache Qpid's Proton
  AMQP 1.0 toolkit.  The blueprint uses the Python bindings for this
  toolkit, which are available on Pypi.  These bindings, however, include
  a C extension that depends on the Proton toolkit development libraries
  in order to build and install.  The lack of this toolkit is the cause
  of the blueprint's current CI failures.
 
  This toolkit is written in C, and thus requires platform-specific
  libraries.
 
  Now here's the problem: packages for Proton are not included by
  default in most distro's base repositories (yet).  The Apache Qpid
  team has provided packages for EPEL, and has a PPA available for
  Ubuntu.  Packages for Debian are also being proposed.
 
  I'm proposing this patch to openstack-infra/config to address the
  dependency problem [2].  It adds the proton toolkit packages to the
  common slave configuration.  Does this make sense?  Are there any
  better alternatives?

 For other cases where we need more native packages, we tyically
 use devstack to ensure they are installed. This is preferrable
 since it works for ordinary developers as well as the CI system.


Thanks Daniel.  It was my understanding - which may be wrong - that
having devstack install the 'out of band' packages would only help in
the case of the devstack-based integration tests, not in the case of
CI running the unit tests.  Is that indeed the case?

At this point, there are no integration tests that exercise the
driver.  However, the new unit tests include a test 'broker', which
allow the unit tests to fully exercise the new driver, right down to
the network.  That's a bonus of AMQP 1.0 - it can support peer-2-peer
messaging.

So its the new unit tests that have the 'hard' requirement of the
proton libraries.And mocking-out the proton libraries really
doesn't allow us to do any meaningful tests of the driver.

But if devstack is the preferred method for getting 'special case'
packages installed, would it be acceptable to have the new unit tests
run as a separate oslo.messaging integration test, and remove them
from the unit tests?

I'm open to any thoughts on how best to solve this, thanks.

 Regards,
 Daniel
 --
 |: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
 |: http://libvirt.org  -o- http://virt-manager.org :|
 |: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
 |: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Oslo] [Oslo.messaging] RPC failover handling in rabbitmq driver

[Ken Giusti]

On Mon, 28 Jul 2014 10:58:02 +0100, Gordon Sim wrote:
 On 07/28/2014 09:20 AM, Bogdan Dobrelya wrote:
  Hello.
  I'd like to bring your attention to major RPC failover issue in
  impl_rabbit.py [0]. There are several *related* patches and a number of
  concerns should be considered as well:
  - Passive exchanges fix [1] (looks like the problem is much deeper than
  it seems though).
  - the first version of the fix [2] which makes the producer to declare a
  queue and bind it to exchange as well as consumer does.
  - Making all RPC involved reply_* queues durable in order to preserve
  them in RabbitMQ after failover (there could be a TTL for such a queues
  as well)
  - RPC throughput tuning patch [3]
 
  I believe the issue [0] should be at least prioritized and assigned to
  some milestone.

 I think the real issue is the lack of clarity around what guarantees are
 made by the API.


Wholeheartedly agree!  This lack of explicitness makes it very
difficult to add new messaging backends (drivers) to oslo.messaging
and expect the API to function uniformly from the application's point
of view.  The end result is that oslo.messaging's API behavior is
somewhat implicitly defined by the characteristics of the rpc backend
(broker), rather than olso.messaging itself.

In other words: we need to solve this problem in general, not just for
the rabbit driver.

 Is it the case that an RPC call should never fail (i.e. never time out)
 due to failover? Either way, the answer to this should be very clear.

 If failures may occur, then the calling code needs to handle that. If
 eliminating failures is part of the 'contract' then the library should
 have a clear strategy for ensuring (and testing) this.

 Another possible scenario is that the connection is lost immediately
 after writing the request message to the socket (but before it is
 processed by the rabbit broker). In this case the issue is that the
 request is not confirmed, so it can complete before it is 'safe'. In
 other words requests are unreliable.

 My own view is that if you want to avoid time outs on failover, the best
 approach is to have olso.messaging retry the entire request regardless
 of the point it had reached in the previous attempt. I.e. rather than
 trying to make delivery of responses reliable, assume that both requests
 and responses are unreliable and re-issue the request immediately on
 failover.

I like this suggestion. By assuming limited reliability from the
underlying messaging system, we reduce oslo.messaging's reliance on
features provided by any particular messaging implementation
(driver/broker).

 (The retry logic could even be made independent of any driver
 if desired).

Exactly!  Having all QOS related code outside of the drivers would
guarantee that the behavior of the API is _uniform_ across all
drivers.


 This is perhaps a bigger change, but I think it is more easy to get
 right and will also be more scalable and performant since it doesn't
 require replication of every queue and every message.


 
  [0] https://bugs.launchpad.net/oslo.messaging/+bug/1338732
  [1] https://review.openstack.org/#/c/109373/
  [2]
  https://github.com/noelbk/oslo.messaging/commit/960fc26ff050ca3073ad90eccbef1ca95712e82e
  [3] https://review.openstack.org/#/c/109143/



-- 
Ken Giusti  (kgiu...@gmail.com)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [devstack][oslo.messaging] Adding a new RPC backend for testing AMQP 1.0

[Ken Giusti]

Hi,

I'd like some help with $SUBJECT.  I've got a WIP patch up for review:

https://review.openstack.org/#/c/109118/

My goal is to have an RPC backend that I can use to test the new AMQP
1.0 oslo.messaging driver against.  I suspect this new backend would
initially only be used by tests specifically written against the
driver, but I'm hoping for wider adoption as the driver stabilizes and
AMQP 1.0 adoption increases.

As I said, this is only a WIP and doesn't completely work yet (though
it shouldn't break support for the existing backends).  I'm just
looking for some early feedback on whether or not this is the correct
approach.

thanks!

-- 
Ken Giusti  (kgiu...@gmail.com)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging][infra] New package dependencies for oslo.messaging's AMQP 1.0 support.

[Ken Giusti]

On 07/15/2014 10:58:50 +0200, Flavio Percoco wrote:
On 07/15/2014 07:16 PM, Doug Hellmann wrote:
 On Tue, Jul 15, 2014 at 1:03 PM, Ken Giusti kgiu...@gmail.com wrote:

 These packages may be obtained via EPEL for Centos/RHEL systems
 (qpid-proton-c-devel), and via the Qpid project's PPA [3]
 (libqpid-proton2-dev) for Debian/Ubuntu.  They are also available for
 Fedora via the default yum repos.  Otherwise, the source can be pulled
 directly from the Qpid project and built/installed manually [4].

 Do you know the timeline for having those added to the Ubuntu cloud
 archives? I think we try not to add PPAs in devstack, but I'm not sure
 if that's a hard policy.

IIUC, the package has been accepted in Debian - Ken, correct me if I'm
wrong. Here's the link to the Debian's mentor page:

http://mentors.debian.net/package/qpid-proton


No, it hasn't been accepted yet - it is still pending approval by the
sponsor.  That's one of the reasons the Qpid project has set up its
own PPA.



 I'd like to get the blueprint accepted, but I'll have to address these
 new dependencies first.  What is the best way to get these new
 packages into CI, devstack, etc?  And will developers be willing to
 install the proton development libraries, or can this be done
 automagically?

 To set up integration tests we'll need an option in devstack to set
 the messaging driver to this new one. That flag should also trigger
 setting up the dependencies needed. Before you spend time implementing
 that, though, we should clarify the policy on PPAs.

Agreed. FWIW, the work on devstack is on the works but it's being held
off while we clarify the policy on PPAs.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo.messaging][infra] New package dependencies for oslo.messaging's AMQP 1.0 support.

[Ken Giusti]

Hi,

The AMQP 1.0 blueprint proposed for oslo.messaging Juno [0] introduces
dependencies on a few packages that provide AMQP functionality.

These packages are:

* pyngus - a client API
* python-qpid-proton - the python bindings for the Proton AMQP library
* qpid-proton: the AMQP 1.0 library.

pyngus is a pure-python module available at pypi [1].

python-qpid-proton is also available at pypi [2], but it contains a C
extension.  This C extension requires that the qpid-proton development
libraries are installed in order to build the extension when
installing python-qpid-proton.

So this means that oslo.messaging developers, as well as the CI
systems, etc, will need to have the qpid-proton development packages
installed.

These packages may be obtained via EPEL for Centos/RHEL systems
(qpid-proton-c-devel), and via the Qpid project's PPA [3]
(libqpid-proton2-dev) for Debian/Ubuntu.  They are also available for
Fedora via the default yum repos.  Otherwise, the source can be pulled
directly from the Qpid project and built/installed manually [4].

I'd like to get the blueprint accepted, but I'll have to address these
new dependencies first.  What is the best way to get these new
packages into CI, devstack, etc?  And will developers be willing to
install the proton development libraries, or can this be done
automagically?

thanks for your help,


[0] 
https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation
[1] https://pypi.python.org/pypi/pyngus
[2] https://pypi.python.org/pypi/python-qpid-proton/0.7-0
[3] https://launchpad.net/~qpid
[4] http://qpid.apache.org/download.html

-- 
Ken Giusti  (kgiu...@gmail.com)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev