[openstack-dev] [Congress] Mid-cycle sprint debrief
Hi all, TL;DR. We had a great midcycle sprint. If you want to help us move Congress toward its new distributed architecture, there are a list of items you can help with below. 1. We had a productive mid-cycle sprint last week! Here were the topics we covered... - Design discussions about implementing the distributed architecture using oslo-messaging. - Discussion about integrating Monasca so that when people write policy Congress can use Monasca alarms to push information to us. - Discussion with Su Zhang about his experiences using Congress at Symantec. - Code sprint aimed at migrating off of our current, in-process message bus to oslo-messaging. For details about the discussions, check out the etherpad... https://etherpad.openstack.org/p/congress-mitaka-sprint 2. Moving forward in the short-term, we are focusing on migrating from our existing in-process message bus (DSE) to a small wrapper around oslo-messaging (DSE2). - As we migrate from DSE to DSE2, we're leaving the mainline code in place and minimizing the changes so that it runs on both the old arch and the new arch. We use the flag 'distributed_architecture' to signal which version we are running. - Ideally the tests will all continue to function without modification in both the old and new architectures. But for those test files that don't pass, we are copying them to congress/tests2 and modifying them there. - We have disabled a few tests temporarily and marked them with TODO(dse2) and an explanation as to why they are disabled so we can easily grep for them later. - tox -enew_arch will (soon) run all the tests in congress/tests2. 3. For those of you looking to help out, here are a few items you can sign up for. 3.1. Work on porting congress/tests/test_congress.py to the new distributed arch. See congress/tests2/test_dse2.py for an example. In fact, there may be tests commented out in tests2/dse2/test_dse2.py that we should re-enable/port. https://bugs.launchpad.net/congress/+bug/1541008 3.2. Work on porting the remaining API models to the new arch. See my recent changesets for the basic idea. https://review.openstack.org/#/c/274957/ https://bugs.launchpad.net/congress/+bug/1541001 https://bugs.launchpad.net/congress/+bug/1541002 https://bugs.launchpad.net/congress/+bug/1541003 https://bugs.launchpad.net/congress/+bug/1541004 3.3. Create a non-voting gate job for tox -enew_arch https://bugs.launchpad.net/congress/+bug/1540990 3.4. Try out the scripts/start_process.py script to see how we would use it for the new arch. Eventually we'll want a non-voting job in the gate that runs all the tempest tests on the new architecture. https://bugs.launchpad.net/congress/+bug/1541019 Questions/comments? Tim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Congress] Mid-cycle sprint and next week's IRC
Just a reminder that next week Jan 26-Jan 28 is the Congress mid-cycle Sprint in the Bay Area. It's not too late to attend. It'd be best to register, but we'd love to have you in any case. More details here: https://wiki.openstack.org/wiki/Sprints/CongressMitakaSprint Also, since we're at the Sprint next week, there will be NO IRC meeting Jan 27. Tim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Congress] Mid-cycle sprint summary
Hi all, We just finished up a great 2 day sprint focusing on a new distributed architecture for Congress. Details can be found in the etherpad: https://etherpad.openstack.org/p/congress-liberty-sprint Here's the summary. 1. Architecture. Each datasource driver will run in its own process; each policy engine will run in its own process; the API will run in its own process. All processes will communicate using oslo-messaging. We decided to remove the functionality for creating/deleting datasources, since that functionality will be handled by the operating system. 2. Blueprints. The blueprints we created all start with "dist-". Please sign up if you're interested. If you attended the sprint and volunteered for a blueprint but did not sign up, please do so. https://blueprints.launchpad.net/congress 3. Code commits. We're making changes in place on master. The plan is to make most of the changes without breaking existing functionality. Then there will be one or two changes at the end that swap out the current, single-process architecture for the new, multi-process architecture. 4. Timelines. We're hoping to have the basic functionality in place by Tokyo. We will release the current architecture for liberty and the new architecture for M. Thanks for a great sprint, everyone! Let me know if you have questions. Tim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [congress] mid cycle Sprint dates
I added Congress to the list of Liberty sprints and built a wiki page with basic info: https://wiki.openstack.org/wiki/Sprints/CongressLibertySprint Please RSVP using eventbrite: http://www.eventbrite.com/e/congress-liberty-midcycle-sprint-tickets-17654731778 Tim On Wed, Jul 1, 2015 at 8:58 AM Jeremy Stanley wrote: > On 2015-07-01 15:40:31 + (+), Tim Hinrichs wrote: > > We settled on dates and location for the Congress mid cycle sprint: > [...] > > Invoking the spirit of Thierry, please remember to add it to the > list at: > > https://wiki.openstack.org/wiki/Sprints#Liberty_sprints > > -- > Jeremy Stanley > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [congress] mid cycle Sprint dates
On 2015-07-01 15:40:31 + (+), Tim Hinrichs wrote: > We settled on dates and location for the Congress mid cycle sprint: [...] Invoking the spirit of Thierry, please remember to add it to the list at: https://wiki.openstack.org/wiki/Sprints#Liberty_sprints -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [congress] mid cycle Sprint dates
Hi all! We settled on dates and location for the Congress mid cycle sprint: Aug 6-7 VMware campus, Palo Alto, CA Please RSVP if you plan to come so we can get a headcount. Hope to see you there! Tim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Congress] Mid-cycle sprint
Hi Zhou, The requirements for the request_refresh API call came from a customer. It wasn't intended as a replacement for the functionality we'd get by integrating with oslo.messaging. But it was a simple feature to add, with nice properties from Congress's perspective, driven by a real use case. It'd be great to support a more standard kind of streaming as well, such as with oslo.messaging. Tim On Thu, Jun 18, 2015 at 11:10 PM Zhou, Zhenzan wrote: > Hi, Tim > > > > I have looked at the request_fresh_action. One big question is that it > requires external services to queue up changes and then call this Congress > API at some point. I’m not sure they would buy in this design as many > projects have notification mechanism ready now and Ceilometer heavily > depends on these notifications. So basically I prefer to use > oslo.messaging. But this API is still useful for other external services > that don’t use oslo.messaging notification to publish changes and are > willing to integrate with Congress this way. > > Anyway, I can upload my draft bp for review at first. > > Thanks. > > > > BR > > Zhou Zhenzan > > > > *From:* Tim Hinrichs [mailto:t...@styra.com] > *Sent:* Wednesday, June 17, 2015 21:57 > > > *To:* OpenStack Development Mailing List (not for usage questions) > > *Subject:* Re: [openstack-dev] [Congress] Mid-cycle sprint > > > > Hi Zhenzan, > > Yes the oslo.messaging integration task is relevant--oslo.messaging is one > way of achieving cross-process, cross-host messaging. So I'll count you as > interested for the mid-cycle sprint. > > > > Have you looked at the API call that forces a datasource driver to pull > immediately? See > congress/api/datasource_model.py:DatasourceModel.request_refresh_action. We > had envisioned using that to implement a kind of notification from external > services as follows. The external service queues up a list of changes and > when the queue is long enough runs the API call to force the datasource > driver hooked up to that service to pull those changes and then publish > them on the bus. So it's not exactly streaming updates from the external > service (which is good so that Congress can easily rate-limit updates), but > it has almost the same effect. > > > > Tim > > > > On Tue, Jun 16, 2015 at 6:02 PM Zhou, Zhenzan > wrote: > > Hi, Tim > > > > Is this the oslo.messaging integration task? I’m interested in > participating. Actually I am working on a bp to receive notifications from > external services in datasource driver at first. I’m ok to change if the > direction is to integrate oslo.messaging thoroughly (even replacing DSE). > > Thanks. > > > > BR > > Zhou Zhenzan > > > > *From:* Tim Hinrichs [mailto:t...@styra.com] > *Sent:* Wednesday, June 17, 2015 05:14 > *To:* OpenStack Development Mailing List (not for usage questions) > *Subject:* [openstack-dev] [Congress] Mid-cycle sprint > > > > Hi all, > > > > In the last couple of IRCs we've been talking about running a mid-cycle > sprint focused on enabling our message bus to span multiple processes and > multiple hosts. The message bus is what allows the Congress policy engine > to communicate with the Congress wrappers around external services like > Nova, Neutron. This cross-process, cross-host message bus is the platform > we'll use to build version 2.0 of our distributed architecture. > > > > If you're interested in participating, drop me a note. Once we know who's > interested we'll work out date/time/location details. > > > > Thanks! > > Tim > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Congress] Mid-cycle sprint
Hi, Tim I have looked at the request_fresh_action. One big question is that it requires external services to queue up changes and then call this Congress API at some point. I’m not sure they would buy in this design as many projects have notification mechanism ready now and Ceilometer heavily depends on these notifications. So basically I prefer to use oslo.messaging. But this API is still useful for other external services that don’t use oslo.messaging notification to publish changes and are willing to integrate with Congress this way. Anyway, I can upload my draft bp for review at first. Thanks. BR Zhou Zhenzan From: Tim Hinrichs [mailto:t...@styra.com] Sent: Wednesday, June 17, 2015 21:57 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Congress] Mid-cycle sprint Hi Zhenzan, Yes the oslo.messaging integration task is relevant--oslo.messaging is one way of achieving cross-process, cross-host messaging. So I'll count you as interested for the mid-cycle sprint. Have you looked at the API call that forces a datasource driver to pull immediately? See congress/api/datasource_model.py:DatasourceModel.request_refresh_action. We had envisioned using that to implement a kind of notification from external services as follows. The external service queues up a list of changes and when the queue is long enough runs the API call to force the datasource driver hooked up to that service to pull those changes and then publish them on the bus. So it's not exactly streaming updates from the external service (which is good so that Congress can easily rate-limit updates), but it has almost the same effect. Tim On Tue, Jun 16, 2015 at 6:02 PM Zhou, Zhenzan mailto:zhenzan.z...@intel.com>> wrote: Hi, Tim Is this the oslo.messaging integration task? I’m interested in participating. Actually I am working on a bp to receive notifications from external services in datasource driver at first. I’m ok to change if the direction is to integrate oslo.messaging thoroughly (even replacing DSE). Thanks. BR Zhou Zhenzan From: Tim Hinrichs [mailto:t...@styra.com<mailto:t...@styra.com>] Sent: Wednesday, June 17, 2015 05:14 To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [Congress] Mid-cycle sprint Hi all, In the last couple of IRCs we've been talking about running a mid-cycle sprint focused on enabling our message bus to span multiple processes and multiple hosts. The message bus is what allows the Congress policy engine to communicate with the Congress wrappers around external services like Nova, Neutron. This cross-process, cross-host message bus is the platform we'll use to build version 2.0 of our distributed architecture. If you're interested in participating, drop me a note. Once we know who's interested we'll work out date/time/location details. Thanks! Tim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Congress] Mid-cycle sprint
Hi Adam, Thanks for the offer! It'd definitely be great to join you all in Boston, but I'm guessing the logistics aren't going to work out. Most of the people I've heard from are in the Bay Area, which makes it hard to host in Boston.But I'll keep it in mind and float it by everyone. Tim On Wed, Jun 17, 2015 at 7:20 PM Adam Young wrote: > How many people do you think you will have? We have a midcycle in > Boston University, July 15-17 and you are welcome to join in. I am pretty > sure we will have more than enough capacity, considering the size of the > Congress team. > > Hotels might be a bit of an issue, as we are getting close, and they are > starting to book up, but the BU admin has let us know that we can get dorm > space is people so desire. > > > > On 06/16/2015 05:13 PM, Tim Hinrichs wrote: > > Hi all, > > In the last couple of IRCs we've been talking about running a mid-cycle > sprint focused on enabling our message bus to span multiple processes and > multiple hosts. The message bus is what allows the Congress policy engine > to communicate with the Congress wrappers around external services like > Nova, Neutron. This cross-process, cross-host message bus is the platform > we'll use to build version 2.0 of our distributed architecture. > > If you're interested in participating, drop me a note. Once we know > who's interested we'll work out date/time/location details. > > Thanks! > Tim > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Congress] Mid-cycle sprint
How many people do you think you will have? We have a midcycle in Boston University, July 15-17 and you are welcome to join in. I am pretty sure we will have more than enough capacity, considering the size of the Congress team. Hotels might be a bit of an issue, as we are getting close, and they are starting to book up, but the BU admin has let us know that we can get dorm space is people so desire. On 06/16/2015 05:13 PM, Tim Hinrichs wrote: Hi all, In the last couple of IRCs we've been talking about running a mid-cycle sprint focused on enabling our message bus to span multiple processes and multiple hosts. The message bus is what allows the Congress policy engine to communicate with the Congress wrappers around external services like Nova, Neutron. This cross-process, cross-host message bus is the platform we'll use to build version 2.0 of our distributed architecture. If you're interested in participating, drop me a note. Once we know who's interested we'll work out date/time/location details. Thanks! Tim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Congress] Mid-cycle sprint
Hi Zhenzan, Yes the oslo.messaging integration task is relevant--oslo.messaging is one way of achieving cross-process, cross-host messaging. So I'll count you as interested for the mid-cycle sprint. Have you looked at the API call that forces a datasource driver to pull immediately? See congress/api/datasource_model.py:DatasourceModel.request_refresh_action. We had envisioned using that to implement a kind of notification from external services as follows. The external service queues up a list of changes and when the queue is long enough runs the API call to force the datasource driver hooked up to that service to pull those changes and then publish them on the bus. So it's not exactly streaming updates from the external service (which is good so that Congress can easily rate-limit updates), but it has almost the same effect. Tim On Tue, Jun 16, 2015 at 6:02 PM Zhou, Zhenzan wrote: > Hi, Tim > > > > Is this the oslo.messaging integration task? I’m interested in > participating. Actually I am working on a bp to receive notifications from > external services in datasource driver at first. I’m ok to change if the > direction is to integrate oslo.messaging thoroughly (even replacing DSE). > > Thanks. > > > > BR > > Zhou Zhenzan > > > > *From:* Tim Hinrichs [mailto:t...@styra.com] > *Sent:* Wednesday, June 17, 2015 05:14 > *To:* OpenStack Development Mailing List (not for usage questions) > *Subject:* [openstack-dev] [Congress] Mid-cycle sprint > > > > Hi all, > > > > In the last couple of IRCs we've been talking about running a mid-cycle > sprint focused on enabling our message bus to span multiple processes and > multiple hosts. The message bus is what allows the Congress policy engine > to communicate with the Congress wrappers around external services like > Nova, Neutron. This cross-process, cross-host message bus is the platform > we'll use to build version 2.0 of our distributed architecture. > > > > If you're interested in participating, drop me a note. Once we know who's > interested we'll work out date/time/location details. > > > > Thanks! > > Tim > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Congress] Mid-cycle sprint
Hi, Tim Is this the oslo.messaging integration task? I’m interested in participating. Actually I am working on a bp to receive notifications from external services in datasource driver at first. I’m ok to change if the direction is to integrate oslo.messaging thoroughly (even replacing DSE). Thanks. BR Zhou Zhenzan From: Tim Hinrichs [mailto:t...@styra.com] Sent: Wednesday, June 17, 2015 05:14 To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [Congress] Mid-cycle sprint Hi all, In the last couple of IRCs we've been talking about running a mid-cycle sprint focused on enabling our message bus to span multiple processes and multiple hosts. The message bus is what allows the Congress policy engine to communicate with the Congress wrappers around external services like Nova, Neutron. This cross-process, cross-host message bus is the platform we'll use to build version 2.0 of our distributed architecture. If you're interested in participating, drop me a note. Once we know who's interested we'll work out date/time/location details. Thanks! Tim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Congress] Mid-cycle sprint
Hi all, In the last couple of IRCs we've been talking about running a mid-cycle sprint focused on enabling our message bus to span multiple processes and multiple hosts. The message bus is what allows the Congress policy engine to communicate with the Congress wrappers around external services like Nova, Neutron. This cross-process, cross-host message bus is the platform we'll use to build version 2.0 of our distributed architecture. If you're interested in participating, drop me a note. Once we know who's interested we'll work out date/time/location details. Thanks! Tim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
