Re: [openstack-dev] [Nova] Turbo hipster problems
Thanks. Yes, it is up and running again! From: Joshua Hesketh joshua.hesk...@rackspace.commailto:joshua.hesk...@rackspace.com Reply-To: OpenStack List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Monday, October 20, 2014 at 3:45 AM To: OpenStack List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Nova] Turbo hipster problems Hi Gary, Sorry we had a mis-hap over the weekend. The Database CI should be back up and running now. Let me know if you see any more problems. Cheers, Josh Rackspace Australia On 10/18/14 1:55 AM, Gary Kotton wrote: Hi, Anyone aware why Turbo his peter is failing with: real-db-upgrade_nova_percona_user_002:th-perconahttps://urldefense.proofpoint.com/v1/url?u=http://localhostk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0Am=HSN3f4I4FiDfec3IQQMovHsZInTYKJB%2BnFTFljbiNsg%3D%0As=63ef5a3f5bd643d5565a9fa0b8df3c47dceafc269f5682f6b63a92ebd076902dException: [Errno 2] No such file or directory: '/var/lib/turbo-hipster/datasets_user_002' in 0s Thanks Gary ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Turbo hipster problems
Hi Gary, Sorry we had a mis-hap over the weekend. The Database CI should be back up and running now. Let me know if you see any more problems. Cheers, Josh Rackspace Australia On 10/18/14 1:55 AM, Gary Kotton wrote: Hi, Anyone aware why Turbo his peter is failing with: real-db-upgrade_nova_percona_user_002:th-percona http://localhost Exception: [Errno 2] No such file or directory: '/var/lib/turbo-hipster/datasets_user_002' in 0s Thanks Gary ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Nova] Turbo hipster problems
Hi, Anyone aware why Turbo his peter is failing with: real-db-upgrade_nova_percona_user_002:th-perconahttp://localhost Exception: [Errno 2] No such file or directory: '/var/lib/turbo-hipster/datasets_user_002' in 0s Thanks Gary ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Nova] Turbo hipster
Hi, This appears to be broken over the last 24 hours. Thanks Gary ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Turbo hipster
Hi Gary, Thanks for the notification. Our nodepool had built a corrupt image which was returning false positives. I have fixed this up and rerun tests on changes that had negative votes where Jenkins passed. If you notice any changes that seem like a false negative please issue a recheck migrations. If it fails a second time please let us know at rc...@rcbops.com. FYI, you can see our testing progress at http://zuul.rcbops.com Cheers, Josh Rackspace Australia On 6/29/14 5:48 PM, Gary Kotton wrote: Hi, This appears to be broken over the last 24 hours. Thanks Gary ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova] Turbo-Hipster bad votes
Hi all, Unfortunately turbo-hipster has been leaving bad votes on nova db migrations. I've disabled voting and we're looking into the problem. Sorry for the inconvenience. If you have any questions please feel free to ask. Regards, Josh ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][turbo hipster] unable to rebase
On 8 January 2014 12:07, Gary Kotton gkot...@vmware.com wrote: Hi, Patch sets that are cascaded has have the following errors: Database migration testing failed either due to migrations unable to be applied correctly or taking too long. This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patch set. What do you suggest? If you rebase you code, would that not fix everything? It seems for the tests to pass, that change needs to be able to rebase on trunk these days. John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][turbo hipster] unable to rebase
Hi! Sorry the slow response. I've been at a conference this week, so I'm a bit behind. I apologise for that. The root cause here is a failure to understand the relatively undocumented git behaviour of zuul. We think we've now got a fix, and will deploy it soon. Michael On Thu, Jan 9, 2014 at 8:30 PM, John Garbutt j...@johngarbutt.com wrote: On 8 January 2014 12:07, Gary Kotton gkot...@vmware.com wrote: Hi, Patch sets that are cascaded has have the following errors: Database migration testing failed either due to migrations unable to be applied correctly or taking too long. This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patch set. What do you suggest? If you rebase you code, would that not fix everything? It seems for the tests to pass, that change needs to be able to rebase on trunk these days. John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Rackspace Australia ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Turbo-hipster
Dan Prince dpri...@redhat.com writes: - Original Message - From: Michael Still mi...@stillhq.com - commenting recheck .* - commenting recheck migrations With the growing interest in 3rd party testing systems would using 'recheck turbo-hipster' make more sense here? I'm fine with 'recheck migrations' in addition for turbo-hipster but it would make sense to align the recheck naming scheme with the title of the reviewer for the 3rd party testing system. This is the can of worms I was hoping we would not open. Or try to get them all back into the can and close it again is perhaps the better metaphor. I do not think that system-specific recheck commands will be actually that useful or important. As I mentioned, I understand the theoretical usefulness of being able to say oops, I can tell this one system messed up, let's ask it to try again. But given that case is covered by asking all systems to recheck, it seems harmless to say all systems to recheck. I just don't think that asking developers to learn a micro-language of commands left in gerrit comments is the best use of time. Maybe I'm wrong about that, but I was hoping we could try not creating it first to see if there's really a need. Dealing with the expected errors of a system first coming online doesn't, in my mind, demonstrate that need. If it turns out that asking programmers not to create a new language is futile, yes, I'd rather we have some predictability. Most third party CI systems have descriptive names, so rather than saying recheck turbo-hipster, perhaps we should change turbo-hipster's display name to something related to migrations. So _if_ we want to have system-specific recheck commands, then I think we should ask operators to make them in the form recheck systemname, and that they output a brief sentence of help text to remind people of that in the report message in Gerrit. -Jim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Turbo-hipster
On 12/31/2013 3:58 PM, Michael Still wrote: Hi. So while turbo hipster is new, I've been reading every failure message it produces to make sure its not too badly wrong. There were four failures posted last night while I slept: https://review.openstack.org/#/c/64521 This one is a TH bug. We shouldn't be testing stable branches. bug/1265238 has been filed to track this. https://review.openstack.org/#/c/61753 This is your review. The failed run's log is https://ssl.rcbops.com/turbo_hipster/logviewer/?q=/turbo_hipster/results/61/61753/8/check/gate-real-db-upgrade_nova_percona_user_001/1326092/user_001.log and you can see from the failure message that migrations 152 and 206 took too long. Migration 152 took 326 seconds, where our historical data of 2,846 test migrations says it should take 222 seconds. Migration 206 took 81 seconds, where we think it should take 56 seconds based on 2,940 test runs. Whilst I can't explain why those migrations took too long this time around, they are certainly exactly the sort of thing TH is meant to catch. If you think your patch isn't responsible (perhaps the machine is just being slow or something), you can always retest by leaving a review comment of recheck migrations. I have done this for you on this patch. Michael, is recheck migrations something that is going to be added to the wiki for test failures here? https://wiki.openstack.org/wiki/GerritJenkinsGit#Test_Failures https://review.openstack.org/#/c/61717 This review also had similar unexplained slowness, but has already been rechecked by someone else and now passes. I note that the slowness in both cases was from the same TH worker node, and I will keep an eye on that node today. https://review.openstack.org/#/c/56420 This review also had slowness in migration 206, but has been rechecked by the developer and now passes. It wasn't on the percona-001 worker that the other two were on, so perhaps this indicates that we need to relax the timing requirements for migration 206. Hope this helps, Michael On Wed, Jan 1, 2014 at 12:34 AM, Gary Kotton gkot...@vmware.com wrote: Hi, It seems that she/he is behaving oddly again. I have posted a patch that does not have any database changes and it has give me a –1…. Happy new year Gary ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Turbo-hipster
Heh, I didn't know that wiki page existed. I've added an entry to the checklist. There's also some talk of adding some help text to the vote message turbo-hipster leaves in gerrit, but we haven't gotten around to doing that yet. Cheers, Michael On Fri, Jan 3, 2014 at 12:57 AM, Matt Riedemann mrie...@linux.vnet.ibm.com wrote: On 12/31/2013 3:58 PM, Michael Still wrote: Hi. So while turbo hipster is new, I've been reading every failure message it produces to make sure its not too badly wrong. There were four failures posted last night while I slept: https://review.openstack.org/#/c/64521 This one is a TH bug. We shouldn't be testing stable branches. bug/1265238 has been filed to track this. https://review.openstack.org/#/c/61753 This is your review. The failed run's log is https://ssl.rcbops.com/turbo_hipster/logviewer/?q=/turbo_hipster/results/61/61753/8/check/gate-real-db-upgrade_nova_percona_user_001/1326092/user_001.log and you can see from the failure message that migrations 152 and 206 took too long. Migration 152 took 326 seconds, where our historical data of 2,846 test migrations says it should take 222 seconds. Migration 206 took 81 seconds, where we think it should take 56 seconds based on 2,940 test runs. Whilst I can't explain why those migrations took too long this time around, they are certainly exactly the sort of thing TH is meant to catch. If you think your patch isn't responsible (perhaps the machine is just being slow or something), you can always retest by leaving a review comment of recheck migrations. I have done this for you on this patch. Michael, is recheck migrations something that is going to be added to the wiki for test failures here? https://wiki.openstack.org/wiki/GerritJenkinsGit#Test_Failures https://review.openstack.org/#/c/61717 This review also had similar unexplained slowness, but has already been rechecked by someone else and now passes. I note that the slowness in both cases was from the same TH worker node, and I will keep an eye on that node today. https://review.openstack.org/#/c/56420 This review also had slowness in migration 206, but has been rechecked by the developer and now passes. It wasn't on the percona-001 worker that the other two were on, so perhaps this indicates that we need to relax the timing requirements for migration 206. Hope this helps, Michael On Wed, Jan 1, 2014 at 12:34 AM, Gary Kotton gkot...@vmware.com wrote: Hi, It seems that she/he is behaving oddly again. I have posted a patch that does not have any database changes and it has give me a –1…. Happy new year Gary ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Rackspace Australia ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Turbo-hipster
On 01/02/2014 04:29 PM, Michael Still wrote: Heh, I didn't know that wiki page existed. I've added an entry to the checklist. There's also some talk of adding some help text to the vote message turbo-hipster leaves in gerrit, but we haven't gotten around to doing that yet. Cheers, Michael So was there enough countable slowness earlier in the run that you could have predicted these runs would be slower overall? My experience looking at Tempest run data is there can be as much as an +60% variance from fastest and slowest nodes (same instance type) within the same cloud provider, which is the reason we've never tried to performance gate on it. However if there was some earlier benchmark that would let you realize that the whole run was slow, so give it more of a buffer, that would probably be useful. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Turbo-hipster
Michael Still mi...@stillhq.com writes: Heh, I didn't know that wiki page existed. I've added an entry to the checklist. There's also some talk of adding some help text to the vote message turbo-hipster leaves in gerrit, but we haven't gotten around to doing that yet. I would rather not mention it on that page, which is the documentation for the project gating system and developer workflow (Zuul links to it when it leaves a failure message) so I have removed it. I _do_ think adding help text to the messages third-party tools leave, and/or linking to specific documentation (ideally also in the OpenStack wiki) from there is a good idea. However, there are _a lot_ of third-party test systems coming on-line, and I'm not sure that expanding the recheck language to support ever more complexity is a good idea. I can see how being able to say recheck foo would be useful in some circumstances, but given that just saying recheck will suffice, I'd prefer that we kept the general recommendation simple so developers can worry about something else. Certainly at a minimum, recheck should recheck all the systems; that's one of the proposed requirements here: https://review.openstack.org/#/c/63478/5/doc/source/third_party.rst I think it would be best if we stopped there. But if you still feel very strongly that you want a private extension to the syntax, please consider how necessary it is for most developers to know about it when you decide how prominently to feature it in messages or documentation about your tools. -Jim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Turbo-hipster
Sean Dague s...@dague.net writes: On 01/02/2014 04:29 PM, Michael Still wrote: Heh, I didn't know that wiki page existed. I've added an entry to the checklist. There's also some talk of adding some help text to the vote message turbo-hipster leaves in gerrit, but we haven't gotten around to doing that yet. Cheers, Michael So was there enough countable slowness earlier in the run that you could have predicted these runs would be slower overall? My experience looking at Tempest run data is there can be as much as an +60% variance from fastest and slowest nodes (same instance type) within the same cloud provider, which is the reason we've never tried to performance gate on it. However if there was some earlier benchmark that would let you realize that the whole run was slow, so give it more of a buffer, that would probably be useful. If you are able to do this and benchmark the performance of a cloud server reliably enough, we might be able to make progress on performance testing, which has been long desired. The large ops test is (somewhat accidentally) a performance test, and predictably, it has failed when we change cloud node provider configurations. A benchmark could make this test more reliable and other tests more feasible. -Jim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Turbo-hipster
On 3 January 2014 11:26, James E. Blair jebl...@openstack.org wrote: If you are able to do this and benchmark the performance of a cloud server reliably enough, we might be able to make progress on performance testing, which has been long desired. The large ops test is (somewhat accidentally) a performance test, and predictably, it has failed when we change cloud node provider configurations. A benchmark could make this test more reliable and other tests more feasible. In bzr we found it much more reliable to do tests that isolate and capture the *effort*, not the time: most [not all] performance issues have both a time and effort domain, and the effort domain is usually correlated with time in a particular environment, but itself approximately constant across environments. For instance, MB sent in a request, or messages on the message bus, or writes to the file system, or queries sent to the DB. So the structure we ended up with - which was quite successful - was: - a cron job based benchmark that ran several versions through functional scenarios and reporting timing data - gating tests that tested effort for operations - a human process whereby someone wanting to put a ratchet on some aspect of performance would write an effort based test or three to capture the status quo, then make it better and update the tests with their improvements. I think this would work well for OpenStack too - and infact we have some things that are in this general direction already. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Turbo-hipster
On Fri, Jan 3, 2014 at 9:24 AM, James E. Blair jebl...@openstack.org wrote: However, there are _a lot_ of third-party test systems coming on-line, and I'm not sure that expanding the recheck language to support ever more complexity is a good idea. I can see how being able to say recheck foo would be useful in some circumstances, but given that just saying recheck will suffice, I'd prefer that we kept the general recommendation simple so developers can worry about something else. Fair enough. I feel like you and I should sit down and chat about this stuff at the LCA meetup next week. If we need to make tweaks based on that, then we will. Cheers, Michael -- Rackspace Australia ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Turbo-hipster
On 01/02/2014 05:39 PM, Michael Still wrote: On Fri, Jan 3, 2014 at 9:24 AM, James E. Blair jebl...@openstack.org wrote: However, there are _a lot_ of third-party test systems coming on-line, and I'm not sure that expanding the recheck language to support ever more complexity is a good idea. I can see how being able to say recheck foo would be useful in some circumstances, but given that just saying recheck will suffice, I'd prefer that we kept the general recommendation simple so developers can worry about something else. Fair enough. I feel like you and I should sit down and chat about this stuff at the LCA meetup next week. If we need to make tweaks based on that, then we will. Cheers, Michael I'd love to attend this chat, if possible. A number of the coming on-line third-party test systems are motivated by neutron plugins. I'd get a lot from hearing this discussion. Thanks, Anita. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Turbo-hipster
On Fri, Jan 3, 2014 at 9:46 AM, Anita Kuno ante...@anteaya.info wrote: I'd love to attend this chat, if possible. A number of the coming on-line third-party test systems are motivated by neutron plugins. I'd get a lot from hearing this discussion. I've created a BoF on Monday night straight after the CI miniconf in the same room. Let's all get together and have a chat and then go to dinner. Michael -- Rackspace Australia ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Turbo-hipster
On 01/02/2014 05:58 PM, Michael Still wrote: On Fri, Jan 3, 2014 at 9:46 AM, Anita Kuno ante...@anteaya.info wrote: I'd love to attend this chat, if possible. A number of the coming on-line third-party test systems are motivated by neutron plugins. I'd get a lot from hearing this discussion. I've created a BoF on Monday night straight after the CI miniconf in the same room. Let's all get together and have a chat and then go to dinner. Michael Great, thank you Michael. Anita. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Turbo-hipster
On Fri, Jan 3, 2014 at 9:39 AM, Michael Still mi...@stillhq.com wrote: On Fri, Jan 3, 2014 at 9:24 AM, James E. Blair jebl...@openstack.org wrote: However, there are _a lot_ of third-party test systems coming on-line, and I'm not sure that expanding the recheck language to support ever more complexity is a good idea. I can see how being able to say recheck foo would be useful in some circumstances, but given that just saying recheck will suffice, I'd prefer that we kept the general recommendation simple so developers can worry about something else. Fair enough. I feel like you and I should sit down and chat about this stuff at the LCA meetup next week. If we need to make tweaks based on that, then we will. Further to this, I have just reloaded our zuul with a rules change. The following events will all cause a turbo hipster check run: - uploading a patchset - restoring a patchset - commenting recheck .* - commenting recheck migrations Cheers, Michael -- Rackspace Australia ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova] Turbo-hipster
Hi, It seems that she/he is behaving oddly again. I have posted a patch that does not have any database changes and it has give me a –1…. Happy new year Gary ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Turbo-hipster
Hi. So while turbo hipster is new, I've been reading every failure message it produces to make sure its not too badly wrong. There were four failures posted last night while I slept: https://review.openstack.org/#/c/64521 This one is a TH bug. We shouldn't be testing stable branches. bug/1265238 has been filed to track this. https://review.openstack.org/#/c/61753 This is your review. The failed run's log is https://ssl.rcbops.com/turbo_hipster/logviewer/?q=/turbo_hipster/results/61/61753/8/check/gate-real-db-upgrade_nova_percona_user_001/1326092/user_001.log and you can see from the failure message that migrations 152 and 206 took too long. Migration 152 took 326 seconds, where our historical data of 2,846 test migrations says it should take 222 seconds. Migration 206 took 81 seconds, where we think it should take 56 seconds based on 2,940 test runs. Whilst I can't explain why those migrations took too long this time around, they are certainly exactly the sort of thing TH is meant to catch. If you think your patch isn't responsible (perhaps the machine is just being slow or something), you can always retest by leaving a review comment of recheck migrations. I have done this for you on this patch. https://review.openstack.org/#/c/61717 This review also had similar unexplained slowness, but has already been rechecked by someone else and now passes. I note that the slowness in both cases was from the same TH worker node, and I will keep an eye on that node today. https://review.openstack.org/#/c/56420 This review also had slowness in migration 206, but has been rechecked by the developer and now passes. It wasn't on the percona-001 worker that the other two were on, so perhaps this indicates that we need to relax the timing requirements for migration 206. Hope this helps, Michael On Wed, Jan 1, 2014 at 12:34 AM, Gary Kotton gkot...@vmware.com wrote: Hi, It seems that she/he is behaving oddly again. I have posted a patch that does not have any database changes and it has give me a –1…. Happy new year Gary ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Rackspace Australia ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
