Re: [Openstack] Steps that can help stabilize Nova's trunk
:) On Thu, Feb 24, 2011 at 7:36 PM, Jay Pipes wrote: > All done, and waiting to be almost immediately out of date. :) > > Cheers, > jay > > On Thu, Feb 24, 2011 at 7:06 PM, Andy Smith wrote: > > Please summarize these on the wiki and add your information the wiki, > that > > is what the wiki page was made to do and what I asked you to do. > > --andy > > > > On Thu, Feb 24, 2011 at 1:27 PM, Jay Pipes wrote: > >> > >> Andy has listed a few things on the wiki. I'll summarize the known > efforts > >> here: > >> > >> * Anso has created some Vagrant scripts that test multi-node > >> functionality of the EC2 API, libvirt + KVM, and nova-objectstore > >> * Vishy/Devin have been refactored Nova's existing smoketests/ and > >> updated to include netadmin tests. Still only testing EC2 API > >> * Trey has been "volunteered" to write an OpenStack API smoketest for > >> XenServer functionality (https://bugs.launchpad.net/nova/+bug/720941) > >> * Jordan Rinke has been working on a 10-machine test cluster for > >> testing deployments and running smoketests on > >> * Other rackers (Pvo, Ant?) have been working on getting a much larger > >> production-level test cluster for running longer, more complex tests > >> on > >> > >> Stuff we need to do: > >> > >> * Create a staging/testing branch, have Openstack Hudson LP user own it > >> * Get the test cluster machines entered into Hudson > >> * On each merge proposal into trunk, have Tarmac pull the branch, run > >> unit tests automatically, fire off smoketests/ against the test > >> machines automatically, and notify the merge proposal that tests pass > >> or don't pass. > >> * For merge proposals that pass the merge into staging and all tests > >> that also have 2 Approves from core devs, have Tarmac merge into trunk > >> * Create long-running functional tests that are essentially re-playing > >> large Apache/nginx log files of existing Nebula and Cloud Servers API > >> nodes against Nova staging branch with various configurations > >> > >> -jay > >> > >> On Thu, Feb 24, 2011 at 3:49 PM, Sandy Walsh > > >> wrote: > >> > Jay, > >> > Nice to see this issue being addressed ... it's a big deal. > >> > From reading this (long) thread, my biggest source of confusion was so > >> > many "we're doing something on this front too ..." messages. > >> > Would it be possible to get a summary of the various integration > testing > >> > efforts underway so we can find out where the > >> > commonality/biggest-bang-for-the-buck/overlap is? Perhaps a wiki page > >> > already exists for this? > >> > Thx, > >> > -S > >> > > >> > > >> > > >> > From: << a cast of thousands ... delete >> > >> > > >> > Confidentiality Notice: This e-mail message (including any attached or > >> > embedded documents) is intended for the exclusive and confidential use > >> > of > >> > the > >> > individual or entity to which this message is addressed, and unless > >> > otherwise > >> > expressly indicated, is confidential and privileged information of > >> > Rackspace. > >> > Any dissemination, distribution or copying of the enclosed material is > >> > prohibited. > >> > If you receive this transmission in error, please notify us > immediately > >> > by > >> > e-mail > >> > at ab...@rackspace.com, and delete the original message. > >> > Your cooperation is appreciated. > >> > > >> > ___ > >> > Mailing list: https://launchpad.net/~openstack > >> > Post to : openstack@lists.launchpad.net > >> > Unsubscribe : https://launchpad.net/~openstack > >> > More help : https://help.launchpad.net/ListHelp > >> > > >> > > >> > >> ___ > >> Mailing list: https://launchpad.net/~openstack > >> Post to : openstack@lists.launchpad.net > >> Unsubscribe : https://launchpad.net/~openstack > >> More help : https://help.launchpad.net/ListHelp > > > > > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
All done, and waiting to be almost immediately out of date. :) Cheers, jay On Thu, Feb 24, 2011 at 7:06 PM, Andy Smith wrote: > Please summarize these on the wiki and add your information the wiki, that > is what the wiki page was made to do and what I asked you to do. > --andy > > On Thu, Feb 24, 2011 at 1:27 PM, Jay Pipes wrote: >> >> Andy has listed a few things on the wiki. I'll summarize the known efforts >> here: >> >> * Anso has created some Vagrant scripts that test multi-node >> functionality of the EC2 API, libvirt + KVM, and nova-objectstore >> * Vishy/Devin have been refactored Nova's existing smoketests/ and >> updated to include netadmin tests. Still only testing EC2 API >> * Trey has been "volunteered" to write an OpenStack API smoketest for >> XenServer functionality (https://bugs.launchpad.net/nova/+bug/720941) >> * Jordan Rinke has been working on a 10-machine test cluster for >> testing deployments and running smoketests on >> * Other rackers (Pvo, Ant?) have been working on getting a much larger >> production-level test cluster for running longer, more complex tests >> on >> >> Stuff we need to do: >> >> * Create a staging/testing branch, have Openstack Hudson LP user own it >> * Get the test cluster machines entered into Hudson >> * On each merge proposal into trunk, have Tarmac pull the branch, run >> unit tests automatically, fire off smoketests/ against the test >> machines automatically, and notify the merge proposal that tests pass >> or don't pass. >> * For merge proposals that pass the merge into staging and all tests >> that also have 2 Approves from core devs, have Tarmac merge into trunk >> * Create long-running functional tests that are essentially re-playing >> large Apache/nginx log files of existing Nebula and Cloud Servers API >> nodes against Nova staging branch with various configurations >> >> -jay >> >> On Thu, Feb 24, 2011 at 3:49 PM, Sandy Walsh >> wrote: >> > Jay, >> > Nice to see this issue being addressed ... it's a big deal. >> > From reading this (long) thread, my biggest source of confusion was so >> > many "we're doing something on this front too ..." messages. >> > Would it be possible to get a summary of the various integration testing >> > efforts underway so we can find out where the >> > commonality/biggest-bang-for-the-buck/overlap is? Perhaps a wiki page >> > already exists for this? >> > Thx, >> > -S >> > >> > >> > >> > From: << a cast of thousands ... delete >> >> > >> > Confidentiality Notice: This e-mail message (including any attached or >> > embedded documents) is intended for the exclusive and confidential use >> > of >> > the >> > individual or entity to which this message is addressed, and unless >> > otherwise >> > expressly indicated, is confidential and privileged information of >> > Rackspace. >> > Any dissemination, distribution or copying of the enclosed material is >> > prohibited. >> > If you receive this transmission in error, please notify us immediately >> > by >> > e-mail >> > at ab...@rackspace.com, and delete the original message. >> > Your cooperation is appreciated. >> > >> > ___ >> > Mailing list: https://launchpad.net/~openstack >> > Post to : openstack@lists.launchpad.net >> > Unsubscribe : https://launchpad.net/~openstack >> > More help : https://help.launchpad.net/ListHelp >> > >> > >> >> ___ >> Mailing list: https://launchpad.net/~openstack >> Post to : openstack@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp > > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
That url is http://wiki.openstack.org/TestingBrainstorm btw On Thu, Feb 24, 2011 at 4:06 PM, Andy Smith wrote: > Please summarize these on the wiki and add your information the wiki, that > is what the wiki page was made to do and what I asked you to do. > > --andy > > > On Thu, Feb 24, 2011 at 1:27 PM, Jay Pipes wrote: > >> Andy has listed a few things on the wiki. I'll summarize the known efforts >> here: >> >> * Anso has created some Vagrant scripts that test multi-node >> functionality of the EC2 API, libvirt + KVM, and nova-objectstore >> * Vishy/Devin have been refactored Nova's existing smoketests/ and >> updated to include netadmin tests. Still only testing EC2 API >> * Trey has been "volunteered" to write an OpenStack API smoketest for >> XenServer functionality (https://bugs.launchpad.net/nova/+bug/720941) >> * Jordan Rinke has been working on a 10-machine test cluster for >> testing deployments and running smoketests on >> * Other rackers (Pvo, Ant?) have been working on getting a much larger >> production-level test cluster for running longer, more complex tests >> on >> >> Stuff we need to do: >> >> * Create a staging/testing branch, have Openstack Hudson LP user own it >> * Get the test cluster machines entered into Hudson >> * On each merge proposal into trunk, have Tarmac pull the branch, run >> unit tests automatically, fire off smoketests/ against the test >> machines automatically, and notify the merge proposal that tests pass >> or don't pass. >> * For merge proposals that pass the merge into staging and all tests >> that also have 2 Approves from core devs, have Tarmac merge into trunk >> * Create long-running functional tests that are essentially re-playing >> large Apache/nginx log files of existing Nebula and Cloud Servers API >> nodes against Nova staging branch with various configurations >> >> -jay >> >> On Thu, Feb 24, 2011 at 3:49 PM, Sandy Walsh >> wrote: >> > Jay, >> > Nice to see this issue being addressed ... it's a big deal. >> > From reading this (long) thread, my biggest source of confusion was so >> > many "we're doing something on this front too ..." messages. >> > Would it be possible to get a summary of the various integration testing >> > efforts underway so we can find out where the >> > commonality/biggest-bang-for-the-buck/overlap is? Perhaps a wiki page >> > already exists for this? >> > Thx, >> > -S >> > >> > >> > >> > From: << a cast of thousands ... delete >> >> > >> > Confidentiality Notice: This e-mail message (including any attached or >> > embedded documents) is intended for the exclusive and confidential use >> of >> > the >> > individual or entity to which this message is addressed, and unless >> > otherwise >> > expressly indicated, is confidential and privileged information of >> > Rackspace. >> > Any dissemination, distribution or copying of the enclosed material is >> > prohibited. >> > If you receive this transmission in error, please notify us immediately >> by >> > e-mail >> > at ab...@rackspace.com, and delete the original message. >> > Your cooperation is appreciated. >> > >> > ___ >> > Mailing list: https://launchpad.net/~openstack >> > Post to : openstack@lists.launchpad.net >> > Unsubscribe : https://launchpad.net/~openstack >> > More help : https://help.launchpad.net/ListHelp >> > >> > >> >> ___ >> Mailing list: https://launchpad.net/~openstack >> Post to : openstack@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp >> > > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Please summarize these on the wiki and add your information the wiki, that is what the wiki page was made to do and what I asked you to do. --andy On Thu, Feb 24, 2011 at 1:27 PM, Jay Pipes wrote: > Andy has listed a few things on the wiki. I'll summarize the known efforts > here: > > * Anso has created some Vagrant scripts that test multi-node > functionality of the EC2 API, libvirt + KVM, and nova-objectstore > * Vishy/Devin have been refactored Nova's existing smoketests/ and > updated to include netadmin tests. Still only testing EC2 API > * Trey has been "volunteered" to write an OpenStack API smoketest for > XenServer functionality (https://bugs.launchpad.net/nova/+bug/720941) > * Jordan Rinke has been working on a 10-machine test cluster for > testing deployments and running smoketests on > * Other rackers (Pvo, Ant?) have been working on getting a much larger > production-level test cluster for running longer, more complex tests > on > > Stuff we need to do: > > * Create a staging/testing branch, have Openstack Hudson LP user own it > * Get the test cluster machines entered into Hudson > * On each merge proposal into trunk, have Tarmac pull the branch, run > unit tests automatically, fire off smoketests/ against the test > machines automatically, and notify the merge proposal that tests pass > or don't pass. > * For merge proposals that pass the merge into staging and all tests > that also have 2 Approves from core devs, have Tarmac merge into trunk > * Create long-running functional tests that are essentially re-playing > large Apache/nginx log files of existing Nebula and Cloud Servers API > nodes against Nova staging branch with various configurations > > -jay > > On Thu, Feb 24, 2011 at 3:49 PM, Sandy Walsh > wrote: > > Jay, > > Nice to see this issue being addressed ... it's a big deal. > > From reading this (long) thread, my biggest source of confusion was so > > many "we're doing something on this front too ..." messages. > > Would it be possible to get a summary of the various integration testing > > efforts underway so we can find out where the > > commonality/biggest-bang-for-the-buck/overlap is? Perhaps a wiki page > > already exists for this? > > Thx, > > -S > > > > > > > > From: << a cast of thousands ... delete >> > > > > Confidentiality Notice: This e-mail message (including any attached or > > embedded documents) is intended for the exclusive and confidential use of > > the > > individual or entity to which this message is addressed, and unless > > otherwise > > expressly indicated, is confidential and privileged information of > > Rackspace. > > Any dissemination, distribution or copying of the enclosed material is > > prohibited. > > If you receive this transmission in error, please notify us immediately > by > > e-mail > > at ab...@rackspace.com, and delete the original message. > > Your cooperation is appreciated. > > > > ___ > > Mailing list: https://launchpad.net/~openstack > > Post to : openstack@lists.launchpad.net > > Unsubscribe : https://launchpad.net/~openstack > > More help : https://help.launchpad.net/ListHelp > > > > > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Andy has listed a few things on the wiki. I'll summarize the known efforts here: * Anso has created some Vagrant scripts that test multi-node functionality of the EC2 API, libvirt + KVM, and nova-objectstore * Vishy/Devin have been refactored Nova's existing smoketests/ and updated to include netadmin tests. Still only testing EC2 API * Trey has been "volunteered" to write an OpenStack API smoketest for XenServer functionality (https://bugs.launchpad.net/nova/+bug/720941) * Jordan Rinke has been working on a 10-machine test cluster for testing deployments and running smoketests on * Other rackers (Pvo, Ant?) have been working on getting a much larger production-level test cluster for running longer, more complex tests on Stuff we need to do: * Create a staging/testing branch, have Openstack Hudson LP user own it * Get the test cluster machines entered into Hudson * On each merge proposal into trunk, have Tarmac pull the branch, run unit tests automatically, fire off smoketests/ against the test machines automatically, and notify the merge proposal that tests pass or don't pass. * For merge proposals that pass the merge into staging and all tests that also have 2 Approves from core devs, have Tarmac merge into trunk * Create long-running functional tests that are essentially re-playing large Apache/nginx log files of existing Nebula and Cloud Servers API nodes against Nova staging branch with various configurations -jay On Thu, Feb 24, 2011 at 3:49 PM, Sandy Walsh wrote: > Jay, > Nice to see this issue being addressed ... it's a big deal. > From reading this (long) thread, my biggest source of confusion was so > many "we're doing something on this front too ..." messages. > Would it be possible to get a summary of the various integration testing > efforts underway so we can find out where the > commonality/biggest-bang-for-the-buck/overlap is? Perhaps a wiki page > already exists for this? > Thx, > -S > > > > From: << a cast of thousands ... delete >> > > Confidentiality Notice: This e-mail message (including any attached or > embedded documents) is intended for the exclusive and confidential use of > the > individual or entity to which this message is addressed, and unless > otherwise > expressly indicated, is confidential and privileged information of > Rackspace. > Any dissemination, distribution or copying of the enclosed material is > prohibited. > If you receive this transmission in error, please notify us immediately by > e-mail > at ab...@rackspace.com, and delete the original message. > Your cooperation is appreciated. > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Jay, Nice to see this issue being addressed ... it's a big deal. >From reading this (long) thread, my biggest source of confusion was so many >"we're doing something on this front too ..." messages. Would it be possible to get a summary of the various integration testing efforts underway so we can find out where the commonality/biggest-bang-for-the-buck/overlap is? Perhaps a wiki page already exists for this? Thx, -S From: << a cast of thousands ... delete >> Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
On Mon, Feb 21, 2011 at 6:37 AM, Jay Pipes wrote: > On Fri, Feb 18, 2011 at 5:30 PM, Andy Smith wrote: > > A few emails back (I have been in meetings and travel for the past two > weeks > > so I am just catching up on email now), Jay pretty much described our > plan > > of setting up a bunch of machines in multiple configurations for use as a > > test cluster. > > Towards that goal I'd love to start compiling the various systems > mentioned > > in this thread so far into a place so that we can look at the > configurations > > people are already using. My vote is just a wiki page linking to Jenkins > > deployments and tarballs of Jenkins build configurations and so forth. > > http://wiki.openstack.org/TestingBrainstorm > > OK. > > > Soren, based on your blog post I suspect you have a bunch of this > material > > already, could you add it to that wiki? > > Brian Schott, could you add your deployment and existing testing strategy > to > > that wiki? > > I'll take care of getting the Anso/NASA pieces (we have a bunch of > > vagrant-based stuff) in there. > > --andy > > There is no documentation as to how the Vagrant/Chef stuff works, how > to kick it off, or how it should be linked into Hudson. It also seems > to be very Anso/Nebula specific. Did we expect that the Vagrant/Chef > stuff would be used by the OpenStack community to test OpenStack? > Also, if we did, why isn't it in the Nova project itself? I thought > one of your goals was not to have the testing stuff in different > projects (and now, on totally different repositories...) > There was no suggestion about making a new repository, I am simply trying to get a list of all the existing repositories and ideas people are using in testing. Please add yours to http://wiki.openstack.org/TestingBrainstorm > > For those of us a little tentative to learn Yet Another Programming > Language, even some basic documentation would be useful to see if the > Vagrant/Chef stuff is something that could be applicable to OpenStack > continuous integration testing. > > As I've stated in previous emails on this thread, the problem we have > is *not* that we don't have the ability to run tests. In other words, > we don't need another test-running platform. What we need is tests for > stuff that isn't Anso-specific, which is why I've asked Trey to work > on creating a smoketest for the valid work patterns that have to do > with OpenStack and not EC2. > > In the meantime, to test the EC2-specific code paths, I would think it > would be fairly trivial to simply fire the smoketests against a test > cluster. I know that one of the Rackspace test clusters (a large, > 188-machine one) should be online in the next week or so, but there > was also word of another smaller test cluster that seems more > appropriate for simply kicking smoketests against. Jordan, what's the > status of that smaller cluster? Are we ready to add those machines to > our set of Hudson builders? > In case Jordan missed the question since it was at the end of a paragraph: Jordan, any news on machines? --andy > > -jay > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Perhaps the articles can be added to docs somewhere to nova.openstack.org, the wiki, or docs.openstack.org? Vishy is going to be working on the chef scripts integrating changes by Mike Ray. (We'd rather not have two active chef recipe sets) On Feb 21, 2011 12:29 PM, "Jay Pipes" wrote: ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Hi Jesse, thanks for the links to those articles. Would be great to add Anso's article feed to the OpenStack planet so these great articles will show up for folks following that. :) -jay On Mon, Feb 21, 2011 at 3:13 PM, Jesse Andrews wrote: > Jay, > > Agree that openstack api coverage top priority in added smoketests. > > The work needed done on vagrant (vish led it and wrote about it a bit at > ansolabs.com/deploy) was a start to making something that worked outside > nasa - the multinode work was so it would work on any *nix box. Nasa's > testing cluster uses bare metal provioning, not vagrant. The goal of the > chef recipes is something that works for vagrant and physical deployments on > real hardware. > > More work needs done on both deploying test clusters and testing them. > > Jesse > > On Feb 21, 2011 6:39 AM, "Jay Pipes" wrote: >> On Fri, Feb 18, 2011 at 5:30 PM, Andy Smith wrote: >>> A few emails back (I have been in meetings and travel for the past two >>> weeks >>> so I am just catching up on email now), Jay pretty much described our >>> plan >>> of setting up a bunch of machines in multiple configurations for use as a >>> test cluster. >>> Towards that goal I'd love to start compiling the various systems >>> mentioned >>> in this thread so far into a place so that we can look at the >>> configurations >>> people are already using. My vote is just a wiki page linking to Jenkins >>> deployments and tarballs of Jenkins build configurations and so forth. >>> http://wiki.openstack.org/TestingBrainstorm >> >> OK. >> >>> Soren, based on your blog post I suspect you have a bunch of this >>> material >>> already, could you add it to that wiki? >>> Brian Schott, could you add your deployment and existing testing strategy >>> to >>> that wiki? >>> I'll take care of getting the Anso/NASA pieces (we have a bunch of >>> vagrant-based stuff) in there. >>> --andy >> >> There is no documentation as to how the Vagrant/Chef stuff works, how >> to kick it off, or how it should be linked into Hudson. It also seems >> to be very Anso/Nebula specific. Did we expect that the Vagrant/Chef >> stuff would be used by the OpenStack community to test OpenStack? >> Also, if we did, why isn't it in the Nova project itself? I thought >> one of your goals was not to have the testing stuff in different >> projects (and now, on totally different repositories...) >> >> For those of us a little tentative to learn Yet Another Programming >> Language, even some basic documentation would be useful to see if the >> Vagrant/Chef stuff is something that could be applicable to OpenStack >> continuous integration testing. >> >> As I've stated in previous emails on this thread, the problem we have >> is *not* that we don't have the ability to run tests. In other words, >> we don't need another test-running platform. What we need is tests for >> stuff that isn't Anso-specific, which is why I've asked Trey to work >> on creating a smoketest for the valid work patterns that have to do >> with OpenStack and not EC2. >> >> In the meantime, to test the EC2-specific code paths, I would think it >> would be fairly trivial to simply fire the smoketests against a test >> cluster. I know that one of the Rackspace test clusters (a large, >> 188-machine one) should be online in the next week or so, but there >> was also word of another smaller test cluster that seems more >> appropriate for simply kicking smoketests against. Jordan, what's the >> status of that smaller cluster? Are we ready to add those machines to >> our set of Hudson builders? >> >> -jay >> >> ___ >> Mailing list: https://launchpad.net/~openstack >> Post to : openstack@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Jay, Agree that openstack api coverage top priority in added smoketests. The work needed done on vagrant (vish led it and wrote about it a bit at ansolabs.com/deploy) was a start to making something that worked outside nasa - the multinode work was so it would work on any *nix box. Nasa's testing cluster uses bare metal provioning, not vagrant. The goal of the chef recipes is something that works for vagrant and physical deployments on real hardware. More work needs done on both deploying test clusters and testing them. Jesse On Feb 21, 2011 6:39 AM, "Jay Pipes" wrote: > On Fri, Feb 18, 2011 at 5:30 PM, Andy Smith wrote: >> A few emails back (I have been in meetings and travel for the past two weeks >> so I am just catching up on email now), Jay pretty much described our plan >> of setting up a bunch of machines in multiple configurations for use as a >> test cluster. >> Towards that goal I'd love to start compiling the various systems mentioned >> in this thread so far into a place so that we can look at the configurations >> people are already using. My vote is just a wiki page linking to Jenkins >> deployments and tarballs of Jenkins build configurations and so forth. >> http://wiki.openstack.org/TestingBrainstorm > > OK. > >> Soren, based on your blog post I suspect you have a bunch of this material >> already, could you add it to that wiki? >> Brian Schott, could you add your deployment and existing testing strategy to >> that wiki? >> I'll take care of getting the Anso/NASA pieces (we have a bunch of >> vagrant-based stuff) in there. >> --andy > > There is no documentation as to how the Vagrant/Chef stuff works, how > to kick it off, or how it should be linked into Hudson. It also seems > to be very Anso/Nebula specific. Did we expect that the Vagrant/Chef > stuff would be used by the OpenStack community to test OpenStack? > Also, if we did, why isn't it in the Nova project itself? I thought > one of your goals was not to have the testing stuff in different > projects (and now, on totally different repositories...) > > For those of us a little tentative to learn Yet Another Programming > Language, even some basic documentation would be useful to see if the > Vagrant/Chef stuff is something that could be applicable to OpenStack > continuous integration testing. > > As I've stated in previous emails on this thread, the problem we have > is *not* that we don't have the ability to run tests. In other words, > we don't need another test-running platform. What we need is tests for > stuff that isn't Anso-specific, which is why I've asked Trey to work > on creating a smoketest for the valid work patterns that have to do > with OpenStack and not EC2. > > In the meantime, to test the EC2-specific code paths, I would think it > would be fairly trivial to simply fire the smoketests against a test > cluster. I know that one of the Rackspace test clusters (a large, > 188-machine one) should be online in the next week or so, but there > was also word of another smaller test cluster that seems more > appropriate for simply kicking smoketests against. Jordan, what's the > status of that smaller cluster? Are we ready to add those machines to > our set of Hudson builders? > > -jay > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
On Fri, Feb 18, 2011 at 5:30 PM, Andy Smith wrote: > A few emails back (I have been in meetings and travel for the past two weeks > so I am just catching up on email now), Jay pretty much described our plan > of setting up a bunch of machines in multiple configurations for use as a > test cluster. > Towards that goal I'd love to start compiling the various systems mentioned > in this thread so far into a place so that we can look at the configurations > people are already using. My vote is just a wiki page linking to Jenkins > deployments and tarballs of Jenkins build configurations and so forth. > http://wiki.openstack.org/TestingBrainstorm OK. > Soren, based on your blog post I suspect you have a bunch of this material > already, could you add it to that wiki? > Brian Schott, could you add your deployment and existing testing strategy to > that wiki? > I'll take care of getting the Anso/NASA pieces (we have a bunch of > vagrant-based stuff) in there. > --andy There is no documentation as to how the Vagrant/Chef stuff works, how to kick it off, or how it should be linked into Hudson. It also seems to be very Anso/Nebula specific. Did we expect that the Vagrant/Chef stuff would be used by the OpenStack community to test OpenStack? Also, if we did, why isn't it in the Nova project itself? I thought one of your goals was not to have the testing stuff in different projects (and now, on totally different repositories...) For those of us a little tentative to learn Yet Another Programming Language, even some basic documentation would be useful to see if the Vagrant/Chef stuff is something that could be applicable to OpenStack continuous integration testing. As I've stated in previous emails on this thread, the problem we have is *not* that we don't have the ability to run tests. In other words, we don't need another test-running platform. What we need is tests for stuff that isn't Anso-specific, which is why I've asked Trey to work on creating a smoketest for the valid work patterns that have to do with OpenStack and not EC2. In the meantime, to test the EC2-specific code paths, I would think it would be fairly trivial to simply fire the smoketests against a test cluster. I know that one of the Rackspace test clusters (a large, 188-machine one) should be online in the next week or so, but there was also word of another smaller test cluster that seems more appropriate for simply kicking smoketests against. Jordan, what's the status of that smaller cluster? Are we ready to add those machines to our set of Hudson builders? -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Catching up as well... On Thu, Feb 17, 2011 at 2:57 PM, Jay Pipes wrote: > The Anso team has told me that they run the smoketests regularly, but > I would assume they only run these smoketests against a test cluster > that mimicks the Nebula environment. Devin and termie can correct me > if I'm wrong on that. We've taken a few approaches as the code has evolved (automatic deployment and testing was always crucial) While preparing for the Nebula beta release our system was as follows. Automatically deploying a fresh cluster every morning using a shell script that would: * pxe/preseed -> deploy and configure base ubuntu os * puppet -> deploy and configure nova * run a few shell commands on the head node to setup users/whatnot Then we would run the smoketests against the cluster, recording errors and exceptions (to sentry using nova-sentry). Then through the day users and QA would test against the cluster, reporting bugs. If required anyone could re-deploy (pxe -> setup) by running a single shell command. (it would take about 30 minutes from reboot to ready to run) Running a cloud is more than just nova. We've had critical issues with configuration (intel 10G cards + LRO/MRO + linux network bridging brought network performance for guests down to 20Kbps) that were completely external to Nova, but crucial for running an openstack deployment. Have a set of "golden systems": OS + configurations (network model + api + hypervisors) + relevant smoketests that run with every trunk merge. Having the system that runs these tests open source so others can setup labs to run tests for their equipment/configurations would be preferable as well. I also agree that "machines are cheaper than devs" and think a good phase two would be able to run this against arbitrary branches. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
A few emails back (I have been in meetings and travel for the past two weeks so I am just catching up on email now), Jay pretty much described our plan of setting up a bunch of machines in multiple configurations for use as a test cluster. Towards that goal I'd love to start compiling the various systems mentioned in this thread so far into a place so that we can look at the configurations people are already using. My vote is just a wiki page linking to Jenkins deployments and tarballs of Jenkins build configurations and so forth. http://wiki.openstack.org/TestingBrainstorm Soren, based on your blog post I suspect you have a bunch of this material already, could you add it to that wiki? Brian Schott, could you add your deployment and existing testing strategy to that wiki? I'll take care of getting the Anso/NASA pieces (we have a bunch of vagrant-based stuff) in there. --andy On Fri, Feb 18, 2011 at 8:40 AM, Trey Morris wrote: > yeah I agree with jay, makes me feel more comfortable with this process. > also, re jay: guess i had better get to work on that bug > > -tr3buchet > > > On Fri, Feb 18, 2011 at 7:29 AM, Jay Pipes wrote: > >> This was an excellent explanation. Thank you, Thierry. >> >> -jay >> >> On Fri, Feb 18, 2011 at 2:55 AM, Thierry Carrez >> wrote: >> > Trey Morris wrote: >> >> I don't like that it currently only runs on ubuntu + the ppa. If it >> >> doesn't work with existing versions I think we're doing something >> wrong. >> >> Even when natty comes out, I don't like the idea of having to ensure I >> >> have latest ubuntu for openstack to run. >> > >> > Maybe with my Ubuntu core developer hat on, I can try to clarify. >> > >> > Once an Ubuntu version is released, you can't push new versions of >> > libraries or new features in it, only critical bugfixes. >> > >> > So when for openstack we happen to need to new version of a library, it >> > won't run on the current stable version. We have a choice between >> > avoiding new versions of libraries altogether, or use a PPA to add the >> > required version. >> > >> > The route we follow is to provide the new version of the library in a >> > PPA for the current stable releases (Lucid/Maverick), and update the >> > development release (Natty) proper. >> > >> > The end goal being, when Natty goes out, the corresponding Openstack >> > release (Cactus) runs on it without the need for a PPA. For those that >> > prefer to run it on an older stable release (or an LTS), the "release" >> > PPA can be used. >> > >> > Regards, >> > >> > -- >> > Thierry Carrez (ttx) >> > Release Manager, OpenStack >> > Core developer, Ubuntu >> > >> > ___ >> > Mailing list: https://launchpad.net/~openstack >> > Post to : openstack@lists.launchpad.net >> > Unsubscribe : https://launchpad.net/~openstack >> > More help : https://help.launchpad.net/ListHelp >> > >> >> ___ >> Mailing list: https://launchpad.net/~openstack >> Post to : openstack@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp >> > > Confidentiality Notice: This e-mail message (including any attached or > embedded documents) is intended for the exclusive and confidential use of the > individual or entity to which this message is addressed, and unless otherwise > expressly indicated, is confidential and privileged information of Rackspace. > Any dissemination, distribution or copying of the enclosed material is > prohibited. > If you receive this transmission in error, please notify us immediately by > e-mail > at ab...@rackspace.com, and delete the original message. > Your cooperation is appreciated. > > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
yeah I agree with jay, makes me feel more comfortable with this process. also, re jay: guess i had better get to work on that bug -tr3buchet On Fri, Feb 18, 2011 at 7:29 AM, Jay Pipes wrote: > This was an excellent explanation. Thank you, Thierry. > > -jay > > On Fri, Feb 18, 2011 at 2:55 AM, Thierry Carrez > wrote: > > Trey Morris wrote: > >> I don't like that it currently only runs on ubuntu + the ppa. If it > >> doesn't work with existing versions I think we're doing something wrong. > >> Even when natty comes out, I don't like the idea of having to ensure I > >> have latest ubuntu for openstack to run. > > > > Maybe with my Ubuntu core developer hat on, I can try to clarify. > > > > Once an Ubuntu version is released, you can't push new versions of > > libraries or new features in it, only critical bugfixes. > > > > So when for openstack we happen to need to new version of a library, it > > won't run on the current stable version. We have a choice between > > avoiding new versions of libraries altogether, or use a PPA to add the > > required version. > > > > The route we follow is to provide the new version of the library in a > > PPA for the current stable releases (Lucid/Maverick), and update the > > development release (Natty) proper. > > > > The end goal being, when Natty goes out, the corresponding Openstack > > release (Cactus) runs on it without the need for a PPA. For those that > > prefer to run it on an older stable release (or an LTS), the "release" > > PPA can be used. > > > > Regards, > > > > -- > > Thierry Carrez (ttx) > > Release Manager, OpenStack > > Core developer, Ubuntu > > > > ___ > > Mailing list: https://launchpad.net/~openstack > > Post to : openstack@lists.launchpad.net > > Unsubscribe : https://launchpad.net/~openstack > > More help : https://help.launchpad.net/ListHelp > > > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
This was an excellent explanation. Thank you, Thierry. -jay On Fri, Feb 18, 2011 at 2:55 AM, Thierry Carrez wrote: > Trey Morris wrote: >> I don't like that it currently only runs on ubuntu + the ppa. If it >> doesn't work with existing versions I think we're doing something wrong. >> Even when natty comes out, I don't like the idea of having to ensure I >> have latest ubuntu for openstack to run. > > Maybe with my Ubuntu core developer hat on, I can try to clarify. > > Once an Ubuntu version is released, you can't push new versions of > libraries or new features in it, only critical bugfixes. > > So when for openstack we happen to need to new version of a library, it > won't run on the current stable version. We have a choice between > avoiding new versions of libraries altogether, or use a PPA to add the > required version. > > The route we follow is to provide the new version of the library in a > PPA for the current stable releases (Lucid/Maverick), and update the > development release (Natty) proper. > > The end goal being, when Natty goes out, the corresponding Openstack > release (Cactus) runs on it without the need for a PPA. For those that > prefer to run it on an older stable release (or an LTS), the "release" > PPA can be used. > > Regards, > > -- > Thierry Carrez (ttx) > Release Manager, OpenStack > Core developer, Ubuntu > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Trey Morris wrote: > I don't like that it currently only runs on ubuntu + the ppa. If it > doesn't work with existing versions I think we're doing something wrong. > Even when natty comes out, I don't like the idea of having to ensure I > have latest ubuntu for openstack to run. Maybe with my Ubuntu core developer hat on, I can try to clarify. Once an Ubuntu version is released, you can't push new versions of libraries or new features in it, only critical bugfixes. So when for openstack we happen to need to new version of a library, it won't run on the current stable version. We have a choice between avoiding new versions of libraries altogether, or use a PPA to add the required version. The route we follow is to provide the new version of the library in a PPA for the current stable releases (Lucid/Maverick), and update the development release (Natty) proper. The end goal being, when Natty goes out, the corresponding Openstack release (Cactus) runs on it without the need for a PPA. For those that prefer to run it on an older stable release (or an LTS), the "release" PPA can be used. Regards, -- Thierry Carrez (ttx) Release Manager, OpenStack Core developer, Ubuntu ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Soren, thank you for the explanation! Jim On Feb 17, 2011, at 5:11 PM, Soren Hansen wrote: > 2011/2/17 Jim Curry : >> By defining an unbreakable reference platform, are we necessarily limiting >> its ability to integrate on other platforms? That is my underlying >> question. I understand the need for a reference platform but am trying to >> understand to what extent that results in us not being able to easily >> support other platforms (or not). > > Oh, I see. Thanks for clarifying. I don't think this limits us in that > respect at all. I don't think we've ever rejected patches that would > allow us to run on other platforms (and I can't imagine we ever > would). > > That said, Ubuntu offers a very up-to-date selection of software that > we can build upon. Not being limited by a static, out-of-date set of > dependencies is very much the reason we can move forward at the pace > that we do. Other distros might not have all the dependencies we need. > Yet, at least. It's possible they will in their next release. > > If we take the example from my previous e-mail in this thread.. There > was a race condition that was caused by a bug in Eventlet. There were > two[1] ways to fix it: 1) Fix it in Eventlet or 2) accept Eventlet's > brokenness and attempt to work around it. Once the problem was > identified and I was at a point where I knew I had to make that > decision, option 1) took probably about an hour or so, includiing > filing the bug with eventlet, providing them with a test case, writing > a patch, uploading it to Ubuntu, and backporting it to multiple > version of Ubuntu in our PPA's. It's not entirely inconceivable that > I'd still be working on option 2 right now (we rely pretty heavily on > Eventlet). If someone wants to put Nova into some other distribution > they "just" need to make sure that the eventlet in said distro has > this patch applied. > > [1]: Well, there's a third: Monkey patch eventlet. I'm just afraid I'd > be caught in an infinite loop of irony. > > -- > Soren Hansen > Ubuntu Developerhttp://www.ubuntu.com/ > OpenStack Developer http://www.openstack.org/ Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
On Feb 17, 2011, at 5:55 PM, Soren Hansen wrote: > 2011/2/17 Lorin Hochstein : >> For those who deploy on platforms other than Ubuntu, are these >> customizations recorded somewhere in the OpenStack documentation? > > Hmm.. No. I submitted it upstream, applied it in Ubuntu, and uploaded > backported packages to the PPA's. I guess we could also add a patches/ > directory to nova and stick stuff in there. Would that address your > concerns? > Yeah, I think that should do. > > >> Having access to a comprehensive set of the dependency packages versions for >> the reference implementation platform would be very helpful for debugging >> problems on other platforms. Is it possible automatically pull this info >> from the PPA, extract it from the package metadata, and then add it to the >> admin guide docs? > > I'd hate to have to update admin docs for this sort of stuff. A > general pointer to the (not-yet-existing) patches/ directory would be > a good idea, though. I think that's fine, something like "Some dependencies contain bugs that prevent nova working correctly. In those cases nova relies on a customized version of the dependency with a bugfix. These bugfixes to dependencies are located in the patches/ directory". Take care, Lorin -- Lorin Hochstein, Computer Scientist USC Information Sciences Institute 703.812.3710 http://www.east.isi.edu/~lorin smime.p7s Description: S/MIME cryptographic signature ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Agreed. I don't think bleeding-edge pip should block a merge either. It could cause a lot of false-positives. I do think up-to-date Natty, Maverick, and Lucid with build-depends from PPA should be tested, probably CentOS and SUSE if that can be automated. You want apt-get dist-upgrade ; run_test.sh -N to work. How about some automation to "Score your Branch"? Do that BEFORE a branch gets proposed for review. Encourage committers to post their branch score in the bug. At least we when see what is broken. Brian Schott bfsch...@gmail.com On Feb 17, 2011, at 4:21 PM, Soren Hansen wrote: > 2011/2/17 Jay Pipes : >>> One thing we saw in the list and experienced first hand is that your Hudson >>> server uses a pre-configured environment and ours pulls virtual env every >>> time. We had failures on trunk that yours missed because of new pip pulled >>> versions. >>> >>> A fresh run_tests.sh -f needs to work. It is the only guaranteed sanity >>> test everybody can replicate. It might pull upstream bugs, but better to >>> be ahead of that curve than behind. >> Although Soren adequately explained why he thinks that running >> run_tests.sh on Hudson should *not* be against a pip/virtualenv setup, >> I would like to state that I think it would be a good idea to have >> Hudson do *both*. >> >> In other words, for each pre-merge into trunk, Hudson would run two things: >> >> * run_tests.sh -N >> * run_tests.sh -V -f > > I understand the motivation, I'm just not sure I want the latter to > actually block a merge. As an example, the recent race condition I > spotted and fixed required a patch to land in eventlet. If the latter > was allowed to block a merge, we'd have to keep a known race condition > in Nova until upstream decides to do a release so that pip could fetch > it. That could take a *long* time. In the mean time, I stuck a fixed > Eventlet package in our PPA (and in Ubuntu Natty proper) so that we > could move on with our lives and get rid of the race condition. > There's simply a flexibility in this approach that I don't see how we > can obtain with the pip approach. > > Again, though, I would be *delighted* to have automated testing on a > load of different platforms. I just don't believe they should all be > allowed to block us from moving forward. > > -- > Soren Hansen > Ubuntu Developerhttp://www.ubuntu.com/ > OpenStack Developer http://www.openstack.org/ PGP.sig Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Believe, me we know. Our team is bringing up OpenStack on our heterogeneous testbed, a 1TB 256core SGI UltraViolet machine, three Nvidia S2050 boards connected to three SGI XE500 boxes (exposing the GPUs to new instance types using a modified gVirtuS driver), and a nova-compute proxy managing 10 Tilera TILEmpower systems with 64-core TilePro64 processors (that are bare-metal PXE booting the images from an x86 box). They all have dual 10GbE interfaces and it's all SUSE Enterprise. Configuration management is a joy. I was under the impression that "runtest.sh -V -f" was what committers should strive to achieve before submitting a branch for review. That means you might win the booby prize and hit python_migrate bug totally unrelated to your patch. Brian Schott bfsch...@gmail.com On Feb 17, 2011, at 2:18 PM, Soren Hansen wrote: > One other thing I forgot to point out is that there's a world outside > of Python, too. You can pull all the stuff you want from pip, but > there's still a whole bunch of things that aren't managed by pip that > you need to rely on your distro for anyways. Like, say, a kernel, > Python itself, libvirt, iscsi tools, etc., etc. > > -- > Soren Hansen > Ubuntu Developerhttp://www.ubuntu.com/ > OpenStack Developer http://www.openstack.org/ PGP.sig Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
2011/2/17 Jim Curry : > By defining an unbreakable reference platform, are we necessarily limiting > its ability to integrate on other platforms? That is my underlying question. > I understand the need for a reference platform but am trying to understand > to what extent that results in us not being able to easily support other > platforms (or not). Oh, I see. Thanks for clarifying. I don't think this limits us in that respect at all. I don't think we've ever rejected patches that would allow us to run on other platforms (and I can't imagine we ever would). That said, Ubuntu offers a very up-to-date selection of software that we can build upon. Not being limited by a static, out-of-date set of dependencies is very much the reason we can move forward at the pace that we do. Other distros might not have all the dependencies we need. Yet, at least. It's possible they will in their next release. If we take the example from my previous e-mail in this thread.. There was a race condition that was caused by a bug in Eventlet. There were two[1] ways to fix it: 1) Fix it in Eventlet or 2) accept Eventlet's brokenness and attempt to work around it. Once the problem was identified and I was at a point where I knew I had to make that decision, option 1) took probably about an hour or so, includiing filing the bug with eventlet, providing them with a test case, writing a patch, uploading it to Ubuntu, and backporting it to multiple version of Ubuntu in our PPA's. It's not entirely inconceivable that I'd still be working on option 2 right now (we rely pretty heavily on Eventlet). If someone wants to put Nova into some other distribution they "just" need to make sure that the eventlet in said distro has this patch applied. [1]: Well, there's a third: Monkey patch eventlet. I'm just afraid I'd be caught in an infinite loop of irony. -- Soren Hansen Ubuntu Developer http://www.ubuntu.com/ OpenStack Developer http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Hi Dan! Sorry you were the last of my reply emails on this thread. Email overload today :) Comments inline. On Thu, Feb 17, 2011 at 9:43 AM, Dan Prince wrote: > I see the 'smoketests' directory in the nova code. Is anyone running these > tests on a regular basis (every commit)? Is this the best place to further > build out integration tests? The Anso team has told me that they run the smoketests regularly, but I would assume they only run these smoketests against a test cluster that mimicks the Nebula environment. Devin and termie can correct me if I'm wrong on that. As far as I know, nobody outside of Anso has run the smoketests, and they are not integrated into Hudson. Yet. :) termie and I had a brief email conversation about using the existing Nova smoketests as the basis for some continuous integration testing. I was kind of waiting for him to email the mailing list about that topic, but I'll guess I'll pre-empt his email here. :) As far as I know, the Anso plan (which I am in favour of) is to set up a multi-node test cluster (Jordan Rinke may have already gotten this done) that can have a number of these "target environments" (Nebula at NASA, Rackspace Cloud Servers setup, etc) ready to run tests on. Then, we'd install the Hudson agent on some test machine that would have a nova.conf that represented the test environments and fire the smoketests against these test environments. The smoketests are perfectly fine, but I think most agree that they represent a limited set of actions that a user and an admin would execute against the Nova API. In fact, the current smoketests only test the EC2 API, since that is the API that the Nebula cluster uses. So, AFAIK, there are no smoketests at all for the OpenStack API. This is a major problem for Racker contributors, as I think you'd agree, and one that we need to address ASAP. :) I've created a bug for this, the lack of OpenStack API integration test (smoketest): https://bugs.launchpad.net/nova/+bug/720941 Trey, by virtue of responding to my prior email suggesting the need for an OpenStack API smoketest, volunteered to create the smoketest. And by "volunteered", I mean of course that I simply assigned Trey to the bug ;) > Regarding environment/setup tools: I been working on an Openstack VPC toolkit > project that we are using in Blacksburg to stage test some things in the > cloud. I'm using Chef along with the Anso/Opscode Openstack cookbooks to > setup Rackspace Cloud Servers with the latest trunk PPA packages. This is great news. How can we integrate this work into these things? a) Nova's smoketest files b) Hudson > This setup works well however I can only test with Qemu (no Xenserver) and > using network managers that have DHCP (I use FlatDHCPManager since Cloud > Servers kernels don't currently have the 'ndb' kernel module which would > support the network injection stuff). That is a hardware problem that Paul Voccio and others have been working on. There is a cluster of machines that are being set up that will have XenServer (5.6p2 Callie?) installed on them that will be available to run tests against. I'll let Paul give details on this test cluster that will shortly (?) be integrated into our Hudson setup. > Using this setup I'm able to create multi node installations where instances > on different machines can ping each other. While this isn't what I would call > a true production setup it is fully functional and can easily be run in > parallel. The only limitations are the limits on your Cloud Servers Account. OK. Are you executing a specific set of pre-determined Cloud Servers API requests against Nova? If so, how are those requests created? Are they in a test case that we could integrate into the existing Nova smoketests files? > If you have bare metal then you can simply swap out the Cloud Servers API > layer with something that interfaces with your PXE imaging system. I'm a big > fan of slicking the machines between each test run to avoid the buildup of > cruft in the system. Yep, AFAIK, Rackspace's internal Autotools team has been working with Paul to enable this kind of thing with the afore-mentioned test cluster to "wipe" the cluster back to an original XenServer setting between test runs. > This would integrate as a Hudson job nicely as well. We've done some similar > setups in Rackspace Email and Apps using a single Hudson server. The hudson > server runs a simple Bash script that invokes the toolkit to create the cloud > servers, chef them up with the latest PPA packages (or your branch code), and > then uses Torque (an HPC'ish resource manager) to run schedule test jobs on > some of the machines. I use Chef recipes to install Torque along with a REST > interface to schedule and monitor the jobs on a "head" node. The Hudson job > the waits for the Torque jobs to finish. The last thing the Hudson script > does is 'scp' the unit test results XML file back to the Hudson server where > you can use somethi
Re: [Openstack] Steps that can help stabilize Nova's trunk
2011/2/17 Lorin Hochstein : > For those who deploy on platforms other than Ubuntu, are these > customizations recorded somewhere in the OpenStack documentation? Hmm.. No. I submitted it upstream, applied it in Ubuntu, and uploaded backported packages to the PPA's. I guess we could also add a patches/ directory to nova and stick stuff in there. Would that address your concerns? > If I wanted to deploy to SUSE, it would help to know that, say, I can use the > stock version of greenlet 0.3.1, but I need to use a customized version of > eventlet 0.9.12 with the patches that correspond to python-eventlet in the > nova-core/release PPA. Sure. > Having access to a comprehensive set of the dependency packages versions for > the reference implementation platform would be very helpful for debugging > problems on other platforms. Is it possible automatically pull this info > from the PPA, extract it from the package metadata, and then add it to the > admin guide docs? I'd hate to have to update admin docs for this sort of stuff. A general pointer to the (not-yet-existing) patches/ directory would be a good idea, though. Extracting the info automatically from the PPA isn't really feasible. These things are not quite rigidly structured enough to allows such things. -- Soren Hansen Ubuntu Developer http://www.ubuntu.com/ OpenStack Developer http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
By defining an unbreakable reference platform, are we necessarily limiting its ability to integrate on other platforms? That is my underlying question. I understand the need for a reference platform but am trying to understand to what extent that results in us not being able to easily support other platforms (or not). Jim On Feb 17, 2011, at 4:46 PM, Soren Hansen wrote: > 2011/2/17 Jim Curry : >> Got it. But the primary tradeoff is simply that we need to make sure any >> changes we make to another platforms don't break the Ubuntu integration? >> And in general that should not be a major issue...? > > I don't think I understand the question, I'm afraid. :( From where I > see it, there's no trade-off involved. We've decided to define a > reference platform. Support for other platforms is very welcome > indeed, but there's just the one "blessed" platform, where we "can't" > break. It's the platform we test everything on. > > -- > Soren Hansen > Ubuntu Developerhttp://www.ubuntu.com/ > OpenStack Developer http://www.openstack.org/ Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
2011/2/17 Jim Curry : > Got it. But the primary tradeoff is simply that we need to make sure any > changes we make to another platforms don't break the Ubuntu integration? And > in general that should not be a major issue...? I don't think I understand the question, I'm afraid. :( From where I see it, there's no trade-off involved. We've decided to define a reference platform. Support for other platforms is very welcome indeed, but there's just the one "blessed" platform, where we "can't" break. It's the platform we test everything on. -- Soren Hansen Ubuntu Developer http://www.ubuntu.com/ OpenStack Developer http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Got it. But the primary tradeoff is simply that we need to make sure any changes we make to another platforms don't break the Ubuntu integration? And in general that should not be a major issue...? Jim On Feb 17, 2011, at 1:32 PM, Soren Hansen wrote: > 2011/2/17 Jim Curry : >> Soren, can you clarify what you mean by Ubuntu being the primary platform? >> Why is that the reference? What limitations does this introduce? > > It's the primary platform because it's the platform we test everything > on, it's the platform we spend time integrating with, etc. > > It started out as the reference because that's what both the Swift > devs as well as the Anso guys had chosen it as their reference > platform. It's an excellent choice, so there has been no motivation to > change that decision as far as I know. > > I can think of a long list of consequences of this choice. Attempting > to phrase any of them as limitations would be contrived. > > Having a reference platform means we can reasonably make a lot of very > useful assumptions about the environment in which Nova runs. Having > that reference platform be Ubuntu means that there's a straightforward > way to modify this environment if we have special needs. I've patched > a number of packages in Ubuntu already to accomodate our needs, > including the Linux kernel, libvirt, Eventlet, user-mode-linux, etc. > > -- > Soren Hansen > Ubuntu Developerhttp://www.ubuntu.com/ > OpenStack Developer http://www.openstack.org/ Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
On Feb 17, 2011, at 4:21 PM, Soren Hansen wrote: > > I understand the motivation, I'm just not sure I want the latter to > actually block a merge. As an example, the recent race condition I > spotted and fixed required a patch to land in eventlet. If the latter > was allowed to block a merge, we'd have to keep a known race condition > in Nova until upstream decides to do a release so that pip could fetch > it. That could take a *long* time. In the mean time, I stuck a fixed > Eventlet package in our PPA (and in Ubuntu Natty proper) so that we > could move on with our lives and get rid of the race condition. > There's simply a flexibility in this approach that I don't see how we > can obtain with the pip approach. > For those who deploy on platforms other than Ubuntu, are these customizations recorded somewhere in the OpenStack documentation? If I wanted to deploy to SUSE, it would help to know that, say, I can use the stock version of greenlet 0.3.1, but I need to use a customized version of eventlet 0.9.12 with the patches that correspond to python-eventlet in the nova-core/release PPA. Having access to a comprehensive set of the dependency packages versions for the reference implementation platform would be very helpful for debugging problems on other platforms. Is it possible automatically pull this info from the PPA, extract it from the package metadata, and then add it to the admin guide docs? Lorin -- Lorin Hochstein, Computer Scientist USC Information Sciences Institute 703.812.3710 http://www.east.isi.edu/~lorin smime.p7s Description: S/MIME cryptographic signature ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
2011/2/17 Jay Pipes : >> One thing we saw in the list and experienced first hand is that your Hudson >> server uses a pre-configured environment and ours pulls virtual env every >> time. We had failures on trunk that yours missed because of new pip pulled >> versions. >> >> A fresh run_tests.sh -f needs to work. It is the only guaranteed sanity >> test everybody can replicate. It might pull upstream bugs, but better to be >> ahead of that curve than behind. > Although Soren adequately explained why he thinks that running > run_tests.sh on Hudson should *not* be against a pip/virtualenv setup, > I would like to state that I think it would be a good idea to have > Hudson do *both*. > > In other words, for each pre-merge into trunk, Hudson would run two things: > > * run_tests.sh -N > * run_tests.sh -V -f I understand the motivation, I'm just not sure I want the latter to actually block a merge. As an example, the recent race condition I spotted and fixed required a patch to land in eventlet. If the latter was allowed to block a merge, we'd have to keep a known race condition in Nova until upstream decides to do a release so that pip could fetch it. That could take a *long* time. In the mean time, I stuck a fixed Eventlet package in our PPA (and in Ubuntu Natty proper) so that we could move on with our lives and get rid of the race condition. There's simply a flexibility in this approach that I don't see how we can obtain with the pip approach. Again, though, I would be *delighted* to have automated testing on a load of different platforms. I just don't believe they should all be allowed to block us from moving forward. -- Soren Hansen Ubuntu Developer http://www.ubuntu.com/ OpenStack Developer http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
+1, this is a great suggestion. On Feb 17, 2011, at 1:06 PM, Jay Pipes wrote: > Although Soren adequately explained why he thinks that running > run_tests.sh on Hudson should *not* be against a pip/virtualenv setup, > I would like to state that I think it would be a good idea to have > Hudson do *both*. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Great thoughts Jay - I think a push to improve test coverage is a great goal for Cactus. It seems like we are getting new contributors faster than ever these days. I would like to suggest that we create a blueprint called something like "improve test coverage" and create a number of bug reports that reference the blueprint. Then label them as "low hanging fruit", which will encourage everyone (especially new contributors) to look at them. As we all know, the best way to learn a codebase is to write tests for it. On Feb 17, 2011, at 12:57 PM, Jay Pipes wrote: > On Wed, Feb 16, 2011 at 11:44 PM, Paul Voccio > wrote: >> Jay, >> >> Thanks for throwing this out. How would we build this with Hudson? What >> would a "standard deploy" of Nova even look like for integration tests? > > I replied with some specifics to Trey, who had a similar question, and > created a bug report (that I subsequently assigned to Trey): > > https://bugs.launchpad.net/nova/+bug/720941 > > Let me know if that answered your questions and if you'd like some > more explanation. > >> We've also bounced the idea within our team of not allowing code commits >> if the code to test ratio decreases but I'm not sure if that would work >> for such a large project like this one. > > This is a good idea, but even if we were to add code to test ratios, > the ratio would be mostly (only?) looking at unit tests. And I think > we've seen that unit tests pass and functional tests blow up because > of all the mocks/stubs in the unit tests that don't adequately > represent a real-world deployment. > > Nothing wrong, of course, with exploring code to test ratio > (basically, automated code coverage tests), but I think good > functional and integration tests are more productive at this time. > > Just my two cents, though, nothing more, > > -jay > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
On Thu, Feb 17, 2011 at 12:53 AM, Brian Schott wrote: > One thing we saw in the list and experienced first hand is that your Hudson > server uses a pre-configured environment and ours pulls virtual env every > time. We had failures on trunk that yours missed because of new pip pulled > versions. > > A fresh run_tests.sh -f needs to work. It is the only guaranteed sanity test > everybody can replicate. It might pull upstream bugs, but better to be ahead > of that curve than behind. Although Soren adequately explained why he thinks that running run_tests.sh on Hudson should *not* be against a pip/virtualenv setup, I would like to state that I think it would be a good idea to have Hudson do *both*. In other words, for each pre-merge into trunk, Hudson would run two things: * run_tests.sh -N * run_tests.sh -V -f The former would verify that the commit will not break an *existing* installation. The latter would verify that the commit will not break a *new* installation. I don't see why the two cannot be run for each commit into trunk. The run_tests.sh -V -f would, of course, take a lot longer than run_tests.sh -N because the virtualenv needs to be destroyed, recreated, and all dependencies in tools/pip-requires need to be downloaded and installed. However, I feel the run_tests.sh -V -f is an important test to run, as it would pick up issues that have bitten a lot of developers -- for instance, the recent "from migrate import versioning" bug... > We are now triggering on trunk commit emails and running Hudson on both trunk > and our hpc-trunk. We can forward to a list or autogenerate bug report > somehow. Can you open up an SSH port/user on your build box and install the Hudson agent on it so we can link it to hudson.openstack.org's build machines? -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
No disagreement with anything you say below, Matt. More testing of all kinds, including unit tests, should be a priority. Cheers, jay On Wed, Feb 16, 2011 at 11:37 PM, Matt Dietz wrote: > These are all great points Jay. > > I'd like to re-echo the comment about unit tests. Obviously they aren't > the panacea, but they can protect against some of the dumber errors that > have made their way into trunk. One particular bug stopped one developer > on my team dead in his tracks, and it ended up being a semi-colon in place > of a colon. There's a lot of utility in simply "exercising" code... > > I think we may want to consider everyone's favorite topic of code coverage > as well in all of this. Specifically, we may want to take note of code > coverage on any given merge, and if subsequent merges reduce that number, > we throw a fit/reject the patch. I know that won't be a popular solution, > but it would definitely put a stop to the lack of unit tests. > > On 2/16/11 4:27 PM, "Jay Pipes" wrote: > >>Hey all, >> >>It's come to my attention that a number of folks are not happy that >>Nova's trunk branch (lp:nova) is, shall we say, "less than stable". :) >> >>First, before going into some suggestions on keeping trunk more >>stable, I'd like to point out that trunk is, by nature, an actively >>developed source tree. Nobody should have an expectation that they can >>simply bzr branch lp:nova and everything will magically work with a) >>their existing installations of software packages, b) whatever code >>commits they have made locally, or c) whatever specific >>hypervisor/volume/network environment that they test their local code >>with. The trunk branch is, after all, in active development. >> >>That said, there's *no* reason we can't *improve* the relative >>stability of the trunk branch to make life less stressful for >>contributors. Here are a few suggestions on how to keep trunk a bit >>more stable for those developers who actively develop from trunk. >> >>1) Participate fully in code reviews. If you suspect a proposed branch >>merge will "mess everything up for you", then you should notify >>reviewers and developers about your concerns. Be proactive. >> >>2) If you pull trunk and something breaks, don't just complain about >>it. Log a bug immediately and talk to the reviewers/approvers of the >>patch that broke your environment. Be constructive in your criticism, >>and be clear about why the patch should have been more thoroughly or >>carefully reviewed. If you don't, we're bound to repeat mistakes. >> >>3) Help us to write functional and integration tests. It's become >>increasingly clear from the frequency of breakages in trunk (and other >>branches) that our unit tests are nowhere near sufficient to catch a >>large portion of bugs. This is to be expected. Our unit tests use >>mocks and stubs for virtually everything, and they only really test >>code interfaces, and they don't even test that very well. We're >>working on adding functional tests to Hudson that will run, as the >>unit test do, before any merge into trunk, with any failure resulting >>in a failed merge. However, we need your help to create functional >>tests and integration tests (tests that various *real* components work >>together properly). We also need help writing test cases that ensure >>software library dependencies and other packaging issues are handled >>properly and don't break with minor patches. >> >>4) If you have a specific environment/setup that you use (Rackers and >>Anso guys, here...), then we need your assistance to set up test >>clusters that will pull trunk into a wiped test environment and ensure >>that a series of realistic calls to the Nova API are properly handled. >>I know some of you are working on getting hardware ready. We need help >>from the software teams to ensure that these environments are >>initialized with the exact setups you use. >> >>The more testing we fire off against each potential merge into trunk, >>and the more those tests are hitting real-life deployment >>environments, the more stable trunk will become and the easier your >>life as a contributor will be. >> >>Thanks in advance for your assistance, and please don't hesitate to >>expand on any more suggestions you might have to stabilize trunk. >> >>-jay >> >>___ >>Mailing list: https://launchpad.net/~openstack >>Post to : openstack@lists.launchpad.net >>Unsubscribe : https://launchpad.net/~openstack >>More help : https://help.launchpad.net/ListHelp > > > > Confidentiality Notice: This e-mail message (including any attached or > embedded documents) is intended for the exclusive and confidential use of the > individual or entity to which this message is addressed, and unless otherwise > expressly indicated, is confidential and privileged information of Rackspace. > Any dissemination, distribution or copying of the enclosed material is > prohibited. > If you receive this transmission in error, ple
Re: [Openstack] Steps that can help stabilize Nova's trunk
On Wed, Feb 16, 2011 at 11:44 PM, Paul Voccio wrote: > Jay, > > Thanks for throwing this out. How would we build this with Hudson? What > would a "standard deploy" of Nova even look like for integration tests? I replied with some specifics to Trey, who had a similar question, and created a bug report (that I subsequently assigned to Trey): https://bugs.launchpad.net/nova/+bug/720941 Let me know if that answered your questions and if you'd like some more explanation. > We've also bounced the idea within our team of not allowing code commits > if the code to test ratio decreases but I'm not sure if that would work > for such a large project like this one. This is a good idea, but even if we were to add code to test ratios, the ratio would be mostly (only?) looking at unit tests. And I think we've seen that unit tests pass and functional tests blow up because of all the mocks/stubs in the unit tests that don't adequately represent a real-world deployment. Nothing wrong, of course, with exploring code to test ratio (basically, automated code coverage tests), but I think good functional and integration tests are more productive at this time. Just my two cents, though, nothing more, -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
On Thu, Feb 17, 2011 at 1:58 PM, Trey Morris wrote: > sounds like a good plan to me :) Awesome. I'm glad you're taking the lead on this ;) https://bugs.launchpad.net/nova/+bug/720941 Cheers! jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
2011/2/17 Jim Curry : > Soren, can you clarify what you mean by Ubuntu being the primary platform? > Why is that the reference? What limitations does this introduce? It's the primary platform because it's the platform we test everything on, it's the platform we spend time integrating with, etc. It started out as the reference because that's what both the Swift devs as well as the Anso guys had chosen it as their reference platform. It's an excellent choice, so there has been no motivation to change that decision as far as I know. I can think of a long list of consequences of this choice. Attempting to phrase any of them as limitations would be contrived. Having a reference platform means we can reasonably make a lot of very useful assumptions about the environment in which Nova runs. Having that reference platform be Ubuntu means that there's a straightforward way to modify this environment if we have special needs. I've patched a number of packages in Ubuntu already to accomodate our needs, including the Linux kernel, libvirt, Eventlet, user-mode-linux, etc. -- Soren Hansen Ubuntu Developer http://www.ubuntu.com/ OpenStack Developer http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
One other thing I forgot to point out is that there's a world outside of Python, too. You can pull all the stuff you want from pip, but there's still a whole bunch of things that aren't managed by pip that you need to rely on your distro for anyways. Like, say, a kernel, Python itself, libvirt, iscsi tools, etc., etc. -- Soren Hansen Ubuntu Developer http://www.ubuntu.com/ OpenStack Developer http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
sounds like a good plan to me :) On Thu, Feb 17, 2011 at 11:44 AM, Jay Pipes wrote: > On Thu, Feb 17, 2011 at 12:19 PM, Trey Morris > wrote: > > I don't like that it currently only runs on ubuntu + the ppa. If it > doesn't > > work with existing versions I think we're doing something wrong. Even > when > > natty comes out, I don't like the idea of having to ensure I have latest > > ubuntu for openstack to run. > > It runs on the platform that people have spent the time and effort to > integrate into Hudson. This happens to be Ubuntu because that is what > Soren (the person who has spent all that time and effort) is > comfortable with. > > I encourage you to set up an integration test for the specific > platform you want to ensure does not break with commits to trunk :) > > This could be as simple as doing the following: > > * Find Jordan Rinke or another Racker who has access to machines that > can be linked to Hudson > * SSH into the said machine and ensure that the machines have all your > environment's necessary components installed. In your case, Trey, I'll > presume that you want XenServer installed on the compute nodes and > MySQL installed on one of the other machines to act as the main > database > * Find soren, mtaylor, myself or others on IRC to help install the > Jenkins/Hudson agent on the machines. The Hudson agent will be > responsible for pulling lp:nova and installing all the necessary > pieces on the machines in your test environment > * Place a /etc/nova/nova.conf file on the machines in question that > matches your target environment > * Create a simple functional test script that runs through a basic set > of API requests that exercise the parts of the Nova API that are > critical to you (XenAPI, Glance integration, etc) > * Have Hudson fire said script against the test environment after > starting up Nova on the relevant nodes > > So, we're ready to help. But we need help from yourself and others on > your team, as well as good communication with folks like Jordan to > make sure we're all on the same page. Together, we can get this done. > > > As far as stability goes, i think integration testing is a great > solution. > > Hudson should run integration tests before it allows code into trunk. I > also > > think that code should be integration tested before it is reviewed since > > hardware is cheaper than core-devs. "I'll review that once I'm sure it > > works." The only issue I can see with the hudson running integration > tests > > is the way the testing environments are set up. There are so many options > an > > exhaustive list would just take forever. > > We have to start somewhere, and a good "somewhere" would be the > internal Rackspace Cloud Servers setup, since that's clearly a target > platform. ;) Add to that NASA's environment and anyone else's > environment who is willing to spend the time and effort to set up the > test environments. > > -jay > Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
On Thu, Feb 17, 2011 at 12:19 PM, Trey Morris wrote: > I don't like that it currently only runs on ubuntu + the ppa. If it doesn't > work with existing versions I think we're doing something wrong. Even when > natty comes out, I don't like the idea of having to ensure I have latest > ubuntu for openstack to run. It runs on the platform that people have spent the time and effort to integrate into Hudson. This happens to be Ubuntu because that is what Soren (the person who has spent all that time and effort) is comfortable with. I encourage you to set up an integration test for the specific platform you want to ensure does not break with commits to trunk :) This could be as simple as doing the following: * Find Jordan Rinke or another Racker who has access to machines that can be linked to Hudson * SSH into the said machine and ensure that the machines have all your environment's necessary components installed. In your case, Trey, I'll presume that you want XenServer installed on the compute nodes and MySQL installed on one of the other machines to act as the main database * Find soren, mtaylor, myself or others on IRC to help install the Jenkins/Hudson agent on the machines. The Hudson agent will be responsible for pulling lp:nova and installing all the necessary pieces on the machines in your test environment * Place a /etc/nova/nova.conf file on the machines in question that matches your target environment * Create a simple functional test script that runs through a basic set of API requests that exercise the parts of the Nova API that are critical to you (XenAPI, Glance integration, etc) * Have Hudson fire said script against the test environment after starting up Nova on the relevant nodes So, we're ready to help. But we need help from yourself and others on your team, as well as good communication with folks like Jordan to make sure we're all on the same page. Together, we can get this done. > As far as stability goes, i think integration testing is a great solution. > Hudson should run integration tests before it allows code into trunk. I also > think that code should be integration tested before it is reviewed since > hardware is cheaper than core-devs. "I'll review that once I'm sure it > works." The only issue I can see with the hudson running integration tests > is the way the testing environments are set up. There are so many options an > exhaustive list would just take forever. We have to start somewhere, and a good "somewhere" would be the internal Rackspace Cloud Servers setup, since that's clearly a target platform. ;) Add to that NASA's environment and anyone else's environment who is willing to spend the time and effort to set up the test environments. -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
I don't like that it currently only runs on ubuntu + the ppa. If it doesn't work with existing versions I think we're doing something wrong. Even when natty comes out, I don't like the idea of having to ensure I have latest ubuntu for openstack to run. As far as stability goes, i think integration testing is a great solution. Hudson should run integration tests before it allows code into trunk. I also think that code should be integration tested before it is reviewed since hardware is cheaper than core-devs. "I'll review that once I'm sure it works." The only issue I can see with the hudson running integration tests is the way the testing environments are set up. There are so many options an exhaustive list would just take forever. OP's 4th bullet point makes a lot of sense here for catching bugs but I'm not sure how we could use these environments to prevent bad code from landing in trunk. Possibly we end up with an array of test clusters owned by different entities and hudson kicks off the tests in parallel. This would be difficult to manage and we'd probably end up with false negatives due to environments not being set up correctly. On Thu, Feb 17, 2011 at 8:43 AM, Dan Prince wrote: > Hey Jay, > > I like what you propose here. I have a couple of comments and questions. > > I see the 'smoketests' directory in the nova code. Is anyone running these > tests on a regular basis (every commit)? Is this the best place to further > build out integration tests? > > --- > > Regarding environment/setup tools: I been working on an Openstack VPC > toolkit project that we are using in Blacksburg to stage test some things in > the cloud. I'm using Chef along with the Anso/Opscode Openstack cookbooks to > setup Rackspace Cloud Servers with the latest trunk PPA packages. > > This setup works well however I can only test with Qemu (no Xenserver) and > using network managers that have DHCP (I use FlatDHCPManager since Cloud > Servers kernels don't currently have the 'ndb' kernel module which would > support the network injection stuff). Using this setup I'm able to create > multi node installations where instances on different machines can ping each > other. While this isn't what I would call a true production setup it is > fully functional and can easily be run in parallel. The only limitations are > the limits on your Cloud Servers Account. > > If you have bare metal then you can simply swap out the Cloud Servers API > layer with something that interfaces with your PXE imaging system. I'm a big > fan of slicking the machines between each test run to avoid the buildup of > cruft in the system. > > This would integrate as a Hudson job nicely as well. We've done some > similar setups in Rackspace Email and Apps using a single Hudson server. The > hudson server runs a simple Bash script that invokes the toolkit to create > the cloud servers, chef them up with the latest PPA packages (or your branch > code), and then uses Torque (an HPC'ish resource manager) to run schedule > test jobs on some of the machines. I use Chef recipes to install Torque > along with a REST interface to schedule and monitor the jobs on a "head" > node. The Hudson job the waits for the Torque jobs to finish. The last thing > the Hudson script does is 'scp' the unit test results XML file back to the > Hudson server where you can use something like the xUnit plugin to display > and graph the results over time. > > To summarize: > > -testing in the cloud provides a low barrier of entry that anyone can use > -testing on bare metal is more expensive but gets us extra coverage > (XenServer, etc.) > -we should do both as often as possible > -the same set of tools and tests should work in both environments > > Dan > > -Original Message- > From: "Jay Pipes" > Sent: Wednesday, February 16, 2011 5:27pm > To: openstack@lists.launchpad.net > Subject: [Openstack] Steps that can help stabilize Nova's trunk > > Hey all, > > It's come to my attention that a number of folks are not happy that > Nova's trunk branch (lp:nova) is, shall we say, "less than stable". :) > > First, before going into some suggestions on keeping trunk more > stable, I'd like to point out that trunk is, by nature, an actively > developed source tree. Nobody should have an expectation that they can > simply bzr branch lp:nova and everything will magically work with a) > their existing installations of software packages, b) whatever code > commits they have made locally, or c) whatever specific > hypervisor/volume/network environment that they test their local code > with. The trunk branch is, after all, in act
Re: [Openstack] Steps that can help stabilize Nova's trunk
Soren, can you clarify what you mean by Ubuntu being the primary platform? Why is that the reference? What limitations does this introduce? Jim On Feb 17, 2011, at 4:31 AM, Soren Hansen wrote: > 2011/2/17 Brian Schott : >> One thing we saw in the list and experienced first hand is that your Hudson >> server uses a pre-configured environment and ours pulls virtual env every >> time. We had failures on trunk that yours missed because of new pip pulled >> versions. > >> A fresh run_tests.sh -f needs to work. It is the only guaranteed sanity >> test everybody can replicate. > > That's not technically accurate. Every part of the environment in > which we run the tests can be rebuilt in a reproducible manner. > Everything on that box is packaged and available in a PPA from which > anyone can build an identical test system. > >> It might pull upstream bugs, but better to be ahead of that curve than >> behind. > > I agree that the status quo is not good enough. However, I don't agree > that we should pull stuff from pip directly. Ubuntu is our primary > target platform, that's the reference, that's where we absolutely > *must* function. I'd be *delighted* if we tested our stuff more > broadly, and I welcome efforts to do so, but whether this works on > Ubuntu or not is the deal breaker. > > That said, our tests right now are run on Maverick with a set of > backported packages (all available in a PPA, though), but really ought > to run on Natty. We should look into that ASAP. > > -- > Soren Hansen > Ubuntu Developerhttp://www.ubuntu.com/ > OpenStack Developer http://www.openstack.org/ > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Hey Jay, I like what you propose here. I have a couple of comments and questions. I see the 'smoketests' directory in the nova code. Is anyone running these tests on a regular basis (every commit)? Is this the best place to further build out integration tests? --- Regarding environment/setup tools: I been working on an Openstack VPC toolkit project that we are using in Blacksburg to stage test some things in the cloud. I'm using Chef along with the Anso/Opscode Openstack cookbooks to setup Rackspace Cloud Servers with the latest trunk PPA packages. This setup works well however I can only test with Qemu (no Xenserver) and using network managers that have DHCP (I use FlatDHCPManager since Cloud Servers kernels don't currently have the 'ndb' kernel module which would support the network injection stuff). Using this setup I'm able to create multi node installations where instances on different machines can ping each other. While this isn't what I would call a true production setup it is fully functional and can easily be run in parallel. The only limitations are the limits on your Cloud Servers Account. If you have bare metal then you can simply swap out the Cloud Servers API layer with something that interfaces with your PXE imaging system. I'm a big fan of slicking the machines between each test run to avoid the buildup of cruft in the system. This would integrate as a Hudson job nicely as well. We've done some similar setups in Rackspace Email and Apps using a single Hudson server. The hudson server runs a simple Bash script that invokes the toolkit to create the cloud servers, chef them up with the latest PPA packages (or your branch code), and then uses Torque (an HPC'ish resource manager) to run schedule test jobs on some of the machines. I use Chef recipes to install Torque along with a REST interface to schedule and monitor the jobs on a "head" node. The Hudson job the waits for the Torque jobs to finish. The last thing the Hudson script does is 'scp' the unit test results XML file back to the Hudson server where you can use something like the xUnit plugin to display and graph the results over time. To summarize: -testing in the cloud provides a low barrier of entry that anyone can use -testing on bare metal is more expensive but gets us extra coverage (XenServer, etc.) -we should do both as often as possible -the same set of tools and tests should work in both environments Dan -Original Message- From: "Jay Pipes" Sent: Wednesday, February 16, 2011 5:27pm To: openstack@lists.launchpad.net Subject: [Openstack] Steps that can help stabilize Nova's trunk Hey all, It's come to my attention that a number of folks are not happy that Nova's trunk branch (lp:nova) is, shall we say, "less than stable". :) First, before going into some suggestions on keeping trunk more stable, I'd like to point out that trunk is, by nature, an actively developed source tree. Nobody should have an expectation that they can simply bzr branch lp:nova and everything will magically work with a) their existing installations of software packages, b) whatever code commits they have made locally, or c) whatever specific hypervisor/volume/network environment that they test their local code with. The trunk branch is, after all, in active development. That said, there's *no* reason we can't *improve* the relative stability of the trunk branch to make life less stressful for contributors. Here are a few suggestions on how to keep trunk a bit more stable for those developers who actively develop from trunk. 1) Participate fully in code reviews. If you suspect a proposed branch merge will "mess everything up for you", then you should notify reviewers and developers about your concerns. Be proactive. 2) If you pull trunk and something breaks, don't just complain about it. Log a bug immediately and talk to the reviewers/approvers of the patch that broke your environment. Be constructive in your criticism, and be clear about why the patch should have been more thoroughly or carefully reviewed. If you don't, we're bound to repeat mistakes. 3) Help us to write functional and integration tests. It's become increasingly clear from the frequency of breakages in trunk (and other branches) that our unit tests are nowhere near sufficient to catch a large portion of bugs. This is to be expected. Our unit tests use mocks and stubs for virtually everything, and they only really test code interfaces, and they don't even test that very well. We're working on adding functional tests to Hudson that will run, as the unit test do, before any merge into trunk, with any failure resulting in a failed merge. However, we need your help to create functional tests and integration tests (tests that various *real* components work together properly). We a
Re: [Openstack] Steps that can help stabilize Nova's trunk
2011/2/17 Brian Schott : > One thing we saw in the list and experienced first hand is that your Hudson > server uses a pre-configured environment and ours pulls virtual env every > time. We had failures on trunk that yours missed because of new pip pulled > versions. > A fresh run_tests.sh -f needs to work. It is the only guaranteed sanity test > everybody can replicate. That's not technically accurate. Every part of the environment in which we run the tests can be rebuilt in a reproducible manner. Everything on that box is packaged and available in a PPA from which anyone can build an identical test system. > It might pull upstream bugs, but better to be ahead of that curve than behind. I agree that the status quo is not good enough. However, I don't agree that we should pull stuff from pip directly. Ubuntu is our primary target platform, that's the reference, that's where we absolutely *must* function. I'd be *delighted* if we tested our stuff more broadly, and I welcome efforts to do so, but whether this works on Ubuntu or not is the deal breaker. That said, our tests right now are run on Maverick with a set of backported packages (all available in a PPA, though), but really ought to run on Natty. We should look into that ASAP. -- Soren Hansen Ubuntu Developer http://www.ubuntu.com/ OpenStack Developer http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
One thing we saw in the list and experienced first hand is that your Hudson server uses a pre-configured environment and ours pulls virtual env every time. We had failures on trunk that yours missed because of new pip pulled versions. A fresh run_tests.sh -f needs to work. It is the only guaranteed sanity test everybody can replicate. It might pull upstream bugs, but better to be ahead of that curve than behind. We are now triggering on trunk commit emails and running Hudson on both trunk and our hpc-trunk. We can forward to a list or autogenerate bug report somehow. Suggestions: 1. Let other teams setup their own trunk Hudson servers with preferred configs (os, virt, db, net) and doc how to forward results up (new list?). This way you quickly find commits that break mysql or Xen before you forgot what you did. 2. Jay already listed my other suggestions... Sent from my iPhone On Feb 16, 2011, at 11:44 PM, Paul Voccio wrote: > Jay, > > Thanks for throwing this out. How would we build this with Hudson? What > would a "standard deploy" of Nova even look like for integration tests? > We've also bounced the idea within our team of not allowing code commits > if the code to test ratio decreases but I'm not sure if that would work > for such a large project like this one. > > pvo > > On 2/16/11 4:27 PM, "Jay Pipes" wrote: > >> Hey all, >> >> It's come to my attention that a number of folks are not happy that >> Nova's trunk branch (lp:nova) is, shall we say, "less than stable". :) >> >> First, before going into some suggestions on keeping trunk more >> stable, I'd like to point out that trunk is, by nature, an actively >> developed source tree. Nobody should have an expectation that they can >> simply bzr branch lp:nova and everything will magically work with a) >> their existing installations of software packages, b) whatever code >> commits they have made locally, or c) whatever specific >> hypervisor/volume/network environment that they test their local code >> with. The trunk branch is, after all, in active development. >> >> That said, there's *no* reason we can't *improve* the relative >> stability of the trunk branch to make life less stressful for >> contributors. Here are a few suggestions on how to keep trunk a bit >> more stable for those developers who actively develop from trunk. >> >> 1) Participate fully in code reviews. If you suspect a proposed branch >> merge will "mess everything up for you", then you should notify >> reviewers and developers about your concerns. Be proactive. >> >> 2) If you pull trunk and something breaks, don't just complain about >> it. Log a bug immediately and talk to the reviewers/approvers of the >> patch that broke your environment. Be constructive in your criticism, >> and be clear about why the patch should have been more thoroughly or >> carefully reviewed. If you don't, we're bound to repeat mistakes. >> >> 3) Help us to write functional and integration tests. It's become >> increasingly clear from the frequency of breakages in trunk (and other >> branches) that our unit tests are nowhere near sufficient to catch a >> large portion of bugs. This is to be expected. Our unit tests use >> mocks and stubs for virtually everything, and they only really test >> code interfaces, and they don't even test that very well. We're >> working on adding functional tests to Hudson that will run, as the >> unit test do, before any merge into trunk, with any failure resulting >> in a failed merge. However, we need your help to create functional >> tests and integration tests (tests that various *real* components work >> together properly). We also need help writing test cases that ensure >> software library dependencies and other packaging issues are handled >> properly and don't break with minor patches. >> >> 4) If you have a specific environment/setup that you use (Rackers and >> Anso guys, here...), then we need your assistance to set up test >> clusters that will pull trunk into a wiped test environment and ensure >> that a series of realistic calls to the Nova API are properly handled. >> I know some of you are working on getting hardware ready. We need help >> from the software teams to ensure that these environments are >> initialized with the exact setups you use. > > Still working on this. We're hoping to have something built out in the > next couple of weeks. Man, someone should write some hardware emulation > stuff so I don't have to wait on real gear. ;) > >> >> The more testing we fire off against each potential merge into trunk, >> and the more those tests are hitting real-life deployment >> environments, the more stable trunk will become and the easier your >> life as a contributor will be. >> >> Thanks in advance for your assistance, and please don't hesitate to >> expand on any more suggestions you might have to stabilize trunk. >> >> -jay >> >> ___ >> Mailing list: https://launchp
Re: [Openstack] Steps that can help stabilize Nova's trunk
Jay, Thanks for throwing this out. How would we build this with Hudson? What would a "standard deploy" of Nova even look like for integration tests? We've also bounced the idea within our team of not allowing code commits if the code to test ratio decreases but I'm not sure if that would work for such a large project like this one. pvo On 2/16/11 4:27 PM, "Jay Pipes" wrote: >Hey all, > >It's come to my attention that a number of folks are not happy that >Nova's trunk branch (lp:nova) is, shall we say, "less than stable". :) > >First, before going into some suggestions on keeping trunk more >stable, I'd like to point out that trunk is, by nature, an actively >developed source tree. Nobody should have an expectation that they can >simply bzr branch lp:nova and everything will magically work with a) >their existing installations of software packages, b) whatever code >commits they have made locally, or c) whatever specific >hypervisor/volume/network environment that they test their local code >with. The trunk branch is, after all, in active development. > >That said, there's *no* reason we can't *improve* the relative >stability of the trunk branch to make life less stressful for >contributors. Here are a few suggestions on how to keep trunk a bit >more stable for those developers who actively develop from trunk. > >1) Participate fully in code reviews. If you suspect a proposed branch >merge will "mess everything up for you", then you should notify >reviewers and developers about your concerns. Be proactive. > >2) If you pull trunk and something breaks, don't just complain about >it. Log a bug immediately and talk to the reviewers/approvers of the >patch that broke your environment. Be constructive in your criticism, >and be clear about why the patch should have been more thoroughly or >carefully reviewed. If you don't, we're bound to repeat mistakes. > >3) Help us to write functional and integration tests. It's become >increasingly clear from the frequency of breakages in trunk (and other >branches) that our unit tests are nowhere near sufficient to catch a >large portion of bugs. This is to be expected. Our unit tests use >mocks and stubs for virtually everything, and they only really test >code interfaces, and they don't even test that very well. We're >working on adding functional tests to Hudson that will run, as the >unit test do, before any merge into trunk, with any failure resulting >in a failed merge. However, we need your help to create functional >tests and integration tests (tests that various *real* components work >together properly). We also need help writing test cases that ensure >software library dependencies and other packaging issues are handled >properly and don't break with minor patches. > >4) If you have a specific environment/setup that you use (Rackers and >Anso guys, here...), then we need your assistance to set up test >clusters that will pull trunk into a wiped test environment and ensure >that a series of realistic calls to the Nova API are properly handled. >I know some of you are working on getting hardware ready. We need help >from the software teams to ensure that these environments are >initialized with the exact setups you use. Still working on this. We're hoping to have something built out in the next couple of weeks. Man, someone should write some hardware emulation stuff so I don't have to wait on real gear. ;) > >The more testing we fire off against each potential merge into trunk, >and the more those tests are hitting real-life deployment >environments, the more stable trunk will become and the easier your >life as a contributor will be. > >Thanks in advance for your assistance, and please don't hesitate to >expand on any more suggestions you might have to stabilize trunk. > >-jay > >___ >Mailing list: https://launchpad.net/~openstack >Post to : openstack@lists.launchpad.net >Unsubscribe : https://launchpad.net/~openstack >More help : https://help.launchpad.net/ListHelp Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Steps that can help stabilize Nova's trunk
These are all great points Jay. I'd like to re-echo the comment about unit tests. Obviously they aren't the panacea, but they can protect against some of the dumber errors that have made their way into trunk. One particular bug stopped one developer on my team dead in his tracks, and it ended up being a semi-colon in place of a colon. There's a lot of utility in simply "exercising" code... I think we may want to consider everyone's favorite topic of code coverage as well in all of this. Specifically, we may want to take note of code coverage on any given merge, and if subsequent merges reduce that number, we throw a fit/reject the patch. I know that won't be a popular solution, but it would definitely put a stop to the lack of unit tests. On 2/16/11 4:27 PM, "Jay Pipes" wrote: >Hey all, > >It's come to my attention that a number of folks are not happy that >Nova's trunk branch (lp:nova) is, shall we say, "less than stable". :) > >First, before going into some suggestions on keeping trunk more >stable, I'd like to point out that trunk is, by nature, an actively >developed source tree. Nobody should have an expectation that they can >simply bzr branch lp:nova and everything will magically work with a) >their existing installations of software packages, b) whatever code >commits they have made locally, or c) whatever specific >hypervisor/volume/network environment that they test their local code >with. The trunk branch is, after all, in active development. > >That said, there's *no* reason we can't *improve* the relative >stability of the trunk branch to make life less stressful for >contributors. Here are a few suggestions on how to keep trunk a bit >more stable for those developers who actively develop from trunk. > >1) Participate fully in code reviews. If you suspect a proposed branch >merge will "mess everything up for you", then you should notify >reviewers and developers about your concerns. Be proactive. > >2) If you pull trunk and something breaks, don't just complain about >it. Log a bug immediately and talk to the reviewers/approvers of the >patch that broke your environment. Be constructive in your criticism, >and be clear about why the patch should have been more thoroughly or >carefully reviewed. If you don't, we're bound to repeat mistakes. > >3) Help us to write functional and integration tests. It's become >increasingly clear from the frequency of breakages in trunk (and other >branches) that our unit tests are nowhere near sufficient to catch a >large portion of bugs. This is to be expected. Our unit tests use >mocks and stubs for virtually everything, and they only really test >code interfaces, and they don't even test that very well. We're >working on adding functional tests to Hudson that will run, as the >unit test do, before any merge into trunk, with any failure resulting >in a failed merge. However, we need your help to create functional >tests and integration tests (tests that various *real* components work >together properly). We also need help writing test cases that ensure >software library dependencies and other packaging issues are handled >properly and don't break with minor patches. > >4) If you have a specific environment/setup that you use (Rackers and >Anso guys, here...), then we need your assistance to set up test >clusters that will pull trunk into a wiped test environment and ensure >that a series of realistic calls to the Nova API are properly handled. >I know some of you are working on getting hardware ready. We need help >from the software teams to ensure that these environments are >initialized with the exact setups you use. > >The more testing we fire off against each potential merge into trunk, >and the more those tests are hitting real-life deployment >environments, the more stable trunk will become and the easier your >life as a contributor will be. > >Thanks in advance for your assistance, and please don't hesitate to >expand on any more suggestions you might have to stabilize trunk. > >-jay > >___ >Mailing list: https://launchpad.net/~openstack >Post to : openstack@lists.launchpad.net >Unsubscribe : https://launchpad.net/~openstack >More help : https://help.launchpad.net/ListHelp Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpa
[Openstack] Steps that can help stabilize Nova's trunk
Hey all, It's come to my attention that a number of folks are not happy that Nova's trunk branch (lp:nova) is, shall we say, "less than stable". :) First, before going into some suggestions on keeping trunk more stable, I'd like to point out that trunk is, by nature, an actively developed source tree. Nobody should have an expectation that they can simply bzr branch lp:nova and everything will magically work with a) their existing installations of software packages, b) whatever code commits they have made locally, or c) whatever specific hypervisor/volume/network environment that they test their local code with. The trunk branch is, after all, in active development. That said, there's *no* reason we can't *improve* the relative stability of the trunk branch to make life less stressful for contributors. Here are a few suggestions on how to keep trunk a bit more stable for those developers who actively develop from trunk. 1) Participate fully in code reviews. If you suspect a proposed branch merge will "mess everything up for you", then you should notify reviewers and developers about your concerns. Be proactive. 2) If you pull trunk and something breaks, don't just complain about it. Log a bug immediately and talk to the reviewers/approvers of the patch that broke your environment. Be constructive in your criticism, and be clear about why the patch should have been more thoroughly or carefully reviewed. If you don't, we're bound to repeat mistakes. 3) Help us to write functional and integration tests. It's become increasingly clear from the frequency of breakages in trunk (and other branches) that our unit tests are nowhere near sufficient to catch a large portion of bugs. This is to be expected. Our unit tests use mocks and stubs for virtually everything, and they only really test code interfaces, and they don't even test that very well. We're working on adding functional tests to Hudson that will run, as the unit test do, before any merge into trunk, with any failure resulting in a failed merge. However, we need your help to create functional tests and integration tests (tests that various *real* components work together properly). We also need help writing test cases that ensure software library dependencies and other packaging issues are handled properly and don't break with minor patches. 4) If you have a specific environment/setup that you use (Rackers and Anso guys, here...), then we need your assistance to set up test clusters that will pull trunk into a wiped test environment and ensure that a series of realistic calls to the Nova API are properly handled. I know some of you are working on getting hardware ready. We need help from the software teams to ensure that these environments are initialized with the exact setups you use. The more testing we fire off against each potential merge into trunk, and the more those tests are hitting real-life deployment environments, the more stable trunk will become and the easier your life as a contributor will be. Thanks in advance for your assistance, and please don't hesitate to expand on any more suggestions you might have to stabilize trunk. -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
