Re: [Openstack] [ubuntu-cloud] Update on Ubuntu automated testing and CI of Openstack
Adam, you've outdone yourself :-) That was a phenomenal reply with tons of good information. You've provided our team with all the info we need now -- thanks! d On Mon, Feb 27, 2012 at 10:36 PM, Adam Gandelman ad...@canonical.com wrote: On 02/27/2012 09:22 AM, Duncan McGreggor wrote: It's been an invaluable source for not only information, but also planning for the cloud work here at DreamHost. This is great to hear! I'm happy to hear other people are benefiting from this as much as Ubuntu To the point of this email, though, I have a question for you that I wasn't able to parse an explicit answer to from your post: For the packages that are built in the PPA linked above, are they only built after all the components of OpenStack have been confirmed working as a whole? Or are they built just after individual testing? The jenkins trunk build jobs are the ones responsible for uploading to the PPA. It basically stuffs the newly built package in the local repository and uploads it to the PPA (if it fails to build, it doesnt upload). The nova trunk build also triggers the deployment + testing. So, to answer, those PPA packages are *pre* deployment + smoke testing. The idea with that is that we can use the PPA externally to debug specific issues that might turn up during testing without blocking the deployment/smoke testing queue. We're limited by the number of machines we have and can not currently run deployment testing in parallel. My question comes from this concern: if we're building out a product based on this PPA, (before Precise is delivered) we want to make sure that when we bring up new systems by installing the packages from the PPA, all of those work together properly. If the latest code from keystone, for example, hasn't been building due to testing errors, we want to make sure that the presence of the older keystone package in the PPA won't be causing issues with the newer builds of the rest of OpenStack. So you might have noticed the version of Keystone we are testing is getting a bit dusty. We decided to freeze the version of Keystone we're testing just before the KSL/redux branch was merged. There are still some features that need to land before we can modify our Juju charms to deploy the new version (specifically SQL persistence for the service catalog). I'm hoping these will land before e4 (tomorrow) so we can begin testing the new branch this week. To clarify: in your blog post, you explicitly mention the validation process per component, starting with the upstream git repos. In the deploy phase, you verify that the system as a whole (all of OpenStack) works as expected. But what happens when one or more of those components don't work? Are packages rolled back in the PPA until the PPA only provides packages that will result in, once installed, a complete working system? So far, we haven't rolled anything back in the PPA or the local repository. I believe Keystone is the only package we've had to freeze while things stabilize upstream. Fortunately, we've found the CI (even with the limited test coverage we currently run) is picking up new bugs almost instantaneously. In most cases we're able to either get a fix into gerrit same day or find a proposed fix that we can cherry pick into our debian/patches/ repository and carry temporarily in our lp:~openstack-ubuntu-testing packages branches until its been fixed proper upstream. Please understand the PPA is for testing purposes only, and we're not making any promise that what installs from there is stable/usable/working. We're using the work around that PPA and this CI to essentially gate what goes into the Ubuntu archive. This cycle, Chuck has been uploading a weekly snapshot of Openstack (usually every Friday). With the CI in place, we can essentially verify that those packages build clean and install okay, and provide something usable given the last weeks Gerrit churn. Keep in mind this is still a developing process. I expect this will evolve over the next few months. We've just begun trigger pre-commit testing to the Diablo stable branch against Oneiric. Still working out some kinks, but we'll hopefully be going into Folsom+Precise with coverage of both trunk and stable updates to Essex/Precise LTS. Also note that, at this point in the Ubuntu development cycle, it can often take *a long time* before an upload to a PPA is built and ready for use. Its probably safe to assume that what you're using from the PPA is what we were testing yesterday (check the version of the package for a timestamp) Keep up the great work, guys -- you have fans out here in the wild, wild world of OpenStack :-) :) Cheers, Adam -- Ubuntu-cloud mailing list ubuntu-cl...@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-cloud ___ Mailing list:
Re: [Openstack] [ubuntu-cloud] Update on Ubuntu automated testing and CI of Openstack
On Wed, Feb 8, 2012 at 6:57 PM, Adam Gandelman ad...@canonical.com wrote: As promised for anyone who was interested when we announced to the last last week, here is a blog post James Page and I put together describing our Openstack testing efforts and infrastructure in greater detail: http://javacruft.wordpress.com/2012/02/08/automating-openstack-testing-on-ubuntu/ Adam, thanks for this great write up :-) Part of my morning ritual involves hitting these pages every day when I get up: * https://jenkins.qa.ubuntu.com/view/Precise%20OpenStack%20Testing/ * https://launchpad.net/~openstack-ubuntu-testing/+archive/openstack-trunk-testing It's been an invaluable source for not only information, but also planning for the cloud work here at DreamHost. To the point of this email, though, I have a question for you that I wasn't able to parse an explicit answer to from your post: For the packages that are built in the PPA linked above, are they only built after all the components of OpenStack have been confirmed working as a whole? Or are they built just after individual testing? My question comes from this concern: if we're building out a product based on this PPA, (before Precise is delivered) we want to make sure that when we bring up new systems by installing the packages from the PPA, all of those work together properly. If the latest code from keystone, for example, hasn't been building due to testing errors, we want to make sure that the presence of the older keystone package in the PPA won't be causing issues with the newer builds of the rest of OpenStack. To clarify: in your blog post, you explicitly mention the validation process per component, starting with the upstream git repos. In the deploy phase, you verify that the system as a whole (all of OpenStack) works as expected. But what happens when one or more of those components don't work? Are packages rolled back in the PPA until the PPA only provides packages that will result in, once installed, a complete working system? So there's that practical side of it, but to be honest, it's also simply an interesting question :-) I find the logistics of automated testing a great source of interest and fascination... Keep up the great work, guys -- you have fans out here in the wild, wild world of OpenStack :-) d ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [ubuntu-cloud] Update on Ubuntu automated testing and CI of Openstack
On 02/27/2012 09:22 AM, Duncan McGreggor wrote: It's been an invaluable source for not only information, but also planning for the cloud work here at DreamHost. This is great to hear! I'm happy to hear other people are benefiting from this as much as Ubuntu To the point of this email, though, I have a question for you that I wasn't able to parse an explicit answer to from your post: For the packages that are built in the PPA linked above, are they only built after all the components of OpenStack have been confirmed working as a whole? Or are they built just after individual testing? The jenkins trunk build jobs are the ones responsible for uploading to the PPA. It basically stuffs the newly built package in the local repository and uploads it to the PPA (if it fails to build, it doesnt upload). The nova trunk build also triggers the deployment + testing. So, to answer, those PPA packages are *pre* deployment + smoke testing. The idea with that is that we can use the PPA externally to debug specific issues that might turn up during testing without blocking the deployment/smoke testing queue. We're limited by the number of machines we have and can not currently run deployment testing in parallel. My question comes from this concern: if we're building out a product based on this PPA, (before Precise is delivered) we want to make sure that when we bring up new systems by installing the packages from the PPA, all of those work together properly. If the latest code from keystone, for example, hasn't been building due to testing errors, we want to make sure that the presence of the older keystone package in the PPA won't be causing issues with the newer builds of the rest of OpenStack. So you might have noticed the version of Keystone we are testing is getting a bit dusty. We decided to freeze the version of Keystone we're testing just before the KSL/redux branch was merged. There are still some features that need to land before we can modify our Juju charms to deploy the new version (specifically SQL persistence for the service catalog). I'm hoping these will land before e4 (tomorrow) so we can begin testing the new branch this week. To clarify: in your blog post, you explicitly mention the validation process per component, starting with the upstream git repos. In the deploy phase, you verify that the system as a whole (all of OpenStack) works as expected. But what happens when one or more of those components don't work? Are packages rolled back in the PPA until the PPA only provides packages that will result in, once installed, a complete working system? So far, we haven't rolled anything back in the PPA or the local repository. I believe Keystone is the only package we've had to freeze while things stabilize upstream. Fortunately, we've found the CI (even with the limited test coverage we currently run) is picking up new bugs almost instantaneously. In most cases we're able to either get a fix into gerrit same day or find a proposed fix that we can cherry pick into our debian/patches/ repository and carry temporarily in our lp:~openstack-ubuntu-testing packages branches until its been fixed proper upstream. Please understand the PPA is for testing purposes only, and we're not making any promise that what installs from there is stable/usable/working. We're using the work around that PPA and this CI to essentially gate what goes into the Ubuntu archive. This cycle, Chuck has been uploading a weekly snapshot of Openstack (usually every Friday). With the CI in place, we can essentially verify that those packages build clean and install okay, and provide something usable given the last weeks Gerrit churn. Keep in mind this is still a developing process. I expect this will evolve over the next few months. We've just begun trigger pre-commit testing to the Diablo stable branch against Oneiric. Still working out some kinks, but we'll hopefully be going into Folsom+Precise with coverage of both trunk and stable updates to Essex/Precise LTS. Also note that, at this point in the Ubuntu development cycle, it can often take *a long time* before an upload to a PPA is built and ready for use. Its probably safe to assume that what you're using from the PPA is what we were testing yesterday (check the version of the package for a timestamp) Keep up the great work, guys -- you have fans out here in the wild, wild world of OpenStack :-) :) Cheers, Adam ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [ubuntu-cloud] Update on Ubuntu automated testing and CI of Openstack
On Thu, 09 Feb 2012 07:57:31 -0800, Robbie Williamson rob...@ubuntu.com wrote: On 02/09/2012 05:21 AM, Ghe Rivero wrote: Wow! Openstack is really getting to much love from Ubuntu, nice! I wish we have so many resources to the same with Debian. Keep the good work guys! Ghe Rivero I could be 100% wrong here, but I think Debian has agreed (or are in talks) to use Ubuntu as the upstream for OpenStack, thus allowing our testing to trickle down. Can you please cite a reference for this? It would be a significant change to Debian policy and packaging. Afaik, the package(s) would still need to be mentored and sponsored - in this case who does this, Debian or Ubuntu or? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [ubuntu-cloud] Update on Ubuntu automated testing and CI of Openstack
On 08/02/12 23:57, Adam Gandelman wrote: As promised for anyone who was interested when we announced to the last last week, here is a blog post James Page and I put together describing our Openstack testing efforts and infrastructure in greater detail: http://javacruft.wordpress.com/2012/02/08/automating-openstack-testing-on-ubuntu/ Awesome work guys! ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [ubuntu-cloud] Update on Ubuntu automated testing and CI of Openstack
Wow! Openstack is really getting to much love from Ubuntu, nice! I wish we have so many resources to the same with Debian. Keep the good work guys! Ghe Rivero On Thu, Feb 9, 2012 at 12:09 PM, Mark Shuttleworth m...@ubuntu.com wrote: On 08/02/12 23:57, Adam Gandelman wrote: As promised for anyone who was interested when we announced to the last last week, here is a blog post James Page and I put together describing our Openstack testing efforts and infrastructure in greater detail: http://javacruft.wordpress.**com/2012/02/08/automating-** openstack-testing-on-ubuntu/http://javacruft.wordpress.com/2012/02/08/automating-openstack-testing-on-ubuntu/ Awesome work guys! __**_ Mailing list: https://launchpad.net/~**openstackhttps://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~**openstackhttps://launchpad.net/~openstack More help : https://help.launchpad.net/**ListHelphttps://help.launchpad.net/ListHelp -- Ghe Rivero *OpenStack Distribution Engineer **www.stackops.com | * ghe.riv...@stackops.com diego.parri...@stackops.com ** | +34 625 63 45 23 | skype:ghe.rivero* * http://www.stackops.com/ * * ADVERTENCIA LEGAL Le informamos, como destinatario de este mensaje, que el correo electrónico y las comunicaciones por medio de Internet no permiten asegurar ni garantizar la confidencialidad de los mensajes transmitidos, así como tampoco su integridad o su correcta recepción, por lo que STACKOPS TECHNOLOGIES S.L. no asume responsabilidad alguna por tales circunstancias. Si no consintiese en la utilización del correo electrónico o de las comunicaciones vía Internet le rogamos nos lo comunique y ponga en nuestro conocimiento de manera inmediata. Este mensaje va dirigido, de manera exclusiva, a su destinatario y contiene información confidencial y sujeta al secreto profesional, cuya divulgación no está permitida por la ley. En caso de haber recibido este mensaje por error, le rogamos que, de forma inmediata, nos lo comunique mediante correo electrónico remitido a nuestra atención y proceda a su eliminación, así como a la de cualquier documento adjunto al mismo. Asimismo, le comunicamos que la distribución, copia o utilización de este mensaje, o de cualquier documento adjunto al mismo, cualquiera que fuera su finalidad, están prohibidas por la ley. * PRIVILEGED AND CONFIDENTIAL We hereby inform you, as addressee of this message, that e-mail and Internet do not guarantee the confidentiality, nor the completeness or proper reception of the messages sent and, thus, STACKOPS TECHNOLOGIES S.L. does not assume any liability for those circumstances. Should you not agree to the use of e-mail or to communications via Internet, you are kindly requested to notify us immediately. This message is intended exclusively for the person to whom it is addressed and contains privileged and confidential information protected from disclosure by law. If you are not the addressee indicated in this message, you should immediately delete it and any attachments and notify the sender by reply e-mail. In such case, you are hereby notified that any dissemination, distribution, copying or use of this message or any attachments, for any purpose, is strictly prohibited by law. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [ubuntu-cloud] Update on Ubuntu automated testing and CI of Openstack
On 02/09/2012 05:21 AM, Ghe Rivero wrote: Wow! Openstack is really getting to much love from Ubuntu, nice! I wish we have so many resources to the same with Debian. Keep the good work guys! Ghe Rivero I could be 100% wrong here, but I think Debian has agreed (or are in talks) to use Ubuntu as the upstream for OpenStack, thus allowing our testing to trickle down. -- Robbie Williamson rob...@ubuntu.com robbiew[irc.freenode.net] Don't make me angry...you wouldn't like me when I'm angry. -Bruce Banner ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [ubuntu-cloud] Update on Ubuntu automated testing and CI of Openstack
Yes, we all agree that Ubuntu is the reference platform for OpenStack. And I wasn't complaining about it, just making a wish trying to get some help :). Sorry about the misunderstanding. Ghe Rivero On Thu, Feb 9, 2012 at 4:57 PM, Robbie Williamson rob...@ubuntu.com wrote: On 02/09/2012 05:21 AM, Ghe Rivero wrote: Wow! Openstack is really getting to much love from Ubuntu, nice! I wish we have so many resources to the same with Debian. Keep the good work guys! Ghe Rivero I could be 100% wrong here, but I think Debian has agreed (or are in talks) to use Ubuntu as the upstream for OpenStack, thus allowing our testing to trickle down. -- Robbie Williamson rob...@ubuntu.com robbiew[irc.freenode.net] Don't make me angry...you wouldn't like me when I'm angry. -Bruce Banner -- Ghe Rivero *OpenStack Distribution Engineer **www.stackops.com | * ghe.riv...@stackops.com diego.parri...@stackops.com ** | +34 625 63 45 23 | skype:ghe.rivero* * http://www.stackops.com/ * * ADVERTENCIA LEGAL Le informamos, como destinatario de este mensaje, que el correo electrónico y las comunicaciones por medio de Internet no permiten asegurar ni garantizar la confidencialidad de los mensajes transmitidos, así como tampoco su integridad o su correcta recepción, por lo que STACKOPS TECHNOLOGIES S.L. no asume responsabilidad alguna por tales circunstancias. Si no consintiese en la utilización del correo electrónico o de las comunicaciones vía Internet le rogamos nos lo comunique y ponga en nuestro conocimiento de manera inmediata. Este mensaje va dirigido, de manera exclusiva, a su destinatario y contiene información confidencial y sujeta al secreto profesional, cuya divulgación no está permitida por la ley. En caso de haber recibido este mensaje por error, le rogamos que, de forma inmediata, nos lo comunique mediante correo electrónico remitido a nuestra atención y proceda a su eliminación, así como a la de cualquier documento adjunto al mismo. Asimismo, le comunicamos que la distribución, copia o utilización de este mensaje, o de cualquier documento adjunto al mismo, cualquiera que fuera su finalidad, están prohibidas por la ley. * PRIVILEGED AND CONFIDENTIAL We hereby inform you, as addressee of this message, that e-mail and Internet do not guarantee the confidentiality, nor the completeness or proper reception of the messages sent and, thus, STACKOPS TECHNOLOGIES S.L. does not assume any liability for those circumstances. Should you not agree to the use of e-mail or to communications via Internet, you are kindly requested to notify us immediately. This message is intended exclusively for the person to whom it is addressed and contains privileged and confidential information protected from disclosure by law. If you are not the addressee indicated in this message, you should immediately delete it and any attachments and notify the sender by reply e-mail. In such case, you are hereby notified that any dissemination, distribution, copying or use of this message or any attachments, for any purpose, is strictly prohibited by law. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [ubuntu-cloud] Update on Ubuntu automated testing and CI of Openstack
- Original message - On Thu, Feb 9, 2012 at 4:57 PM, Robbie Williamson rob...@ubuntu.com wrote: On 02/09/2012 05:21 AM, Ghe Rivero wrote: Wow! Openstack is really getting to much love from Ubuntu, nice! I wish we have so many resources to the same with Debian. Keep the good work guys! Ghe Rivero I could be 100% wrong here, but I think Debian has agreed (or are in talks) to use Ubuntu as the upstream for OpenStack, thus allowing our testing to trickle down. There's not really such thing anymore. Our Debian packages aren't the same as from Ubuntu now, and we take the code from Github. So openstack on github is our upstream (even though many developers sending code there are from Ubuntu). Thomas ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
