Re: [openstack-dev] [TripleO] test environment requirements
- Original Message - From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Sent: Sunday, March 23, 2014 9:02:23 PM Subject: Re: [openstack-dev] [TripleO] test environment requirements Excerpts from Dan Prince's message of 2014-03-21 09:25:42 -0700: - Original Message - From: Robert Collins robe...@robertcollins.net To: OpenStack Development Mailing List openstack-dev@lists.openstack.org Sent: Thursday, March 13, 2014 5:51:30 AM Subject: [openstack-dev] [TripleO] test environment requirements So we already have pretty high requirements - its basically a 16G workstation as minimum. Specifically to test the full story: - a seed VM - an undercloud VM (bm deploy infra) - 1 overcloud control VM - 2 overcloud hypervisor VMs 5 VMs with 2+G RAM each. To test the overcloud alone against the seed we save 1 VM, to skip the overcloud we save 3. However, as HA matures we're about to add 4 more VMs: we need a HA control plane for both the under and overclouds: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud control VMs (HA) - 2 overcloud hypervisor VMs 9 VMs with 2+G RAM each == 18GB What should we do about this? A few thoughts to kick start discussion: - use Ironic to test across multiple machines (involves tunnelling brbm across machines, fairly easy) - shrink the VM sizes (causes thrashing) - tell folk to toughen up and get bigger machines (ahahahahaha, no) - make the default configuration inline the hypervisors on the overcloud with the control plane: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud all-in-one VMs (HA) 7 VMs with 2+G RAM each == 14GB I think its important that we exercise features like HA and live migration regularly by developers, so I'm quite keen to have a fairly solid systematic answer that will let us catch things like bad firewall rules on the control node preventing network tunnelling etc... I'm all for supporting HA development and testing within devtest. I'm *against* forcing it on all users as a default. I can imaging wanting to cut corners and have configurations flexible on both ends (undercloud and overcloud). I may for example deploy a single all-in-one undercloud when I'm testing overcloud HA. Or vice versa. I think I'm one of the few (if not the only) developer who uses almost exclusive baremetal (besides seed VM) when test/developing TripleO. Forcing users who want to do this to have 6-7 real machines is a bit much I think. Arguably wasteful even. By requiring more machines to run through devtest you actually make it harder for people to test it on real hardware which is usually harder to come by. Given deployment on real bare metal is sort of the point or TripleO I'd very much like to see more developers using it rather than less. So by all means lets support HA... but lets do it in a way that is configurable (i.e. not forcing people to be wasters) Dan I don't think anybody wants to force it on _users_. But a predominance of end users will require HA, and thus we need our developers to be able to develop with HA. This is for the _benefit_ of developers. I imagine we've all been in situations where our dev environment is almost nothing like CI, and then when CI runs you find that you have missed huge problems.. and now to test for those problems you either have to re-do your dev environment, or wait.. a lot.. for CI. All good points. Running an exactly copy of the upstream CI environment seems to be getting more and more costly though. My goal is that I'd like developers to be able to choose what they want to test as much as they can. Streamline things where appropriate. Take the overcloud today: I actually like to idea of going the other way here and running an all-in-one version of it from time to time to save resources. If I know I need to dev test on an HA cloud then by all means I'll try to do that. But in many cases I may not need to go to such lengths. Furthermore, some testing is better than no testing at all because we've set the resources bar too high. Again, all for supporting the HA test and dev path in the devtest scripts. Ideally making it as configurable as we can... Dan I don't have any really clever answers to this problem. We're testing an end-to-end cloud deployment. If we can't run a small, accurate simulation of such an environment as developers, then we will end up going very slow. The problem is that this small simulation is still massive compared to the usual development paradigm which involves at most two distinct virtual machines. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi
Re: [openstack-dev] [TripleO] test environment requirements
- Original Message - From: Robert Collins robe...@robertcollins.net To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Sunday, March 23, 2014 11:18:19 PM Subject: Re: [openstack-dev] [TripleO] test environment requirements On 24 March 2014 14:06, Clint Byrum cl...@fewbar.com wrote: Excerpts from Robert Collins's message of 2014-03-13 02:51:30 -0700: So we already have pretty high requirements - its basically a 16G workstation as minimum. Specifically to test the full story: - a seed VM - an undercloud VM (bm deploy infra) - 1 overcloud control VM - 2 overcloud hypervisor VMs 5 VMs with 2+G RAM each. To test the overcloud alone against the seed we save 1 VM, to skip the overcloud we save 3. However, as HA matures we're about to add 4 more VMs: we need a HA control plane for both the under and overclouds: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud control VMs (HA) - 2 overcloud hypervisor VMs 9 VMs with 2+G RAM each == 18GB If we switch end-user-vm tests to cirros, and use a flavor that is really really tiny, like 128M RAM, then I think we can downsize the development environment hypervisors from 2G to 1G. That at least drops us to 16G. If we can also turn off the seed as soon as the undercloud boots, that's another 2G saved. Finally if we can turn off 1 of the undercloud VMs and run degraded, that is another 2G saved. We can't turn off the seed unless we change the network topology; we need to make sure if we do that that we don't invalidate the test structure. This should make changing the network topology in the test env's easier: https://review.openstack.org/#/c/82327/ If we use something else (besides the seed VM) as the gateway for the baremetal network I think it should work fine right? Small potatoes I know, but it wouldn't be complicated to do any of these and it would also help test real scenarios we want to test (seed going away, undercloud node going away). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I like this. I also like much of the spread of ideas that came up - lets capture them to an etherpad and link that from a blueprint for future selves finding it. In particular I'd like to make being able to use OpenStack a J series goal, though it doesn't help with local dev overheads. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] test environment requirements
On 2014-03-23 22:18, Robert Collins wrote: On 24 March 2014 14:06, Clint Byrum cl...@fewbar.com wrote: Excerpts from Robert Collins's message of 2014-03-13 02:51:30 -0700: So we already have pretty high requirements - its basically a 16G workstation as minimum. Specifically to test the full story: - a seed VM - an undercloud VM (bm deploy infra) - 1 overcloud control VM - 2 overcloud hypervisor VMs 5 VMs with 2+G RAM each. To test the overcloud alone against the seed we save 1 VM, to skip the overcloud we save 3. However, as HA matures we're about to add 4 more VMs: we need a HA control plane for both the under and overclouds: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud control VMs (HA) - 2 overcloud hypervisor VMs 9 VMs with 2+G RAM each == 18GB If we switch end-user-vm tests to cirros, and use a flavor that is really really tiny, like 128M RAM, then I think we can downsize the development environment hypervisors from 2G to 1G. That at least drops us to 16G. If we can also turn off the seed as soon as the undercloud boots, that's another 2G saved. Finally if we can turn off 1 of the undercloud VMs and run degraded, that is another 2G saved. We can't turn off the seed unless we change the network topology; we need to make sure if we do that that we don't invalidate the test structure. Small potatoes I know, but it wouldn't be complicated to do any of these and it would also help test real scenarios we want to test (seed going away, undercloud node going away). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I like this. I also like much of the spread of ideas that came up - lets capture them to an etherpad and link that from a blueprint for future selves finding it. I created an etherpad here: https://etherpad.openstack.org/p/devtest-env-reqs And linked it from the blueprint here: https://blueprints.launchpad.net/tripleo/+spec/test-environment-requirements I only added some details about devtest on openstack for now since I figured everyone else could add their thoughts in their own words. In particular I'd like to make being able to use OpenStack a J series goal, though it doesn't help with local dev overheads. -Rob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] test environment requirements
On 25 March 2014 06:28, Ben Nemec openst...@nemebean.com wrote: I created an etherpad here: https://etherpad.openstack.org/p/devtest-env-reqs And linked it from the blueprint here: https://blueprints.launchpad.net/tripleo/+spec/test-environment-requirements I only added some details about devtest on openstack for now since I figured everyone else could add their thoughts in their own words. I think it would be super useful if you could capture all the things in one place - it will mean we don't need to chase folk up to ensure their idea is present in the etherpad. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] test environment requirements
On 2014-03-24 14:43, Robert Collins wrote: On 25 March 2014 06:28, Ben Nemec openst...@nemebean.com wrote: I created an etherpad here: https://etherpad.openstack.org/p/devtest-env-reqs And linked it from the blueprint here: https://blueprints.launchpad.net/tripleo/+spec/test-environment-requirements I only added some details about devtest on openstack for now since I figured everyone else could add their thoughts in their own words. I think it would be super useful if you could capture all the things in one place - it will mean we don't need to chase folk up to ensure their idea is present in the etherpad. -Rob Okay, updated the etherpad with the main points I got from the thread. -Ben ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] test environment requirements
Excerpts from Ben Nemec's message of 2014-03-21 09:38:00 -0700: On 2014-03-21 10:57, Derek Higgins wrote: On 14/03/14 20:16, Ben Nemec wrote: On 2014-03-13 11:12, James Slagle wrote: On Thu, Mar 13, 2014 at 2:51 AM, Robert Collins robe...@robertcollins.net wrote: So we already have pretty high requirements - its basically a 16G workstation as minimum. Specifically to test the full story: - a seed VM - an undercloud VM (bm deploy infra) - 1 overcloud control VM - 2 overcloud hypervisor VMs 5 VMs with 2+G RAM each. To test the overcloud alone against the seed we save 1 VM, to skip the overcloud we save 3. However, as HA matures we're about to add 4 more VMs: we need a HA control plane for both the under and overclouds: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud control VMs (HA) - 2 overcloud hypervisor VMs 9 VMs with 2+G RAM each == 18GB What should we do about this? A few thoughts to kick start discussion: - use Ironic to test across multiple machines (involves tunnelling brbm across machines, fairly easy) - shrink the VM sizes (causes thrashing) - tell folk to toughen up and get bigger machines (ahahahahaha, no) - make the default configuration inline the hypervisors on the overcloud with the control plane: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud all-in-one VMs (HA) 7 VMs with 2+G RAM each == 14GB I think its important that we exercise features like HA and live migration regularly by developers, so I'm quite keen to have a fairly solid systematic answer that will let us catch things like bad firewall rules on the control node preventing network tunnelling etc... e.g. we benefit the more things are split out like scale deployments are. OTOH testing the micro-cloud that folk may start with is also a really good idea The idea I was thinking was to make a testenv host available to tripleo atc's. Or, perhaps make it a bit more locked down and only available to a new group of tripleo folk, existing somewhere between the privileges of tripleo atc's and tripleo-cd-admins. We could document how you use the cloud (Red Hat's or HP's) rack to start up a instance to run devtest on one of the compute hosts, request and lock yourself a testenv environment on one of the testenv hosts, etc. Basically, how our CI works. Although I think we'd want different testenv hosts for development vs what runs the CI, and would need to make sure everything was locked down appropriately security-wise. Some other ideas: - Allow an option to get rid of the seed VM, or make it so that you can shut it down after the Undercloud is up. This only really gets rid of 1 VM though, so it doesn't buy you much nor solve any long term problem. - Make it easier to see how you'd use virsh against any libvirt host you might have lying around. We already have the setting exposed, but make it a bit more public and call it out more in the docs. I've actually never tried it myself, but have been meaning to. - I'm really reaching now, and this may be entirely unrealistic :), butsomehow use the fake baremetal driver and expose a mechanism to let the developer specify the already setup undercloud/overcloud environment ahead of time. For example: * Build your undercloud images with the vm element since you won't be PXE booting it * Upload your images to a public cloud, and boot instances for them. * Use this new mechanism when you run devtest (presumably running from another instance in the same cloud) to say I'm using the fake baremetal driver, and here are the IP's of the undercloud instances. * Repeat steps for the overcloud (e.g., configure undercloud to use fake baremetal driver, etc). * Maybe it's not the fake baremetal driver, and instead a new driver that is a noop for the pxe stuff, and the power_on implementation powers on the cloud instances. * Obviously if your aim is to test the pxe and disk deploy process itself, this wouldn't work for you. * Presumably said public cloud is OpenStack, so we've also achieved another layer of On OpenStack. I actually spent quite a while looking into something like this last option when I first started on TripleO, because I had only one big server locally and it was running my OpenStack installation. I was hoping to use it for my TripleO instances, and even went so far as to add support for OpenStack to the virtual power driver in baremetal. I was never completely successful, but I did work through a number of problems: 1. Neutron didn't like allowing the DHCP/PXE traffic to let my seed serve to the undercloud. I was able to get around this by using flat networking with a local bridge on the OpenStack system, but I'm not sure if that's going to be possible on most public cloud providers.
Re: [openstack-dev] [TripleO] test environment requirements
Excerpts from Dan Prince's message of 2014-03-21 09:25:42 -0700: - Original Message - From: Robert Collins robe...@robertcollins.net To: OpenStack Development Mailing List openstack-dev@lists.openstack.org Sent: Thursday, March 13, 2014 5:51:30 AM Subject: [openstack-dev] [TripleO] test environment requirements So we already have pretty high requirements - its basically a 16G workstation as minimum. Specifically to test the full story: - a seed VM - an undercloud VM (bm deploy infra) - 1 overcloud control VM - 2 overcloud hypervisor VMs 5 VMs with 2+G RAM each. To test the overcloud alone against the seed we save 1 VM, to skip the overcloud we save 3. However, as HA matures we're about to add 4 more VMs: we need a HA control plane for both the under and overclouds: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud control VMs (HA) - 2 overcloud hypervisor VMs 9 VMs with 2+G RAM each == 18GB What should we do about this? A few thoughts to kick start discussion: - use Ironic to test across multiple machines (involves tunnelling brbm across machines, fairly easy) - shrink the VM sizes (causes thrashing) - tell folk to toughen up and get bigger machines (ahahahahaha, no) - make the default configuration inline the hypervisors on the overcloud with the control plane: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud all-in-one VMs (HA) 7 VMs with 2+G RAM each == 14GB I think its important that we exercise features like HA and live migration regularly by developers, so I'm quite keen to have a fairly solid systematic answer that will let us catch things like bad firewall rules on the control node preventing network tunnelling etc... I'm all for supporting HA development and testing within devtest. I'm *against* forcing it on all users as a default. I can imaging wanting to cut corners and have configurations flexible on both ends (undercloud and overcloud). I may for example deploy a single all-in-one undercloud when I'm testing overcloud HA. Or vice versa. I think I'm one of the few (if not the only) developer who uses almost exclusive baremetal (besides seed VM) when test/developing TripleO. Forcing users who want to do this to have 6-7 real machines is a bit much I think. Arguably wasteful even. By requiring more machines to run through devtest you actually make it harder for people to test it on real hardware which is usually harder to come by. Given deployment on real bare metal is sort of the point or TripleO I'd very much like to see more developers using it rather than less. So by all means lets support HA... but lets do it in a way that is configurable (i.e. not forcing people to be wasters) Dan I don't think anybody wants to force it on _users_. But a predominance of end users will require HA, and thus we need our developers to be able to develop with HA. This is for the _benefit_ of developers. I imagine we've all been in situations where our dev environment is almost nothing like CI, and then when CI runs you find that you have missed huge problems.. and now to test for those problems you either have to re-do your dev environment, or wait.. a lot.. for CI. I don't have any really clever answers to this problem. We're testing an end-to-end cloud deployment. If we can't run a small, accurate simulation of such an environment as developers, then we will end up going very slow. The problem is that this small simulation is still massive compared to the usual development paradigm which involves at most two distinct virtual machines. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] test environment requirements
Excerpts from Robert Collins's message of 2014-03-13 02:51:30 -0700: So we already have pretty high requirements - its basically a 16G workstation as minimum. Specifically to test the full story: - a seed VM - an undercloud VM (bm deploy infra) - 1 overcloud control VM - 2 overcloud hypervisor VMs 5 VMs with 2+G RAM each. To test the overcloud alone against the seed we save 1 VM, to skip the overcloud we save 3. However, as HA matures we're about to add 4 more VMs: we need a HA control plane for both the under and overclouds: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud control VMs (HA) - 2 overcloud hypervisor VMs 9 VMs with 2+G RAM each == 18GB If we switch end-user-vm tests to cirros, and use a flavor that is really really tiny, like 128M RAM, then I think we can downsize the development environment hypervisors from 2G to 1G. That at least drops us to 16G. If we can also turn off the seed as soon as the undercloud boots, that's another 2G saved. Finally if we can turn off 1 of the undercloud VMs and run degraded, that is another 2G saved. Small potatoes I know, but it wouldn't be complicated to do any of these and it would also help test real scenarios we want to test (seed going away, undercloud node going away). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] test environment requirements
On 24 March 2014 14:06, Clint Byrum cl...@fewbar.com wrote: Excerpts from Robert Collins's message of 2014-03-13 02:51:30 -0700: So we already have pretty high requirements - its basically a 16G workstation as minimum. Specifically to test the full story: - a seed VM - an undercloud VM (bm deploy infra) - 1 overcloud control VM - 2 overcloud hypervisor VMs 5 VMs with 2+G RAM each. To test the overcloud alone against the seed we save 1 VM, to skip the overcloud we save 3. However, as HA matures we're about to add 4 more VMs: we need a HA control plane for both the under and overclouds: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud control VMs (HA) - 2 overcloud hypervisor VMs 9 VMs with 2+G RAM each == 18GB If we switch end-user-vm tests to cirros, and use a flavor that is really really tiny, like 128M RAM, then I think we can downsize the development environment hypervisors from 2G to 1G. That at least drops us to 16G. If we can also turn off the seed as soon as the undercloud boots, that's another 2G saved. Finally if we can turn off 1 of the undercloud VMs and run degraded, that is another 2G saved. We can't turn off the seed unless we change the network topology; we need to make sure if we do that that we don't invalidate the test structure. Small potatoes I know, but it wouldn't be complicated to do any of these and it would also help test real scenarios we want to test (seed going away, undercloud node going away). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I like this. I also like much of the spread of ideas that came up - lets capture them to an etherpad and link that from a blueprint for future selves finding it. In particular I'd like to make being able to use OpenStack a J series goal, though it doesn't help with local dev overheads. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] test environment requirements
On 13/03/14 16:12, James Slagle wrote: On Thu, Mar 13, 2014 at 2:51 AM, Robert Collins robe...@robertcollins.net wrote: So we already have pretty high requirements - its basically a 16G workstation as minimum. Specifically to test the full story: - a seed VM - an undercloud VM (bm deploy infra) - 1 overcloud control VM - 2 overcloud hypervisor VMs 5 VMs with 2+G RAM each. To test the overcloud alone against the seed we save 1 VM, to skip the overcloud we save 3. However, as HA matures we're about to add 4 more VMs: we need a HA control plane for both the under and overclouds: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud control VMs (HA) - 2 overcloud hypervisor VMs 9 VMs with 2+G RAM each == 18GB What should we do about this? A few thoughts to kick start discussion: - use Ironic to test across multiple machines (involves tunnelling brbm across machines, fairly easy) - shrink the VM sizes (causes thrashing) - tell folk to toughen up and get bigger machines (ahahahahaha, no) - make the default configuration inline the hypervisors on the overcloud with the control plane: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud all-in-one VMs (HA) 7 VMs with 2+G RAM each == 14GB I think its important that we exercise features like HA and live migration regularly by developers, so I'm quite keen to have a fairly solid systematic answer that will let us catch things like bad firewall rules on the control node preventing network tunnelling etc... e.g. we benefit the more things are split out like scale deployments are. OTOH testing the micro-cloud that folk may start with is also a really good idea The idea I was thinking was to make a testenv host available to tripleo atc's. Or, perhaps make it a bit more locked down and only available to a new group of tripleo folk, existing somewhere between the privileges of tripleo atc's and tripleo-cd-admins. We could document how you use the cloud (Red Hat's or HP's) rack to start up a instance to run devtest on one of the compute hosts, request and lock yourself a testenv environment on one of the testenv hosts, etc. Basically, how our CI works. Although I think we'd want different testenv hosts for development vs what runs the CI, and would need to make sure everything was locked down appropriately security-wise. I like this idea, I think it could work, my only concern is the extra capacity we would need to pull it off. At the moment we are probably falling short on capacity to do what we want for CI so adding to this would make the situation worse (how much worse I don't know). So unless we get to the point where we have spare hardware doing nothing I think its a non runner. Some other ideas: - Allow an option to get rid of the seed VM, or make it so that you can shut it down after the Undercloud is up. This only really gets rid of 1 VM though, so it doesn't buy you much nor solve any long term problem. - Make it easier to see how you'd use virsh against any libvirt host you might have lying around. We already have the setting exposed, but make it a bit more public and call it out more in the docs. I've actually never tried it myself, but have been meaning to. this could work as an option - I'm really reaching now, and this may be entirely unrealistic :), butsomehow use the fake baremetal driver and expose a mechanism to let the developer specify the already setup undercloud/overcloud environment ahead of time. For example: * Build your undercloud images with the vm element since you won't be PXE booting it * Upload your images to a public cloud, and boot instances for them. * Use this new mechanism when you run devtest (presumably running from another instance in the same cloud) to say I'm using the fake baremetal driver, and here are the IP's of the undercloud instances. * Repeat steps for the overcloud (e.g., configure undercloud to use fake baremetal driver, etc). * Maybe it's not the fake baremetal driver, and instead a new driver that is a noop for the pxe stuff, and the power_on implementation powers on the cloud instances. * Obviously if your aim is to test the pxe and disk deploy process itself, this wouldn't work for you. * Presumably said public cloud is OpenStack, so we've also achieved another layer of On OpenStack. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] test environment requirements
On 13/03/14 09:51, Robert Collins wrote: So we already have pretty high requirements - its basically a 16G workstation as minimum. Specifically to test the full story: - a seed VM - an undercloud VM (bm deploy infra) - 1 overcloud control VM - 2 overcloud hypervisor VMs 5 VMs with 2+G RAM each. To test the overcloud alone against the seed we save 1 VM, to skip the overcloud we save 3. However, as HA matures we're about to add 4 more VMs: we need a HA control plane for both the under and overclouds: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud control VMs (HA) - 2 overcloud hypervisor VMs 9 VMs with 2+G RAM each == 18GB What should we do about this? A few thoughts to kick start discussion: - use Ironic to test across multiple machines (involves tunnelling brbm across machines, fairly easy) - shrink the VM sizes (causes thrashing) - tell folk to toughen up and get bigger machines (ahahahahaha, no) - make the default configuration inline the hypervisors on the overcloud with the control plane: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud all-in-one VMs (HA) 7 VMs with 2+G RAM each == 14GB I think its important that we exercise features like HA and live migration regularly by developers, so I'm quite keen to have a fairly solid systematic answer that will let us catch things like bad firewall rules on the control node preventing network tunnelling etc... e.g. we benefit the more things are split out like scale deployments are. OTOH testing the micro-cloud that folk may start with is also a really good idea I'd vote for an optional (non default) inline cloud setup like what you mention above, and maybe a non HA setup aswell. This would allow a lower entry bar to people who only want to worry about a specific component. We then would need to cover all supported setups in CI (adding to capacity needs). and of course we then wouldn't have everybody exercising HA but it may be necessary to encourage uptake. -Rob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] test environment requirements
On 14/03/14 20:16, Ben Nemec wrote: On 2014-03-13 11:12, James Slagle wrote: On Thu, Mar 13, 2014 at 2:51 AM, Robert Collins robe...@robertcollins.net wrote: So we already have pretty high requirements - its basically a 16G workstation as minimum. Specifically to test the full story: - a seed VM - an undercloud VM (bm deploy infra) - 1 overcloud control VM - 2 overcloud hypervisor VMs 5 VMs with 2+G RAM each. To test the overcloud alone against the seed we save 1 VM, to skip the overcloud we save 3. However, as HA matures we're about to add 4 more VMs: we need a HA control plane for both the under and overclouds: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud control VMs (HA) - 2 overcloud hypervisor VMs 9 VMs with 2+G RAM each == 18GB What should we do about this? A few thoughts to kick start discussion: - use Ironic to test across multiple machines (involves tunnelling brbm across machines, fairly easy) - shrink the VM sizes (causes thrashing) - tell folk to toughen up and get bigger machines (ahahahahaha, no) - make the default configuration inline the hypervisors on the overcloud with the control plane: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud all-in-one VMs (HA) 7 VMs with 2+G RAM each == 14GB I think its important that we exercise features like HA and live migration regularly by developers, so I'm quite keen to have a fairly solid systematic answer that will let us catch things like bad firewall rules on the control node preventing network tunnelling etc... e.g. we benefit the more things are split out like scale deployments are. OTOH testing the micro-cloud that folk may start with is also a really good idea The idea I was thinking was to make a testenv host available to tripleo atc's. Or, perhaps make it a bit more locked down and only available to a new group of tripleo folk, existing somewhere between the privileges of tripleo atc's and tripleo-cd-admins. We could document how you use the cloud (Red Hat's or HP's) rack to start up a instance to run devtest on one of the compute hosts, request and lock yourself a testenv environment on one of the testenv hosts, etc. Basically, how our CI works. Although I think we'd want different testenv hosts for development vs what runs the CI, and would need to make sure everything was locked down appropriately security-wise. Some other ideas: - Allow an option to get rid of the seed VM, or make it so that you can shut it down after the Undercloud is up. This only really gets rid of 1 VM though, so it doesn't buy you much nor solve any long term problem. - Make it easier to see how you'd use virsh against any libvirt host you might have lying around. We already have the setting exposed, but make it a bit more public and call it out more in the docs. I've actually never tried it myself, but have been meaning to. - I'm really reaching now, and this may be entirely unrealistic :), butsomehow use the fake baremetal driver and expose a mechanism to let the developer specify the already setup undercloud/overcloud environment ahead of time. For example: * Build your undercloud images with the vm element since you won't be PXE booting it * Upload your images to a public cloud, and boot instances for them. * Use this new mechanism when you run devtest (presumably running from another instance in the same cloud) to say I'm using the fake baremetal driver, and here are the IP's of the undercloud instances. * Repeat steps for the overcloud (e.g., configure undercloud to use fake baremetal driver, etc). * Maybe it's not the fake baremetal driver, and instead a new driver that is a noop for the pxe stuff, and the power_on implementation powers on the cloud instances. * Obviously if your aim is to test the pxe and disk deploy process itself, this wouldn't work for you. * Presumably said public cloud is OpenStack, so we've also achieved another layer of On OpenStack. I actually spent quite a while looking into something like this last option when I first started on TripleO, because I had only one big server locally and it was running my OpenStack installation. I was hoping to use it for my TripleO instances, and even went so far as to add support for OpenStack to the virtual power driver in baremetal. I was never completely successful, but I did work through a number of problems: 1. Neutron didn't like allowing the DHCP/PXE traffic to let my seed serve to the undercloud. I was able to get around this by using flat networking with a local bridge on the OpenStack system, but I'm not sure if that's going to be possible on most public cloud providers. There may very well be a less invasive way to configure Neutron to allow that, but I don't know how to do it. 2. Last time I checked, Nova doesn't support PXE booting instances so I had to use iPXE images to do
Re: [openstack-dev] [TripleO] test environment requirements
- Original Message - From: Robert Collins robe...@robertcollins.net To: OpenStack Development Mailing List openstack-dev@lists.openstack.org Sent: Thursday, March 13, 2014 5:51:30 AM Subject: [openstack-dev] [TripleO] test environment requirements So we already have pretty high requirements - its basically a 16G workstation as minimum. Specifically to test the full story: - a seed VM - an undercloud VM (bm deploy infra) - 1 overcloud control VM - 2 overcloud hypervisor VMs 5 VMs with 2+G RAM each. To test the overcloud alone against the seed we save 1 VM, to skip the overcloud we save 3. However, as HA matures we're about to add 4 more VMs: we need a HA control plane for both the under and overclouds: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud control VMs (HA) - 2 overcloud hypervisor VMs 9 VMs with 2+G RAM each == 18GB What should we do about this? A few thoughts to kick start discussion: - use Ironic to test across multiple machines (involves tunnelling brbm across machines, fairly easy) - shrink the VM sizes (causes thrashing) - tell folk to toughen up and get bigger machines (ahahahahaha, no) - make the default configuration inline the hypervisors on the overcloud with the control plane: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud all-in-one VMs (HA) 7 VMs with 2+G RAM each == 14GB I think its important that we exercise features like HA and live migration regularly by developers, so I'm quite keen to have a fairly solid systematic answer that will let us catch things like bad firewall rules on the control node preventing network tunnelling etc... I'm all for supporting HA development and testing within devtest. I'm *against* forcing it on all users as a default. I can imaging wanting to cut corners and have configurations flexible on both ends (undercloud and overcloud). I may for example deploy a single all-in-one undercloud when I'm testing overcloud HA. Or vice versa. I think I'm one of the few (if not the only) developer who uses almost exclusive baremetal (besides seed VM) when test/developing TripleO. Forcing users who want to do this to have 6-7 real machines is a bit much I think. Arguably wasteful even. By requiring more machines to run through devtest you actually make it harder for people to test it on real hardware which is usually harder to come by. Given deployment on real bare metal is sort of the point or TripleO I'd very much like to see more developers using it rather than less. So by all means lets support HA... but lets do it in a way that is configurable (i.e. not forcing people to be wasters) Dan e.g. we benefit the more things are split out like scale deployments are. OTOH testing the micro-cloud that folk may start with is also a really good idea -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] test environment requirements
On 2014-03-21 10:57, Derek Higgins wrote: On 14/03/14 20:16, Ben Nemec wrote: On 2014-03-13 11:12, James Slagle wrote: On Thu, Mar 13, 2014 at 2:51 AM, Robert Collins robe...@robertcollins.net wrote: So we already have pretty high requirements - its basically a 16G workstation as minimum. Specifically to test the full story: - a seed VM - an undercloud VM (bm deploy infra) - 1 overcloud control VM - 2 overcloud hypervisor VMs 5 VMs with 2+G RAM each. To test the overcloud alone against the seed we save 1 VM, to skip the overcloud we save 3. However, as HA matures we're about to add 4 more VMs: we need a HA control plane for both the under and overclouds: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud control VMs (HA) - 2 overcloud hypervisor VMs 9 VMs with 2+G RAM each == 18GB What should we do about this? A few thoughts to kick start discussion: - use Ironic to test across multiple machines (involves tunnelling brbm across machines, fairly easy) - shrink the VM sizes (causes thrashing) - tell folk to toughen up and get bigger machines (ahahahahaha, no) - make the default configuration inline the hypervisors on the overcloud with the control plane: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud all-in-one VMs (HA) 7 VMs with 2+G RAM each == 14GB I think its important that we exercise features like HA and live migration regularly by developers, so I'm quite keen to have a fairly solid systematic answer that will let us catch things like bad firewall rules on the control node preventing network tunnelling etc... e.g. we benefit the more things are split out like scale deployments are. OTOH testing the micro-cloud that folk may start with is also a really good idea The idea I was thinking was to make a testenv host available to tripleo atc's. Or, perhaps make it a bit more locked down and only available to a new group of tripleo folk, existing somewhere between the privileges of tripleo atc's and tripleo-cd-admins. We could document how you use the cloud (Red Hat's or HP's) rack to start up a instance to run devtest on one of the compute hosts, request and lock yourself a testenv environment on one of the testenv hosts, etc. Basically, how our CI works. Although I think we'd want different testenv hosts for development vs what runs the CI, and would need to make sure everything was locked down appropriately security-wise. Some other ideas: - Allow an option to get rid of the seed VM, or make it so that you can shut it down after the Undercloud is up. This only really gets rid of 1 VM though, so it doesn't buy you much nor solve any long term problem. - Make it easier to see how you'd use virsh against any libvirt host you might have lying around. We already have the setting exposed, but make it a bit more public and call it out more in the docs. I've actually never tried it myself, but have been meaning to. - I'm really reaching now, and this may be entirely unrealistic :), butsomehow use the fake baremetal driver and expose a mechanism to let the developer specify the already setup undercloud/overcloud environment ahead of time. For example: * Build your undercloud images with the vm element since you won't be PXE booting it * Upload your images to a public cloud, and boot instances for them. * Use this new mechanism when you run devtest (presumably running from another instance in the same cloud) to say I'm using the fake baremetal driver, and here are the IP's of the undercloud instances. * Repeat steps for the overcloud (e.g., configure undercloud to use fake baremetal driver, etc). * Maybe it's not the fake baremetal driver, and instead a new driver that is a noop for the pxe stuff, and the power_on implementation powers on the cloud instances. * Obviously if your aim is to test the pxe and disk deploy process itself, this wouldn't work for you. * Presumably said public cloud is OpenStack, so we've also achieved another layer of On OpenStack. I actually spent quite a while looking into something like this last option when I first started on TripleO, because I had only one big server locally and it was running my OpenStack installation. I was hoping to use it for my TripleO instances, and even went so far as to add support for OpenStack to the virtual power driver in baremetal. I was never completely successful, but I did work through a number of problems: 1. Neutron didn't like allowing the DHCP/PXE traffic to let my seed serve to the undercloud. I was able to get around this by using flat networking with a local bridge on the OpenStack system, but I'm not sure if that's going to be possible on most public cloud providers. There may very well be a less invasive way to configure Neutron to allow that, but I don't know how to do it. 2. Last time I checked, Nova doesn't support PXE booting
Re: [openstack-dev] [TripleO] test environment requirements
On 2014-03-13 11:12, James Slagle wrote: On Thu, Mar 13, 2014 at 2:51 AM, Robert Collins robe...@robertcollins.net wrote: So we already have pretty high requirements - its basically a 16G workstation as minimum. Specifically to test the full story: - a seed VM - an undercloud VM (bm deploy infra) - 1 overcloud control VM - 2 overcloud hypervisor VMs 5 VMs with 2+G RAM each. To test the overcloud alone against the seed we save 1 VM, to skip the overcloud we save 3. However, as HA matures we're about to add 4 more VMs: we need a HA control plane for both the under and overclouds: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud control VMs (HA) - 2 overcloud hypervisor VMs 9 VMs with 2+G RAM each == 18GB What should we do about this? A few thoughts to kick start discussion: - use Ironic to test across multiple machines (involves tunnelling brbm across machines, fairly easy) - shrink the VM sizes (causes thrashing) - tell folk to toughen up and get bigger machines (ahahahahaha, no) - make the default configuration inline the hypervisors on the overcloud with the control plane: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud all-in-one VMs (HA) 7 VMs with 2+G RAM each == 14GB I think its important that we exercise features like HA and live migration regularly by developers, so I'm quite keen to have a fairly solid systematic answer that will let us catch things like bad firewall rules on the control node preventing network tunnelling etc... e.g. we benefit the more things are split out like scale deployments are. OTOH testing the micro-cloud that folk may start with is also a really good idea The idea I was thinking was to make a testenv host available to tripleo atc's. Or, perhaps make it a bit more locked down and only available to a new group of tripleo folk, existing somewhere between the privileges of tripleo atc's and tripleo-cd-admins. We could document how you use the cloud (Red Hat's or HP's) rack to start up a instance to run devtest on one of the compute hosts, request and lock yourself a testenv environment on one of the testenv hosts, etc. Basically, how our CI works. Although I think we'd want different testenv hosts for development vs what runs the CI, and would need to make sure everything was locked down appropriately security-wise. Some other ideas: - Allow an option to get rid of the seed VM, or make it so that you can shut it down after the Undercloud is up. This only really gets rid of 1 VM though, so it doesn't buy you much nor solve any long term problem. - Make it easier to see how you'd use virsh against any libvirt host you might have lying around. We already have the setting exposed, but make it a bit more public and call it out more in the docs. I've actually never tried it myself, but have been meaning to. - I'm really reaching now, and this may be entirely unrealistic :), butsomehow use the fake baremetal driver and expose a mechanism to let the developer specify the already setup undercloud/overcloud environment ahead of time. For example: * Build your undercloud images with the vm element since you won't be PXE booting it * Upload your images to a public cloud, and boot instances for them. * Use this new mechanism when you run devtest (presumably running from another instance in the same cloud) to say I'm using the fake baremetal driver, and here are the IP's of the undercloud instances. * Repeat steps for the overcloud (e.g., configure undercloud to use fake baremetal driver, etc). * Maybe it's not the fake baremetal driver, and instead a new driver that is a noop for the pxe stuff, and the power_on implementation powers on the cloud instances. * Obviously if your aim is to test the pxe and disk deploy process itself, this wouldn't work for you. * Presumably said public cloud is OpenStack, so we've also achieved another layer of On OpenStack. I actually spent quite a while looking into something like this last option when I first started on TripleO, because I had only one big server locally and it was running my OpenStack installation. I was hoping to use it for my TripleO instances, and even went so far as to add support for OpenStack to the virtual power driver in baremetal. I was never completely successful, but I did work through a number of problems: 1. Neutron didn't like allowing the DHCP/PXE traffic to let my seed serve to the undercloud. I was able to get around this by using flat networking with a local bridge on the OpenStack system, but I'm not sure if that's going to be possible on most public cloud providers. There may very well be a less invasive way to configure Neutron to allow that, but I don't know how to do it. 2. Last time I checked, Nova doesn't support PXE booting instances so I had to use iPXE images to do the booting. This doesn't work since we PXE boot every time an instance reboots and the iPXE image gets
Re: [openstack-dev] [TripleO] test environment requirements
On Thu, Mar 13, 2014 at 2:51 AM, Robert Collins robe...@robertcollins.net wrote: So we already have pretty high requirements - its basically a 16G workstation as minimum. Specifically to test the full story: - a seed VM - an undercloud VM (bm deploy infra) - 1 overcloud control VM - 2 overcloud hypervisor VMs 5 VMs with 2+G RAM each. To test the overcloud alone against the seed we save 1 VM, to skip the overcloud we save 3. However, as HA matures we're about to add 4 more VMs: we need a HA control plane for both the under and overclouds: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud control VMs (HA) - 2 overcloud hypervisor VMs 9 VMs with 2+G RAM each == 18GB What should we do about this? A few thoughts to kick start discussion: - use Ironic to test across multiple machines (involves tunnelling brbm across machines, fairly easy) - shrink the VM sizes (causes thrashing) - tell folk to toughen up and get bigger machines (ahahahahaha, no) - make the default configuration inline the hypervisors on the overcloud with the control plane: - a seed VM - 3 undercloud VMs (HA bm deploy infra) - 3 overcloud all-in-one VMs (HA) 7 VMs with 2+G RAM each == 14GB I think its important that we exercise features like HA and live migration regularly by developers, so I'm quite keen to have a fairly solid systematic answer that will let us catch things like bad firewall rules on the control node preventing network tunnelling etc... e.g. we benefit the more things are split out like scale deployments are. OTOH testing the micro-cloud that folk may start with is also a really good idea The idea I was thinking was to make a testenv host available to tripleo atc's. Or, perhaps make it a bit more locked down and only available to a new group of tripleo folk, existing somewhere between the privileges of tripleo atc's and tripleo-cd-admins. We could document how you use the cloud (Red Hat's or HP's) rack to start up a instance to run devtest on one of the compute hosts, request and lock yourself a testenv environment on one of the testenv hosts, etc. Basically, how our CI works. Although I think we'd want different testenv hosts for development vs what runs the CI, and would need to make sure everything was locked down appropriately security-wise. Some other ideas: - Allow an option to get rid of the seed VM, or make it so that you can shut it down after the Undercloud is up. This only really gets rid of 1 VM though, so it doesn't buy you much nor solve any long term problem. - Make it easier to see how you'd use virsh against any libvirt host you might have lying around. We already have the setting exposed, but make it a bit more public and call it out more in the docs. I've actually never tried it myself, but have been meaning to. - I'm really reaching now, and this may be entirely unrealistic :), butsomehow use the fake baremetal driver and expose a mechanism to let the developer specify the already setup undercloud/overcloud environment ahead of time. For example: * Build your undercloud images with the vm element since you won't be PXE booting it * Upload your images to a public cloud, and boot instances for them. * Use this new mechanism when you run devtest (presumably running from another instance in the same cloud) to say I'm using the fake baremetal driver, and here are the IP's of the undercloud instances. * Repeat steps for the overcloud (e.g., configure undercloud to use fake baremetal driver, etc). * Maybe it's not the fake baremetal driver, and instead a new driver that is a noop for the pxe stuff, and the power_on implementation powers on the cloud instances. * Obviously if your aim is to test the pxe and disk deploy process itself, this wouldn't work for you. * Presumably said public cloud is OpenStack, so we've also achieved another layer of On OpenStack. -- -- James Slagle -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
