Re: [openstack-dev] [infra][nova] Running NFV tests in CI
On 24 July 2018 at 19:47, Clark Boylan wrote: > > On Tue, Jul 24, 2018, at 10:21 AM, Artom Lifshitz wrote: > > On Tue, Jul 24, 2018 at 12:30 PM, Clark Boylan wrote: > > > On Tue, Jul 24, 2018, at 9:23 AM, Artom Lifshitz wrote: > > >> Hey all, > > >> > > >> tl;dr Humbly requesting a handful of nodes to run NFV tests in CI > > >> > > >> Intel has their NFV tests tempest plugin [1] and manages a third party > > >> CI for Nova. Two of the cores on that project (Stephen Finucane and > > >> Sean Mooney) have now moved to Red Hat, but the point still stands > > >> that there's a need and a use case for testing things like NUMA > > >> topologies, CPU pinning and hugepages. > > >> > > >> At Red Hat, we also have a similar tempest plugin project [2] that we > > >> use for downstream whitebox testing. The scope is a bit bigger than > > >> just NFV, but the main use case is still testing NFV code in an > > >> automated way. > > >> > > >> Given that there's a clear need for this sort of whitebox testing, I > > >> would like to humbly request a handful of nodes (in the 3 to 5 range) > > >> from infra to run an "official" Nova NFV CI. The code doing the > > >> testing would initially be the current Intel plugin, bug we could have > > >> a separate discussion about keeping "Intel" in the name or forking > > >> and/or renaming it to something more vendor-neutral. > > > > > > The way you request nodes from Infra is through your Zuul configuration. > > > Add jobs to a project to run tests on the node labels that you want. > > > > Aha, thanks, I'll look into that. I was coming from a place of > > complete ignorance about infra. > > > > > > I'm guessing this process doesn't work for NFV tests because you have > > > specific hardware requirements that are not met by our current VM > > > resources? > > > If that is the case it would probably be best to start by documenting > > > what is required and where the existing VM resources fall > > > short. > > > > Well, it should be possible to do most of what we'd like with nested > > virt and virtual NUMA topologies, though things like hugepages will > > need host configuration, specifically the kernel boot command [1]. Is > > that possible with the nodes we have? > > https://docs.openstack.org/infra/manual/testing.html attempts to give you an > idea for what is currently available via the test environments. > > > Nested virt has historically been painful because not all clouds support it > and those that do did not do so in a reliable way (VMs and possibly > hypervisors would crash). This has gotten better recently as nested virt is > something more people have an interest in >getting working but it is still > hit and miss particularly as you use newer kernels in guests. I think if we > can continue to work together with our clouds (thank you limestone, OVH, and > vexxhost!) we may be able to work out nested virt that is redundant across > multiple >clouds. We will likely need individuals willing to keep caring for > that though and debug problems when the next release of your favorite distro > shows up. Can you get by with qemu or is nested virt required? for what its worth the intel nfv ci has alway ran with nested virt since we first set it up on ubuntu 12.04 all the way through the time we ran it fedora 20- fedora 21 and it continue to use nested virt on ubuntu 16.04 we have never had any issue with nested virt but the key to using it correctly is you should always set the nova cpu mode to host-passthrough if you use nested virt. because of how we currently do cpu pinning/ hugepanges and numa affinity in nova today to do this testign we have a hard requiremetn on running kvm in devstack which mean we have a hard requirement for nested virt. there ware ways around that but the nova core team has previously express there view that adding the code changes reqiured to allow the use of qemu is not warrented for ci since we would also not be testing the normal config e.g. these feature are normaly only used when performance matters whcih means you will be useing kvm not qemu. i have tried to test ovs-dpdk in the upstream ci on 3 ocation in the past (this being the most recent https://review.openstack.org/#/c/433491/) but without nested virt that didnt get very far. > > As for hugepages, I've done a quick survey of cpuinfo across our clouds and > all seem to have pse available but not all have pdpe1gb available. Are you > using 1GB hugepages? Keep in mind that the test VMs only have 8GB of memory > total. As for booting with special kernel parameters you can have your job > make those modifications to the test environment then reboot the test > environment within the job. There is some Zuul specific housekeeping that > needs to be done post reboot, we can figure that out if we decide to go down > this route. Would your setup work with 2M hugepages? the host vm does not need to be backed by hungepages at all. it does need to have the cpuflags set to allow
Re: [openstack-dev] [infra][nova] Running NFV tests in CI
On 07/24/2018 12:47 PM, Clark Boylan wrote: Can you get by with qemu or is nested virt required? Pretty sure that nested virt is needed in order to test CPU pinning. As for hugepages, I've done a quick survey of cpuinfo across our clouds and all seem to have pse available but not all have pdpe1gb available. Are you using 1GB hugepages? If we want to test nova's handling of 1G hugepages then I think we'd need pdpe1gb. Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra][nova] Running NFV tests in CI
On Tue, Jul 24, 2018, at 10:21 AM, Artom Lifshitz wrote: > On Tue, Jul 24, 2018 at 12:30 PM, Clark Boylan wrote: > > On Tue, Jul 24, 2018, at 9:23 AM, Artom Lifshitz wrote: > >> Hey all, > >> > >> tl;dr Humbly requesting a handful of nodes to run NFV tests in CI > >> > >> Intel has their NFV tests tempest plugin [1] and manages a third party > >> CI for Nova. Two of the cores on that project (Stephen Finucane and > >> Sean Mooney) have now moved to Red Hat, but the point still stands > >> that there's a need and a use case for testing things like NUMA > >> topologies, CPU pinning and hugepages. > >> > >> At Red Hat, we also have a similar tempest plugin project [2] that we > >> use for downstream whitebox testing. The scope is a bit bigger than > >> just NFV, but the main use case is still testing NFV code in an > >> automated way. > >> > >> Given that there's a clear need for this sort of whitebox testing, I > >> would like to humbly request a handful of nodes (in the 3 to 5 range) > >> from infra to run an "official" Nova NFV CI. The code doing the > >> testing would initially be the current Intel plugin, bug we could have > >> a separate discussion about keeping "Intel" in the name or forking > >> and/or renaming it to something more vendor-neutral. > > > > The way you request nodes from Infra is through your Zuul configuration. > > Add jobs to a project to run tests on the node labels that you want. > > Aha, thanks, I'll look into that. I was coming from a place of > complete ignorance about infra. > > > > I'm guessing this process doesn't work for NFV tests because you have > > specific hardware requirements that are not met by our current VM resources? > > If that is the case it would probably be best to start by documenting what > > is required and where the existing VM resources fall > > short. > > Well, it should be possible to do most of what we'd like with nested > virt and virtual NUMA topologies, though things like hugepages will > need host configuration, specifically the kernel boot command [1]. Is > that possible with the nodes we have? https://docs.openstack.org/infra/manual/testing.html attempts to give you an idea for what is currently available via the test environments. Nested virt has historically been painful because not all clouds support it and those that do did not do so in a reliable way (VMs and possibly hypervisors would crash). This has gotten better recently as nested virt is something more people have an interest in getting working but it is still hit and miss particularly as you use newer kernels in guests. I think if we can continue to work together with our clouds (thank you limestone, OVH, and vexxhost!) we may be able to work out nested virt that is redundant across multiple clouds. We will likely need individuals willing to keep caring for that though and debug problems when the next release of your favorite distro shows up. Can you get by with qemu or is nested virt required? As for hugepages, I've done a quick survey of cpuinfo across our clouds and all seem to have pse available but not all have pdpe1gb available. Are you using 1GB hugepages? Keep in mind that the test VMs only have 8GB of memory total. As for booting with special kernel parameters you can have your job make those modifications to the test environment then reboot the test environment within the job. There is some Zuul specific housekeeping that needs to be done post reboot, we can figure that out if we decide to go down this route. Would your setup work with 2M hugepages? > > > In general though we operate on top of donated cloud resources, and if > > those do not work we will have to identify a source of resources that would > > work. > > Right, as always it comes down to resources and money. I believe > historically Red Hat has been opposed to running an upstream third > party CI (this is by no means an official Red Hat position, just > remembering what I think I heard), but I can always see what I can do. > > [1] > https://docs.openstack.org/nova/latest/admin/huge-pages.html#enabling-huge-pages-on-the-host __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra][nova] Running NFV tests in CI
On Tue, Jul 24, 2018 at 12:30 PM, Clark Boylan wrote: > On Tue, Jul 24, 2018, at 9:23 AM, Artom Lifshitz wrote: >> Hey all, >> >> tl;dr Humbly requesting a handful of nodes to run NFV tests in CI >> >> Intel has their NFV tests tempest plugin [1] and manages a third party >> CI for Nova. Two of the cores on that project (Stephen Finucane and >> Sean Mooney) have now moved to Red Hat, but the point still stands >> that there's a need and a use case for testing things like NUMA >> topologies, CPU pinning and hugepages. >> >> At Red Hat, we also have a similar tempest plugin project [2] that we >> use for downstream whitebox testing. The scope is a bit bigger than >> just NFV, but the main use case is still testing NFV code in an >> automated way. >> >> Given that there's a clear need for this sort of whitebox testing, I >> would like to humbly request a handful of nodes (in the 3 to 5 range) >> from infra to run an "official" Nova NFV CI. The code doing the >> testing would initially be the current Intel plugin, bug we could have >> a separate discussion about keeping "Intel" in the name or forking >> and/or renaming it to something more vendor-neutral. > > The way you request nodes from Infra is through your Zuul configuration. Add > jobs to a project to run tests on the node labels that you want. Aha, thanks, I'll look into that. I was coming from a place of complete ignorance about infra. > > I'm guessing this process doesn't work for NFV tests because you have > specific hardware requirements that are not met by our current VM resources? > If that is the case it would probably be best to start by documenting what is > required and where the existing VM resources fall > short. Well, it should be possible to do most of what we'd like with nested virt and virtual NUMA topologies, though things like hugepages will need host configuration, specifically the kernel boot command [1]. Is that possible with the nodes we have? > In general though we operate on top of donated cloud resources, and if those > do not work we will have to identify a source of resources that would work. Right, as always it comes down to resources and money. I believe historically Red Hat has been opposed to running an upstream third party CI (this is by no means an official Red Hat position, just remembering what I think I heard), but I can always see what I can do. [1] https://docs.openstack.org/nova/latest/admin/huge-pages.html#enabling-huge-pages-on-the-host __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra][nova] Running NFV tests in CI
On Tue, Jul 24, 2018, at 9:23 AM, Artom Lifshitz wrote: > Hey all, > > tl;dr Humbly requesting a handful of nodes to run NFV tests in CI > > Intel has their NFV tests tempest plugin [1] and manages a third party > CI for Nova. Two of the cores on that project (Stephen Finucane and > Sean Mooney) have now moved to Red Hat, but the point still stands > that there's a need and a use case for testing things like NUMA > topologies, CPU pinning and hugepages. > > At Red Hat, we also have a similar tempest plugin project [2] that we > use for downstream whitebox testing. The scope is a bit bigger than > just NFV, but the main use case is still testing NFV code in an > automated way. > > Given that there's a clear need for this sort of whitebox testing, I > would like to humbly request a handful of nodes (in the 3 to 5 range) > from infra to run an "official" Nova NFV CI. The code doing the > testing would initially be the current Intel plugin, bug we could have > a separate discussion about keeping "Intel" in the name or forking > and/or renaming it to something more vendor-neutral. The way you request nodes from Infra is through your Zuul configuration. Add jobs to a project to run tests on the node labels that you want. I'm guessing this process doesn't work for NFV tests because you have specific hardware requirements that are not met by our current VM resources? If that is the case it would probably be best to start by documenting what is required and where the existing VM resources fall short. In general though we operate on top of donated cloud resources, and if those do not work we will have to identify a source of resources that would work. > > I won't be at PTG (conflict with personal travel), so I'm kindly > asking Stephen and Sean to represent this idea in Denver. > > Cheers! > > [1] https://github.com/openstack/intel-nfv-ci-tests > [2] > https://review.rdoproject.org/r/#/admin/projects/openstack/whitebox-tempest-plugin __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [infra][nova] Running NFV tests in CI
Hey all, tl;dr Humbly requesting a handful of nodes to run NFV tests in CI Intel has their NFV tests tempest plugin [1] and manages a third party CI for Nova. Two of the cores on that project (Stephen Finucane and Sean Mooney) have now moved to Red Hat, but the point still stands that there's a need and a use case for testing things like NUMA topologies, CPU pinning and hugepages. At Red Hat, we also have a similar tempest plugin project [2] that we use for downstream whitebox testing. The scope is a bit bigger than just NFV, but the main use case is still testing NFV code in an automated way. Given that there's a clear need for this sort of whitebox testing, I would like to humbly request a handful of nodes (in the 3 to 5 range) from infra to run an "official" Nova NFV CI. The code doing the testing would initially be the current Intel plugin, bug we could have a separate discussion about keeping "Intel" in the name or forking and/or renaming it to something more vendor-neutral. I won't be at PTG (conflict with personal travel), so I'm kindly asking Stephen and Sean to represent this idea in Denver. Cheers! [1] https://github.com/openstack/intel-nfv-ci-tests [2] https://review.rdoproject.org/r/#/admin/projects/openstack/whitebox-tempest-plugin __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
