Re: [openstack-dev] [nova][neutron] numa aware vswitch
On Thu, 2018-08-30 at 00:15 +, Guo, Ruijing wrote: > Hi, Stephen, Sean > > It worked as expected. Good to hear. What were you missing? If there are other gaps in the documentation, I'd be happy to fix them. Stephen > Thanks, > -Ruijing > > -Original Message- > From: Stephen Finucane [mailto:sfinu...@redhat.com] > Sent: Monday, August 27, 2018 5:37 PM > To: OpenStack Development Mailing List (not for usage questions) > > Subject: Re: [openstack-dev] [nova][neutron] numa aware vswitch > > On Mon, 2018-08-27 at 10:24 +0100, Sean Mooney wrote: > > > > > > On Mon 27 Aug 2018, 04:20 Guo, Ruijing, wrote: > > > Hi, Stephen, > > > > > > After setting flavor, VM was created in node 0 (expect in node1). How to > > > debug it? > > > > > > Nova.conf > > > [neutron] > > > physnets = physnet0,physnet1 > > > > > > [neutron_physnet_physnet1] > > > numa_nodes = 1 > > > > Have you enabled the numa topology filter its off by default and without it > > the numa aware vswitch code is disabled. > > Yeah, make sure this is enabled. You should turn on debug-level logging as > this will give you additional information about how things are being > scheduled. Also, is this a new deployment? If not, you're going to need to > upgrade and restart all the nova-* services since there are object changes > which will need to be propagated. > > Stephen > > > > openstack network create net1 --external > > > --provider-network-type=vlan --provider-physical-network=physnet1 > > > --provider-segment=200 ... > > > openstack server create --flavor 1 --image=cirros-0.3.5-x86_64-disk > > > --nic net-id=net1 vm1 > > > > > > > > > 1024 > > > > > > > > > > > > > > > available: 2 nodes (0-1) > > > node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 node 0 size: > > > 64412 MB node 0 free: 47658 MB node 1 cpus: 8 9 10 11 12 13 14 15 24 > > > 25 26 27 28 29 30 31 node 1 size: 64502 MB node 1 free: 44945 MB > > > node distances: > > > node 0 1 > > > 0: 10 21 > > > 1: 21 10 > > > > > > Thanks, > > > -Ruijing > > > > > > -Original Message- > > > From: Stephen Finucane [mailto:sfinu...@redhat.com] > > > Sent: Saturday, August 25, 2018 12:15 AM > > > To: OpenStack Development Mailing List (not for usage questions) > > > > > > Subject: Re: [openstack-dev] [nova][neutron] numa aware vswitch > > > > > > On Fri, 2018-08-24 at 09:13 -0500, Matt Riedemann wrote: > > > > On 8/24/2018 8:58 AM, Stephen Finucane wrote: > > > > > Using this won't add a NUMA topology - it'll just control how > > > > > any topology present will be mapped to the guest. You need to > > > > > enable dedicated CPUs or a explicitly request a NUMA topology > > > > > for this to work. > > > > > > > > > > openstack flavor set --property hw:numa_nodes=1 1 > > > > > > > > > > > > > > > > > > > > openstack flavor set --property hw:cpu_policy=dedicated 1 > > > > > > > > > > > > > > > This is perhaps something that we could change in the future, > > > > > though I haven't given it much thought yet. > > > > > > > > Looks like the admin guide [1] should be updated to at least refer > > > > to the flavor user guide on setting up these types of flavors? > > > > > > > > [1] > > > > https://docs.openstack.org/nova/latest/admin/networking.html#numa- > > > > affi > > > > nity > > > > > > Good idea. > > > > > > https://review.openstack.org/596393 > > > > > > Stephen > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron] numa aware vswitch
Hi, Stephen, Sean It worked as expected. Thanks, -Ruijing -Original Message- From: Stephen Finucane [mailto:sfinu...@redhat.com] Sent: Monday, August 27, 2018 5:37 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova][neutron] numa aware vswitch On Mon, 2018-08-27 at 10:24 +0100, Sean Mooney wrote: > > > On Mon 27 Aug 2018, 04:20 Guo, Ruijing, wrote: > > Hi, Stephen, > > > > After setting flavor, VM was created in node 0 (expect in node1). How to > > debug it? > > > > Nova.conf > > [neutron] > > physnets = physnet0,physnet1 > > > > [neutron_physnet_physnet1] > > numa_nodes = 1 > > Have you enabled the numa topology filter its off by default and without it > the numa aware vswitch code is disabled. Yeah, make sure this is enabled. You should turn on debug-level logging as this will give you additional information about how things are being scheduled. Also, is this a new deployment? If not, you're going to need to upgrade and restart all the nova-* services since there are object changes which will need to be propagated. Stephen > > openstack network create net1 --external > > --provider-network-type=vlan --provider-physical-network=physnet1 > > --provider-segment=200 ... > > openstack server create --flavor 1 --image=cirros-0.3.5-x86_64-disk > > --nic net-id=net1 vm1 > > > > > > 1024 > > > > > > > > > > available: 2 nodes (0-1) > > node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 node 0 size: > > 64412 MB node 0 free: 47658 MB node 1 cpus: 8 9 10 11 12 13 14 15 24 > > 25 26 27 28 29 30 31 node 1 size: 64502 MB node 1 free: 44945 MB > > node distances: > > node 0 1 > > 0: 10 21 > > 1: 21 10 > > > > Thanks, > > -Ruijing > > > > -----Original Message----- > > From: Stephen Finucane [mailto:sfinu...@redhat.com] > > Sent: Saturday, August 25, 2018 12:15 AM > > To: OpenStack Development Mailing List (not for usage questions) > > > > Subject: Re: [openstack-dev] [nova][neutron] numa aware vswitch > > > > On Fri, 2018-08-24 at 09:13 -0500, Matt Riedemann wrote: > > > On 8/24/2018 8:58 AM, Stephen Finucane wrote: > > > > Using this won't add a NUMA topology - it'll just control how > > > > any topology present will be mapped to the guest. You need to > > > > enable dedicated CPUs or a explicitly request a NUMA topology > > > > for this to work. > > > > > > > > openstack flavor set --property hw:numa_nodes=1 1 > > > > > > > > > > > > > > > > openstack flavor set --property hw:cpu_policy=dedicated 1 > > > > > > > > > > > > This is perhaps something that we could change in the future, > > > > though I haven't given it much thought yet. > > > > > > Looks like the admin guide [1] should be updated to at least refer > > > to the flavor user guide on setting up these types of flavors? > > > > > > [1] > > > https://docs.openstack.org/nova/latest/admin/networking.html#numa- > > > affi > > > nity > > > > Good idea. > > > > https://review.openstack.org/596393 > > > > Stephen __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron] numa aware vswitch
On Mon, 2018-08-27 at 10:24 +0100, Sean Mooney wrote: > > > On Mon 27 Aug 2018, 04:20 Guo, Ruijing, wrote: > > Hi, Stephen, > > > > After setting flavor, VM was created in node 0 (expect in node1). How to > > debug it? > > > > Nova.conf > > [neutron] > > physnets = physnet0,physnet1 > > > > [neutron_physnet_physnet1] > > numa_nodes = 1 > > Have you enabled the numa topology filter its off by default and without it > the numa aware vswitch code is disabled. Yeah, make sure this is enabled. You should turn on debug-level logging as this will give you additional information about how things are being scheduled. Also, is this a new deployment? If not, you're going to need to upgrade and restart all the nova-* services since there are object changes which will need to be propagated. Stephen > > openstack network create net1 --external --provider-network-type=vlan > > --provider-physical-network=physnet1 --provider-segment=200 > > ... > > openstack server create --flavor 1 --image=cirros-0.3.5-x86_64-disk --nic > > net-id=net1 vm1 > > > > > > 1024 > > > > > > > > > > available: 2 nodes (0-1) > > node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 > > node 0 size: 64412 MB > > node 0 free: 47658 MB > > node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 > > node 1 size: 64502 MB > > node 1 free: 44945 MB > > node distances: > > node 0 1 > > 0: 10 21 > > 1: 21 10 > > > > Thanks, > > -Ruijing > > > > -Original Message- > > From: Stephen Finucane [mailto:sfinu...@redhat.com] > > Sent: Saturday, August 25, 2018 12:15 AM > > To: OpenStack Development Mailing List (not for usage questions) > > > > Subject: Re: [openstack-dev] [nova][neutron] numa aware vswitch > > > > On Fri, 2018-08-24 at 09:13 -0500, Matt Riedemann wrote: > > > On 8/24/2018 8:58 AM, Stephen Finucane wrote: > > > > Using this won't add a NUMA topology - it'll just control how any > > > > topology present will be mapped to the guest. You need to enable > > > > dedicated CPUs or a explicitly request a NUMA topology for this to > > > > work. > > > > > > > > openstack flavor set --property hw:numa_nodes=1 1 > > > > > > > > > > > > > > > > openstack flavor set --property hw:cpu_policy=dedicated 1 > > > > > > > > > > > > This is perhaps something that we could change in the future, though > > > > I haven't given it much thought yet. > > > > > > Looks like the admin guide [1] should be updated to at least refer to > > > the flavor user guide on setting up these types of flavors? > > > > > > [1] > > > https://docs.openstack.org/nova/latest/admin/networking.html#numa-affi > > > nity > > > > Good idea. > > > > https://review.openstack.org/596393 > > > > Stephen __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron] numa aware vswitch
On Mon 27 Aug 2018, 04:20 Guo, Ruijing, wrote: > Hi, Stephen, > > After setting flavor, VM was created in node 0 (expect in node1). How to > debug it? > > Nova.conf > [neutron] > physnets = physnet0,physnet1 > > [neutron_physnet_physnet1] > numa_nodes = 1 > Have you enabled the numa topology filter its off by default and without it the numa aware vswitch code is disabled. > > openstack network create net1 --external --provider-network-type=vlan > --provider-physical-network=physnet1 --provider-segment=200 > ... > openstack server create --flavor 1 --image=cirros-0.3.5-x86_64-disk --nic > net-id=net1 vm1 > > > 1024 > > > > > available: 2 nodes (0-1) > node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 > node 0 size: 64412 MB > node 0 free: 47658 MB > node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 > node 1 size: 64502 MB > node 1 free: 44945 MB > node distances: > node 0 1 > 0: 10 21 > 1: 21 10 > > Thanks, > -Ruijing > > -Original Message- > From: Stephen Finucane [mailto:sfinu...@redhat.com] > Sent: Saturday, August 25, 2018 12:15 AM > To: OpenStack Development Mailing List (not for usage questions) < > openstack-dev@lists.openstack.org> > Subject: Re: [openstack-dev] [nova][neutron] numa aware vswitch > > On Fri, 2018-08-24 at 09:13 -0500, Matt Riedemann wrote: > > On 8/24/2018 8:58 AM, Stephen Finucane wrote: > > > Using this won't add a NUMA topology - it'll just control how any > > > topology present will be mapped to the guest. You need to enable > > > dedicated CPUs or a explicitly request a NUMA topology for this to > > > work. > > > > > > openstack flavor set --property hw:numa_nodes=1 1 > > > > > > > > > > > > openstack flavor set --property hw:cpu_policy=dedicated 1 > > > > > > > > > This is perhaps something that we could change in the future, though > > > I haven't given it much thought yet. > > > > Looks like the admin guide [1] should be updated to at least refer to > > the flavor user guide on setting up these types of flavors? > > > > [1] > > https://docs.openstack.org/nova/latest/admin/networking.html#numa-affi > > nity > > Good idea. > > https://review.openstack.org/596393 > > Stephen > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron] numa aware vswitch
Hi, Stephen, After setting flavor, VM was created in node 0 (expect in node1). How to debug it? Nova.conf [neutron] physnets = physnet0,physnet1 [neutron_physnet_physnet1] numa_nodes = 1 openstack network create net1 --external --provider-network-type=vlan --provider-physical-network=physnet1 --provider-segment=200 ... openstack server create --flavor 1 --image=cirros-0.3.5-x86_64-disk --nic net-id=net1 vm1 1024 available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 node 0 size: 64412 MB node 0 free: 47658 MB node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 node 1 size: 64502 MB node 1 free: 44945 MB node distances: node 0 1 0: 10 21 1: 21 10 Thanks, -Ruijing -Original Message- From: Stephen Finucane [mailto:sfinu...@redhat.com] Sent: Saturday, August 25, 2018 12:15 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova][neutron] numa aware vswitch On Fri, 2018-08-24 at 09:13 -0500, Matt Riedemann wrote: > On 8/24/2018 8:58 AM, Stephen Finucane wrote: > > Using this won't add a NUMA topology - it'll just control how any > > topology present will be mapped to the guest. You need to enable > > dedicated CPUs or a explicitly request a NUMA topology for this to > > work. > > > > openstack flavor set --property hw:numa_nodes=1 1 > > > > > > > > openstack flavor set --property hw:cpu_policy=dedicated 1 > > > > > > This is perhaps something that we could change in the future, though > > I haven't given it much thought yet. > > Looks like the admin guide [1] should be updated to at least refer to > the flavor user guide on setting up these types of flavors? > > [1] > https://docs.openstack.org/nova/latest/admin/networking.html#numa-affi > nity Good idea. https://review.openstack.org/596393 Stephen __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron] numa aware vswitch
On Fri, 2018-08-24 at 09:13 -0500, Matt Riedemann wrote: > On 8/24/2018 8:58 AM, Stephen Finucane wrote: > > Using this won't add a NUMA topology - it'll just control how any > > topology present will be mapped to the guest. You need to enable > > dedicated CPUs or a explicitly request a NUMA topology for this to > > work. > > > > openstack flavor set --property hw:numa_nodes=1 1 > > > > > > > > openstack flavor set --property hw:cpu_policy=dedicated 1 > > > > > > This is perhaps something that we could change in the future, though I > > haven't given it much thought yet. > > Looks like the admin guide [1] should be updated to at least refer to > the flavor user guide on setting up these types of flavors? > > [1] https://docs.openstack.org/nova/latest/admin/networking.html#numa-affinity Good idea. https://review.openstack.org/596393 Stephen __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron] numa aware vswitch
On 8/24/2018 8:58 AM, Stephen Finucane wrote: Using this won't add a NUMA topology - it'll just control how any topology present will be mapped to the guest. You need to enable dedicated CPUs or a explicitly request a NUMA topology for this to work. openstack flavor set --property hw:numa_nodes=1 1 openstack flavor set --property hw:cpu_policy=dedicated 1 This is perhaps something that we could change in the future, though I haven't given it much thought yet. Looks like the admin guide [1] should be updated to at least refer to the flavor user guide on setting up these types of flavors? [1] https://docs.openstack.org/nova/latest/admin/networking.html#numa-affinity -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron] numa aware vswitch
On Fri, 2018-08-24 at 07:55 +, Guo, Ruijing wrote: > Hi, All, > > I am verifying numa aware vwitch features > (https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/numa-aware-vswitches.html). > But the result is not my expectation. > > What I missing? > > > Nova configuration: > > [filter_scheduler] > track_instance_changes = False > enabled_filters = > RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter,DifferentHostFilter,NUMATopologyFilter > > [neutron] > physnets = physnet0,physnet1 > > [neutron_physnet_physnet0] > numa_nodes = 0 > > [neutron_physnet_physnet1] > numa_nodes = 1 > > > ml2 configuration: > > [ml2_type_vlan] > network_vlan_ranges = physnet0,physnet1 > [ovs] > vhostuser_socket_dir = /var/lib/libvirt/qemu > bridge_mappings = physnet0:br-physnet0,physnet1:br-physnet1 > > > command list: > > openstack network create net0 --external --provider-network-type=vlan > --provider-physical-network=physnet0 --provider-segment=100 > openstack network create net1 --external --provider-network-type=vlan > --provider-physical-network=physnet1 --provider-segment=200 > openstack subnet create --network=net0 --subnet-range=192.168.1.0/24 > --allocation-pool start=192.168.1.200,end=192.168.1.250 --gateway 192.168.1.1 > subnet0 > openstack subnet create --network=net1 --subnet-range=192.168.2.0/24 > --allocation-pool start=192.168.2.200,end=192.168.2.250 --gateway 192.168.2.1 > subnet1 > openstack server create --flavor 1 --image=cirros-0.3.5-x86_64-disk --nic > net-id=net0 vm0 > openstack server create --flavor 1 --image=cirros-0.3.5-x86_64-disk --nic > net-id=net1 vm1 > > vm0 and vm1 are created but numa is not enabled: > 1 > > 1024 > Using this won't add a NUMA topology - it'll just control how any topology present will be mapped to the guest. You need to enable dedicated CPUs or a explicitly request a NUMA topology for this to work. openstack flavor set --property hw:numa_nodes=1 1 openstack flavor set --property hw:cpu_policy=dedicated 1 This is perhaps something that we could change in the future, though I haven't given it much thought yet. Regards, Stephen __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova][neutron] numa aware vswitch
Hi, All, I am verifying numa aware vwitch features (https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/numa-aware-vswitches.html). But the result is not my expectation. What I missing? Nova configuration: [filter_scheduler] track_instance_changes = False enabled_filters = RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter,DifferentHostFilter,NUMATopologyFilter [neutron] physnets = physnet0,physnet1 [neutron_physnet_physnet0] numa_nodes = 0 [neutron_physnet_physnet1] numa_nodes = 1 ml2 configuration: [ml2_type_vlan] network_vlan_ranges = physnet0,physnet1 [ovs] vhostuser_socket_dir = /var/lib/libvirt/qemu bridge_mappings = physnet0:br-physnet0,physnet1:br-physnet1 command list: openstack network create net0 --external --provider-network-type=vlan --provider-physical-network=physnet0 --provider-segment=100 openstack network create net1 --external --provider-network-type=vlan --provider-physical-network=physnet1 --provider-segment=200 openstack subnet create --network=net0 --subnet-range=192.168.1.0/24 --allocation-pool start=192.168.1.200,end=192.168.1.250 --gateway 192.168.1.1 subnet0 openstack subnet create --network=net1 --subnet-range=192.168.2.0/24 --allocation-pool start=192.168.2.200,end=192.168.2.250 --gateway 192.168.2.1 subnet1 openstack server create --flavor 1 --image=cirros-0.3.5-x86_64-disk --nic net-id=net0 vm0 openstack server create --flavor 1 --image=cirros-0.3.5-x86_64-disk --nic net-id=net1 vm1 vm0 and vm1 are created but numa is not enabled: 1 1024 Thanks, -Ruijing __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
