Public bug reported: Hi all,
I have a Centos 7.1 kilo deployment, using sr-iov ports to my instances. I'm trying to configure NUMA topology and CPU pinning for some telco based workloads. I have 3 compute nodes, I'm trying to use one of them to use cpu pinning. I've configured it like this: Compute Node (total 24 cpus) /etc/nova/nova.conf vcpu_pin_set=2,3,4,5,6,7,8,9,10,11,12,13,14,15,18,19,22,23 Changed grub to isolate my cpus: #grubby --update-kernel=ALL --args="isolcpus=2,3,4,5,6,7,8,9,10,11,12,13,14,15,18,19,22,23" #grub2-install /dev/sda Controller Nodes: /etc/nova/nova.conf scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,PciPassthroughFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter scheduler_available_filters = nova.scheduler.filters.all_filters scheduler_available_filters = nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter Created host aggregate performance #nova aggregate-create performance #nova aggregate-set-metadata 1 pinned=true #nova aggregate-add-host 1 compute03 Created host aggregate normal #nova aggregate-create normal #nova aggregate-set-metadata 2 pinned=false #nova aggregate-add-host 2 compute01 #nova aggregate-add-host 2 compute02 Created the flavor with cpu pinning #nova flavor-create m1.performance 6 2048 20 4 #nova flavor-key 6 set hw:cpu_policy=dedicated #nova flavor-key 6 set aggregate_instance_extra_specs:pinned=true The issue is: With SR-IOV ports it only let's me create instances with 6 vcpus in total with the conf described above. Without SR-IOV, using OVS, I don't have that limitation. Is this a bug or something? I've seen this: https://bugs.launchpad.net/nova/+bug/1441169, however I have the patch, and as I said it works for the first 6 vcpus with my configuration. Some relevant logs: /var/log/nova/nova-scheduler.log 2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Starting with 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:70 2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter RetryFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter AvailabilityZoneFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter RamFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ComputeFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ComputeCapabilitiesFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ImagePropertiesFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ServerGroupAntiAffinityFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ServerGroupAffinityFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.957 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter PciPassthroughFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.959 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter NUMATopologyFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1513808 Title: cannot schedule instances in NUMA topology with more than 6 vcpus using SR-IOV ports Status in OpenStack Compute (nova): New Bug description: Hi all, I have a Centos 7.1 kilo deployment, using sr-iov ports to my instances. I'm trying to configure NUMA topology and CPU pinning for some telco based workloads. I have 3 compute nodes, I'm trying to use one of them to use cpu pinning. I've configured it like this: Compute Node (total 24 cpus) /etc/nova/nova.conf vcpu_pin_set=2,3,4,5,6,7,8,9,10,11,12,13,14,15,18,19,22,23 Changed grub to isolate my cpus: #grubby --update-kernel=ALL --args="isolcpus=2,3,4,5,6,7,8,9,10,11,12,13,14,15,18,19,22,23" #grub2-install /dev/sda Controller Nodes: /etc/nova/nova.conf scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,PciPassthroughFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter scheduler_available_filters = nova.scheduler.filters.all_filters scheduler_available_filters = nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter Created host aggregate performance #nova aggregate-create performance #nova aggregate-set-metadata 1 pinned=true #nova aggregate-add-host 1 compute03 Created host aggregate normal #nova aggregate-create normal #nova aggregate-set-metadata 2 pinned=false #nova aggregate-add-host 2 compute01 #nova aggregate-add-host 2 compute02 Created the flavor with cpu pinning #nova flavor-create m1.performance 6 2048 20 4 #nova flavor-key 6 set hw:cpu_policy=dedicated #nova flavor-key 6 set aggregate_instance_extra_specs:pinned=true The issue is: With SR-IOV ports it only let's me create instances with 6 vcpus in total with the conf described above. Without SR-IOV, using OVS, I don't have that limitation. Is this a bug or something? I've seen this: https://bugs.launchpad.net/nova/+bug/1441169, however I have the patch, and as I said it works for the first 6 vcpus with my configuration. Some relevant logs: /var/log/nova/nova-scheduler.log 2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Starting with 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:70 2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter RetryFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter AvailabilityZoneFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.955 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter RamFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ComputeFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ComputeCapabilitiesFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ImagePropertiesFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ServerGroupAntiAffinityFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.956 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter ServerGroupAffinityFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.957 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter PciPassthroughFilter returned 3 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 2015-11-06 11:18:17.959 59494 DEBUG nova.filters [req-9e20f8a9-384f-45c2-aa99-2d7b3100c98d 9340dc4e70a14aeb82013e5a1631de80 d5ecb0eea96f4996b565fd983a768b11 - - -] Filter NUMATopologyFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:84 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1513808/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
