Re: [openstack-dev] problems with huge pages and libvirt
On Tue, Feb 03, 2015 at 03:05:24PM +0100, Sahid Orentino Ferdjaoui wrote: > On Mon, Feb 02, 2015 at 11:44:37AM -0600, Chris Friesen wrote: > > On 02/02/2015 11:00 AM, Sahid Orentino Ferdjaoui wrote: > > >On Mon, Feb 02, 2015 at 10:44:09AM -0600, Chris Friesen wrote: > > >>Hi, > > >> > > >>I'm trying to make use of huge pages as described in > > >>"http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-large-pages.html";. > > >>I'm running kilo as of Jan 27th. > > >>I've allocated 1 2MB pages on a compute node. "virsh capabilities" > > >>on that node contains: > > >> > > >> > > >> > > >> > > >> 67028244 > > >> 16032069 > > >> 5000 > > >> 1 > > >>... > > >> > > >> 67108864 > > >> 16052224 > > >> 5000 > > >> 1 > > >> > > >> > > >>I then restarted nova-compute, I set "hw:mem_page_size=large" on a > > >>flavor, and then tried to boot up an instance with that flavor. I > > >>got the error logs below in nova-scheduler. Is this a bug? > > > > > >Hello, > > > > > >Launchpad.net could be more appropriate to > > >discuss on something which looks like a bug. > > > > > > https://bugs.launchpad.net/nova/+filebug > > > > Just wanted to make sure I wasn't missing something. Bug has been opened at > > https://bugs.launchpad.net/nova/+bug/1417201 > > > > I added some additional logs to the bug report of what the numa topology > > looks like on the compute node and in NUMATopologyFilter.host_passes(). > > > > >According to your trace I would say you are running different versions > > >of Nova services. > > > > nova should all be the same version. I'm running juno versions of other > > openstack components though. > > Hum if I understand well and according your issue reported to > launchpad.net > > https://bugs.launchpad.net/nova/+bug/1417201 > > You are trying to test hugepages under kilo which it is not possible > since it has been implemented in this release (Juno, not yet > published) Please ignore this point. > I have tried to reproduce your issue with trunk but I have not been > able to do it. Please reopen the bug with more information of your env > if still present. I should received any notification from it. > > Thanks, > s. > > > >BTW please verify your version of libvirt. Hugepages is supported > > >start to 1.2.8 (but this should difinitly not failed so badly like > > >that) > > > > Libvirt is 1.2.8. > > Chris > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] problems with huge pages and libvirt
On Mon, Feb 02, 2015 at 11:44:37AM -0600, Chris Friesen wrote: > On 02/02/2015 11:00 AM, Sahid Orentino Ferdjaoui wrote: > >On Mon, Feb 02, 2015 at 10:44:09AM -0600, Chris Friesen wrote: > >>Hi, > >> > >>I'm trying to make use of huge pages as described in > >>"http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-large-pages.html";. > >>I'm running kilo as of Jan 27th. > >>I've allocated 1 2MB pages on a compute node. "virsh capabilities" on > >>that node contains: > >> > >> > >> > >> > >> 67028244 > >> 16032069 > >> 5000 > >> 1 > >>... > >> > >> 67108864 > >> 16052224 > >> 5000 > >> 1 > >> > >> > >>I then restarted nova-compute, I set "hw:mem_page_size=large" on a > >>flavor, and then tried to boot up an instance with that flavor. I > >>got the error logs below in nova-scheduler. Is this a bug? > > > >Hello, > > > >Launchpad.net could be more appropriate to > >discuss on something which looks like a bug. > > > > https://bugs.launchpad.net/nova/+filebug > > Just wanted to make sure I wasn't missing something. Bug has been opened at > https://bugs.launchpad.net/nova/+bug/1417201 > > I added some additional logs to the bug report of what the numa topology > looks like on the compute node and in NUMATopologyFilter.host_passes(). > > >According to your trace I would say you are running different versions > >of Nova services. > > nova should all be the same version. I'm running juno versions of other > openstack components though. Hum if I understand well and according your issue reported to launchpad.net https://bugs.launchpad.net/nova/+bug/1417201 You are trying to test hugepages under kilo which it is not possible since it has been implemented in this release (Juno, not yet published) I have tried to reproduce your issue with trunk but I have not been able to do it. Please reopen the bug with more information of your env if still present. I should received any notification from it. Thanks, s. > >BTW please verify your version of libvirt. Hugepages is supported > >start to 1.2.8 (but this should difinitly not failed so badly like > >that) > > Libvirt is 1.2.8. > Chris > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] problems with huge pages and libvirt
- Original Message - > From: "Ian Wells" > To: "OpenStack Development Mailing List (not for usage questions)" > > > On 2 February 2015 at 09:49, Chris Friesen > wrote: > > > On 02/02/2015 10:51 AM, Jay Pipes wrote: > > > >> This is a bug that I discovered when fixing some of the NUMA related nova > >> objects. I have a patch that should fix it up shortly. > >> > > > > Any chance you could point me at it or send it to me? > > > > This is what happens when we don't have any functional testing of stuff > >> that is > >> merged into master... > >> > > > > Indeed. Does tempest support hugepages/NUMA/pinning? > > > > This is a running discussion, but largely no - because this is ited to the > capabilities of the host, there's no guarantee for a given scenario what > result you would get (because Tempest will run on any hardware). > > If you have test cases that should pass or fail on a NUMA-capable node, can > you write them up? We're working on NUMA-specific testing right now > (though I'm not sure who, specifically, is working on the test case side of > that). Vladik and Sean (CC'd) are working on these. Thanks, Steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] problems with huge pages and libvirt
On 02/02/2015 12:13 PM, Ian Wells wrote: On 2 February 2015 at 09:49, Chris Friesen Indeed. Does tempest support hugepages/NUMA/pinning? This is a running discussion, but largely no - because this is ited to the capabilities of the host, there's no guarantee for a given scenario what result you would get (because Tempest will run on any hardware). If you have test cases that should pass or fail on a NUMA-capable node, can you write them up? We're working on NUMA-specific testing right now (though I'm not sure who, specifically, is working on the test case side of that). I don't really have time to write up individual testcases right now, but I think a good start would be to test the following features: http://specs.openstack.org/openstack/nova-specs/specs/juno/implemented/virt-driver-numa-placement.html http://specs.openstack.org/openstack/nova-specs/specs/juno/implemented/virt-driver-vcpu-topology.html http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-large-pages.html http://specs.openstack.org/openstack/nova-specs/specs/kilo/approved/input-output-based-numa-scheduling.html http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-cpu-pinning.html Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] problems with huge pages and libvirt
On 2 February 2015 at 09:49, Chris Friesen wrote: > On 02/02/2015 10:51 AM, Jay Pipes wrote: > >> This is a bug that I discovered when fixing some of the NUMA related nova >> objects. I have a patch that should fix it up shortly. >> > > Any chance you could point me at it or send it to me? > > This is what happens when we don't have any functional testing of stuff >> that is >> merged into master... >> > > Indeed. Does tempest support hugepages/NUMA/pinning? > This is a running discussion, but largely no - because this is ited to the capabilities of the host, there's no guarantee for a given scenario what result you would get (because Tempest will run on any hardware). If you have test cases that should pass or fail on a NUMA-capable node, can you write them up? We're working on NUMA-specific testing right now (though I'm not sure who, specifically, is working on the test case side of that). __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] problems with huge pages and libvirt
On Mon, Feb 02, 2015 at 11:49:26AM -0600, Chris Friesen wrote: > On 02/02/2015 10:51 AM, Jay Pipes wrote: > >This is a bug that I discovered when fixing some of the NUMA related nova > >objects. I have a patch that should fix it up shortly. > > Any chance you could point me at it or send it to me? > > >This is what happens when we don't have any functional testing of stuff that > >is > >merged into master... > > Indeed. Does tempest support hugepages/NUMA/pinning? The short answer is not explicitly. The longer answer is that there are 2 patches[1][2] up for review right now that add basic checks to tempest. But, they haven't been able to merge because the nova support hasn't worked and the tests fail... Aside from those 2 basic checks I don't expect any other direct numa, hugepage, etc. tests to be in tempest. Testing anything besides these basic cases would require knowledge of the underlying hardware for the deployment, which is out of scope for tempest. There really needs to be lower level functional testing of these features. That being said the other thing you could do using tempest is to configure tempest to use flavors which are created to use numa. That would at least implicitly test that the functionality would work. But, that really isn't a replacement for the functional testing which is sorely needed here. [1] https://review.openstack.org/143540 [2] https://review.openstack.org/#/c/143541/ -Matt Treinish pgpYTjxjBs7qI.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] problems with huge pages and libvirt
On 02/02/2015 10:51 AM, Jay Pipes wrote: This is a bug that I discovered when fixing some of the NUMA related nova objects. I have a patch that should fix it up shortly. Any chance you could point me at it or send it to me? This is what happens when we don't have any functional testing of stuff that is merged into master... Indeed. Does tempest support hugepages/NUMA/pinning? Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] problems with huge pages and libvirt
On 02/02/2015 11:00 AM, Sahid Orentino Ferdjaoui wrote: On Mon, Feb 02, 2015 at 10:44:09AM -0600, Chris Friesen wrote: Hi, I'm trying to make use of huge pages as described in "http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-large-pages.html";. I'm running kilo as of Jan 27th. I've allocated 1 2MB pages on a compute node. "virsh capabilities" on that node contains: 67028244 16032069 5000 1 ... 67108864 16052224 5000 1 I then restarted nova-compute, I set "hw:mem_page_size=large" on a flavor, and then tried to boot up an instance with that flavor. I got the error logs below in nova-scheduler. Is this a bug? Hello, Launchpad.net could be more appropriate to discuss on something which looks like a bug. https://bugs.launchpad.net/nova/+filebug Just wanted to make sure I wasn't missing something. Bug has been opened at https://bugs.launchpad.net/nova/+bug/1417201 I added some additional logs to the bug report of what the numa topology looks like on the compute node and in NUMATopologyFilter.host_passes(). According to your trace I would say you are running different versions of Nova services. nova should all be the same version. I'm running juno versions of other openstack components though. BTW please verify your version of libvirt. Hugepages is supported start to 1.2.8 (but this should difinitly not failed so badly like that) Libvirt is 1.2.8. Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] problems with huge pages and libvirt
On Mon, Feb 02, 2015 at 11:51:47AM -0500, Jay Pipes wrote: > This is a bug that I discovered when fixing some of the NUMA related nova > objects. I have a patch that should fix it up shortly. Never seen this issue, could be great to have a bug repported. > This is what happens when we don't have any functional testing of stuff that > is merged into master... > Best, > -jay Thanks, s. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] problems with huge pages and libvirt
On Mon, Feb 02, 2015 at 10:44:09AM -0600, Chris Friesen wrote: > Hi, > > I'm trying to make use of huge pages as described in > "http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-large-pages.html";. > I'm running kilo as of Jan 27th. > I've allocated 1 2MB pages on a compute node. "virsh capabilities" on > that node contains: > > > > > 67028244 > 16032069 > 5000 > 1 > ... > > 67108864 > 16052224 > 5000 > 1 > > > I then restarted nova-compute, I set "hw:mem_page_size=large" on a > flavor, and then tried to boot up an instance with that flavor. I > got the error logs below in nova-scheduler. Is this a bug? Hello, Launchpad.net could be more appropriate to discuss on something which looks like a bug. https://bugs.launchpad.net/nova/+filebug According to your trace I would say you are running different versions of Nova services. BTW please verify your version of libvirt. Hugepages is supported start to 1.2.8 (but this should difinitly not failed so badly like that) s. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] problems with huge pages and libvirt
This is a bug that I discovered when fixing some of the NUMA related nova objects. I have a patch that should fix it up shortly. This is what happens when we don't have any functional testing of stuff that is merged into master... Best, -jay On 02/02/2015 11:44 AM, Chris Friesen wrote: Hi, I'm trying to make use of huge pages as described in "http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-large-pages.html";. I'm running kilo as of Jan 27th. I've allocated 1 2MB pages on a compute node. "virsh capabilities" on that node contains: 67028244 16032069 5000 1 ... 67108864 16052224 5000 1 I then restarted nova-compute, I set "hw:mem_page_size=large" on a flavor, and then tried to boot up an instance with that flavor. I got the error logs below in nova-scheduler. Is this a bug? Feb 2 16:23:10 controller-0 nova-scheduler Exception during message handling: Cannot load 'mempages' in the base class 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args) 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher return func(*args, **kwargs) 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/scheduler/manager.py", line 86, in select_destinations 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher filter_properties) 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 67, in select_destinations 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher filter_properties) 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 138, in _schedule 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher filter_properties, index=num) 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/scheduler/host_manager.py", line 391, in get_filtered_hosts 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher hosts, filter_properties, index) 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/filters.py", line 77, in get_filtered_objects 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher list_objs = list(objs) 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/filters.py", line 43, in filter_all 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher if self._filter_one(obj, filter_properties): 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/scheduler/filters/__init__.py", line 27, in _filter_one 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher return self.host_passes(obj, filter_properties) 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/scheduler/filters/numa_topology_filter.py", line 45, in host_passes 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher limits_topology=limits)) 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/virt/hardware.py", line 1161, in numa_fit_instance_to_host 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher host_cell, instance_cell, limit_cell) 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/virt/hardware.py", line 851, in _numa_fit_instance_cell 2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.
