Re: [openstack-dev] problems with huge pages and libvirt

2015-02-03 Thread Sahid Orentino Ferdjaoui
On Tue, Feb 03, 2015 at 03:05:24PM +0100, Sahid Orentino Ferdjaoui wrote:
 On Mon, Feb 02, 2015 at 11:44:37AM -0600, Chris Friesen wrote:
  On 02/02/2015 11:00 AM, Sahid Orentino Ferdjaoui wrote:
  On Mon, Feb 02, 2015 at 10:44:09AM -0600, Chris Friesen wrote:
  Hi,
  
  I'm trying to make use of huge pages as described in
  http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-large-pages.html;.
  I'm running kilo as of Jan 27th.
  I've allocated 1 2MB pages on a compute node.  virsh capabilities 
  on that node contains:
  
   topology
 cells num='2'
   cell id='0'
 memory unit='KiB'67028244/memory
 pages unit='KiB' size='4'16032069/pages
 pages unit='KiB' size='2048'5000/pages
 pages unit='KiB' size='1048576'1/pages
  ...
   cell id='1'
 memory unit='KiB'67108864/memory
 pages unit='KiB' size='4'16052224/pages
 pages unit='KiB' size='2048'5000/pages
 pages unit='KiB' size='1048576'1/pages
  
  
  I then restarted nova-compute, I set hw:mem_page_size=large on a
  flavor, and then tried to boot up an instance with that flavor.  I
  got the error logs below in nova-scheduler.  Is this a bug?
  
  Hello,
  
  Launchpad.net could be more appropriate to
  discuss on something which looks like a bug.
  
 https://bugs.launchpad.net/nova/+filebug
  
  Just wanted to make sure I wasn't missing something.  Bug has been opened at
  https://bugs.launchpad.net/nova/+bug/1417201
  
  I added some additional logs to the bug report of what the numa topology
  looks like on the compute node and in NUMATopologyFilter.host_passes().
  
  According to your trace I would say you are running different versions
  of Nova services.
  
  nova should all be the same version.  I'm running juno versions of other
  openstack components though.
 
 Hum if I understand well and according your issue reported to
 launchpad.net
 
   https://bugs.launchpad.net/nova/+bug/1417201
 
 You are trying to test hugepages under kilo which it is not possible
 since it has been implemented in this release (Juno, not yet
 published)

Please ignore this point.

 I have tried to reproduce your issue with trunk but I have not been
 able to do it. Please reopen the bug with more information of your env
 if still present. I should received any notification from it.
 
 Thanks,
 s.
 
  BTW please verify your version of libvirt. Hugepages is supported
  start to 1.2.8 (but this should difinitly not failed so badly like
  that)
  
  Libvirt is 1.2.8.
  Chris
  
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] problems with huge pages and libvirt

2015-02-03 Thread Sahid Orentino Ferdjaoui
On Mon, Feb 02, 2015 at 11:44:37AM -0600, Chris Friesen wrote:
 On 02/02/2015 11:00 AM, Sahid Orentino Ferdjaoui wrote:
 On Mon, Feb 02, 2015 at 10:44:09AM -0600, Chris Friesen wrote:
 Hi,
 
 I'm trying to make use of huge pages as described in
 http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-large-pages.html;.
 I'm running kilo as of Jan 27th.
 I've allocated 1 2MB pages on a compute node.  virsh capabilities on 
 that node contains:
 
  topology
cells num='2'
  cell id='0'
memory unit='KiB'67028244/memory
pages unit='KiB' size='4'16032069/pages
pages unit='KiB' size='2048'5000/pages
pages unit='KiB' size='1048576'1/pages
 ...
  cell id='1'
memory unit='KiB'67108864/memory
pages unit='KiB' size='4'16052224/pages
pages unit='KiB' size='2048'5000/pages
pages unit='KiB' size='1048576'1/pages
 
 
 I then restarted nova-compute, I set hw:mem_page_size=large on a
 flavor, and then tried to boot up an instance with that flavor.  I
 got the error logs below in nova-scheduler.  Is this a bug?
 
 Hello,
 
 Launchpad.net could be more appropriate to
 discuss on something which looks like a bug.
 
https://bugs.launchpad.net/nova/+filebug
 
 Just wanted to make sure I wasn't missing something.  Bug has been opened at
 https://bugs.launchpad.net/nova/+bug/1417201
 
 I added some additional logs to the bug report of what the numa topology
 looks like on the compute node and in NUMATopologyFilter.host_passes().
 
 According to your trace I would say you are running different versions
 of Nova services.
 
 nova should all be the same version.  I'm running juno versions of other
 openstack components though.

Hum if I understand well and according your issue reported to
launchpad.net

  https://bugs.launchpad.net/nova/+bug/1417201

You are trying to test hugepages under kilo which it is not possible
since it has been implemented in this release (Juno, not yet
published)

I have tried to reproduce your issue with trunk but I have not been
able to do it. Please reopen the bug with more information of your env
if still present. I should received any notification from it.

Thanks,
s.

 BTW please verify your version of libvirt. Hugepages is supported
 start to 1.2.8 (but this should difinitly not failed so badly like
 that)
 
 Libvirt is 1.2.8.
 Chris
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] problems with huge pages and libvirt

2015-02-02 Thread Sahid Orentino Ferdjaoui
On Mon, Feb 02, 2015 at 10:44:09AM -0600, Chris Friesen wrote:
 Hi,
 
 I'm trying to make use of huge pages as described in
 http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-large-pages.html;.
 I'm running kilo as of Jan 27th.
 I've allocated 1 2MB pages on a compute node.  virsh capabilities on 
 that node contains:
 
 topology
   cells num='2'
 cell id='0'
   memory unit='KiB'67028244/memory
   pages unit='KiB' size='4'16032069/pages
   pages unit='KiB' size='2048'5000/pages
   pages unit='KiB' size='1048576'1/pages
 ...
 cell id='1'
   memory unit='KiB'67108864/memory
   pages unit='KiB' size='4'16052224/pages
   pages unit='KiB' size='2048'5000/pages
   pages unit='KiB' size='1048576'1/pages
 
 
 I then restarted nova-compute, I set hw:mem_page_size=large on a
 flavor, and then tried to boot up an instance with that flavor.  I
 got the error logs below in nova-scheduler.  Is this a bug?

Hello,

Launchpad.net could be more appropriate to
discuss on something which looks like a bug.

  https://bugs.launchpad.net/nova/+filebug

According to your trace I would say you are running different versions
of Nova services.

BTW please verify your version of libvirt. Hugepages is supported
start to 1.2.8 (but this should difinitly not failed so badly like
that)

s.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] problems with huge pages and libvirt

2015-02-02 Thread Jay Pipes
This is a bug that I discovered when fixing some of the NUMA related 
nova objects. I have a patch that should fix it up shortly.


This is what happens when we don't have any functional testing of stuff 
that is merged into master...


Best,
-jay

On 02/02/2015 11:44 AM, Chris Friesen wrote:

Hi,

I'm trying to make use of huge pages as described in 
http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-large-pages.html;.
  I'm running kilo as of Jan 27th.

I've allocated 1 2MB pages on a compute node.  virsh capabilities on that 
node contains:

 topology
   cells num='2'
 cell id='0'
   memory unit='KiB'67028244/memory
   pages unit='KiB' size='4'16032069/pages
   pages unit='KiB' size='2048'5000/pages
   pages unit='KiB' size='1048576'1/pages
...
 cell id='1'
   memory unit='KiB'67108864/memory
   pages unit='KiB' size='4'16052224/pages
   pages unit='KiB' size='2048'5000/pages
   pages unit='KiB' size='1048576'1/pages


I then restarted nova-compute, I set hw:mem_page_size=large on a flavor, and 
then tried to boot up an instance with that flavor.  I got the error logs below in 
nova-scheduler.  Is this a bug?


Feb  2 16:23:10 controller-0 nova-scheduler Exception during message handling: 
Cannot load 'mempages' in the base class
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher Traceback 
(most recent call last):
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File 
/usr/lib64/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py, line 
134, in _dispatch_and_reply
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher 
incoming.message))
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File 
/usr/lib64/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py, line 
177, in _dispatch
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher return 
self._do_dispatch(endpoint, method, ctxt, args)
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File 
/usr/lib64/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py, line 
123, in _do_dispatch
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher result = 
getattr(endpoint, method)(ctxt, **new_args)
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File 
/usr/lib64/python2.7/site-packages/oslo/messaging/rpc/server.py, line 139, in 
inner
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher return 
func(*args, **kwargs)
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File 
/usr/lib64/python2.7/site-packages/nova/scheduler/manager.py, line 86, in 
select_destinations
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher 
filter_properties)
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File 
/usr/lib64/python2.7/site-packages/nova/scheduler/filter_scheduler.py, line 
67, in select_destinations
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher 
filter_properties)
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File 
/usr/lib64/python2.7/site-packages/nova/scheduler/filter_scheduler.py, line 
138, in _schedule
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher 
filter_properties, index=num)
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File 
/usr/lib64/python2.7/site-packages/nova/scheduler/host_manager.py, line 391, 
in get_filtered_hosts
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher hosts, 
filter_properties, index)
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File 
/usr/lib64/python2.7/site-packages/nova/filters.py, line 77, in 
get_filtered_objects
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher list_objs 
= list(objs)
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File 
/usr/lib64/python2.7/site-packages/nova/filters.py, line 43, in filter_all
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher if 
self._filter_one(obj, filter_properties):
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File 
/usr/lib64/python2.7/site-packages/nova/scheduler/filters/__init__.py, line 
27, in _filter_one
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher return 
self.host_passes(obj, filter_properties)
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File 
/usr/lib64/python2.7/site-packages/nova/scheduler/filters/numa_topology_filter.py,
 line 45, in host_passes
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher 
limits_topology=limits))
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher   File 
/usr/lib64/python2.7/site-packages/nova/virt/hardware.py, line 1161, in 
numa_fit_instance_to_host
2015-02-02 16:23:10.746 37521 TRACE oslo.messaging.rpc.dispatcher 

Re: [openstack-dev] problems with huge pages and libvirt

2015-02-02 Thread Ian Wells
On 2 February 2015 at 09:49, Chris Friesen chris.frie...@windriver.com
wrote:

 On 02/02/2015 10:51 AM, Jay Pipes wrote:

 This is a bug that I discovered when fixing some of the NUMA related nova
 objects. I have a patch that should fix it up shortly.


 Any chance you could point me at it or send it to me?

  This is what happens when we don't have any functional testing of stuff
 that is
 merged into master...


 Indeed.  Does tempest support hugepages/NUMA/pinning?


This is a running discussion, but largely no - because this is ited to the
capabilities of the host, there's no guarantee for a given scenario what
result you would get (because Tempest will run on any hardware).

If you have test cases that should pass or fail on a NUMA-capable node, can
you write them up?  We're working on NUMA-specific testing right now
(though I'm not sure who, specifically, is working on the test case side of
that).
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] problems with huge pages and libvirt

2015-02-02 Thread Chris Friesen

On 02/02/2015 12:13 PM, Ian Wells wrote:

On 2 February 2015 at 09:49, Chris Friesen chris.frie...@windriver.com



Indeed.  Does tempest support hugepages/NUMA/pinning?


This is a running discussion, but largely no - because this is ited to the
capabilities of the host, there's no guarantee for a given scenario what result
you would get (because Tempest will run on any hardware).

If you have test cases that should pass or fail on a NUMA-capable node, can you
write them up?  We're working on NUMA-specific testing right now (though I'm not
sure who, specifically, is working on the test case side of that).


I don't really have time to write up individual testcases right now, but I think 
a good start would be to test the following features:



http://specs.openstack.org/openstack/nova-specs/specs/juno/implemented/virt-driver-numa-placement.html

http://specs.openstack.org/openstack/nova-specs/specs/juno/implemented/virt-driver-vcpu-topology.html

http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-large-pages.html

http://specs.openstack.org/openstack/nova-specs/specs/kilo/approved/input-output-based-numa-scheduling.html

http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-cpu-pinning.html

Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] problems with huge pages and libvirt

2015-02-02 Thread Sahid Orentino Ferdjaoui
On Mon, Feb 02, 2015 at 11:51:47AM -0500, Jay Pipes wrote:
 This is a bug that I discovered when fixing some of the NUMA related nova
 objects. I have a patch that should fix it up shortly.

Never seen this issue, could be great to have a bug repported.

 This is what happens when we don't have any functional testing of stuff that
 is merged into master...
 Best,
 -jay

Thanks,
s.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] problems with huge pages and libvirt

2015-02-02 Thread Matthew Treinish
On Mon, Feb 02, 2015 at 11:49:26AM -0600, Chris Friesen wrote:
 On 02/02/2015 10:51 AM, Jay Pipes wrote:
 This is a bug that I discovered when fixing some of the NUMA related nova
 objects. I have a patch that should fix it up shortly.
 
 Any chance you could point me at it or send it to me?
 
 This is what happens when we don't have any functional testing of stuff that 
 is
 merged into master...
 
 Indeed.  Does tempest support hugepages/NUMA/pinning?

The short answer is not explicitly. The longer answer is that there are 2
patches[1][2] up for review right now that add basic checks to tempest. But,
they haven't been able to merge because the nova support hasn't worked and the
tests fail...

Aside from those 2 basic checks I don't expect any other direct numa, hugepage,
etc. tests to be in tempest. Testing anything besides these basic cases would
require knowledge of the underlying hardware for the deployment, which is out of
scope for tempest. There really needs to be lower level functional testing of
these features.

That being said the other thing you could do using tempest is to configure 
tempest
to use flavors which are created to use numa. That would at least implicitly
test that the functionality would work. But, that really isn't a replacement for
the functional testing which is sorely needed here.

[1] https://review.openstack.org/143540
[2] https://review.openstack.org/#/c/143541/


-Matt Treinish


pgpYTjxjBs7qI.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] problems with huge pages and libvirt

2015-02-02 Thread Chris Friesen

On 02/02/2015 11:00 AM, Sahid Orentino Ferdjaoui wrote:

On Mon, Feb 02, 2015 at 10:44:09AM -0600, Chris Friesen wrote:

Hi,

I'm trying to make use of huge pages as described in
http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-large-pages.html;.
I'm running kilo as of Jan 27th.
I've allocated 1 2MB pages on a compute node.  virsh capabilities on that 
node contains:

 topology
   cells num='2'
 cell id='0'
   memory unit='KiB'67028244/memory
   pages unit='KiB' size='4'16032069/pages
   pages unit='KiB' size='2048'5000/pages
   pages unit='KiB' size='1048576'1/pages
...
 cell id='1'
   memory unit='KiB'67108864/memory
   pages unit='KiB' size='4'16052224/pages
   pages unit='KiB' size='2048'5000/pages
   pages unit='KiB' size='1048576'1/pages


I then restarted nova-compute, I set hw:mem_page_size=large on a
flavor, and then tried to boot up an instance with that flavor.  I
got the error logs below in nova-scheduler.  Is this a bug?


Hello,

Launchpad.net could be more appropriate to
discuss on something which looks like a bug.

   https://bugs.launchpad.net/nova/+filebug


Just wanted to make sure I wasn't missing something.  Bug has been opened at 
https://bugs.launchpad.net/nova/+bug/1417201


I added some additional logs to the bug report of what the numa topology looks 
like on the compute node and in NUMATopologyFilter.host_passes().



According to your trace I would say you are running different versions
of Nova services.


nova should all be the same version.  I'm running juno versions of other 
openstack components though.



BTW please verify your version of libvirt. Hugepages is supported
start to 1.2.8 (but this should difinitly not failed so badly like
that)


Libvirt is 1.2.8.

Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] problems with huge pages and libvirt

2015-02-02 Thread Chris Friesen

On 02/02/2015 10:51 AM, Jay Pipes wrote:

This is a bug that I discovered when fixing some of the NUMA related nova
objects. I have a patch that should fix it up shortly.


Any chance you could point me at it or send it to me?


This is what happens when we don't have any functional testing of stuff that is
merged into master...


Indeed.  Does tempest support hugepages/NUMA/pinning?

Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] problems with huge pages and libvirt

2015-02-02 Thread Steve Gordon
- Original Message -
 From: Ian Wells ijw.ubu...@cack.org.uk
 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org

 On 2 February 2015 at 09:49, Chris Friesen chris.frie...@windriver.com
 wrote:
 
  On 02/02/2015 10:51 AM, Jay Pipes wrote:
 
  This is a bug that I discovered when fixing some of the NUMA related nova
  objects. I have a patch that should fix it up shortly.
 
 
  Any chance you could point me at it or send it to me?
 
   This is what happens when we don't have any functional testing of stuff
  that is
  merged into master...
 
 
  Indeed.  Does tempest support hugepages/NUMA/pinning?
 
 
 This is a running discussion, but largely no - because this is ited to the
 capabilities of the host, there's no guarantee for a given scenario what
 result you would get (because Tempest will run on any hardware).
 
 If you have test cases that should pass or fail on a NUMA-capable node, can
 you write them up?  We're working on NUMA-specific testing right now
 (though I'm not sure who, specifically, is working on the test case side of
 that).

Vladik and Sean (CC'd) are working on these.

Thanks,

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev