Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

On 06/22/2016 01:56 PM, Paul Michali wrote: I did have a question about the current implementation as described by 292499, 324379, and 292500. Looking at the code, when a NUMAPagesTopology object is create, a new parameter is passed for the "reserved" pages. This reservation comes from a

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

On 06/22/2016 01:56 PM, Paul Michali wrote: I did have a question about the current implementation as described by 292499, 324379, and 292500. Looking at the code, when a NUMAPagesTopology object is create, a new parameter is passed for the "reserved" pages. This reservation comes from a

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

I did have a question about the current implementation as described by 292499, 324379, and 292500. Looking at the code, when a NUMAPagesTopology object is create, a new parameter is passed for the "reserved" pages. This reservation comes from a dictionary, which is populated at LIbvirtDriver init

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

On 6/15/2016 3:10 PM, Paul Michali wrote: Is the plan to back port that change to Mitaka? Thanks, PCM On Wed, Jun 15, 2016 at 1:31 PM Matt Riedemann > wrote: On 6/14/2016 3:09 PM, Jay Pipes wrote: > > Yes. Code

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

Is the plan to back port that change to Mitaka? Thanks, PCM On Wed, Jun 15, 2016 at 1:31 PM Matt Riedemann wrote: > On 6/14/2016 3:09 PM, Jay Pipes wrote: > > > > Yes. Code merged recently from Sahid does this: > > > > https://review.openstack.org/#/c/277422/ > >

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

On 6/14/2016 3:09 PM, Jay Pipes wrote: Yes. Code merged recently from Sahid does this: https://review.openstack.org/#/c/277422/ Best, -jay That was actually reverted out of mitaka: https://review.openstack.org/#/c/292290/ The feature change that got into newton was this:

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

There have been a number of bug fixes to the NUMA code in both Mitaka and Newton. I think you would need to be very careful in your backporting :) -jay On 06/15/2016 09:55 AM, Paul Michali wrote: Yeah, was thinking more of technically vs policy. Wondering if there are other dependencies or if

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

Yeah, was thinking more of technically vs policy. Wondering if there are other dependencies or if I could patch this into a Liberty code base. On Wed, Jun 15, 2016 at 12:38 PM Jay Pipes wrote: > On 06/15/2016 03:58 AM, Paul Michali wrote: > > Awesome Jay! > > > > Do you

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

On 06/15/2016 03:58 AM, Paul Michali wrote: Awesome Jay! Do you think this is something that can be backporting to Liberty w/o other dependencies? We're running Liberty on our system right now. Doubtful, Paul :( The policy for upstream is not to backport feature patches. This would be

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

Awesome Jay! Do you think this is something that can be backporting to Liberty w/o other dependencies? We're running Liberty on our system right now. On Tue, Jun 14, 2016 at 4:10 PM Jay Pipes wrote: > On 06/14/2016 12:30 PM, Paul Michali wrote: > > Well, looks like we

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

On 06/14/2016 12:30 PM, Paul Michali wrote: Well, looks like we figured out what is going on - maybe folks have some ideas on how we could handle this issue. What I see is that for each VM create (small flavor), 1024 huge pages are used and NUMA node 0 used. It appears that, when there is no

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

Well, looks like we figured out what is going on - maybe folks have some ideas on how we could handle this issue. What I see is that for each VM create (small flavor), 1024 huge pages are used and NUMA node 0 used. It appears that, when there is no longer enough huge pages on that NUMA node, Nova

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

Great info Chris and thanks for confirming the assignment of blocks of pages to a numa node. I'm still struggling with why each VM is being assigned to NUMA node 0. Any ideas on where I should look to see why Nova is not using NUMA id 1? Thanks! PCM On Tue, Jun 14, 2016 at 10:29 AM Chris

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

On 06/13/2016 02:17 PM, Paul Michali wrote: Hmm... I tried Friday and again today, and I'm not seeing the VMs being evenly created on the NUMA nodes. Every Cirros VM is created on nodeid 0. I have the m1/small flavor (@GB) selected and am using hw:numa_nodes=1 and hw:mem_page_size=2048

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

Hmm... I tried Friday and again today, and I'm not seeing the VMs being evenly created on the NUMA nodes. Every Cirros VM is created on nodeid 0. I have the m1/small flavor (@GB) selected and am using hw:numa_nodes=1 and hw:mem_page_size=2048 flavor-key settings. Each VM is consuming 1024 huge

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

Actually, I had menm_page_size set to "large" and not "1024". However, it seemed like it was using 1024 pages per (small VM creation). Is there possibly some issue with large not using one of the supported values? I would have guessed it would have chosen 2M or 1G for the size. Any thoughts? PCM

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

Thanks Daniel and Chris! I think that was the problem, I had configured Nova flavor with a mem_page_size of 1024, and it should have been one of the supported values. I'll go through and check things out one more time, but I think that is the problem. I still need to figure out what is going on

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

openstack-dev@lists.openstack.org> > > Sent: Tuesday, June 7, 2016 11:00:30 AM > > Subject: Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling > > > > Anyone have any thoughts on the two questions below? Namely... > > > > If the huge pages are 2M, w

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

I'll try to reproduce and collect logs for a bug report. Thanks for the info. PCM On Thu, Jun 9, 2016 at 9:43 AM Matt Riedemann wrote: > On 6/9/2016 6:15 AM, Paul Michali wrote: > > > > > > On Wed, Jun 8, 2016 at 11:21 PM Chris Friesen > >

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

On Thu, Jun 09, 2016 at 12:35:06PM -0600, Chris Friesen wrote: > On 06/09/2016 05:15 AM, Paul Michali wrote: > > 1) On the host, I was seeing 32768 huge pages, of 2MB size. > > Please check the number of huge pages _per host numa node_. > > > 2) I changed mem_page_size from 1024 to 2048 in the

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

On 06/09/2016 05:15 AM, Paul Michali wrote: 1) On the host, I was seeing 32768 huge pages, of 2MB size. Please check the number of huge pages _per host numa node_. 2) I changed mem_page_size from 1024 to 2048 in the flavor, and then when VMs were created, they were being evenly assigned to

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

- Original Message - > From: "Paul Michali" <p...@michali.net> > To: "OpenStack Development Mailing List (not for usage questions)" > <openstack-dev@lists.openstack.org> > Sent: Tuesday, June 7, 2016 11:00:30 AM > Subject: Re: [opensta

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

On 6/9/2016 6:15 AM, Paul Michali wrote: On Wed, Jun 8, 2016 at 11:21 PM Chris Friesen > wrote: On 06/03/2016 12:03 PM, Paul Michali wrote: > Thanks for the link Tim! > > Right now, I have two things I'm unsure

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

On Wed, Jun 8, 2016 at 11:21 PM Chris Friesen wrote: > On 06/03/2016 12:03 PM, Paul Michali wrote: > > Thanks for the link Tim! > > > > Right now, I have two things I'm unsure about... > > > > One is that I had 1945 huge pages left (of size 2048k) and tried to >

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

On 06/03/2016 12:03 PM, Paul Michali wrote: Thanks for the link Tim! Right now, I have two things I'm unsure about... One is that I had 1945 huge pages left (of size 2048k) and tried to create a VM with a small flavor (2GB), which should need 1024 pages, but Nova indicated that it wasn't able

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

gt;> >> >> >> *From: *Paul Michali <p...@michali.net> >> *Reply-To: *"OpenStack Development Mailing List (not for usage >> questions)" <openstack-dev@lists.openstack.org> >> *Date: *Friday 3 June 2016 at 15:18 >> *To: *"Danie

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

sage > questions)" <openstack-dev@lists.openstack.org> > *Date: *Friday 3 June 2016 at 15:18 > *To: *"Daniel P. Berrange" <berra...@redhat.com>, "OpenStack Development > Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org > > &g

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

redhat.com>, "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling See PCM inline... On Fri, Jun 3, 2016 at 8:44 AM Daniel P. Berrange <berra...@redhat.com<ma

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

See PCM inline... On Fri, Jun 3, 2016 at 8:44 AM Daniel P. Berrange wrote: > On Fri, Jun 03, 2016 at 12:32:17PM +, Paul Michali wrote: > > Hi! > > > > I've been playing with Liberty code a bit and had some questions that I'm > > hoping Nova folks may be able to provide

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

On Fri, Jun 03, 2016 at 12:32:17PM +, Paul Michali wrote: > Hi! > > I've been playing with Liberty code a bit and had some questions that I'm > hoping Nova folks may be able to provide guidance on... > > If I set up a flavor with hw:mem_page_size=2048, and I'm creating (Cirros) > VMs with