Re: [openstack-dev] [openstack-dev[[nova] Simple question about sorting CPU topologies

2017-06-21 Thread Zhenyu Zheng 
Thanks all for the reply, I guess it will be better to config those
preference using flavor/image according to different hardware then.

On Wed, Jun 21, 2017 at 1:21 AM, Mooney, Sean K 
wrote:

>
>
> > -Original Message-
> > From: Jay Pipes [mailto:jaypi...@gmail.com]
> > Sent: Tuesday, June 20, 2017 5:59 PM
> > To: openstack-dev@lists.openstack.org
> > Subject: Re: [openstack-dev] [openstack-dev[[nova] Simple question
> > about sorting CPU topologies
> >
> > On 06/20/2017 12:53 PM, Chris Friesen wrote:
> > > On 06/20/2017 06:29 AM, Jay Pipes wrote:
> > >> On 06/19/2017 10:45 PM, Zhenyu Zheng wrote:
> > >>> Sorry, The mail sent accidentally by mis-typing ...
> > >>>
> > >>> My question is, what is the benefit of the above preference?
> > >>
> > >> Hi Kevin!
> > >>
> > >> I believe the benefit is so that the compute node prefers CPU
> > >> topologies that do not have hardware threads over CPU topologies
> > that
> > >> do include hardware threads.
> [Mooney, Sean K] if you have not expressed that you want the require or
> isolate policy
> Then you really cant infer which is better as for some workloads
> preferring hyperthread
> Siblings will improve performance( 2 threads sharing data via l2 cache)
> and other it will reduce it
> (2 thread that do not share data)
> > >>
> > >> I'm not sure exactly of the reason for this preference, but perhaps
> > >> it is due to assumptions that on some hardware, threads will compete
> > >> for the same cache resources as other siblings on a core whereas
> > >> cores may have their own caches (again, on some specific hardware).
> > >
> > > Isn't the definition of hardware threads basically the fact that the
> > > sibling threads share the resources of a single core?
> > >
> > > Are there architectures that OpenStack runs on where hardware threads
> > > don't compete for cache/TLB/execution units?  (And if there are, then
> > > why are they called threads and not cores?)
> [Mooney, Sean K] well on x86 when you turn on hypter threading your L1
> data and instruction cache is
> Partitioned in 2 with each half allocated to a thread sibling. The l2
> cache which is also per core is shared
> Between the 2 thread siblings so on intels x86 implementation the thread
> do not compete for l1 cache but do share l2
> That could easibly change though in new generations.
>
> Pre xen architure I believe amd shared the floating point units between
> each smt thread but had separate integer execution units that
> Were not shared. That meant for integer heavy workloads there smt
> implementation approached 2X performance limited by the
> Shared load and store units and reduced to 0 scaling if both Treads tried
> to access the floating point execution unit concurrently.
>
> So its not quite as clean cut as saying the thread  do or don’t share
> resources
> Each vendor addresses this differently even with in x86 you are not
> required to have the partitioning
> described above for cache as intel did or for the execution units. On
> other architectures im sure they have
> come up with equally inventive ways to make this an interesting shade of
> grey when describing the difference
> between a hardware thread a full core.
>
> >
> > I've learned over the years not to make any assumptions about hardware.
> >
> > Thus my "not sure exactly" bet-hedging ;)
> [Mooney, Sean K] yep hardware is weird and will always find ways to break
> your assumptions :)
> >
> > Best,
> > -jay
> >
> > ___
> > ___
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-
> > requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-dev[[nova] Simple question about sorting CPU topologies

2017-06-20 Thread Mooney, Sean K 


> -Original Message-
> From: Jay Pipes [mailto:jaypi...@gmail.com]
> Sent: Tuesday, June 20, 2017 5:59 PM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [openstack-dev[[nova] Simple question
> about sorting CPU topologies
> 
> On 06/20/2017 12:53 PM, Chris Friesen wrote:
> > On 06/20/2017 06:29 AM, Jay Pipes wrote:
> >> On 06/19/2017 10:45 PM, Zhenyu Zheng wrote:
> >>> Sorry, The mail sent accidentally by mis-typing ...
> >>>
> >>> My question is, what is the benefit of the above preference?
> >>
> >> Hi Kevin!
> >>
> >> I believe the benefit is so that the compute node prefers CPU
> >> topologies that do not have hardware threads over CPU topologies
> that
> >> do include hardware threads.
[Mooney, Sean K] if you have not expressed that you want the require or isolate 
policy
Then you really cant infer which is better as for some workloads preferring 
hyperthread
Siblings will improve performance( 2 threads sharing data via l2 cache) and 
other it will reduce it
(2 thread that do not share data) 
> >>
> >> I'm not sure exactly of the reason for this preference, but perhaps
> >> it is due to assumptions that on some hardware, threads will compete
> >> for the same cache resources as other siblings on a core whereas
> >> cores may have their own caches (again, on some specific hardware).
> >
> > Isn't the definition of hardware threads basically the fact that the
> > sibling threads share the resources of a single core?
> >
> > Are there architectures that OpenStack runs on where hardware threads
> > don't compete for cache/TLB/execution units?  (And if there are, then
> > why are they called threads and not cores?)
[Mooney, Sean K] well on x86 when you turn on hypter threading your L1 data and 
instruction cache is
Partitioned in 2 with each half allocated to a thread sibling. The l2 cache 
which is also per core is shared
Between the 2 thread siblings so on intels x86 implementation the thread do not 
compete for l1 cache but do share l2
That could easibly change though in new generations. 

Pre xen architure I believe amd shared the floating point units between each 
smt thread but had separate integer execution units that
Were not shared. That meant for integer heavy workloads there smt 
implementation approached 2X performance limited by the
Shared load and store units and reduced to 0 scaling if both Treads tried to 
access the floating point execution unit concurrently.

So its not quite as clean cut as saying the thread  do or don’t share resources
Each vendor addresses this differently even with in x86 you are not required to 
have the partitioning
described above for cache as intel did or for the execution units. On other 
architectures im sure they have
come up with equally inventive ways to make this an interesting shade of grey 
when describing the difference
between a hardware thread a full core. 

> 
> I've learned over the years not to make any assumptions about hardware.
> 
> Thus my "not sure exactly" bet-hedging ;)
[Mooney, Sean K] yep hardware is weird and will always find ways to break your 
assumptions :)
> 
> Best,
> -jay
> 
> ___
> ___
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-dev[[nova] Simple question about sorting CPU topologies

2017-06-20 Thread Jay Pipes 

On 06/20/2017 12:53 PM, Chris Friesen wrote:

On 06/20/2017 06:29 AM, Jay Pipes wrote:

On 06/19/2017 10:45 PM, Zhenyu Zheng wrote:

Sorry, The mail sent accidentally by mis-typing ...

My question is, what is the benefit of the above preference?


Hi Kevin!

I believe the benefit is so that the compute node prefers CPU 
topologies that do
not have hardware threads over CPU topologies that do include hardware 
threads.


I'm not sure exactly of the reason for this preference, but perhaps it 
is due to
assumptions that on some hardware, threads will compete for the same 
cache
resources as other siblings on a core whereas cores may have their own 
caches

(again, on some specific hardware).


Isn't the definition of hardware threads basically the fact that the 
sibling threads share the resources of a single core?


Are there architectures that OpenStack runs on where hardware threads 
don't compete for cache/TLB/execution units?  (And if there are, then 
why are they called threads and not cores?)


I've learned over the years not to make any assumptions about hardware.

Thus my "not sure exactly" bet-hedging ;)

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-dev[[nova] Simple question about sorting CPU topologies

2017-06-20 Thread Chris Friesen 

On 06/20/2017 06:29 AM, Jay Pipes wrote:

On 06/19/2017 10:45 PM, Zhenyu Zheng wrote:

Sorry, The mail sent accidentally by mis-typing ...

My question is, what is the benefit of the above preference?


Hi Kevin!

I believe the benefit is so that the compute node prefers CPU topologies that do
not have hardware threads over CPU topologies that do include hardware threads.

I'm not sure exactly of the reason for this preference, but perhaps it is due to
assumptions that on some hardware, threads will compete for the same cache
resources as other siblings on a core whereas cores may have their own caches
(again, on some specific hardware).


Isn't the definition of hardware threads basically the fact that the sibling 
threads share the resources of a single core?


Are there architectures that OpenStack runs on where hardware threads don't 
compete for cache/TLB/execution units?  (And if there are, then why are they 
called threads and not cores?)


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-dev[[nova] Simple question about sorting CPU topologies

2017-06-20 Thread Jay Pipes 

On 06/19/2017 10:45 PM, Zhenyu Zheng wrote:

Sorry, The mail sent accidentally by mis-typing ...

My question is, what is the benefit of the above preference?


Hi Kevin!

I believe the benefit is so that the compute node prefers CPU topologies 
that do not have hardware threads over CPU topologies that do include 
hardware threads.


I'm not sure exactly of the reason for this preference, but perhaps it 
is due to assumptions that on some hardware, threads will compete for 
the same cache resources as other siblings on a core whereas cores may 
have their own caches (again, on some specific hardware).


Best,
-jay

On Tue, Jun 20, 2017 at 10:43 AM, Zhenyu Zheng 
mailto:zhengzhenyul...@gmail.com>> wrote:


Hi,

In
https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L396

we calculated every possible CPU topologies and sorted by:
# We want to
# - Minimize threads (ie larger sockets * cores is best)
# - Prefer sockets over cores
possible = sorted(possible, reverse=True,
key=lambda x: (x.sockets * x.cores,
x.sockets,
x.threads))




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-dev[[nova] Simple question about sorting CPU topologies

2017-06-19 Thread Zhenyu Zheng 
Sorry, The mail sent accidentally by mis-typing ...

My question is, what is the benefit of the above preference?

BR,
Kevin

On Tue, Jun 20, 2017 at 10:43 AM, Zhenyu Zheng 
wrote:

> Hi,
>
> In https://github.com/openstack/nova/blob/master/
> nova/virt/hardware.py#L396 we calculated every possible CPU topologies
> and sorted by:
> # We want to
> # - Minimize threads (ie larger sockets * cores is best)
> # - Prefer sockets over cores
> possible = sorted(possible, reverse=True,
> key=lambda x: (x.sockets * x.cores,
> x.sockets,
> x.threads))
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [openstack-dev[[nova] Simple question about sorting CPU topologies

2017-06-19 Thread Zhenyu Zheng 
Hi,

In https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L396
we calculated every possible CPU topologies and sorted by:
# We want to
# - Minimize threads (ie larger sockets * cores is best)
# - Prefer sockets over cores
possible = sorted(possible, reverse=True,
key=lambda x: (x.sockets * x.cores,
x.sockets,
x.threads))
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev