Tuomas, > I haven't but I will write a blueprint for the core pinning part. Can’t wait to see it!
> Are you using extra specs for carrying cpuset attributes in your > implementation? Yes, exactly. Although we're using slightly different syntax to update flavor, for example: $ nova flavor-key set <flavor> set vcpupin:0=1-5,12-17 In here ‘0’ is vCPU, and '1-5,12-17' - pCPUs. Basically this command results in the following libvirt xml: <cputune> <vcpupin vcpu=‘0’ cpuset=‘1-5,12-17’/> </cputune> We’re also using the ‘placement’ attribute of <vcpu> set to ‘static’: $ nova flavor-key <flavor> set vcpu:placement=static Which results in the following libvirt xml: <vcpu placement=‘static’>…</vcpu> Otherwise, the functionality and implementation seem to be identical. [offtopic] Apologies for delayed answer, for some reason I thought your email would arrive to my personal mailbox [/offtopic] -Roman On Nov 19, 2013, at 14:35, Tuomas Paappanen <tuomas.paappa...@tieto.com> wrote: > Hi Roman, > > I haven't but I will write a blueprint for the core pinning part. > I considered vcpu element usage as well but in that case you can not set e.g. > vcpu-0 to run on pcpu-0. Vcpus and emulator are sharing all pcpus defined in > cpuset so I decided to use cputune element. > > Are you using extra specs for carrying cpuset attributes in your > implementation? > > Br,Tuomas > > On 18.11.2013 17:14, Roman Verchikov wrote: >> Tuomas, >> >> Have you published your code/blueprints anywhere? Looks like we’re working >> on the same stuff. I have implemented almost the same feature set (haven’t >> published anything yet because of this thread), except for the scheduler >> part. The main goal is to be able to pin VCPUs in NUMA environment. >> >> Have you considered adding placement and cpuset attributes to <vcpu> >> element? For example: >> <vcpu placement=‘static’ cpuset=‘%whatever%’> >> >> Thanks, >> Roman >> >> On Nov 13, 2013, at 14:46, Tuomas Paappanen <tuomas.paappa...@tieto.com> >> wrote: >> >>> Hi all, >>> >>> I would like to hear your thoughts about core pinning in Openstack. >>> Currently nova(with qemu-kvm) supports usage of cpu set of PCPUs what can >>> be used by instances. I didn't find blueprint, but I think this feature is >>> for isolate cpus used by host from cpus used by instances(VCPUs). >>> >>> But, from performance point of view it is better to exclusively dedicate >>> PCPUs for VCPUs and emulator. In some cases you may want to guarantee that >>> only one instance(and its VCPUs) is using certain PCPUs. By using core >>> pinning you can optimize instance performance based on e.g. cache sharing, >>> NUMA topology, interrupt handling, pci pass through(SR-IOV) in multi socket >>> hosts etc. >>> >>> We have already implemented feature like this(PoC with limitations) to Nova >>> Grizzly version and would like to hear your opinion about it. >>> >>> The current implementation consists of three main parts: >>> - Definition of pcpu-vcpu maps for instances and instance spawning >>> - (optional) Compute resource and capability advertising including free >>> pcpus and NUMA topology. >>> - (optional) Scheduling based on free cpus and NUMA topology. >>> >>> The implementation is quite simple: >>> >>> (additional/optional parts) >>> Nova-computes are advertising free pcpus and NUMA topology in same manner >>> than host capabilities. Instances are scheduled based on this information. >>> >>> (core pinning) >>> admin can set PCPUs for VCPUs and for emulator process, or select NUMA cell >>> for instance vcpus, by adding key:value pairs to flavor's extra specs. >>> >>> EXAMPLE: >>> instance has 4 vcpus >>> <key>:<value> >>> vcpus:1,2,3,4 --> vcpu0 pinned to pcpu1, vcpu1 pinned to pcpu2... >>> emulator:5 --> emulator pinned to pcpu5 >>> or >>> numacell:0 --> all vcpus are pinned to pcpus in numa cell 0. >>> >>> In nova-compute, core pinning information is read from extra specs and >>> added to domain xml same way as cpu quota values(cputune). >>> >>> <cputune> >>> <vcpupin vcpu='0' cpuset='1'/> >>> <vcpupin vcpu='1' cpuset='2'/> >>> <vcpupin vcpu='2' cpuset='3'/> >>> <vcpupin vcpu='3' cpuset='4'/> >>> <emulatorpin cpuset='5'/> >>> </cputune> >>> >>> What do you think? Implementation alternatives? Is this worth of blueprint? >>> All related comments are welcome! >>> >>> Regards, >>> Tuomas >>> >>> >>> >>> >>> >>> _______________________________________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
