[libvirt] vCPU hotplug roadmap (was: Minutes for KVM call 2013-01-15)
Am 15.01.2013 17:16, schrieb Juan Quintela: * cpu hot plug - use qdev propierties conected to a set of socket objects (anthony) - cpusets are the wrong interface (anthony) - make a link between cpu - socket instead of a propierty? - how far are we from being able to describe a cpu with -device? (didn't heare the answer, andreas?) - perhaps the best approach? - After soft-freeze, exceptions depend on the maintainer - After hard-freeze, no exceptions -device don't require a bus, just an implementation detail, we can change that - use cpuset as an intermediate step until full vision is implemented - several approaches from where we are now, to have something before we get a full solution At this point, Andreas agreed to write a better summary of the discussion and suggestions O:-) Got buried, here we go: == vCPU hot-plug user interfaces == === cpu_set === Previously available in qemu-kvm.git: `cpu_set n+1 online` via HMP Pros: * Hides QOM/qdev implementation details (afaerber) * Thus: Doesn't depend on QOM CPUState refactoring (imammedo) * Opens a fast route to implementing vCPU unplug in KVM (imammedo) * Unintrusive to add and easy to obsolete/remove in future (imammedo) * Existing virt-test cases (afaerber) * Supported by libvirt (imammedo) * Prevents confusing guests by hot-plugging random mix of CPUs (agraf) Cons: * Cannot express topologies (ehabkost) === device_add === `device_add driver=Haswell-x86_64-cpu id=qdevid` [You can try this today and see it failing / not working.] Pros: * QMP/HMP command available today and known to users (afaerber) * Unified command for device and CPU hot-plug (imammedo) * Would allow first doing thread-level vCPU hotplug (imammedo) * Could be extended to support socket-level hot-plug (aliguori/imammedo) Cons: * Operates on raw QOM type name unlike -cpu (afaerber) * Needs support in libvirt for device_add driver=CPU (imammedo) * libvirt needs means to enumerate CPU types (imammedo) = QMP? (AF) Challenges: * No CPU qbus (afaerber) = should work without (aliguori) * CPU subclasses needed for identifying type name (afaerber/imammedo) = Haswell-x86_64-cpu does not exist yet, just x86_64-cpu * CPU class_init for -cpu host requires KVM init (imammedo) [suggestion by ehabkost to use kvm_arch_vcpu_init, WIP by afaerber] * Conversion of CPU features to static properties needed (imammedo) = device_add driver=foo,level=x,xlevel=y,... * Alternatively conversion to global properties (imammedo) * Cements type names - rename for 1.4? (afaerber) = permissable (alig.) [patches for arm, m68k, openrisc, unicore32 on list] === qom-set === `qom-set` via QMP w/ linkCPUSocket property (aliguori) Topology represented in QOM: CPUSocket has-aCPUCore has-aCPUThread a.k.a. CPUState, or CPUSocket links-to CPUCore links-to CPUThread a.k.a. CPUState Challenges (afaerber): * No CPUSocket/CPUCore objects yet and may take a while to get there... topology fields being moved to CPUState for 1.4 [done, more WIP] * No decisions on canonical paths for CPUs: CPU? machine? unassigned? * Duality of thread-level device types and socket-level? (afaerber) = fine to have, e.g., quad-core Xeon 500 device (aliguori) * CPUState is no_user (afaerber) = need to generally drop no_user for QOM (aliguori) === libvirt === libvirt's XML topology modelling is closer to today's -smp than to the desired QOM modelling: http://www.libvirt.org/formatcaps.html `virsh setvcpus domain n` http://libvirt.org/sources/virshcmdref/html/sect-setvcpus.html == qom-cpu course of action (afaerber) == It was requested to have vCPU hot-plug in v1.5. For device_add we need to move code from cpu_init() into QOM facilities. = QOM realize support would help [applied by aliguori] = cleanups piggy-backed onto CPU realizefn [applied to qom-cpu-next] Agreement on goal of X86CPU subclasses, but conflicts how to get there: * Refactor x86_def_t to X86CPUInfo for X86CPUClass class_init? (AF 2012) * Refactor x86_def_t to X86CPU instance_init as done for arm? * Refactor x86_def_t to class_inits? (afaerber) - heavy merge conflicts due to bug fixes / cleanups Pro: We can get things into a consistent QOM'ish state across targets. Con: We will refactor again on top for machine-compat properties. * Keep x86_def_t within X86CPUClass as done for ppc? (WIP: afaerber) = smallest common denominator, separates x86 from cross-target work APIC ID topology fixes are being reviewed for 1.4. [merged] X86CPU wave 4 cleanups by Igor are being reviewed for 1.4. [merged] Rename CPU types according to unified name-arch-cpu scheme for 1.4? (aliguori: permissable) [patches on list] VMState series by Juan being rebased - subset for 1.4, rest for 1.5. [1.4 part on list, WIP for 1.5] Remainder is considered 1.5 material, qom-cpu-next avail. during Freeze. == Common issues (imammedo) == - back-port CPU hot-plug ACPI notification - hot-plug is not allowed on SysBus: - APIC that
Re: [libvirt] vCPU hotplug roadmap (was: Minutes for KVM call 2013-01-15)
On Wed, Jan 30, 2013 at 11:58:56AM +0100, Andreas Färber wrote: Am 15.01.2013 17:16, schrieb Juan Quintela: * cpu hot plug - use qdev propierties conected to a set of socket objects (anthony) - cpusets are the wrong interface (anthony) - make a link between cpu - socket instead of a propierty? - how far are we from being able to describe a cpu with -device? (didn't heare the answer, andreas?) - perhaps the best approach? - After soft-freeze, exceptions depend on the maintainer - After hard-freeze, no exceptions -device don't require a bus, just an implementation detail, we can change that - use cpuset as an intermediate step until full vision is implemented - several approaches from where we are now, to have something before we get a full solution At this point, Andreas agreed to write a better summary of the discussion and suggestions O:-) Got buried, here we go: == vCPU hot-plug user interfaces == === cpu_set === Previously available in qemu-kvm.git: `cpu_set n+1 online` via HMP Pros: * Hides QOM/qdev implementation details (afaerber) * Thus: Doesn't depend on QOM CPUState refactoring (imammedo) * Opens a fast route to implementing vCPU unplug in KVM (imammedo) * Unintrusive to add and easy to obsolete/remove in future (imammedo) * Existing virt-test cases (afaerber) * Supported by libvirt (imammedo) * Prevents confusing guests by hot-plugging random mix of CPUs (agraf) Cons: * Cannot express topologies (ehabkost) Actually, I believe this is not the main problem (we will have exactly the same limitation if using thread-level device_add). To me, the main problem is that we are creating a new QMP command that should be eventually obsoleted by device_add. === device_add === `device_add driver=Haswell-x86_64-cpu id=qdevid` [You can try this today and see it failing / not working.] Pros: * QMP/HMP command available today and known to users (afaerber) * Unified command for device and CPU hot-plug (imammedo) * Would allow first doing thread-level vCPU hotplug (imammedo) * Could be extended to support socket-level hot-plug (aliguori/imammedo) Cons: * Operates on raw QOM type name unlike -cpu (afaerber) * Needs support in libvirt for device_add driver=CPU (imammedo) * libvirt needs means to enumerate CPU types (imammedo) = QMP? (AF) Challenges: * No CPU qbus (afaerber) = should work without (aliguori) * CPU subclasses needed for identifying type name (afaerber/imammedo) = Haswell-x86_64-cpu does not exist yet, just x86_64-cpu * CPU class_init for -cpu host requires KVM init (imammedo) [suggestion by ehabkost to use kvm_arch_vcpu_init, WIP by afaerber] I don't know what you mean by use kvm_arch_vcpu_init(). I sent a RFC following somebody's suggestion of simply make kvm_arch_init() call a function to finish the -cpu host initialization, as we can't initialize everything inside class_init. See x86_cpu_finish_host_class_init() at: Message-Id: 1357329382-20944-7-git-send-email-ehabk...@redhat.com http://article.gmane.org/gmane.comp.emulators.qemu/186778 * Conversion of CPU features to static properties needed (imammedo) = device_add driver=foo,level=x,xlevel=y,... * Alternatively conversion to global properties (imammedo) * Cements type names - rename for 1.4? (afaerber) = permissable (alig.) [patches for arm, m68k, openrisc, unicore32 on list] === qom-set === `qom-set` via QMP w/ linkCPUSocket property (aliguori) Topology represented in QOM: CPUSocket has-aCPUCore has-aCPUThread a.k.a. CPUState, or CPUSocket links-to CPUCore links-to CPUThread a.k.a. CPUState Challenges (afaerber): * No CPUSocket/CPUCore objects yet and may take a while to get there... topology fields being moved to CPUState for 1.4 [done, more WIP] * No decisions on canonical paths for CPUs: CPU? machine? unassigned? * Duality of thread-level device types and socket-level? (afaerber) = fine to have, e.g., quad-core Xeon 500 device (aliguori) * CPUState is no_user (afaerber) = need to generally drop no_user for QOM (aliguori) I would like to drop no_user on 1.5 even if we don't manage to finish CPU hotplug, as exposing the CPU objects and classes will be very useful to allow libvirt to probe for the available CPU models and features. === libvirt === libvirt's XML topology modelling is closer to today's -smp than to the desired QOM modelling: http://www.libvirt.org/formatcaps.html `virsh setvcpus domain n` http://libvirt.org/sources/virshcmdref/html/sect-setvcpus.html == qom-cpu course of action (afaerber) == It was requested to have vCPU hot-plug in v1.5. For device_add we need to move code from cpu_init() into QOM facilities. = QOM realize support would help [applied by aliguori] = cleanups piggy-backed onto CPU realizefn [applied to qom-cpu-next] Agreement on goal of X86CPU subclasses, but conflicts how to get there: * Refactor
Re: [libvirt] vCPU hotplug roadmap
Am 30.01.2013 13:49, schrieb Eduardo Habkost: On Wed, Jan 30, 2013 at 11:58:56AM +0100, Andreas Färber wrote: * CPU class_init for -cpu host requires KVM init (imammedo) [suggestion by ehabkost to use kvm_arch_vcpu_init, WIP by afaerber] I don't know what you mean by use kvm_arch_vcpu_init(). Sorry, scratch the _vcpu. I.e., the x86-specific KVM init hook. I sent a RFC following somebody's suggestion of simply make kvm_arch_init() call a function to finish the -cpu host initialization, as we can't initialize everything inside class_init. See x86_cpu_finish_host_class_init() at: Message-Id: 1357329382-20944-7-git-send-email-ehabk...@redhat.com http://article.gmane.org/gmane.comp.emulators.qemu/186778 ...and I have been working on making it even simpler for the still-x86_def_t-based approach. I'm still busy looking at 1.4 issues currently though. Andreas -- SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
