Re: [Qemu-devel] [PATCH 00/16 RFC v6] ACPI memory hotplug
On Wed, 11 Sep 2013 12:01:44 +0800 Hu Tao hu...@cn.fujitsu.com wrote: On Tue, Jul 23, 2013 at 06:22:56PM +0200, Igor Mammedov wrote: As opposed to previous approach, This series allows to hotplug 'arbitrary' DIMM devices specifying size, NUMA node mapping, slot and address where to map it, at runtime. Due to ACPI limitation there is need to specify a number of possible DIMM devices. For this task -m option was extended to support following format: -m [mem=]RamSize[,slots=N,maxmem=M] To allow memory hotplug user must specify a pair additional parameters: 'slots' - number of possible increments 'maxmem' - max possible total memory size QEMU is allowed to use, including RamSize. minimal monitor command syntax to hotplug DIMM device: device_add dimm,id=dimmX DIMM device provides following properties that could be used with device_add / -device to alter default behavior: id- unique string identifying device [mandatory] slot - number in range [0-slots) [optional], if not specified the first free slot is used node - NUMA node id [optional] (default: 0) size - amount of memory to add [optional] (default: 1Gb) start - guest's physical address where to plug DIMM [optional], if not specified the first gap in hotplug memory region that fits DIMM is used -device option could be used for adding potentially hotunplugable DIMMs and also for specifying hotplugged DIMMs in migration case (not tested). Current implementation supports only x86-64 variant and places hotplug memory region above 4Gb before 64-bit PCI hole. Tested guests: - Fedora 19x64 - Windows 2012DCx64 - Windows 2008DCx64 Known limitations/bugs/TODOs: - only hot-add supported - q35 is not supported yet - max number of supported DIMM devices 255 (due to ACPI object name limit), could be increased creating several containers and putting DIMMs there. (exercise for future) - failed hotplug action consumes 1 slot (device_add doesn't delete device if realize failed) - e820 table doesn't include DIMM devices added with -device / (or after reboot devices added with device_add) - Windows 2008 remembers DIMM configuration, so if DIMM with other start/size is added into the same slot, it refuses to use it insisting on old mapping. With this series we can hotplug memory of arbitrary size, but Linux expects a minimum size of hotpluggable memory. Take 128M in x86_64 for example, if first hotplug 64M memory (less than 128M) than we can't add another 64M memory. So the question is should we have a lower limit of hotplugged memory in qemu (which is easy but it's not qemu's problem, and different OS/hardware may have different limits)? Or maybe we can fix it in Linux (which is difficult and I'm not sure it'll casue any compatibility problem)? I'd say it's not QEMU problem, in real hardware there is/was 64Mb DIMMs and other OSes might handle this small amount just fine (tested with Windows server 2012). It's upto linux kernel to fix bug or management tools which know what OS is being installed to implement workaround/limit.
Re: [Qemu-devel] [PATCH 00/16 RFC v6] ACPI memory hotplug
On Tue, Jul 23, 2013 at 06:22:56PM +0200, Igor Mammedov wrote: As opposed to previous approach, This series allows to hotplug 'arbitrary' DIMM devices specifying size, NUMA node mapping, slot and address where to map it, at runtime. Due to ACPI limitation there is need to specify a number of possible DIMM devices. For this task -m option was extended to support following format: -m [mem=]RamSize[,slots=N,maxmem=M] To allow memory hotplug user must specify a pair additional parameters: 'slots' - number of possible increments 'maxmem' - max possible total memory size QEMU is allowed to use, including RamSize. minimal monitor command syntax to hotplug DIMM device: device_add dimm,id=dimmX DIMM device provides following properties that could be used with device_add / -device to alter default behavior: id- unique string identifying device [mandatory] slot - number in range [0-slots) [optional], if not specified the first free slot is used node - NUMA node id [optional] (default: 0) size - amount of memory to add [optional] (default: 1Gb) start - guest's physical address where to plug DIMM [optional], if not specified the first gap in hotplug memory region that fits DIMM is used -device option could be used for adding potentially hotunplugable DIMMs and also for specifying hotplugged DIMMs in migration case (not tested). Current implementation supports only x86-64 variant and places hotplug memory region above 4Gb before 64-bit PCI hole. Tested guests: - Fedora 19x64 - Windows 2012DCx64 - Windows 2008DCx64 Known limitations/bugs/TODOs: - only hot-add supported - q35 is not supported yet - max number of supported DIMM devices 255 (due to ACPI object name limit), could be increased creating several containers and putting DIMMs there. (exercise for future) - failed hotplug action consumes 1 slot (device_add doesn't delete device if realize failed) - e820 table doesn't include DIMM devices added with -device / (or after reboot devices added with device_add) - Windows 2008 remembers DIMM configuration, so if DIMM with other start/size is added into the same slot, it refuses to use it insisting on old mapping. With this series we can hotplug memory of arbitrary size, but Linux expects a minimum size of hotpluggable memory. Take 128M in x86_64 for example, if first hotplug 64M memory (less than 128M) than we can't add another 64M memory. So the question is should we have a lower limit of hotplugged memory in qemu (which is easy but it's not qemu's problem, and different OS/hardware may have different limits)? Or maybe we can fix it in Linux (which is difficult and I'm not sure it'll casue any compatibility problem)?
Re: [Qemu-devel] [PATCH 00/16 RFC v6] ACPI memory hotplug
Anthony Liguori aligu...@us.ibm.com writes: Applied. Thanks. Something went bad here... Apologies, it was not applied. Regards, Anthony Liguori Regards, Anthony Liguori
Re: [Qemu-devel] [PATCH 00/16 RFC v6] ACPI memory hotplug
On Fri, Aug 2, 2013 at 9:35 AM, Anthony Liguori aligu...@us.ibm.com wrote: Applied. Thanks. Where can I find this branch? Regards, Anthony Liguori
Re: [Qemu-devel] [PATCH 00/16 RFC v6] ACPI memory hotplug
Applied. Thanks. Regards, Anthony Liguori
Re: [Qemu-devel] [PATCH 00/16 RFC v6] ACPI memory hotplug
v6 doesn't work here, things are going fine until online hotplugged memory in guest. steps: 1. qemu cmd: ./x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 512,maxmem=2G,slots=1 \ -hda /mnt/data/libvirt-images/hut-rhel6.3.img -L ../pc-bios-memhp/ (bios is from MST's acpi tree) 2. hot-plug a dimm: device_adddimm,id=d0,size=1G 3. online hotplugged memory(in guest): echo 'onlone' /sys/devices/system/memory/memory/32/state then after several seconds the console prints error messages like: nommu_map_sg: overflow 107c15000+4096 of device mask ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 ata1.00: cmd ca/00:10:d0:0d:a4/00:00:00:00:00/e0 tag 0 dma 8192 out res 50/00:00:08:09:e0/00:00:00:00:00/e0 Emask 0x40 (internal error) ata1.00: configured for MWDMA2 ata1: EH complete (repeat) and can't do any disk I/O.
Re: [Qemu-devel] [PATCH 00/16 RFC v6] ACPI memory hotplug
On Wed, 24 Jul 2013 17:52:50 +0800 Hu Tao hu...@cn.fujitsu.com wrote: v6 doesn't work here, things are going fine until online hotplugged memory in guest. steps: 1. qemu cmd: ./x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 512,maxmem=2G,slots=1 \ -hda /mnt/data/libvirt-images/hut-rhel6.3.img -L ../pc-bios-memhp/ (bios is from MST's acpi tree) 2. hot-plug a dimm: device_adddimm,id=d0,size=1G 3. online hotplugged memory(in guest): echo 'onlone' /sys/devices/system/memory/memory/32/state then after several seconds the console prints error messages like: nommu_map_sg: overflow 107c15000+4096 of device mask ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 ata1.00: cmd ca/00:10:d0:0d:a4/00:00:00:00:00/e0 tag 0 dma 8192 out res 50/00:00:08:09:e0/00:00:00:00:00/e0 Emask 0x40 (internal error) ata1.00: configured for MWDMA2 ata1: EH complete (repeat) and can't do any disk I/O. Looks like a guest bug where it tries to use high memory but assumes low one. if you boot guest with initial memory 4Gb then it wont hit issue or use FC18 which doesn't have this problem.
Re: [Qemu-devel] [PATCH 00/16 RFC v6] ACPI memory hotplug
On Wed, Jul 24, 2013 at 12:02:46PM +0200, Igor Mammedov wrote: On Wed, 24 Jul 2013 17:52:50 +0800 Hu Tao hu...@cn.fujitsu.com wrote: v6 doesn't work here, things are going fine until online hotplugged memory in guest. steps: 1. qemu cmd: ./x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 512,maxmem=2G,slots=1 \ -hda /mnt/data/libvirt-images/hut-rhel6.3.img -L ../pc-bios-memhp/ (bios is from MST's acpi tree) 2. hot-plug a dimm: device_adddimm,id=d0,size=1G 3. online hotplugged memory(in guest): echo 'onlone' /sys/devices/system/memory/memory/32/state then after several seconds the console prints error messages like: nommu_map_sg: overflow 107c15000+4096 of device mask ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 ata1.00: cmd ca/00:10:d0:0d:a4/00:00:00:00:00/e0 tag 0 dma 8192 out res 50/00:00:08:09:e0/00:00:00:00:00/e0 Emask 0x40 (internal error) ata1.00: configured for MWDMA2 ata1: EH complete (repeat) and can't do any disk I/O. Looks like a guest bug where it tries to use high memory but assumes low one. yes. Iirc booting the guest kernel with swiotlb=force option could also work around this. if you boot guest with initial memory 4Gb then it wont hit issue or use FC18 which doesn't have this problem. thanks, - Vasilis
