Re: NVDIMM sizes and DIMM hot plug
On Thu, Jun 18, 2020 at 11:22:46AM +0200, Milan Zamazal wrote: Peter Krempa writes: On Tue, Jun 16, 2020 at 12:54:29 +0200, Milan Zamazal wrote: Hi, I've found out that NVDIMM size and label size matter for regular (non-NV) DIMM hot plug. If the NVDIMM is not aligned correctly, the guest OS will not accept the hot plugged memory and will complain with messages such as Block size [0x800] unaligned hotplug range: start 0x22500, size 0x1000 The start address above is also reported within element of the hot plugged memory in the domain XML: Apparently, in order to make memory hot plug working in the guest OS, the inserted memory must be aligned to the platform memory alignment (128 MB on x86_64). I'd like to clarify, how libvirt makes the DIMM address above. How is If the address isn't provided in the device XML of the attached device, libvirt attaches the device without any address at all and then refreshes the address from qemu in 'qemuDomainUpdateMemoryDeviceInfo'. OK, I can look into the QEMU source code, but I'd still like to have some official confirmation, especially regarding possible pitfalls or future changes. We can't risk data loss. the NVDIMM memory range determined? According to my experiments, it seems the NVDIMM specified is taken, NVDIMM size is subtracted from it and the resulting value is reduced to the nearest multiple of NVDIMM . Is this observation correct? Is it guaranteed to be stable in future versions? I need to determine the right NVDIMM size to make the subsequent memory modules correctly aligned and then I can't change the NVDIMM size, to not damage data stored in the NVDIMM. Unfortunatelly I didn't implement NVDIMM support so I don't know the intricacies. I've cc'd Martin Kletzander who did that part. Martin, do you know how the QEMU part is supposed to work? I haven't received any response on the QEMU list, do you know who could I ask directly? Sorry, no idea, it was actually Michal (Cc'd) who implemented it if I'm not mistaken. Additionally, when adjusting maxMemory due to NVDIMM presence, should I increase it by the specified NVDIMM or a different value? Thank you, Milan signature.asc Description: PGP signature
Re: NVDIMM sizes and DIMM hot plug
Peter Krempa writes: > On Tue, Jun 16, 2020 at 12:54:29 +0200, Milan Zamazal wrote: >> Hi, >> > >> I've found out that NVDIMM size and label size matter for regular >> (non-NV) DIMM hot plug. If the NVDIMM is not aligned correctly, the >> guest OS will not accept the hot plugged memory and will complain with >> messages such as >> >> Block size [0x800] unaligned hotplug range: start 0x22500, size >> 0x1000 >> >> The start address above is also reported within element of the >> hot plugged memory in the domain XML: >> >> >> >> Apparently, in order to make memory hot plug working in the guest OS, >> the inserted memory must be aligned to the platform memory alignment >> (128 MB on x86_64). >> >> I'd like to clarify, how libvirt makes the DIMM address above. How is > > If the address isn't provided in the device XML of the attached device, > libvirt attaches the device without any address at all and then > refreshes the address from qemu in 'qemuDomainUpdateMemoryDeviceInfo'. OK, I can look into the QEMU source code, but I'd still like to have some official confirmation, especially regarding possible pitfalls or future changes. We can't risk data loss. >> the NVDIMM memory range determined? According to my experiments, it >> seems the NVDIMM specified is taken, NVDIMM size is >> subtracted from it and the resulting value is reduced to the nearest >> multiple of NVDIMM . Is this observation correct? Is it >> guaranteed to be stable in future versions? I need to determine the >> right NVDIMM size to make the subsequent memory modules correctly >> aligned and then I can't change the NVDIMM size, to not damage data >> stored in the NVDIMM. > > > Unfortunatelly I didn't implement NVDIMM support so I don't know the > intricacies. I've cc'd Martin Kletzander who did that part. Martin, do you know how the QEMU part is supposed to work? I haven't received any response on the QEMU list, do you know who could I ask directly? >> Additionally, when adjusting maxMemory due to NVDIMM presence, should I >> increase it by the specified NVDIMM or a different value? >> >> Thank you, >> Milan >>
Re: NVDIMM sizes and DIMM hot plug
Daniel P. Berrangé writes: > On Tue, Jun 16, 2020 at 12:54:29PM +0200, Milan Zamazal wrote: >> Hi, >> > >> I've found out that NVDIMM size and label size matter for regular >> (non-NV) DIMM hot plug. If the NVDIMM is not aligned correctly, the >> guest OS will not accept the hot plugged memory and will complain with >> messages such as >> >> Block size [0x800] unaligned hotplug range: start 0x22500, size >> 0x1000 >> >> The start address above is also reported within element of the >> hot plugged memory in the domain XML: >> >> >> >> Apparently, in order to make memory hot plug working in the guest OS, >> the inserted memory must be aligned to the platform memory alignment >> (128 MB on x86_64). >> >> I'd like to clarify, how libvirt makes the DIMM address above. How is >> the NVDIMM memory range determined? According to my experiments, it >> seems the NVDIMM specified is taken, NVDIMM size is >> subtracted from it and the resulting value is reduced to the nearest >> multiple of NVDIMM . Is this observation correct? Is it >> guaranteed to be stable in future versions? I need to determine the >> right NVDIMM size to make the subsequent memory modules correctly >> aligned and then I can't change the NVDIMM size, to not damage data >> stored in the NVDIMM. > > Libvirt doesn't ever assign a "base" address value itself. We just > start QEMU, and then fill in the XML "base" with the value that QEMU > has assigned. I see, then I'll ask about it on the QEMU list. >> Additionally, when adjusting maxMemory due to NVDIMM presence, should I >> increase it by the specified NVDIMM or a different value? > > IIRC, maxMemory has to allow for the sum of the basic RAM amount, plus > RAM intended to be used for all possible future (NV)DIMMS that will be > hotplugged. OK. Thanks, Milan
Re: NVDIMM sizes and DIMM hot plug
On Tue, Jun 16, 2020 at 12:54:29 +0200, Milan Zamazal wrote: > Hi, > > I've found out that NVDIMM size and label size matter for regular > (non-NV) DIMM hot plug. If the NVDIMM is not aligned correctly, the > guest OS will not accept the hot plugged memory and will complain with > messages such as > > Block size [0x800] unaligned hotplug range: start 0x22500, size > 0x1000 > > The start address above is also reported within element of the > hot plugged memory in the domain XML: > > > > Apparently, in order to make memory hot plug working in the guest OS, > the inserted memory must be aligned to the platform memory alignment > (128 MB on x86_64). > > I'd like to clarify, how libvirt makes the DIMM address above. How is If the address isn't provided in the device XML of the attached device, libvirt attaches the device without any address at all and then refreshes the address from qemu in 'qemuDomainUpdateMemoryDeviceInfo'. > the NVDIMM memory range determined? According to my experiments, it > seems the NVDIMM specified is taken, NVDIMM size is > subtracted from it and the resulting value is reduced to the nearest > multiple of NVDIMM . Is this observation correct? Is it > guaranteed to be stable in future versions? I need to determine the > right NVDIMM size to make the subsequent memory modules correctly > aligned and then I can't change the NVDIMM size, to not damage data > stored in the NVDIMM. Unfortunatelly I didn't implement NVDIMM support so I don't know the intricacies. I've cc'd Martin Kletzander who did that part. > > Additionally, when adjusting maxMemory due to NVDIMM presence, should I > increase it by the specified NVDIMM or a different value? > > Thank you, > Milan >
Re: NVDIMM sizes and DIMM hot plug
On Tue, Jun 16, 2020 at 12:54:29PM +0200, Milan Zamazal wrote: > Hi, > > I've found out that NVDIMM size and label size matter for regular > (non-NV) DIMM hot plug. If the NVDIMM is not aligned correctly, the > guest OS will not accept the hot plugged memory and will complain with > messages such as > > Block size [0x800] unaligned hotplug range: start 0x22500, size > 0x1000 > > The start address above is also reported within element of the > hot plugged memory in the domain XML: > > > > Apparently, in order to make memory hot plug working in the guest OS, > the inserted memory must be aligned to the platform memory alignment > (128 MB on x86_64). > > I'd like to clarify, how libvirt makes the DIMM address above. How is > the NVDIMM memory range determined? According to my experiments, it > seems the NVDIMM specified is taken, NVDIMM size is > subtracted from it and the resulting value is reduced to the nearest > multiple of NVDIMM . Is this observation correct? Is it > guaranteed to be stable in future versions? I need to determine the > right NVDIMM size to make the subsequent memory modules correctly > aligned and then I can't change the NVDIMM size, to not damage data > stored in the NVDIMM. Libvirt doesn't ever assign a "base" address value itself. We just start QEMU, and then fill in the XML "base" with the value that QEMU has assigned. > Additionally, when adjusting maxMemory due to NVDIMM presence, should I > increase it by the specified NVDIMM or a different value? IIRC, maxMemory has to allow for the sum of the basic RAM amount, plus RAM intended to be used for all possible future (NV)DIMMS that will be hotplugged. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
NVDIMM sizes and DIMM hot plug
Hi, I've found out that NVDIMM size and label size matter for regular (non-NV) DIMM hot plug. If the NVDIMM is not aligned correctly, the guest OS will not accept the hot plugged memory and will complain with messages such as Block size [0x800] unaligned hotplug range: start 0x22500, size 0x1000 The start address above is also reported within element of the hot plugged memory in the domain XML: Apparently, in order to make memory hot plug working in the guest OS, the inserted memory must be aligned to the platform memory alignment (128 MB on x86_64). I'd like to clarify, how libvirt makes the DIMM address above. How is the NVDIMM memory range determined? According to my experiments, it seems the NVDIMM specified is taken, NVDIMM size is subtracted from it and the resulting value is reduced to the nearest multiple of NVDIMM . Is this observation correct? Is it guaranteed to be stable in future versions? I need to determine the right NVDIMM size to make the subsequent memory modules correctly aligned and then I can't change the NVDIMM size, to not damage data stored in the NVDIMM. Additionally, when adjusting maxMemory due to NVDIMM presence, should I increase it by the specified NVDIMM or a different value? Thank you, Milan
