Re: arm64: dropping prevent_bootmem_remove_notifier
Hi Anshuman, David, Thanks for all the detailed explanations for the reasoning to have bootmem protected from being removed. Also, I do agree drivers being able to mark memory sections isn't the right thing to do. We went ahead with the approach of using "mem=" as you suggested to limit the bootmem and add remaining blocks using add_memory_driver_managed() so that driver has ownership of these blocks. We do have some follow-up questions regarding this - will initiate a discussion soon. On 2020-10-18 22:37, Anshuman Khandual wrote: Hello Sudarshan, On 10/17/2020 04:41 AM, Sudarshan Rajagopalan wrote: Hello Anshuman, In the patch that enables memory hot-remove (commit bbd6ec605c0f ("arm64/mm: Enable memory hot remove")) for arm64, there’s a notifier put in place that prevents boot memory from being offlined and removed. Also commit text mentions that boot memory on arm64 cannot be removed. We wanted to understand more about the reasoning for this. X86 and other archs doesn’t seem to do this prevention. There’s also comment in the code that this notifier could be dropped in future if and when boot memory can be removed. Right and till then the notifier cannot be dropped. There was a lot of discussions around this topic during multiple iterations of memory hot remove series. Hence, I would just request you to please go through them first. This list here is from one such series (https://lwn.net/Articles/809179/) but might not be exhaustive. - On arm64 platform, it is essential to ensure that the boot time discovered memory couldn't be hot-removed so that, 1. FW data structures used across kexec are idempotent e.g. the EFI memory map. 2. linear map or vmemmap would not have to be dynamically split, and can map boot memory at a large granularity 3. Avoid penalizing paths that have to walk page tables, where we can be certain that the memory is not hot-removable - The primary reason being kexec which would need substantial rework otherwise. The current logic is that only “new” memory blocks which are hot-added can later be offlined and removed. The memory that system booted up with cannot be offlined and removed. But there could be many usercases such as inter-VM memory sharing where a primary VM could offline and hot-remove a block/section of memory and lend it to secondary VM where it could hot-add it. And after usecase is done, the reverse happens where secondary VM hot-removes and gives it back to primary which can hot-add it back. In such cases, the present logic for arm64 doesn’t allow this hot-remove in primary to happen. That is not true. Each VM could just boot with a minimum boot memory which can not be offlined or removed but then a possible larger portion of memory can be hot added during the boot process itself, making them available for any future inter VM sharing purpose. Hence this problem could easily be solved in the user space itself. Also, on systems with movable zone that sort of guarantees pages to be migrated and isolated so that blocks can be offlined, this logic also defeats the purpose of having a movable zone which system can rely on memory hot-plugging, which say virt-io mem also relies on for fully plugged memory blocks. ZONE_MOVABLE does not really guarantee migration, isolation and removal. There are reasons an offline request might just fail. I agree that those reasons are normally not platform related but core memory gives platform an opportunity to decline an offlining request via a notifier. Hence ZONE_MOVABLE offline can be denied. Semantics wise we are still okay. This might look bit inconsistent that movablecore/kernelcore/movable_node with firmware sending in 'hot pluggable' memory (IIRC arm64 does not really support this yet), the system might end up with ZONE_MOVABLE marked boot memory which cannot be offlined or removed. But an offline notifier action is orthogonal. Hence did not block those kernel command line paths that creates ZONE_MOVABLE during boot to preserve existing behavior. I understand that some region of boot RAM shouldn’t be allowed to be removed, but such regions won’t be allowed to be offlined in first place since pages cannot be migrated and isolated, example reserved pages. So we’re trying to understand the reasoning for such a prevention put in place for arm64 arch alone. Primary reason being kexec. During kexec on arm64, next kernel's memory map is derived from firmware and not from current running kernel. So the next kernel will crash if it would access memory that might have been removed in running kernel. Until kexec on arm64 changes substantially and takes into account the real available memory on the current kernel, boot memory cannot be removed. One possible way to solve this is by marking the required sections as “non-early” by removing the SECTION_IS_EARLY bit in its section_mem_map. That is too
Re: arm64: dropping prevent_bootmem_remove_notifier
On 10/17/2020 03:05 PM, David Hildenbrand wrote: > On 17.10.20 01:11, Sudarshan Rajagopalan wrote: >> >> Hello Anshuman, >> > David here, > > in general, if your driver offlines+removes random memory, it is doing > something *very* wrong and dangerous. You shouldn't ever be > offlining+removing memory unless > a) you own that boot memory after boot. E.g., the ACPI driver owns DIMMs > after a reboot. > b) you added that memory via add_memory() and friends. Right. > > Even trusting that offline memory can be used by your driver is wrong. Right. > > Just imagine you racing with actual memory hot(un)plug, you'll be in > *big* trouble. For example, > > 1. You offlined memory and assume you can use it. A DIMM can simply get > unplugged. you're doomed. > 2. You offlined+removed memory and assume you can use it. A DIMM can > simply get unplugged and the whole machine would crash. > > Or imagine your driver running on a system that has virtio-mem, which > will try to remove/offline+remove memory that was added by virtio-mem/ > is under its control. > > Long story short: don't do it. > > There is *one* instance in Linux where we currently allow it for legacy > reasons. It is powernv/memtrace code that offlines+removes boot memory. > But here we are guaranteed to run in an environment (HW) without any > actual memory hot(un)plug. > > I guess you're going to say "but in our environment we don't have ..." - > this is not a valid argument to change such generic things upstream / > introducing such hacks. Agreed. > >> In the patch that enables memory hot-remove (commit bbd6ec605c0f >> ("arm64/mm: Enable memory hot remove")) for arm64, there’s a notifier >> put in place that prevents boot memory from being offlined and removed. >> Also commit text mentions that boot memory on arm64 cannot be removed. >> We wanted to understand more about the reasoning for this. X86 and other >> archs doesn’t seem to do this prevention. There’s also comment in the >> code that this notifier could be dropped in future if and when boot >> memory can be removed. > > The issue is that with *actual* memory hotunplug (for what the whole > machinery should be used for), that memory/DIMM will be gone. And as you > cannot fixup the initial memmap, if you were to reboot that machine, you > would simply crash immediately. Right. > > On x86, you can have that easily: hotplug DIMMs on bare metal and > reboot. The DIMMs will be exposed via e820 during boot, so they are > "early", although if done right (movable_node, movable_core and > similar), they can get hotunplugged later. Important in environments > where you want to hotunplug whole nodes. But has HW on x86 will properly > adjust the initial memmap / e820, there is no such issue as on arm64. That is the primary problem. > >> >> The current logic is that only “new” memory blocks which are hot-added >> can later be offlined and removed. The memory that system booted up with >> cannot be offlined and removed. But there could be many usercases such >> as inter-VM memory sharing where a primary VM could offline and >> hot-remove a block/section of memory and lend it to secondary VM where >> it could hot-add it. And after usecase is done, the reverse happens > > That use case is using the wrong mechanisms. It shouldn't be > offlining+removing memory. Read below. > >> where secondary VM hot-removes and gives it back to primary which can >> hot-add it back. In such cases, the present logic for arm64 doesn’t >> allow this hot-remove in primary to happen. >> >> Also, on systems with movable zone that sort of guarantees pages to be >> migrated and isolated so that blocks can be offlined, this logic also >> defeats the purpose of having a movable zone which system can rely on >> memory hot-plugging, which say virt-io mem also relies on for fully >> plugged memory blocks. > > The MOVABLE_ZONE is *not* just for better guarantees when trying to > hotunplug memory. It also increases the number of THP/huge pages. And > that part works just fine. Right. > >> >> So we’re trying to understand the reasoning for such a prevention put in >> place for arm64 arch alone. >> >> One possible way to solve this is by marking the required sections as >> “non-early” by removing the SECTION_IS_EARLY bit in its section_mem_map. >> This puts these sections in the context of “memory hotpluggable” which >> can be offlined-removed and added-onlined which are part of boot RAM >> itself and doesn’t need any extra blocks to be hot added. This way of >> marking certain sections as “non-early” could be exported so that module >> drivers can set the required number of sections as “memory > > Oh please no. No driver should be doing that. That's just hacking around > the root issue: you're not supposed to do that. > >> hotpluggable”. This could have certain checks put in place to see which >> sections are allowed, example only movable zone sections can be marked >> as “non-early”.
Re: arm64: dropping prevent_bootmem_remove_notifier
Hello Sudarshan, On 10/17/2020 04:41 AM, Sudarshan Rajagopalan wrote: > > Hello Anshuman, > > In the patch that enables memory hot-remove (commit bbd6ec605c0f ("arm64/mm: > Enable memory hot remove")) for arm64, there’s a notifier put in place that > prevents boot memory from being offlined and removed. Also commit text > mentions that boot memory on arm64 cannot be removed. We wanted to understand > more about the reasoning for this. X86 and other archs doesn’t seem to do > this prevention. There’s also comment in the code that this notifier could be > dropped in future if and when boot memory can be removed. Right and till then the notifier cannot be dropped. There was a lot of discussions around this topic during multiple iterations of memory hot remove series. Hence, I would just request you to please go through them first. This list here is from one such series (https://lwn.net/Articles/809179/) but might not be exhaustive. - On arm64 platform, it is essential to ensure that the boot time discovered memory couldn't be hot-removed so that, 1. FW data structures used across kexec are idempotent e.g. the EFI memory map. 2. linear map or vmemmap would not have to be dynamically split, and can map boot memory at a large granularity 3. Avoid penalizing paths that have to walk page tables, where we can be certain that the memory is not hot-removable - The primary reason being kexec which would need substantial rework otherwise. > > The current logic is that only “new” memory blocks which are hot-added can > later be offlined and removed. The memory that system booted up with cannot > be offlined and removed. But there could be many usercases such as inter-VM > memory sharing where a primary VM could offline and hot-remove a > block/section of memory and lend it to secondary VM where it could hot-add > it. And after usecase is done, the reverse happens where secondary VM > hot-removes and gives it back to primary which can hot-add it back. In such > cases, the present logic for arm64 doesn’t allow this hot-remove in primary > to happen. That is not true. Each VM could just boot with a minimum boot memory which can not be offlined or removed but then a possible larger portion of memory can be hot added during the boot process itself, making them available for any future inter VM sharing purpose. Hence this problem could easily be solved in the user space itself. > > Also, on systems with movable zone that sort of guarantees pages to be > migrated and isolated so that blocks can be offlined, this logic also defeats > the purpose of having a movable zone which system can rely on memory > hot-plugging, which say virt-io mem also relies on for fully plugged memory > blocks. ZONE_MOVABLE does not really guarantee migration, isolation and removal. There are reasons an offline request might just fail. I agree that those reasons are normally not platform related but core memory gives platform an opportunity to decline an offlining request via a notifier. Hence ZONE_MOVABLE offline can be denied. Semantics wise we are still okay. This might look bit inconsistent that movablecore/kernelcore/movable_node with firmware sending in 'hot pluggable' memory (IIRC arm64 does not really support this yet), the system might end up with ZONE_MOVABLE marked boot memory which cannot be offlined or removed. But an offline notifier action is orthogonal. Hence did not block those kernel command line paths that creates ZONE_MOVABLE during boot to preserve existing behavior. > > I understand that some region of boot RAM shouldn’t be allowed to be removed, > but such regions won’t be allowed to be offlined in first place since pages > cannot be migrated and isolated, example reserved pages. > > So we’re trying to understand the reasoning for such a prevention put in > place for arm64 arch alone. Primary reason being kexec. During kexec on arm64, next kernel's memory map is derived from firmware and not from current running kernel. So the next kernel will crash if it would access memory that might have been removed in running kernel. Until kexec on arm64 changes substantially and takes into account the real available memory on the current kernel, boot memory cannot be removed. > > One possible way to solve this is by marking the required sections as > “non-early” by removing the SECTION_IS_EARLY bit in its section_mem_map. That is too intrusive from core memory perspective. This puts these sections in the context of “memory hotpluggable” which can be offlined-removed and added-onlined which are part of boot RAM itself and doesn’t need any extra blocks to be hot added. This way of marking certain sections as “non-early” could be exported so that module drivers can set the required number of sections as “memory hotpluggable”. This could have certain checks put in place to see which sections are allowed, example only movable zone sections can be
Re: arm64: dropping prevent_bootmem_remove_notifier
On 17.10.20 01:11, Sudarshan Rajagopalan wrote: > > Hello Anshuman, > David here, in general, if your driver offlines+removes random memory, it is doing something *very* wrong and dangerous. You shouldn't ever be offlining+removing memory unless a) you own that boot memory after boot. E.g., the ACPI driver owns DIMMs after a reboot. b) you added that memory via add_memory() and friends. Even trusting that offline memory can be used by your driver is wrong. Just imagine you racing with actual memory hot(un)plug, you'll be in *big* trouble. For example, 1. You offlined memory and assume you can use it. A DIMM can simply get unplugged. you're doomed. 2. You offlined+removed memory and assume you can use it. A DIMM can simply get unplugged and the whole machine would crash. Or imagine your driver running on a system that has virtio-mem, which will try to remove/offline+remove memory that was added by virtio-mem/ is under its control. Long story short: don't do it. There is *one* instance in Linux where we currently allow it for legacy reasons. It is powernv/memtrace code that offlines+removes boot memory. But here we are guaranteed to run in an environment (HW) without any actual memory hot(un)plug. I guess you're going to say "but in our environment we don't have ..." - this is not a valid argument to change such generic things upstream / introducing such hacks. > In the patch that enables memory hot-remove (commit bbd6ec605c0f > ("arm64/mm: Enable memory hot remove")) for arm64, there’s a notifier > put in place that prevents boot memory from being offlined and removed. > Also commit text mentions that boot memory on arm64 cannot be removed. > We wanted to understand more about the reasoning for this. X86 and other > archs doesn’t seem to do this prevention. There’s also comment in the > code that this notifier could be dropped in future if and when boot > memory can be removed. The issue is that with *actual* memory hotunplug (for what the whole machinery should be used for), that memory/DIMM will be gone. And as you cannot fixup the initial memmap, if you were to reboot that machine, you would simply crash immediately. On x86, you can have that easily: hotplug DIMMs on bare metal and reboot. The DIMMs will be exposed via e820 during boot, so they are "early", although if done right (movable_node, movable_core and similar), they can get hotunplugged later. Important in environments where you want to hotunplug whole nodes. But has HW on x86 will properly adjust the initial memmap / e820, there is no such issue as on arm64. > > The current logic is that only “new” memory blocks which are hot-added > can later be offlined and removed. The memory that system booted up with > cannot be offlined and removed. But there could be many usercases such > as inter-VM memory sharing where a primary VM could offline and > hot-remove a block/section of memory and lend it to secondary VM where > it could hot-add it. And after usecase is done, the reverse happens That use case is using the wrong mechanisms. It shouldn't be offlining+removing memory. Read below. > where secondary VM hot-removes and gives it back to primary which can > hot-add it back. In such cases, the present logic for arm64 doesn’t > allow this hot-remove in primary to happen. > > Also, on systems with movable zone that sort of guarantees pages to be > migrated and isolated so that blocks can be offlined, this logic also > defeats the purpose of having a movable zone which system can rely on > memory hot-plugging, which say virt-io mem also relies on for fully > plugged memory blocks. The MOVABLE_ZONE is *not* just for better guarantees when trying to hotunplug memory. It also increases the number of THP/huge pages. And that part works just fine. > > So we’re trying to understand the reasoning for such a prevention put in > place for arm64 arch alone. > > One possible way to solve this is by marking the required sections as > “non-early” by removing the SECTION_IS_EARLY bit in its section_mem_map. > This puts these sections in the context of “memory hotpluggable” which > can be offlined-removed and added-onlined which are part of boot RAM > itself and doesn’t need any extra blocks to be hot added. This way of > marking certain sections as “non-early” could be exported so that module > drivers can set the required number of sections as “memory Oh please no. No driver should be doing that. That's just hacking around the root issue: you're not supposed to do that. > hotpluggable”. This could have certain checks put in place to see which > sections are allowed, example only movable zone sections can be marked > as “non-early”. > I assume what your use case wants to achieve is, starting VMs with large, contiguous memory backings, not wasting memory for the memmap in the hypervisor. The "traditional" way of doing that is using the "mem=" boot parameter, and starting VMs with memory within the "never
arm64: dropping prevent_bootmem_remove_notifier
Hello Anshuman, In the patch that enables memory hot-remove (commit bbd6ec605c0f ("arm64/mm: Enable memory hot remove")) for arm64, there’s a notifier put in place that prevents boot memory from being offlined and removed. Also commit text mentions that boot memory on arm64 cannot be removed. We wanted to understand more about the reasoning for this. X86 and other archs doesn’t seem to do this prevention. There’s also comment in the code that this notifier could be dropped in future if and when boot memory can be removed. The current logic is that only “new” memory blocks which are hot-added can later be offlined and removed. The memory that system booted up with cannot be offlined and removed. But there could be many usercases such as inter-VM memory sharing where a primary VM could offline and hot-remove a block/section of memory and lend it to secondary VM where it could hot-add it. And after usecase is done, the reverse happens where secondary VM hot-removes and gives it back to primary which can hot-add it back. In such cases, the present logic for arm64 doesn’t allow this hot-remove in primary to happen. Also, on systems with movable zone that sort of guarantees pages to be migrated and isolated so that blocks can be offlined, this logic also defeats the purpose of having a movable zone which system can rely on memory hot-plugging, which say virt-io mem also relies on for fully plugged memory blocks. I understand that some region of boot RAM shouldn’t be allowed to be removed, but such regions won’t be allowed to be offlined in first place since pages cannot be migrated and isolated, example reserved pages. So we’re trying to understand the reasoning for such a prevention put in place for arm64 arch alone. One possible way to solve this is by marking the required sections as “non-early” by removing the SECTION_IS_EARLY bit in its section_mem_map. This puts these sections in the context of “memory hotpluggable” which can be offlined-removed and added-onlined which are part of boot RAM itself and doesn’t need any extra blocks to be hot added. This way of marking certain sections as “non-early” could be exported so that module drivers can set the required number of sections as “memory hotpluggable”. This could have certain checks put in place to see which sections are allowed, example only movable zone sections can be marked as “non-early”. Your thoughts on this? We are also looking for different ways to solve the problem without having to completely dropping this notifier, but just putting out the concern here about the notifier logic that is breaking our usecase which is a generic memory sharing usecase using memory hotplug feature. Sudarshan -- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project