Re: [PATCH v3 4/4] xen: add helpers to allocate unpopulated memory
On Tue, Jul 28, 2020 at 06:12:46PM +0100, Julien Grall wrote: > Hi Roger, > > On 28/07/2020 17:59, Roger Pau Monné wrote: > > On Tue, Jul 28, 2020 at 05:48:23PM +0100, Julien Grall wrote: > > > Hi, > > > > > > On 27/07/2020 10:13, Roger Pau Monne wrote: > > > > To be used in order to create foreign mappings. This is based on the > > > > ZONE_DEVICE facility which is used by persistent memory devices in > > > > order to create struct pages and kernel virtual mappings for the IOMEM > > > > areas of such devices. Note that on kernels without support for > > > > ZONE_DEVICE Xen will fallback to use ballooned pages in order to > > > > create foreign mappings. > > > > > > > > The newly added helpers use the same parameters as the existing > > > > {alloc/free}_xenballooned_pages functions, which allows for in-place > > > > replacement of the callers. Once a memory region has been added to be > > > > used as scratch mapping space it will no longer be released, and pages > > > > returned are kept in a linked list. This allows to have a buffer of > > > > pages and prevents resorting to frequent additions and removals of > > > > regions. > > > > > > > > If enabled (because ZONE_DEVICE is supported) the usage of the new > > > > functionality untangles Xen balloon and RAM hotplug from the usage of > > > > unpopulated physical memory ranges to map foreign pages, which is the > > > > correct thing to do in order to avoid mappings of foreign pages depend > > > > on memory hotplug. > > > I think this is going to break Dom0 on Arm if the kernel has been built > > > with > > > hotplug. This is because you may end up to re-use region that will be used > > > for the 1:1 mapping of a foreign map. > > > > > > Note that I don't know whether hotplug has been tested on Xen on Arm yet. > > > So > > > it might be possible to be already broken. > > > > > > Meanwhile, my suggestion would be to make the use of hotplug in the > > > balloon > > > code conditional (maybe using CONFIG_ARM64 and CONFIG_ARM)? > > > > Right, this feature (allocation of unpopulated memory separated from > > the balloon driver) is currently gated on CONFIG_ZONE_DEVICE, which I > > think could be used on Arm. > > > > IMO the right solution seems to be to subtract the physical memory > > regions that can be used for the identity mappings of foreign pages > > (all RAM on the system AFAICT) from iomem_resource, as that would make > > this and the memory hotplug done in the balloon driver safe? > > Dom0 doesn't know the regions used for the identity mappings as this is only > managed by Xen. So there is nothing you can really do here. OK, I will add the guards to prevent this being built on Arm. > But don't you have the same issue on x86 with "magic pages"? Those are marked as reserved on the memory map, and hence I would expect them to never end up in iomem_resource. Thanks, Roger.
Re: [PATCH v3 4/4] xen: add helpers to allocate unpopulated memory
On Tue, Jul 28, 2020 at 06:06:25PM +0100, Andrew Cooper wrote: > On 28/07/2020 17:59, Roger Pau Monné wrote: > > On Tue, Jul 28, 2020 at 05:48:23PM +0100, Julien Grall wrote: > >> Hi, > >> > >> On 27/07/2020 10:13, Roger Pau Monne wrote: > >>> To be used in order to create foreign mappings. This is based on the > >>> ZONE_DEVICE facility which is used by persistent memory devices in > >>> order to create struct pages and kernel virtual mappings for the IOMEM > >>> areas of such devices. Note that on kernels without support for > >>> ZONE_DEVICE Xen will fallback to use ballooned pages in order to > >>> create foreign mappings. > >>> > >>> The newly added helpers use the same parameters as the existing > >>> {alloc/free}_xenballooned_pages functions, which allows for in-place > >>> replacement of the callers. Once a memory region has been added to be > >>> used as scratch mapping space it will no longer be released, and pages > >>> returned are kept in a linked list. This allows to have a buffer of > >>> pages and prevents resorting to frequent additions and removals of > >>> regions. > >>> > >>> If enabled (because ZONE_DEVICE is supported) the usage of the new > >>> functionality untangles Xen balloon and RAM hotplug from the usage of > >>> unpopulated physical memory ranges to map foreign pages, which is the > >>> correct thing to do in order to avoid mappings of foreign pages depend > >>> on memory hotplug. > >> I think this is going to break Dom0 on Arm if the kernel has been built > >> with > >> hotplug. This is because you may end up to re-use region that will be used > >> for the 1:1 mapping of a foreign map. > >> > >> Note that I don't know whether hotplug has been tested on Xen on Arm yet. > >> So > >> it might be possible to be already broken. > >> > >> Meanwhile, my suggestion would be to make the use of hotplug in the balloon > >> code conditional (maybe using CONFIG_ARM64 and CONFIG_ARM)? > > Right, this feature (allocation of unpopulated memory separated from > > the balloon driver) is currently gated on CONFIG_ZONE_DEVICE, which I > > think could be used on Arm. > > > > IMO the right solution seems to be to subtract the physical memory > > regions that can be used for the identity mappings of foreign pages > > (all RAM on the system AFAICT) from iomem_resource, as that would make > > this and the memory hotplug done in the balloon driver safe? > > The right solution is a mechanism for translated guests to query Xen to > find regions of guest physical address space which are unused, and can > be safely be used for foreign/grant/other mappings. > > Please don't waste any more time applying more duct tape to a broken > system, and instead spend the time organising some proper foundations. The piece added here (using ZONE_DEVICE) will be relevant when Xen can provide the space to map foreign pages, it's just that right now it relies on iomem_resource instead of a Xen specific resource map that should be provided by the hypervisor. It should indeed be fixed, but right now this patch should allow a PVH dom0 to work slightly better. When Xen provides such areas Linux just needs to populate a custom Xen resource with them and use it instead of iomem_resurce. The Arm stuff I'm certainly not familiar with, and can't provide much insight on that. If it's best to just disable it and continue to rely on ballooned out pages that's fine. Roger.
Re: [PATCH v3 4/4] xen: add helpers to allocate unpopulated memory
Hi Roger, On 28/07/2020 17:59, Roger Pau Monné wrote: On Tue, Jul 28, 2020 at 05:48:23PM +0100, Julien Grall wrote: Hi, On 27/07/2020 10:13, Roger Pau Monne wrote: To be used in order to create foreign mappings. This is based on the ZONE_DEVICE facility which is used by persistent memory devices in order to create struct pages and kernel virtual mappings for the IOMEM areas of such devices. Note that on kernels without support for ZONE_DEVICE Xen will fallback to use ballooned pages in order to create foreign mappings. The newly added helpers use the same parameters as the existing {alloc/free}_xenballooned_pages functions, which allows for in-place replacement of the callers. Once a memory region has been added to be used as scratch mapping space it will no longer be released, and pages returned are kept in a linked list. This allows to have a buffer of pages and prevents resorting to frequent additions and removals of regions. If enabled (because ZONE_DEVICE is supported) the usage of the new functionality untangles Xen balloon and RAM hotplug from the usage of unpopulated physical memory ranges to map foreign pages, which is the correct thing to do in order to avoid mappings of foreign pages depend on memory hotplug. I think this is going to break Dom0 on Arm if the kernel has been built with hotplug. This is because you may end up to re-use region that will be used for the 1:1 mapping of a foreign map. Note that I don't know whether hotplug has been tested on Xen on Arm yet. So it might be possible to be already broken. Meanwhile, my suggestion would be to make the use of hotplug in the balloon code conditional (maybe using CONFIG_ARM64 and CONFIG_ARM)? Right, this feature (allocation of unpopulated memory separated from the balloon driver) is currently gated on CONFIG_ZONE_DEVICE, which I think could be used on Arm. IMO the right solution seems to be to subtract the physical memory regions that can be used for the identity mappings of foreign pages (all RAM on the system AFAICT) from iomem_resource, as that would make this and the memory hotplug done in the balloon driver safe? Dom0 doesn't know the regions used for the identity mappings as this is only managed by Xen. So there is nothing you can really do here. But don't you have the same issue on x86 with "magic pages"? Cheers, -- Julien Grall
Re: [PATCH v3 4/4] xen: add helpers to allocate unpopulated memory
On 28/07/2020 17:59, Roger Pau Monné wrote: > On Tue, Jul 28, 2020 at 05:48:23PM +0100, Julien Grall wrote: >> Hi, >> >> On 27/07/2020 10:13, Roger Pau Monne wrote: >>> To be used in order to create foreign mappings. This is based on the >>> ZONE_DEVICE facility which is used by persistent memory devices in >>> order to create struct pages and kernel virtual mappings for the IOMEM >>> areas of such devices. Note that on kernels without support for >>> ZONE_DEVICE Xen will fallback to use ballooned pages in order to >>> create foreign mappings. >>> >>> The newly added helpers use the same parameters as the existing >>> {alloc/free}_xenballooned_pages functions, which allows for in-place >>> replacement of the callers. Once a memory region has been added to be >>> used as scratch mapping space it will no longer be released, and pages >>> returned are kept in a linked list. This allows to have a buffer of >>> pages and prevents resorting to frequent additions and removals of >>> regions. >>> >>> If enabled (because ZONE_DEVICE is supported) the usage of the new >>> functionality untangles Xen balloon and RAM hotplug from the usage of >>> unpopulated physical memory ranges to map foreign pages, which is the >>> correct thing to do in order to avoid mappings of foreign pages depend >>> on memory hotplug. >> I think this is going to break Dom0 on Arm if the kernel has been built with >> hotplug. This is because you may end up to re-use region that will be used >> for the 1:1 mapping of a foreign map. >> >> Note that I don't know whether hotplug has been tested on Xen on Arm yet. So >> it might be possible to be already broken. >> >> Meanwhile, my suggestion would be to make the use of hotplug in the balloon >> code conditional (maybe using CONFIG_ARM64 and CONFIG_ARM)? > Right, this feature (allocation of unpopulated memory separated from > the balloon driver) is currently gated on CONFIG_ZONE_DEVICE, which I > think could be used on Arm. > > IMO the right solution seems to be to subtract the physical memory > regions that can be used for the identity mappings of foreign pages > (all RAM on the system AFAICT) from iomem_resource, as that would make > this and the memory hotplug done in the balloon driver safe? The right solution is a mechanism for translated guests to query Xen to find regions of guest physical address space which are unused, and can be safely be used for foreign/grant/other mappings. Please don't waste any more time applying more duct tape to a broken system, and instead spend the time organising some proper foundations. ~Andrew
Re: [PATCH v3 4/4] xen: add helpers to allocate unpopulated memory
On Tue, Jul 28, 2020 at 05:48:23PM +0100, Julien Grall wrote: > Hi, > > On 27/07/2020 10:13, Roger Pau Monne wrote: > > To be used in order to create foreign mappings. This is based on the > > ZONE_DEVICE facility which is used by persistent memory devices in > > order to create struct pages and kernel virtual mappings for the IOMEM > > areas of such devices. Note that on kernels without support for > > ZONE_DEVICE Xen will fallback to use ballooned pages in order to > > create foreign mappings. > > > > The newly added helpers use the same parameters as the existing > > {alloc/free}_xenballooned_pages functions, which allows for in-place > > replacement of the callers. Once a memory region has been added to be > > used as scratch mapping space it will no longer be released, and pages > > returned are kept in a linked list. This allows to have a buffer of > > pages and prevents resorting to frequent additions and removals of > > regions. > > > > If enabled (because ZONE_DEVICE is supported) the usage of the new > > functionality untangles Xen balloon and RAM hotplug from the usage of > > unpopulated physical memory ranges to map foreign pages, which is the > > correct thing to do in order to avoid mappings of foreign pages depend > > on memory hotplug. > I think this is going to break Dom0 on Arm if the kernel has been built with > hotplug. This is because you may end up to re-use region that will be used > for the 1:1 mapping of a foreign map. > > Note that I don't know whether hotplug has been tested on Xen on Arm yet. So > it might be possible to be already broken. > > Meanwhile, my suggestion would be to make the use of hotplug in the balloon > code conditional (maybe using CONFIG_ARM64 and CONFIG_ARM)? Right, this feature (allocation of unpopulated memory separated from the balloon driver) is currently gated on CONFIG_ZONE_DEVICE, which I think could be used on Arm. IMO the right solution seems to be to subtract the physical memory regions that can be used for the identity mappings of foreign pages (all RAM on the system AFAICT) from iomem_resource, as that would make this and the memory hotplug done in the balloon driver safe? Thanks, Roger.
Re: [PATCH v3 4/4] xen: add helpers to allocate unpopulated memory
Hi, On 27/07/2020 10:13, Roger Pau Monne wrote: To be used in order to create foreign mappings. This is based on the ZONE_DEVICE facility which is used by persistent memory devices in order to create struct pages and kernel virtual mappings for the IOMEM areas of such devices. Note that on kernels without support for ZONE_DEVICE Xen will fallback to use ballooned pages in order to create foreign mappings. The newly added helpers use the same parameters as the existing {alloc/free}_xenballooned_pages functions, which allows for in-place replacement of the callers. Once a memory region has been added to be used as scratch mapping space it will no longer be released, and pages returned are kept in a linked list. This allows to have a buffer of pages and prevents resorting to frequent additions and removals of regions. If enabled (because ZONE_DEVICE is supported) the usage of the new functionality untangles Xen balloon and RAM hotplug from the usage of unpopulated physical memory ranges to map foreign pages, which is the correct thing to do in order to avoid mappings of foreign pages depend on memory hotplug. I think this is going to break Dom0 on Arm if the kernel has been built with hotplug. This is because you may end up to re-use region that will be used for the 1:1 mapping of a foreign map. Note that I don't know whether hotplug has been tested on Xen on Arm yet. So it might be possible to be already broken. Meanwhile, my suggestion would be to make the use of hotplug in the balloon code conditional (maybe using CONFIG_ARM64 and CONFIG_ARM)? Cheers, -- Julien Grall
Re: [PATCH v3 4/4] xen: add helpers to allocate unpopulated memory
Hi Roger, Thank you for the patch! Yet something to improve: [auto build test ERROR on xen-tip/linux-next] [also build test ERROR on linus/master v5.8-rc7 next-20200727] [cannot apply to linux/master] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch] url: https://github.com/0day-ci/linux/commits/Roger-Pau-Monne/xen-balloon-fixes-for-memory-hotplug/20200727-171643 base: https://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git linux-next config: i386-randconfig-a001-20200727 (attached as .config) compiler: gcc-9 (Debian 9.3.0-14) 9.3.0 reproduce (this is a W=1 build): # save the attached .config to linux build tree make W=1 ARCH=i386 If you fix the issue, kindly add following tag as appropriate Reported-by: kernel test robot All errors (new ones prefixed by >>): In file included from include/xen/xen.h:61, from drivers/xen/xen-front-pgdir-shbuf.c:18: include/xen/balloon.h:8:8: error: redefinition of 'struct balloon_stats' 8 | struct balloon_stats { |^ In file included from drivers/xen/xen-front-pgdir-shbuf.c:17: include/xen/balloon.h:8:8: note: originally defined here 8 | struct balloon_stats { |^ In file included from include/xen/xen.h:61, from drivers/xen/xen-front-pgdir-shbuf.c:18: include/xen/balloon.h:23:29: error: conflicting types for 'balloon_stats' 23 | extern struct balloon_stats balloon_stats; | ^ In file included from drivers/xen/xen-front-pgdir-shbuf.c:17: include/xen/balloon.h:23:29: note: previous declaration of 'balloon_stats' was here 23 | extern struct balloon_stats balloon_stats; | ^ In file included from include/xen/xen.h:61, from drivers/xen/xen-front-pgdir-shbuf.c:18: >> include/xen/balloon.h:33:20: error: redefinition of 'xen_balloon_init' 33 | static inline void xen_balloon_init(void) |^~~~ In file included from drivers/xen/xen-front-pgdir-shbuf.c:17: include/xen/balloon.h:33:20: note: previous definition of 'xen_balloon_init' was here 33 | static inline void xen_balloon_init(void) |^~~~ -- In file included from drivers/xen/balloon.c:71: include/xen/balloon.h:8:8: error: redefinition of 'struct balloon_stats' 8 | struct balloon_stats { |^ In file included from include/xen/xen.h:61, from drivers/xen/balloon.c:68: include/xen/balloon.h:8:8: note: originally defined here 8 | struct balloon_stats { |^ In file included from drivers/xen/balloon.c:71: include/xen/balloon.h:23:29: error: conflicting types for 'balloon_stats' 23 | extern struct balloon_stats balloon_stats; | ^ In file included from include/xen/xen.h:61, from drivers/xen/balloon.c:68: include/xen/balloon.h:23:29: note: previous declaration of 'balloon_stats' was here 23 | extern struct balloon_stats balloon_stats; | ^ In file included from drivers/xen/balloon.c:71: >> include/xen/balloon.h:33:20: error: redefinition of 'xen_balloon_init' 33 | static inline void xen_balloon_init(void) |^~~~ In file included from include/xen/xen.h:61, from drivers/xen/balloon.c:68: include/xen/balloon.h:33:20: note: previous definition of 'xen_balloon_init' was here 33 | static inline void xen_balloon_init(void) |^~~~ drivers/xen/balloon.c:138:22: error: conflicting types for 'balloon_stats' 138 | struct balloon_stats balloon_stats; | ^ In file included from include/xen/xen.h:61, from drivers/xen/balloon.c:68: include/xen/balloon.h:23:29: note: previous declaration of 'balloon_stats' was here 23 | extern struct balloon_stats balloon_stats; | ^ In file included from include/linux/linkage.h:7, from include/linux/kernel.h:8, from arch/x86/include/asm/percpu.h:45, from arch/x86/include/asm/current.h:6, from include/linux/sched.h:12, from include/linux/ratelimit.h:6, from include/linux/dev_printk.h:16, from include/linux/device.h:15, from include/linux/node.h:18, from include/linux/cpu.h:17, from drivers/xen/balloon.c:41:
Re: [PATCH v3 4/4] xen: add helpers to allocate unpopulated memory
On 27.07.20 11:13, Roger Pau Monne wrote: To be used in order to create foreign mappings. This is based on the ZONE_DEVICE facility which is used by persistent memory devices in order to create struct pages and kernel virtual mappings for the IOMEM areas of such devices. Note that on kernels without support for ZONE_DEVICE Xen will fallback to use ballooned pages in order to create foreign mappings. The newly added helpers use the same parameters as the existing {alloc/free}_xenballooned_pages functions, which allows for in-place replacement of the callers. Once a memory region has been added to be used as scratch mapping space it will no longer be released, and pages returned are kept in a linked list. This allows to have a buffer of pages and prevents resorting to frequent additions and removals of regions. If enabled (because ZONE_DEVICE is supported) the usage of the new functionality untangles Xen balloon and RAM hotplug from the usage of unpopulated physical memory ranges to map foreign pages, which is the correct thing to do in order to avoid mappings of foreign pages depend on memory hotplug. Signed-off-by: Roger Pau Monné --- I've not added a new memory_type type and just used MEMORY_DEVICE_DEVDAX which seems to be what we want for such memory regions. I'm unsure whether abusing this type is fine, or if I should instead add a specific type, maybe MEMORY_DEVICE_GENERIC? I don't think we should be using a specific Xen type at all. What about introducing MEMORY_DEVICE_VIRT? The comment for MEMORY_DEVICE_DEVDAX doesn't really fit, as there is no character device involved. In the end its the memory management maintainers to decide that. Other than that you can have my Reviewed-by: Juergen Gross Juergen
[PATCH v3 4/4] xen: add helpers to allocate unpopulated memory
To be used in order to create foreign mappings. This is based on the ZONE_DEVICE facility which is used by persistent memory devices in order to create struct pages and kernel virtual mappings for the IOMEM areas of such devices. Note that on kernels without support for ZONE_DEVICE Xen will fallback to use ballooned pages in order to create foreign mappings. The newly added helpers use the same parameters as the existing {alloc/free}_xenballooned_pages functions, which allows for in-place replacement of the callers. Once a memory region has been added to be used as scratch mapping space it will no longer be released, and pages returned are kept in a linked list. This allows to have a buffer of pages and prevents resorting to frequent additions and removals of regions. If enabled (because ZONE_DEVICE is supported) the usage of the new functionality untangles Xen balloon and RAM hotplug from the usage of unpopulated physical memory ranges to map foreign pages, which is the correct thing to do in order to avoid mappings of foreign pages depend on memory hotplug. Signed-off-by: Roger Pau Monné --- I've not added a new memory_type type and just used MEMORY_DEVICE_DEVDAX which seems to be what we want for such memory regions. I'm unsure whether abusing this type is fine, or if I should instead add a specific type, maybe MEMORY_DEVICE_GENERIC? I don't think we should be using a specific Xen type at all. --- Cc: Oleksandr Andrushchenko Cc: David Airlie Cc: Daniel Vetter Cc: Boris Ostrovsky Cc: Juergen Gross Cc: Stefano Stabellini Cc: Dan Carpenter Cc: Roger Pau Monne Cc: Wei Liu Cc: Yan Yankovskyi Cc: dri-de...@lists.freedesktop.org Cc: xen-de...@lists.xenproject.org Cc: linux...@kvack.org Cc: David Hildenbrand Cc: Michal Hocko Cc: Dan Williams --- Changes since v2: - Drop BUILD_BUG_ON regarding PVMMU page sizes. - Use a SPDX license identifier. - Call fill with only the minimum required number of pages. - Include xen.h header in xen_drm_front_gem.c. - Use less generic function names. - Exit early from the init function if not a PV guest. - Don't use all caps for region name. --- drivers/gpu/drm/xen/xen_drm_front_gem.c | 9 +- drivers/xen/Makefile| 1 + drivers/xen/balloon.c | 4 +- drivers/xen/grant-table.c | 4 +- drivers/xen/privcmd.c | 4 +- drivers/xen/unpopulated-alloc.c | 185 drivers/xen/xenbus/xenbus_client.c | 6 +- drivers/xen/xlate_mmu.c | 4 +- include/xen/xen.h | 9 ++ 9 files changed, 211 insertions(+), 15 deletions(-) create mode 100644 drivers/xen/unpopulated-alloc.c diff --git a/drivers/gpu/drm/xen/xen_drm_front_gem.c b/drivers/gpu/drm/xen/xen_drm_front_gem.c index f0b85e094111..270e1bd3e4da 100644 --- a/drivers/gpu/drm/xen/xen_drm_front_gem.c +++ b/drivers/gpu/drm/xen/xen_drm_front_gem.c @@ -18,6 +18,7 @@ #include #include +#include #include "xen_drm_front.h" #include "xen_drm_front_gem.h" @@ -99,8 +100,8 @@ static struct xen_gem_object *gem_create(struct drm_device *dev, size_t size) * allocate ballooned pages which will be used to map * grant references provided by the backend */ - ret = alloc_xenballooned_pages(xen_obj->num_pages, - xen_obj->pages); + ret = xen_alloc_unpopulated_pages(xen_obj->num_pages, + xen_obj->pages); if (ret < 0) { DRM_ERROR("Cannot allocate %zu ballooned pages: %d\n", xen_obj->num_pages, ret); @@ -152,8 +153,8 @@ void xen_drm_front_gem_free_object_unlocked(struct drm_gem_object *gem_obj) } else { if (xen_obj->pages) { if (xen_obj->be_alloc) { - free_xenballooned_pages(xen_obj->num_pages, - xen_obj->pages); + xen_free_unpopulated_pages(xen_obj->num_pages, + xen_obj->pages); gem_free_pages_array(xen_obj); } else { drm_gem_put_pages(_obj->base, diff --git a/drivers/xen/Makefile b/drivers/xen/Makefile index 0d322f3d90cd..788a5d9c8ef0 100644 --- a/drivers/xen/Makefile +++ b/drivers/xen/Makefile @@ -42,3 +42,4 @@ xen-gntdev-$(CONFIG_XEN_GNTDEV_DMABUF)+= gntdev-dmabuf.o xen-gntalloc-y := gntalloc.o xen-privcmd-y := privcmd.o privcmd-buf.o obj-$(CONFIG_XEN_FRONT_PGDIR_SHBUF)+= xen-front-pgdir-shbuf.o +obj-$(CONFIG_ZONE_DEVICE) += unpopulated-alloc.o diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c index b1d8b028bf80..815ef10eb2ff 100644 ---