Re: [RFCv3 0/5] enable migration of driver pages
On Thu, Jul 09, 2015 at 08:55:25AM +0900, Gioh Kim wrote: 2015-07-09 오전 7:47에 Dave Airlie 이(가) 쓴 글: Can the various in-kernel GPU drivers benefit from this? If so, wiring up one or more of those would be helpful? I'm sure that other in-kernel GPU drivers can have benefit. It must be helpful. If I was familiar with other in-kernel GPU drivers code, I tried to patch them. It's too bad. I'll bring dri-devel into the loop here. ARM GPU developers please take a look at this stuff, Laurent, Rob, Eric I suppose. I sent a patch, https://lkml.org/lkml/2015/3/24/1182, and my opinion about compaction to ARM GPU developers via Korea ARM branch. I got a reply that they had no time to review it. I hope they're interested to this patch. i915 gpus would support 64kb and 2mb pages, but we never implemented this. I don't think this would fit for gem based drivers since our backing storage is shmemfs. So if we want to implement page migration (which we'd probably want to make large pages work well) we'd need to pimp shmem to a) hand large pages to us b) forward the migrate calls. Probably that means we need to build our own gemfs reusing shmemfs code. I guess something similar would apply for ttm-based drivers (which use shmemfs just for swap-in/out but otherwise have their own page allocator, at least sometimes). Given all that I'd expect anything implementing migrate to just create a gpufs thing for the backing storage, no need for more hooks. There's also other areas for better code sharing among gpu drivers (e.g. mmu notifiers to get off userspace pages slurped inpinned with gup or shrinker callbacks to get the gpu off it's memory binge). But that would all be helper libraries in drm, not sure we need anything new from the core vm. Also there's a bit a lack of gpu drivers from the arm world in upstream, which is probabyl why this patch series doesn't come with a user. Might be better to first upstream the driver before talking about additional infrastructure that it needs. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFCv3 0/5] enable migration of driver pages
On Thu, Jul 09, 2015 at 03:08:48PM +0200, Daniel Vetter wrote: On Thu, Jul 09, 2015 at 08:55:25AM +0900, Gioh Kim wrote: 2015-07-09 오전 7:47에 Dave Airlie 이(가) 쓴 글: Can the various in-kernel GPU drivers benefit from this? If so, wiring up one or more of those would be helpful? I'm sure that other in-kernel GPU drivers can have benefit. It must be helpful. If I was familiar with other in-kernel GPU drivers code, I tried to patch them. It's too bad. I'll bring dri-devel into the loop here. ARM GPU developers please take a look at this stuff, Laurent, Rob, Eric I suppose. I sent a patch, https://lkml.org/lkml/2015/3/24/1182, and my opinion about compaction to ARM GPU developers via Korea ARM branch. I got a reply that they had no time to review it. I hope they're interested to this patch. i915 gpus would support 64kb and 2mb pages, but we never implemented this. I don't think this would fit for gem based drivers since our backing storage is shmemfs. So if we want to implement page migration (which we'd probably want to make large pages work well) we'd need to pimp shmem to a) hand large pages to us b) forward the migrate calls. Probably that means we need to build our own gemfs reusing shmemfs code. AFAIK there are efforts ongoing to make large pages work with shmem. Kirill, IIRC you mentioned that you're were looking into this a while back? -- Ville Syrjälä Intel OTC ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFCv3 0/5] enable migration of driver pages
On Thu, Jul 09, 2015 at 04:33:01PM +0300, Ville Syrjälä wrote: On Thu, Jul 09, 2015 at 03:08:48PM +0200, Daniel Vetter wrote: On Thu, Jul 09, 2015 at 08:55:25AM +0900, Gioh Kim wrote: 2015-07-09 오전 7:47에 Dave Airlie 이(가) 쓴 글: Can the various in-kernel GPU drivers benefit from this? If so, wiring up one or more of those would be helpful? I'm sure that other in-kernel GPU drivers can have benefit. It must be helpful. If I was familiar with other in-kernel GPU drivers code, I tried to patch them. It's too bad. I'll bring dri-devel into the loop here. ARM GPU developers please take a look at this stuff, Laurent, Rob, Eric I suppose. I sent a patch, https://lkml.org/lkml/2015/3/24/1182, and my opinion about compaction to ARM GPU developers via Korea ARM branch. I got a reply that they had no time to review it. I hope they're interested to this patch. i915 gpus would support 64kb and 2mb pages, but we never implemented this. I don't think this would fit for gem based drivers since our backing storage is shmemfs. So if we want to implement page migration (which we'd probably want to make large pages work well) we'd need to pimp shmem to a) hand large pages to us b) forward the migrate calls. Probably that means we need to build our own gemfs reusing shmemfs code. AFAIK there are efforts ongoing to make large pages work with shmem. Kirill, IIRC you mentioned that you're were looking into this a while back? I work in this direction, but don't have anything to show at the moment. Hugh has published his implementation of huge tmpfs back in February: http://lkml.kernel.org/g/alpine.LSU.2.11.1502201941340.14414@eggly.anvils -- Kirill A. Shutemov ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFCv3 0/5] enable migration of driver pages
2015-07-09 오후 10:08에 Daniel Vetter 이(가) 쓴 글: Also there's a bit a lack of gpu drivers from the arm world in upstream, which is probabyl why this patch series doesn't come with a user. Might be better to first upstream the driver before talking about additional infrastructure that it needs. -Daniel I'm not from ARM but I just got the idea of driver page migration during I worked with ARM gpu driver. I'm sure this patch is good for zram and balloon and hope it can be applied to drivers consuming many pages and generating fragmentation, such as GPU or gfx driver. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFCv3 0/5] enable migration of driver pages
Can the various in-kernel GPU drivers benefit from this? If so, wiring up one or more of those would be helpful? I'm sure that other in-kernel GPU drivers can have benefit. It must be helpful. If I was familiar with other in-kernel GPU drivers code, I tried to patch them. It's too bad. I'll bring dri-devel into the loop here. ARM GPU developers please take a look at this stuff, Laurent, Rob, Eric I suppose. Daniel Vetter you might have some opinions as well. Dave. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFCv3 0/5] enable migration of driver pages
2015-07-09 오전 7:47에 Dave Airlie 이(가) 쓴 글: Can the various in-kernel GPU drivers benefit from this? If so, wiring up one or more of those would be helpful? I'm sure that other in-kernel GPU drivers can have benefit. It must be helpful. If I was familiar with other in-kernel GPU drivers code, I tried to patch them. It's too bad. I'll bring dri-devel into the loop here. ARM GPU developers please take a look at this stuff, Laurent, Rob, Eric I suppose. I sent a patch, https://lkml.org/lkml/2015/3/24/1182, and my opinion about compaction to ARM GPU developers via Korea ARM branch. I got a reply that they had no time to review it. I hope they're interested to this patch. Daniel Vetter you might have some opinions as well. Dave. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFCv3 0/5] enable migration of driver pages
On Tue, Jul 07, 2015 at 01:36:20PM +0900, Gioh Kim wrote: From: Gioh Kim guru...@hanmail.net Hello, This series try to enable migration of non-LRU pages, such as driver's page. My ARM-based platform occured severe fragmentation problem after long-term (several days) test. Sometimes even order-3 page allocation failed. It has memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic processing and 20~30 memory is reserved for zram. I found that many pages of GPU driver and zram are non-movable pages. So I reported Minchan Kim, the maintainer of zram, and he made the internal compaction logic of zram. And I made the internal compaction of GPU driver. They reduced some fragmentation but they are not enough effective. They are activated by its own interface, /sys, so they are not cooperative with kernel compaction. If there is too much fragmentation and kernel starts to compaction, zram and GPU driver cannot work with the kernel compaction. This patch set combines 5 patches. 1. patch 1/5: get inode from anon_inodes This patch adds new interface to create inode from anon_inodes. 2. patch 2/5: framework to isolate/migrate/putback page Add isolatepage, putbackpage into address_space_operations and wrapper function to call them 3. patch 3/5: apply the framework into balloon driver The balloon driver is applied into the framework. It gets a inode from anon_inodes and register operations in the inode. Any other drivers can register operations via inode like this to migrate it's pages. 4. patch 4/5: compaction/migration call the generic interfaces Compaction and migration pages call the generic interfaces of the framework, instead of calling balloon migration directly. 5. patch 5/5: remove direct calling of migration of driver pages Non-lru pages are migrated with lru pages by move_to_new_page(). This patch set is tested: - turn on Ubuntu 14.04 with 1G memory on qemu. - do kernel building - after several seconds check more than 512MB is used with free command - command balloon 512 in qemu monitor - check hundreds MB of pages are migrated My thanks to Konstantin Khlebnikov for his reviews of the v2 patch set. Most of the changes were based on his feedback. Changes since v2: - change the name of page type from migratable page into mobile page - get and lock page to isolate page - add wrapper interfaces for page-mapping-a_ops-isolate/putback - leave balloon pages marked as balloon This patch-set is based on v4.1 Gioh Kim (5): fs/anon_inodes: new interface to create new inode mm/compaction: enable mobile-page migration mm/balloon: apply mobile page migratable into balloon mm/compaction: call generic migration callbacks mm: remove direct calling of migration drivers/virtio/virtio_balloon.c| 3 ++ fs/anon_inodes.c | 6 +++ fs/proc/page.c | 3 ++ include/linux/anon_inodes.h| 1 + include/linux/balloon_compaction.h | 15 +-- include/linux/compaction.h | 76 ++ include/linux/fs.h | 2 + include/linux/page-flags.h | 19 + include/uapi/linux/kernel-page-flags.h | 1 + mm/balloon_compaction.c| 71 ++- mm/compaction.c| 8 ++-- mm/migrate.c | 24 +++ 12 files changed, 154 insertions(+), 75 deletions(-) -- 2.1.4 Acked-by: Rafael Aquini aqu...@redhat.com ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFCv3 0/5] enable migration of driver pages
On Tue, 7 Jul 2015 13:36:20 +0900 Gioh Kim gioh@lge.com wrote: From: Gioh Kim guru...@hanmail.net Hello, This series try to enable migration of non-LRU pages, such as driver's page. My ARM-based platform occured severe fragmentation problem after long-term (several days) test. Sometimes even order-3 page allocation failed. It has memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic processing and 20~30 memory is reserved for zram. I found that many pages of GPU driver and zram are non-movable pages. So I reported Minchan Kim, the maintainer of zram, and he made the internal compaction logic of zram. And I made the internal compaction of GPU driver. They reduced some fragmentation but they are not enough effective. They are activated by its own interface, /sys, so they are not cooperative with kernel compaction. If there is too much fragmentation and kernel starts to compaction, zram and GPU driver cannot work with the kernel compaction. ... This patch set is tested: - turn on Ubuntu 14.04 with 1G memory on qemu. - do kernel building - after several seconds check more than 512MB is used with free command - command balloon 512 in qemu monitor - check hundreds MB of pages are migrated OK, but what happens if the balloon driver is not used to force compaction? Does your test machine successfully compact pages on demand, so those order-3 allocations now succeed? Why are your changes to the GPU driver not included in this patch series? ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFCv3 0/5] enable migration of driver pages
2015-07-08 오전 7:37에 Andrew Morton 이(가) 쓴 글: On Tue, 7 Jul 2015 13:36:20 +0900 Gioh Kim gioh@lge.com wrote: From: Gioh Kim guru...@hanmail.net Hello, This series try to enable migration of non-LRU pages, such as driver's page. My ARM-based platform occured severe fragmentation problem after long-term (several days) test. Sometimes even order-3 page allocation failed. It has memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic processing and 20~30 memory is reserved for zram. I found that many pages of GPU driver and zram are non-movable pages. So I reported Minchan Kim, the maintainer of zram, and he made the internal compaction logic of zram. And I made the internal compaction of GPU driver. They reduced some fragmentation but they are not enough effective. They are activated by its own interface, /sys, so they are not cooperative with kernel compaction. If there is too much fragmentation and kernel starts to compaction, zram and GPU driver cannot work with the kernel compaction. ... This patch set is tested: - turn on Ubuntu 14.04 with 1G memory on qemu. - do kernel building - after several seconds check more than 512MB is used with free command - command balloon 512 in qemu monitor - check hundreds MB of pages are migrated OK, but what happens if the balloon driver is not used to force compaction? Does your test machine successfully compact pages on demand, so those order-3 allocations now succeed? If any driver that has many pages like the balloon driver is forced to compact, the system can get free high-order pages. I have to show how this patch work with a driver existing in the kernel source, for kernel developers' undestanding. So I selected the balloon driver because it has already compaction and working with kernel compaction. I can show how driver pages is compacted with lru-pages together. Actually balloon driver is not best example to show how this patch compacts pages. The balloon driver compaction is decreasing page consumtion, for instance 1024MB - 512MB. I think it is not compaction precisely. It frees pages. Of course there will be many high-order pages after 512MB is freed. Why are your changes to the GPU driver not included in this patch series? My platform is ARM-based and GPU is ARM-Mali. The driver is not open source. It's too bad that I cannot show effect of this patch with the GPU driver. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFCv3 0/5] enable migration of driver pages
On Wed, Jul 08, 2015 at 09:19:50AM +0900, Gioh Kim wrote: 2015-07-08 오전 9:07에 Andrew Morton 이(가) 쓴 글: On Wed, 08 Jul 2015 09:02:59 +0900 Gioh Kim gioh@lge.com wrote: 2015-07-08 __ 7:37___ Andrew Morton ___(___) ___ ___: On Tue, 7 Jul 2015 13:36:20 +0900 Gioh Kim gioh@lge.com wrote: From: Gioh Kim guru...@hanmail.net Hello, This series try to enable migration of non-LRU pages, such as driver's page. My ARM-based platform occured severe fragmentation problem after long-term (several days) test. Sometimes even order-3 page allocation failed. It has memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic processing and 20~30 memory is reserved for zram. I found that many pages of GPU driver and zram are non-movable pages. So I reported Minchan Kim, the maintainer of zram, and he made the internal compaction logic of zram. And I made the internal compaction of GPU driver. They reduced some fragmentation but they are not enough effective. They are activated by its own interface, /sys, so they are not cooperative with kernel compaction. If there is too much fragmentation and kernel starts to compaction, zram and GPU driver cannot work with the kernel compaction. ... This patch set is tested: - turn on Ubuntu 14.04 with 1G memory on qemu. - do kernel building - after several seconds check more than 512MB is used with free command - command balloon 512 in qemu monitor - check hundreds MB of pages are migrated OK, but what happens if the balloon driver is not used to force compaction? Does your test machine successfully compact pages on demand, so those order-3 allocations now succeed? If any driver that has many pages like the balloon driver is forced to compact, the system can get free high-order pages. I have to show how this patch work with a driver existing in the kernel source, for kernel developers' undestanding. So I selected the balloon driver because it has already compaction and working with kernel compaction. I can show how driver pages is compacted with lru-pages together. Actually balloon driver is not best example to show how this patch compacts pages. The balloon driver compaction is decreasing page consumtion, for instance 1024MB - 512MB. I think it is not compaction precisely. It frees pages. Of course there will be many high-order pages after 512MB is freed. Can the various in-kernel GPU drivers benefit from this? If so, wiring up one or more of those would be helpful? I'm sure that other in-kernel GPU drivers can have benefit. It must be helpful. If I was familiar with other in-kernel GPU drivers code, I tried to patch them. It's too bad. Minchan Kim said he had a plan to apply this patch into zram compaction. Many embedded machines use several hundreds MB for zram. The zram can also have benefit with this patch as much as GPU drivers. Hello Gioh, It would be helpful for fork-latency and zra+CMA in small memory system. I will implement zsmalloc.migratepages after I finish current going works. Thanks for the nice work! -- Kind regards, Minchan Kim ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFCv3 0/5] enable migration of driver pages
On Wed, 08 Jul 2015 09:02:59 +0900 Gioh Kim gioh@lge.com wrote: 2015-07-08 __ 7:37___ Andrew Morton ___(___) ___ ___: On Tue, 7 Jul 2015 13:36:20 +0900 Gioh Kim gioh@lge.com wrote: From: Gioh Kim guru...@hanmail.net Hello, This series try to enable migration of non-LRU pages, such as driver's page. My ARM-based platform occured severe fragmentation problem after long-term (several days) test. Sometimes even order-3 page allocation failed. It has memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic processing and 20~30 memory is reserved for zram. I found that many pages of GPU driver and zram are non-movable pages. So I reported Minchan Kim, the maintainer of zram, and he made the internal compaction logic of zram. And I made the internal compaction of GPU driver. They reduced some fragmentation but they are not enough effective. They are activated by its own interface, /sys, so they are not cooperative with kernel compaction. If there is too much fragmentation and kernel starts to compaction, zram and GPU driver cannot work with the kernel compaction. ... This patch set is tested: - turn on Ubuntu 14.04 with 1G memory on qemu. - do kernel building - after several seconds check more than 512MB is used with free command - command balloon 512 in qemu monitor - check hundreds MB of pages are migrated OK, but what happens if the balloon driver is not used to force compaction? Does your test machine successfully compact pages on demand, so those order-3 allocations now succeed? If any driver that has many pages like the balloon driver is forced to compact, the system can get free high-order pages. I have to show how this patch work with a driver existing in the kernel source, for kernel developers' undestanding. So I selected the balloon driver because it has already compaction and working with kernel compaction. I can show how driver pages is compacted with lru-pages together. Actually balloon driver is not best example to show how this patch compacts pages. The balloon driver compaction is decreasing page consumtion, for instance 1024MB - 512MB. I think it is not compaction precisely. It frees pages. Of course there will be many high-order pages after 512MB is freed. Can the various in-kernel GPU drivers benefit from this? If so, wiring up one or more of those would be helpful? ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFCv3 0/5] enable migration of driver pages
2015-07-08 오전 9:07에 Andrew Morton 이(가) 쓴 글: On Wed, 08 Jul 2015 09:02:59 +0900 Gioh Kim gioh@lge.com wrote: 2015-07-08 __ 7:37___ Andrew Morton ___(___) ___ ___: On Tue, 7 Jul 2015 13:36:20 +0900 Gioh Kim gioh@lge.com wrote: From: Gioh Kim guru...@hanmail.net Hello, This series try to enable migration of non-LRU pages, such as driver's page. My ARM-based platform occured severe fragmentation problem after long-term (several days) test. Sometimes even order-3 page allocation failed. It has memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic processing and 20~30 memory is reserved for zram. I found that many pages of GPU driver and zram are non-movable pages. So I reported Minchan Kim, the maintainer of zram, and he made the internal compaction logic of zram. And I made the internal compaction of GPU driver. They reduced some fragmentation but they are not enough effective. They are activated by its own interface, /sys, so they are not cooperative with kernel compaction. If there is too much fragmentation and kernel starts to compaction, zram and GPU driver cannot work with the kernel compaction. ... This patch set is tested: - turn on Ubuntu 14.04 with 1G memory on qemu. - do kernel building - after several seconds check more than 512MB is used with free command - command balloon 512 in qemu monitor - check hundreds MB of pages are migrated OK, but what happens if the balloon driver is not used to force compaction? Does your test machine successfully compact pages on demand, so those order-3 allocations now succeed? If any driver that has many pages like the balloon driver is forced to compact, the system can get free high-order pages. I have to show how this patch work with a driver existing in the kernel source, for kernel developers' undestanding. So I selected the balloon driver because it has already compaction and working with kernel compaction. I can show how driver pages is compacted with lru-pages together. Actually balloon driver is not best example to show how this patch compacts pages. The balloon driver compaction is decreasing page consumtion, for instance 1024MB - 512MB. I think it is not compaction precisely. It frees pages. Of course there will be many high-order pages after 512MB is freed. Can the various in-kernel GPU drivers benefit from this? If so, wiring up one or more of those would be helpful? I'm sure that other in-kernel GPU drivers can have benefit. It must be helpful. If I was familiar with other in-kernel GPU drivers code, I tried to patch them. It's too bad. Minchan Kim said he had a plan to apply this patch into zram compaction. Many embedded machines use several hundreds MB for zram. The zram can also have benefit with this patch as much as GPU drivers. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
