[PATCH 0/4] enable migration of driver pages

[Gioh Kim]

2015-07-29 오후 9:46에 Daniel Vetter 이(가) 쓴 글:
> On Wed, Jul 29, 2015 at 01:16:14PM +0100, Mel Gorman wrote:
>> On Wed, Jul 29, 2015 at 12:55:54PM +0200, Daniel Vetter wrote:
>>> On Wed, Jul 29, 2015 at 11:49:45AM +0100, Mel Gorman wrote:
 On Mon, Jul 13, 2015 at 05:35:15PM +0900, Gioh Kim wrote:
> My ARM-based platform occured severe fragmentation problem after long-term
> (several days) test. Sometimes even order-3 page allocation failed. It has
> memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic 
> processing
> and 20~30 memory is reserved for zram.
>

 The primary motivation of this series is to reduce fragmentation by 
 allowing
 more kernel pages to be moved. Conceptually that is a worthwhile goal but
 there should be at least one major in-kernel user and while balloon
 pages were a good starting point, I think we really need to see what the
 zram changes look like at the same time.
>>>
>>> I think gpu drivers really would be the perfect candidate for compacting
>>> kernel page allocations. And this also seems the primary motivation for
>>> this patch series, so I think that's really what we should use to judge
>>> these patches.
>>>
>>> Of course then there's the seemingly eternal chicken/egg problem of
>>> upstream gpu drivers for SoCs :(
>>
>> I recognised that the driver he had modified was not an in-tree user so
>> it did not really help the review or the design. I did not think it was
>> very fair to ask that an in-tree GPU driver be converted when it would not
>> help the embedded platform of interest. Converting zram is both a useful
>> illustration of the aops requirements and is expected to be beneficial on
>> the embedded platform. Now, if a GPU driver author was willing to convert
>> theirs as an example then that would be useful!
>
> Well my concern is more with merging infrastructure to upstream for
> drivers which aren't upstream and with no plan to make that happen anytime
> soon. Seems like just offload a bit to me ... but in the end core mm isn't
> my thing so not my decision.
> -Daniel
>

I get idea from the out-tree driver but this infrastructure will be useful
for zram and balloon. That is agreed by the maintainers of each driver.

I'm currently accepting feedbacks from
balloon and zram and trying to be applicable for them.
Of course I hope there will be more application. It'll be more useful
if it has more application.

[PATCH 0/4] enable migration of driver pages

[Daniel Vetter]

On Wed, Jul 29, 2015 at 01:16:14PM +0100, Mel Gorman wrote:
> On Wed, Jul 29, 2015 at 12:55:54PM +0200, Daniel Vetter wrote:
> > On Wed, Jul 29, 2015 at 11:49:45AM +0100, Mel Gorman wrote:
> > > On Mon, Jul 13, 2015 at 05:35:15PM +0900, Gioh Kim wrote:
> > > > My ARM-based platform occured severe fragmentation problem after 
> > > > long-term
> > > > (several days) test. Sometimes even order-3 page allocation failed. It 
> > > > has
> > > > memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic 
> > > > processing
> > > > and 20~30 memory is reserved for zram.
> > > > 
> > > 
> > > The primary motivation of this series is to reduce fragmentation by 
> > > allowing
> > > more kernel pages to be moved. Conceptually that is a worthwhile goal but
> > > there should be at least one major in-kernel user and while balloon
> > > pages were a good starting point, I think we really need to see what the
> > > zram changes look like at the same time.
> > 
> > I think gpu drivers really would be the perfect candidate for compacting
> > kernel page allocations. And this also seems the primary motivation for
> > this patch series, so I think that's really what we should use to judge
> > these patches.
> > 
> > Of course then there's the seemingly eternal chicken/egg problem of
> > upstream gpu drivers for SoCs :(
> 
> I recognised that the driver he had modified was not an in-tree user so
> it did not really help the review or the design. I did not think it was
> very fair to ask that an in-tree GPU driver be converted when it would not
> help the embedded platform of interest. Converting zram is both a useful
> illustration of the aops requirements and is expected to be beneficial on
> the embedded platform. Now, if a GPU driver author was willing to convert
> theirs as an example then that would be useful!

Well my concern is more with merging infrastructure to upstream for
drivers which aren't upstream and with no plan to make that happen anytime
soon. Seems like just offload a bit to me ... but in the end core mm isn't
my thing so not my decision.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

[PATCH 0/4] enable migration of driver pages

[Mel Gorman]

On Wed, Jul 29, 2015 at 12:55:54PM +0200, Daniel Vetter wrote:
> On Wed, Jul 29, 2015 at 11:49:45AM +0100, Mel Gorman wrote:
> > On Mon, Jul 13, 2015 at 05:35:15PM +0900, Gioh Kim wrote:
> > > My ARM-based platform occured severe fragmentation problem after long-term
> > > (several days) test. Sometimes even order-3 page allocation failed. It has
> > > memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic 
> > > processing
> > > and 20~30 memory is reserved for zram.
> > > 
> > 
> > The primary motivation of this series is to reduce fragmentation by allowing
> > more kernel pages to be moved. Conceptually that is a worthwhile goal but
> > there should be at least one major in-kernel user and while balloon
> > pages were a good starting point, I think we really need to see what the
> > zram changes look like at the same time.
> 
> I think gpu drivers really would be the perfect candidate for compacting
> kernel page allocations. And this also seems the primary motivation for
> this patch series, so I think that's really what we should use to judge
> these patches.
> 
> Of course then there's the seemingly eternal chicken/egg problem of
> upstream gpu drivers for SoCs :(

I recognised that the driver he had modified was not an in-tree user so
it did not really help the review or the design. I did not think it was
very fair to ask that an in-tree GPU driver be converted when it would not
help the embedded platform of interest. Converting zram is both a useful
illustration of the aops requirements and is expected to be beneficial on
the embedded platform. Now, if a GPU driver author was willing to convert
theirs as an example then that would be useful!

-- 
Mel Gorman
SUSE Labs

[PATCH 0/4] enable migration of driver pages

[Daniel Vetter]

On Wed, Jul 29, 2015 at 11:49:45AM +0100, Mel Gorman wrote:
> On Mon, Jul 13, 2015 at 05:35:15PM +0900, Gioh Kim wrote:
> > My ARM-based platform occured severe fragmentation problem after long-term
> > (several days) test. Sometimes even order-3 page allocation failed. It has
> > memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic 
> > processing
> > and 20~30 memory is reserved for zram.
> > 
> 
> The primary motivation of this series is to reduce fragmentation by allowing
> more kernel pages to be moved. Conceptually that is a worthwhile goal but
> there should be at least one major in-kernel user and while balloon
> pages were a good starting point, I think we really need to see what the
> zram changes look like at the same time.

I think gpu drivers really would be the perfect candidate for compacting
kernel page allocations. And this also seems the primary motivation for
this patch series, so I think that's really what we should use to judge
these patches.

Of course then there's the seemingly eternal chicken/egg problem of
upstream gpu drivers for SoCs :(
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

[PATCH 0/4] enable migration of driver pages

[Mel Gorman]

On Mon, Jul 13, 2015 at 05:35:15PM +0900, Gioh Kim wrote:
> My ARM-based platform occured severe fragmentation problem after long-term
> (several days) test. Sometimes even order-3 page allocation failed. It has
> memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic 
> processing
> and 20~30 memory is reserved for zram.
> 

The primary motivation of this series is to reduce fragmentation by allowing
more kernel pages to be moved. Conceptually that is a worthwhile goal but
there should be at least one major in-kernel user and while balloon
pages were a good starting point, I think we really need to see what the
zram changes look like at the same time.

> I found that many pages of GPU driver and zram are non-movable pages. So I
> reported Minchan Kim, the maintainer of zram, and he made the internal 
> compaction logic of zram. And I made the internal compaction of GPU driver.
> 

I am not familiar with the internals of zram but I took a look at what
it merged.  At a glance the compaction it implements and what you need are
are different in important respects. The core ability to move a zsmalloc
object is useful but the motivation of zram compaction appears to be
reducing the memory footprint. You need to reduce fragmentation which is
not the same. You could be faced with a situation where a full page in an
awkward place. Then there are three choices I can think of quickly and
probably more

1. You can move the whole page to another whole page and update all the
   references. This would play nicely with how compactions migrate and
   free scanner operates. However, you need free memory to move it

2. You could try moving the full page into other zsmalloc pages so that
   memory usage is also potentially reduced. This would work better with
   what Minchan intended but then there is the problem of discovery.
   Potentially it means though that another address space callback is
   required to nominate a target migration page

3. Hybrid approach. First trigger the zsmalloc compaction as it
   currently exists, then kick of compaction and move whole pages
   regardless of their content. The downside here is that it's expensive
   and potentially copies data multiple times but it's going to be
   easier to implement than 2.

1 would be the logical starting point, 3 is probably most effective even
if it's expensive and 2 is probably the best overall if the search costs
can be controlled.

This is a lot more complex than what balloon requires which is why I
would like to see it pinned down before new address_space operations are
created. Once they are created and drivers start using them then we lose
a lot of flexibilty and fixing the design becomes a lot harder.

With that in mind, I'll still read the rest of the series.

-- 
Mel Gorman
SUSE Labs

[PATCH 0/4] enable migration of driver pages

[Gioh Kim]

2015-07-13 오후 6:24에 Konstantin Khlebnikov 이(가) 쓴 글:
> On Mon, Jul 13, 2015 at 11:35 AM, Gioh Kim  wrote:
>> From: Gioh Kim 
>>
>> Hello,
>>
>> This series try to enable migration of non-LRU pages, such as driver's page.
>>
>> My ARM-based platform occured severe fragmentation problem after long-term
>> (several days) test. Sometimes even order-3 page allocation failed. It has
>> memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic 
>> processing
>> and 20~30 memory is reserved for zram.
>>
>> I found that many pages of GPU driver and zram are non-movable pages. So I
>> reported Minchan Kim, the maintainer of zram, and he made the internal
>> compaction logic of zram. And I made the internal compaction of GPU driver.
>>
>> They reduced some fragmentation but they are not enough effective.
>> They are activated by its own interface, /sys, so they are not cooperative
>> with kernel compaction. If there is too much fragmentation and kernel starts
>> to compaction, zram and GPU driver cannot work with the kernel compaction.
>>
>> So I thought there needs a interface to combine driver and kernel compaction.
>> This patch adds a generic isolate/migrate/putback callbacks for page
>> address-space and a new interface to create anon-inode to manage
>> address_space_operation. The zram and GPU, and any other modules can create
>> anon_inode and register its own migration method. The kernel compaction can
>> call the registered migration when it does compaction.
>>
>> My GPU driver source is not in-kernel driver so that I apply the interface
>> into balloon driver. The balloon driver is already merged
>> into the kernel compaction as a corner-case. This patch have the balloon
>> driver migration be called by the generic interface.
>>
>>
>> This patch set combines 4 patches.
>>
>> 1. patch 1/4: get inode from anon_inodes
>> This patch adds new interface to create inode from anon_inodes.
>>
>> 2. patch 2/4: framework to isolate/migrate/putback page
>> Add isolatepage, putbackpage into address_space_operations
>> and wrapper function to call them.
>>
>> 3. patch 3/4: apply the framework into balloon driver
>> The balloon driver is applied into the framework. It gets a inode
>> from anon_inodes and register operations in the inode.
>> The kernel compaction calls generic interfaces, not balloon
>> driver interfaces.
>> Any other drivers can register operations via inode like this
>> to migrate it's pages.
>>
>> 4. patch 4/4: remove direct calling of migration of driver pages
>> Non-lru pages are also migrated with lru pages by move_to_new_page().
>
> The whole patchset looks good.
>
> Reviewed-by: Konstantin Khlebnikov 
>
>>
>> This patch set is tested:
>> - turn on Ubuntu 14.04 with 1G memory on qemu.
>> - do kernel building
>> - after several seconds check more than 512MB is used with free command
>> - command "balloon 512" in qemu monitor
>> - check hundreds MB of pages are migrated
>
> Another simple test is several instances of
> tools/testing/selftests/vm/transhuge-stress.c
> runnng in parallel with balloon inflating/deflating.
> (transparent huge pages must be enabled of course)
> That catched a lot of races in ballooning code.
>

Great!
I'll do it and inform you the result in this week.

[PATCH 0/4] enable migration of driver pages

[Gioh Kim]

From: Gioh Kim 

Hello,

This series try to enable migration of non-LRU pages, such as driver's page.

My ARM-based platform occured severe fragmentation problem after long-term
(several days) test. Sometimes even order-3 page allocation failed. It has
memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic processing
and 20~30 memory is reserved for zram.

I found that many pages of GPU driver and zram are non-movable pages. So I
reported Minchan Kim, the maintainer of zram, and he made the internal 
compaction logic of zram. And I made the internal compaction of GPU driver.

They reduced some fragmentation but they are not enough effective.
They are activated by its own interface, /sys, so they are not cooperative
with kernel compaction. If there is too much fragmentation and kernel starts
to compaction, zram and GPU driver cannot work with the kernel compaction.

So I thought there needs a interface to combine driver and kernel compaction.
This patch adds a generic isolate/migrate/putback callbacks for page
address-space and a new interface to create anon-inode to manage
address_space_operation. The zram and GPU, and any other modules can create
anon_inode and register its own migration method. The kernel compaction can
call the registered migration when it does compaction.

My GPU driver source is not in-kernel driver so that I apply the interface
into balloon driver. The balloon driver is already merged
into the kernel compaction as a corner-case. This patch have the balloon
driver migration be called by the generic interface.


This patch set combines 4 patches.

1. patch 1/4: get inode from anon_inodes
This patch adds new interface to create inode from anon_inodes.

2. patch 2/4: framework to isolate/migrate/putback page
Add isolatepage, putbackpage into address_space_operations
and wrapper function to call them.

3. patch 3/4: apply the framework into balloon driver
The balloon driver is applied into the framework. It gets a inode
from anon_inodes and register operations in the inode.
The kernel compaction calls generic interfaces, not balloon
driver interfaces. 
Any other drivers can register operations via inode like this
to migrate it's pages.

4. patch 4/4: remove direct calling of migration of driver pages
Non-lru pages are also migrated with lru pages by move_to_new_page().

This patch set is tested:
- turn on Ubuntu 14.04 with 1G memory on qemu.
- do kernel building
- after several seconds check more than 512MB is used with free command
- command "balloon 512" in qemu monitor
- check hundreds MB of pages are migrated

My thanks to Konstantin Khlebnikov for his reviews of the RFC patch set.
Most of the changes were based on his feedback.

This patch-set is based on v4.1


Gioh Kim (4):
  fs/anon_inodes: new interface to create new inode
  mm/compaction: enable mobile-page migration
  mm/balloon: apply mobile page migratable into balloon
  mm: remove direct calling of migration

 drivers/virtio/virtio_balloon.c|  3 ++
 fs/anon_inodes.c   |  6 +++
 fs/proc/page.c |  3 ++
 include/linux/anon_inodes.h|  1 +
 include/linux/balloon_compaction.h | 15 +--
 include/linux/compaction.h | 80 ++
 include/linux/fs.h |  2 +
 include/linux/page-flags.h | 19 
 include/uapi/linux/kernel-page-flags.h |  1 +
 mm/balloon_compaction.c| 72 ++
 mm/compaction.c|  8 ++--
 mm/migrate.c   | 24 +++---
 12 files changed, 160 insertions(+), 74 deletions(-)

-- 
2.1.4

[PATCH 0/4] enable migration of driver pages

[Konstantin Khlebnikov]

On Mon, Jul 13, 2015 at 11:35 AM, Gioh Kim  wrote:
> From: Gioh Kim 
>
> Hello,
>
> This series try to enable migration of non-LRU pages, such as driver's page.
>
> My ARM-based platform occured severe fragmentation problem after long-term
> (several days) test. Sometimes even order-3 page allocation failed. It has
> memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic 
> processing
> and 20~30 memory is reserved for zram.
>
> I found that many pages of GPU driver and zram are non-movable pages. So I
> reported Minchan Kim, the maintainer of zram, and he made the internal
> compaction logic of zram. And I made the internal compaction of GPU driver.
>
> They reduced some fragmentation but they are not enough effective.
> They are activated by its own interface, /sys, so they are not cooperative
> with kernel compaction. If there is too much fragmentation and kernel starts
> to compaction, zram and GPU driver cannot work with the kernel compaction.
>
> So I thought there needs a interface to combine driver and kernel compaction.
> This patch adds a generic isolate/migrate/putback callbacks for page
> address-space and a new interface to create anon-inode to manage
> address_space_operation. The zram and GPU, and any other modules can create
> anon_inode and register its own migration method. The kernel compaction can
> call the registered migration when it does compaction.
>
> My GPU driver source is not in-kernel driver so that I apply the interface
> into balloon driver. The balloon driver is already merged
> into the kernel compaction as a corner-case. This patch have the balloon
> driver migration be called by the generic interface.
>
>
> This patch set combines 4 patches.
>
> 1. patch 1/4: get inode from anon_inodes
> This patch adds new interface to create inode from anon_inodes.
>
> 2. patch 2/4: framework to isolate/migrate/putback page
> Add isolatepage, putbackpage into address_space_operations
> and wrapper function to call them.
>
> 3. patch 3/4: apply the framework into balloon driver
> The balloon driver is applied into the framework. It gets a inode
> from anon_inodes and register operations in the inode.
> The kernel compaction calls generic interfaces, not balloon
> driver interfaces.
> Any other drivers can register operations via inode like this
> to migrate it's pages.
>
> 4. patch 4/4: remove direct calling of migration of driver pages
> Non-lru pages are also migrated with lru pages by move_to_new_page().

The whole patchset looks good.

Reviewed-by: Konstantin Khlebnikov 

>
> This patch set is tested:
> - turn on Ubuntu 14.04 with 1G memory on qemu.
> - do kernel building
> - after several seconds check more than 512MB is used with free command
> - command "balloon 512" in qemu monitor
> - check hundreds MB of pages are migrated

Another simple test is several instances of
tools/testing/selftests/vm/transhuge-stress.c
runnng in parallel with balloon inflating/deflating.
(transparent huge pages must be enabled of course)
That catched a lot of races in ballooning code.

>
> My thanks to Konstantin Khlebnikov for his reviews of the RFC patch set.
> Most of the changes were based on his feedback.
>
> This patch-set is based on v4.1
>
>
> Gioh Kim (4):
>   fs/anon_inodes: new interface to create new inode
>   mm/compaction: enable mobile-page migration
>   mm/balloon: apply mobile page migratable into balloon
>   mm: remove direct calling of migration
>
>  drivers/virtio/virtio_balloon.c|  3 ++
>  fs/anon_inodes.c   |  6 +++
>  fs/proc/page.c |  3 ++
>  include/linux/anon_inodes.h|  1 +
>  include/linux/balloon_compaction.h | 15 +--
>  include/linux/compaction.h | 80 
> ++
>  include/linux/fs.h |  2 +
>  include/linux/page-flags.h | 19 
>  include/uapi/linux/kernel-page-flags.h |  1 +
>  mm/balloon_compaction.c| 72 ++
>  mm/compaction.c|  8 ++--
>  mm/migrate.c   | 24 +++---
>  12 files changed, 160 insertions(+), 74 deletions(-)
>
> --
> 2.1.4
>

[PATCH 0/4] enable migration of driver pages

[Rafael Aquini]

On Mon, Jul 13, 2015 at 05:35:15PM +0900, Gioh Kim wrote:
> From: Gioh Kim 
> 
> Hello,
> 
> This series try to enable migration of non-LRU pages, such as driver's page.
> 
> My ARM-based platform occured severe fragmentation problem after long-term
> (several days) test. Sometimes even order-3 page allocation failed. It has
> memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic 
> processing
> and 20~30 memory is reserved for zram.
> 
> I found that many pages of GPU driver and zram are non-movable pages. So I
> reported Minchan Kim, the maintainer of zram, and he made the internal 
> compaction logic of zram. And I made the internal compaction of GPU driver.
> 
> They reduced some fragmentation but they are not enough effective.
> They are activated by its own interface, /sys, so they are not cooperative
> with kernel compaction. If there is too much fragmentation and kernel starts
> to compaction, zram and GPU driver cannot work with the kernel compaction.
> 
> So I thought there needs a interface to combine driver and kernel compaction.
> This patch adds a generic isolate/migrate/putback callbacks for page
> address-space and a new interface to create anon-inode to manage
> address_space_operation. The zram and GPU, and any other modules can create
> anon_inode and register its own migration method. The kernel compaction can
> call the registered migration when it does compaction.
> 
> My GPU driver source is not in-kernel driver so that I apply the interface
> into balloon driver. The balloon driver is already merged
> into the kernel compaction as a corner-case. This patch have the balloon
> driver migration be called by the generic interface.
> 
> 
> This patch set combines 4 patches.
> 
> 1. patch 1/4: get inode from anon_inodes
> This patch adds new interface to create inode from anon_inodes.
> 
> 2. patch 2/4: framework to isolate/migrate/putback page
> Add isolatepage, putbackpage into address_space_operations
> and wrapper function to call them.
> 
> 3. patch 3/4: apply the framework into balloon driver
> The balloon driver is applied into the framework. It gets a inode
> from anon_inodes and register operations in the inode.
> The kernel compaction calls generic interfaces, not balloon
> driver interfaces. 
> Any other drivers can register operations via inode like this
> to migrate it's pages.
> 
> 4. patch 4/4: remove direct calling of migration of driver pages
> Non-lru pages are also migrated with lru pages by move_to_new_page().
> 
> This patch set is tested:
> - turn on Ubuntu 14.04 with 1G memory on qemu.
> - do kernel building
> - after several seconds check more than 512MB is used with free command
> - command "balloon 512" in qemu monitor
> - check hundreds MB of pages are migrated
> 
> My thanks to Konstantin Khlebnikov for his reviews of the RFC patch set.
> Most of the changes were based on his feedback.
> 
> This patch-set is based on v4.1
> 
> 
> Gioh Kim (4):
>   fs/anon_inodes: new interface to create new inode
>   mm/compaction: enable mobile-page migration
>   mm/balloon: apply mobile page migratable into balloon
>   mm: remove direct calling of migration
> 
>  drivers/virtio/virtio_balloon.c|  3 ++
>  fs/anon_inodes.c   |  6 +++
>  fs/proc/page.c |  3 ++
>  include/linux/anon_inodes.h|  1 +
>  include/linux/balloon_compaction.h | 15 +--
>  include/linux/compaction.h | 80 
> ++
>  include/linux/fs.h |  2 +
>  include/linux/page-flags.h | 19 
>  include/uapi/linux/kernel-page-flags.h |  1 +
>  mm/balloon_compaction.c| 72 ++
>  mm/compaction.c|  8 ++--
>  mm/migrate.c   | 24 +++---
>  12 files changed, 160 insertions(+), 74 deletions(-)
> 
> -- 
> 2.1.4
> 
Acked-by: Rafael Aquini