Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
Hello, On Tue, Nov 04, 2014 at 10:29:59AM +0100, Vlastimil Babka wrote: > On 11/04/2014 08:53 AM, Minchan Kim wrote: > >Hello, > > > >On Wed, Oct 29, 2014 at 03:43:33PM +0100, Vlastimil Babka wrote: > >>On 10/16/2014 10:55 AM, Laura Abbott wrote: > >> > >>Hi, > >> > >>did anyone try/suggest the following idea? > >> > >>- keep CMA as fallback to MOVABLE as is is now, i.e. non-agressive > >>- when UNMOVABLE (RECLAIMABLE also?) allocation fails and CMA > >>pageblocks have space, don't OOM immediately, but first try to > >>migrate some MOVABLE pages to CMA pageblocks, to make space for the > >>UNMOVABLE allocation in non-CMA pageblocks > >>- this should keep CMA pageblocks free as long as possible and > >>useful for CMA allocations, but without restricting the non-MOVABLE > >>allocations even though there is free memory (but in CMA pageblocks) > >>- the fact that a MOVABLE page could be successfully migrated to CMA > >>pageblock, means it was not pinned or otherwise non-migratable, so > >>there's a good chance it can be migrated back again if CMA > >>pageblocks need to be used by CMA allocation > > > >I suggested exactly same idea long time ago. > > > >>- it's more complex, but I guess we have most of the necessary > >>infrastructure in compaction already :) > > > >I agree but still, it doesn't solve reclaim problem(ie, VM doesn't > >need to reclaim CMA pages when memory pressure of unmovable pages > >happens). Of course, we could make VM be aware of that via introducing > >new flag of __isolate_lru_page. > > Well, if it relaims CMA pages, then it has to be followed by the > migration. Is that better or worse than breaking LRU assumptions by > reclaiming based on where the page is located? I thought this was > basically what lumpy reclaim did, and it was removed. It would work and it might cost for using for CMA because CMA already can migrate/discard lots of pages, which will hurt LRU assumption. However, I don't think it's optimal. > > >However, I'd like to think CMA design from the beginning. > >It made page allocation logic complicated, even very fragile as we > >had recently and now we need to add new logics to migrate like you said. > >As well, we need to fix reclaim path, too. > > > >It makes mm complicated day by day even though it doesn't do the role > >enough well(ie, big latency and frequent allocation failure) so I really > >want to stop making the mess bloated. > > Yeah that would be great. > > >Long time ago, when I saw Joonsoo's CMA agressive allocation patchset > >(ie, roundrobin allocation between CMA and normal movable pages) > >it was good to me at a first glance but it needs tweak of allocation > >path and doesn't solve reclaim path, either. Yes, reclaim path could > >be solved by another patch but I want to solve it altogether. > > > >At that time, I suggested big surgery to Joonsoo in offline that > >let's move CMA allocation with movable zone allocation. With it, > >we could make allocation/reclaim path simple but thing is we should > > I'm not sure I understand enough from this. You want to introduce a > movable zone instead of CMA pageblocks? But how to size it, resize > it, would it be possible? Why do we need to care of resizing? All of CMA pages are reserved by using memblock during boot. If we can set the zone size after that, maybe we don't need to resize the zone. > > >make VM be aware of overlapping MOVABLE zone which means some of pages > >in the zone could be part of another zones but I think we already have > >logics to handle it when I read comment in isolate_freepages so I think > >the design should work. > > Why would it overlap in the first place? Just because it wouldn't be > sized on pageblock boundary? Or to make (re)sizing simpler? Yeah we > could probably handle that, but it's not completely for free (you > iterate over blocks/pages uselessly). Reserved pages for CMA are spread over the system memory. So zones could overlap each other so we need check that overlapping like pageblock_pfn_to_page while we need to walk pfn in order. It's not free but it would add the overhead pfn-order walking like compaction, which is not hot path. > > >A thing you guys might worry is bigger CMA latency because it makes > >CMA memory usage ratio higher than the approach you mentioned but > >anyone couldn't guarantee it once memory is fully utilized. > >In addition, we have used fair zone allocator policy so it makes > >round robin allocation automatically so I believe it should be way > >to go. > > Yeah maybe it could be simpler in the end. Although a new zone type > could be a disturbing change, with some overhead to per-cpu > structures etc. The allocations in that zone would be somewhat at > disadvantage wrt LRU, as CMA allocation would mostly reclaim them > instead of migrating away (assuming there wouldn't be so much spare > space for migration as when CMA pageblocks are part of a much larger > zone). But I guess the same could be said about the DMA zone... What
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
Hello, On Tue, Nov 04, 2014 at 10:29:59AM +0100, Vlastimil Babka wrote: On 11/04/2014 08:53 AM, Minchan Kim wrote: Hello, On Wed, Oct 29, 2014 at 03:43:33PM +0100, Vlastimil Babka wrote: On 10/16/2014 10:55 AM, Laura Abbott wrote: Hi, did anyone try/suggest the following idea? - keep CMA as fallback to MOVABLE as is is now, i.e. non-agressive - when UNMOVABLE (RECLAIMABLE also?) allocation fails and CMA pageblocks have space, don't OOM immediately, but first try to migrate some MOVABLE pages to CMA pageblocks, to make space for the UNMOVABLE allocation in non-CMA pageblocks - this should keep CMA pageblocks free as long as possible and useful for CMA allocations, but without restricting the non-MOVABLE allocations even though there is free memory (but in CMA pageblocks) - the fact that a MOVABLE page could be successfully migrated to CMA pageblock, means it was not pinned or otherwise non-migratable, so there's a good chance it can be migrated back again if CMA pageblocks need to be used by CMA allocation I suggested exactly same idea long time ago. - it's more complex, but I guess we have most of the necessary infrastructure in compaction already :) I agree but still, it doesn't solve reclaim problem(ie, VM doesn't need to reclaim CMA pages when memory pressure of unmovable pages happens). Of course, we could make VM be aware of that via introducing new flag of __isolate_lru_page. Well, if it relaims CMA pages, then it has to be followed by the migration. Is that better or worse than breaking LRU assumptions by reclaiming based on where the page is located? I thought this was basically what lumpy reclaim did, and it was removed. It would work and it might cost for using for CMA because CMA already can migrate/discard lots of pages, which will hurt LRU assumption. However, I don't think it's optimal. However, I'd like to think CMA design from the beginning. It made page allocation logic complicated, even very fragile as we had recently and now we need to add new logics to migrate like you said. As well, we need to fix reclaim path, too. It makes mm complicated day by day even though it doesn't do the role enough well(ie, big latency and frequent allocation failure) so I really want to stop making the mess bloated. Yeah that would be great. Long time ago, when I saw Joonsoo's CMA agressive allocation patchset (ie, roundrobin allocation between CMA and normal movable pages) it was good to me at a first glance but it needs tweak of allocation path and doesn't solve reclaim path, either. Yes, reclaim path could be solved by another patch but I want to solve it altogether. At that time, I suggested big surgery to Joonsoo in offline that let's move CMA allocation with movable zone allocation. With it, we could make allocation/reclaim path simple but thing is we should I'm not sure I understand enough from this. You want to introduce a movable zone instead of CMA pageblocks? But how to size it, resize it, would it be possible? Why do we need to care of resizing? All of CMA pages are reserved by using memblock during boot. If we can set the zone size after that, maybe we don't need to resize the zone. make VM be aware of overlapping MOVABLE zone which means some of pages in the zone could be part of another zones but I think we already have logics to handle it when I read comment in isolate_freepages so I think the design should work. Why would it overlap in the first place? Just because it wouldn't be sized on pageblock boundary? Or to make (re)sizing simpler? Yeah we could probably handle that, but it's not completely for free (you iterate over blocks/pages uselessly). Reserved pages for CMA are spread over the system memory. So zones could overlap each other so we need check that overlapping like pageblock_pfn_to_page while we need to walk pfn in order. It's not free but it would add the overhead pfn-order walking like compaction, which is not hot path. A thing you guys might worry is bigger CMA latency because it makes CMA memory usage ratio higher than the approach you mentioned but anyone couldn't guarantee it once memory is fully utilized. In addition, we have used fair zone allocator policy so it makes round robin allocation automatically so I believe it should be way to go. Yeah maybe it could be simpler in the end. Although a new zone type could be a disturbing change, with some overhead to per-cpu structures etc. The allocations in that zone would be somewhat at disadvantage wrt LRU, as CMA allocation would mostly reclaim them instead of migrating away (assuming there wouldn't be so much spare space for migration as when CMA pageblocks are part of a much larger zone). But I guess the same could be said about the DMA zone... What do you mean CMA allocation? If you meant movable pages allocation like userspace page, it would be round-robin by fair zone policy. If you meant device request for
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On 11/04/2014 08:53 AM, Minchan Kim wrote: Hello, On Wed, Oct 29, 2014 at 03:43:33PM +0100, Vlastimil Babka wrote: On 10/16/2014 10:55 AM, Laura Abbott wrote: Hi, did anyone try/suggest the following idea? - keep CMA as fallback to MOVABLE as is is now, i.e. non-agressive - when UNMOVABLE (RECLAIMABLE also?) allocation fails and CMA pageblocks have space, don't OOM immediately, but first try to migrate some MOVABLE pages to CMA pageblocks, to make space for the UNMOVABLE allocation in non-CMA pageblocks - this should keep CMA pageblocks free as long as possible and useful for CMA allocations, but without restricting the non-MOVABLE allocations even though there is free memory (but in CMA pageblocks) - the fact that a MOVABLE page could be successfully migrated to CMA pageblock, means it was not pinned or otherwise non-migratable, so there's a good chance it can be migrated back again if CMA pageblocks need to be used by CMA allocation I suggested exactly same idea long time ago. - it's more complex, but I guess we have most of the necessary infrastructure in compaction already :) I agree but still, it doesn't solve reclaim problem(ie, VM doesn't need to reclaim CMA pages when memory pressure of unmovable pages happens). Of course, we could make VM be aware of that via introducing new flag of __isolate_lru_page. Well, if it relaims CMA pages, then it has to be followed by the migration. Is that better or worse than breaking LRU assumptions by reclaiming based on where the page is located? I thought this was basically what lumpy reclaim did, and it was removed. However, I'd like to think CMA design from the beginning. It made page allocation logic complicated, even very fragile as we had recently and now we need to add new logics to migrate like you said. As well, we need to fix reclaim path, too. It makes mm complicated day by day even though it doesn't do the role enough well(ie, big latency and frequent allocation failure) so I really want to stop making the mess bloated. Yeah that would be great. Long time ago, when I saw Joonsoo's CMA agressive allocation patchset (ie, roundrobin allocation between CMA and normal movable pages) it was good to me at a first glance but it needs tweak of allocation path and doesn't solve reclaim path, either. Yes, reclaim path could be solved by another patch but I want to solve it altogether. At that time, I suggested big surgery to Joonsoo in offline that let's move CMA allocation with movable zone allocation. With it, we could make allocation/reclaim path simple but thing is we should I'm not sure I understand enough from this. You want to introduce a movable zone instead of CMA pageblocks? But how to size it, resize it, would it be possible? make VM be aware of overlapping MOVABLE zone which means some of pages in the zone could be part of another zones but I think we already have logics to handle it when I read comment in isolate_freepages so I think the design should work. Why would it overlap in the first place? Just because it wouldn't be sized on pageblock boundary? Or to make (re)sizing simpler? Yeah we could probably handle that, but it's not completely for free (you iterate over blocks/pages uselessly). A thing you guys might worry is bigger CMA latency because it makes CMA memory usage ratio higher than the approach you mentioned but anyone couldn't guarantee it once memory is fully utilized. In addition, we have used fair zone allocator policy so it makes round robin allocation automatically so I believe it should be way to go. Yeah maybe it could be simpler in the end. Although a new zone type could be a disturbing change, with some overhead to per-cpu structures etc. The allocations in that zone would be somewhat at disadvantage wrt LRU, as CMA allocation would mostly reclaim them instead of migrating away (assuming there wouldn't be so much spare space for migration as when CMA pageblocks are part of a much larger zone). But I guess the same could be said about the DMA zone... Thoughts? Vlastimil Thanks, Laura -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majord...@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: mailto:"d...@kvack.org;> em...@kvack.org -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On Tue, Nov 4, 2014 at 3:53 PM, Minchan Kim wrote: > Hello, > > On Wed, Oct 29, 2014 at 03:43:33PM +0100, Vlastimil Babka wrote: >> On 10/16/2014 10:55 AM, Laura Abbott wrote: >> >On 10/15/2014 8:35 PM, Hui Zhu wrote: >> > >> >It's good to see another proposal to fix CMA utilization. Do you have >> >any data about the success rate of CMA contiguous allocation after >> >this patch series? I played around with a similar approach of using >> >CMA for MIGRATE_MOVABLE allocations and found that although utilization >> >did increase, contiguous allocations failed at a higher rate and were >> >much slower. I see what this series is trying to do with avoiding >> >allocation from CMA pages when a contiguous allocation is progress. >> >My concern is that there would still be problems with contiguous >> >allocation after all the MIGRATE_MOVABLE fallback has happened. >> >> Hi, >> >> did anyone try/suggest the following idea? >> >> - keep CMA as fallback to MOVABLE as is is now, i.e. non-agressive >> - when UNMOVABLE (RECLAIMABLE also?) allocation fails and CMA >> pageblocks have space, don't OOM immediately, but first try to >> migrate some MOVABLE pages to CMA pageblocks, to make space for the >> UNMOVABLE allocation in non-CMA pageblocks >> - this should keep CMA pageblocks free as long as possible and >> useful for CMA allocations, but without restricting the non-MOVABLE >> allocations even though there is free memory (but in CMA pageblocks) >> - the fact that a MOVABLE page could be successfully migrated to CMA >> pageblock, means it was not pinned or otherwise non-migratable, so >> there's a good chance it can be migrated back again if CMA >> pageblocks need to be used by CMA allocation > > I suggested exactly same idea long time ago. > >> - it's more complex, but I guess we have most of the necessary >> infrastructure in compaction already :) > > I agree but still, it doesn't solve reclaim problem(ie, VM doesn't > need to reclaim CMA pages when memory pressure of unmovable pages > happens). Of course, we could make VM be aware of that via introducing > new flag of __isolate_lru_page. > > However, I'd like to think CMA design from the beginning. > It made page allocation logic complicated, even very fragile as we > had recently and now we need to add new logics to migrate like you said. > As well, we need to fix reclaim path, too. > > It makes mm complicated day by day even though it doesn't do the role > enough well(ie, big latency and frequent allocation failure) so I really > want to stop making the mess bloated. > > Long time ago, when I saw Joonsoo's CMA agressive allocation patchset > (ie, roundrobin allocation between CMA and normal movable pages) > it was good to me at a first glance but it needs tweak of allocation > path and doesn't solve reclaim path, either. Yes, reclaim path could > be solved by another patch but I want to solve it altogether. > > At that time, I suggested big surgery to Joonsoo in offline that > let's move CMA allocation with movable zone allocation. With it, > we could make allocation/reclaim path simple but thing is we should > make VM be aware of overlapping MOVABLE zone which means some of pages > in the zone could be part of another zones but I think we already have > logics to handle it when I read comment in isolate_freepages so I think > the design should work. Thanks. > > A thing you guys might worry is bigger CMA latency because it makes > CMA memory usage ratio higher than the approach you mentioned but > anyone couldn't guarantee it once memory is fully utilized. > In addition, we have used fair zone allocator policy so it makes > round robin allocation automatically so I believe it should be way > to go. Even if kernel use it to allocate the CMA memory, CMA alloc latency will happen if most of memory is allocated and driver try to get CMA memory. https://lkml.org/lkml/2014/10/17/129 https://lkml.org/lkml/2014/10/17/130 These patches let cma_alloc do a shrink with function shrink_all_memory_for_cma if need. It handle a lot of latency issue in my part. And I think it can be more configurable for example some device use it and others not. Thanks, Hui > >> >> Thoughts? >> Vlastimil >> >> >Thanks, >> >Laura >> > >> >> -- >> To unsubscribe, send a message with 'unsubscribe linux-mm' in >> the body to majord...@kvack.org. For more info on Linux MM, >> see: http://www.linux-mm.org/ . >> Don't email: mailto:"d...@kvack.org;> em...@kvack.org > > -- > Kind regards, > Minchan Kim > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majord...@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: mailto:"d...@kvack.org;> em...@kvack.org -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On Tue, Nov 4, 2014 at 3:53 PM, Minchan Kim minc...@kernel.org wrote: Hello, On Wed, Oct 29, 2014 at 03:43:33PM +0100, Vlastimil Babka wrote: On 10/16/2014 10:55 AM, Laura Abbott wrote: On 10/15/2014 8:35 PM, Hui Zhu wrote: It's good to see another proposal to fix CMA utilization. Do you have any data about the success rate of CMA contiguous allocation after this patch series? I played around with a similar approach of using CMA for MIGRATE_MOVABLE allocations and found that although utilization did increase, contiguous allocations failed at a higher rate and were much slower. I see what this series is trying to do with avoiding allocation from CMA pages when a contiguous allocation is progress. My concern is that there would still be problems with contiguous allocation after all the MIGRATE_MOVABLE fallback has happened. Hi, did anyone try/suggest the following idea? - keep CMA as fallback to MOVABLE as is is now, i.e. non-agressive - when UNMOVABLE (RECLAIMABLE also?) allocation fails and CMA pageblocks have space, don't OOM immediately, but first try to migrate some MOVABLE pages to CMA pageblocks, to make space for the UNMOVABLE allocation in non-CMA pageblocks - this should keep CMA pageblocks free as long as possible and useful for CMA allocations, but without restricting the non-MOVABLE allocations even though there is free memory (but in CMA pageblocks) - the fact that a MOVABLE page could be successfully migrated to CMA pageblock, means it was not pinned or otherwise non-migratable, so there's a good chance it can be migrated back again if CMA pageblocks need to be used by CMA allocation I suggested exactly same idea long time ago. - it's more complex, but I guess we have most of the necessary infrastructure in compaction already :) I agree but still, it doesn't solve reclaim problem(ie, VM doesn't need to reclaim CMA pages when memory pressure of unmovable pages happens). Of course, we could make VM be aware of that via introducing new flag of __isolate_lru_page. However, I'd like to think CMA design from the beginning. It made page allocation logic complicated, even very fragile as we had recently and now we need to add new logics to migrate like you said. As well, we need to fix reclaim path, too. It makes mm complicated day by day even though it doesn't do the role enough well(ie, big latency and frequent allocation failure) so I really want to stop making the mess bloated. Long time ago, when I saw Joonsoo's CMA agressive allocation patchset (ie, roundrobin allocation between CMA and normal movable pages) it was good to me at a first glance but it needs tweak of allocation path and doesn't solve reclaim path, either. Yes, reclaim path could be solved by another patch but I want to solve it altogether. At that time, I suggested big surgery to Joonsoo in offline that let's move CMA allocation with movable zone allocation. With it, we could make allocation/reclaim path simple but thing is we should make VM be aware of overlapping MOVABLE zone which means some of pages in the zone could be part of another zones but I think we already have logics to handle it when I read comment in isolate_freepages so I think the design should work. Thanks. A thing you guys might worry is bigger CMA latency because it makes CMA memory usage ratio higher than the approach you mentioned but anyone couldn't guarantee it once memory is fully utilized. In addition, we have used fair zone allocator policy so it makes round robin allocation automatically so I believe it should be way to go. Even if kernel use it to allocate the CMA memory, CMA alloc latency will happen if most of memory is allocated and driver try to get CMA memory. https://lkml.org/lkml/2014/10/17/129 https://lkml.org/lkml/2014/10/17/130 These patches let cma_alloc do a shrink with function shrink_all_memory_for_cma if need. It handle a lot of latency issue in my part. And I think it can be more configurable for example some device use it and others not. Thanks, Hui Thoughts? Vlastimil Thanks, Laura -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majord...@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: a href=mailto:d...@kvack.org; em...@kvack.org /a -- Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majord...@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: a href=mailto:d...@kvack.org; em...@kvack.org /a -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On 11/04/2014 08:53 AM, Minchan Kim wrote: Hello, On Wed, Oct 29, 2014 at 03:43:33PM +0100, Vlastimil Babka wrote: On 10/16/2014 10:55 AM, Laura Abbott wrote: Hi, did anyone try/suggest the following idea? - keep CMA as fallback to MOVABLE as is is now, i.e. non-agressive - when UNMOVABLE (RECLAIMABLE also?) allocation fails and CMA pageblocks have space, don't OOM immediately, but first try to migrate some MOVABLE pages to CMA pageblocks, to make space for the UNMOVABLE allocation in non-CMA pageblocks - this should keep CMA pageblocks free as long as possible and useful for CMA allocations, but without restricting the non-MOVABLE allocations even though there is free memory (but in CMA pageblocks) - the fact that a MOVABLE page could be successfully migrated to CMA pageblock, means it was not pinned or otherwise non-migratable, so there's a good chance it can be migrated back again if CMA pageblocks need to be used by CMA allocation I suggested exactly same idea long time ago. - it's more complex, but I guess we have most of the necessary infrastructure in compaction already :) I agree but still, it doesn't solve reclaim problem(ie, VM doesn't need to reclaim CMA pages when memory pressure of unmovable pages happens). Of course, we could make VM be aware of that via introducing new flag of __isolate_lru_page. Well, if it relaims CMA pages, then it has to be followed by the migration. Is that better or worse than breaking LRU assumptions by reclaiming based on where the page is located? I thought this was basically what lumpy reclaim did, and it was removed. However, I'd like to think CMA design from the beginning. It made page allocation logic complicated, even very fragile as we had recently and now we need to add new logics to migrate like you said. As well, we need to fix reclaim path, too. It makes mm complicated day by day even though it doesn't do the role enough well(ie, big latency and frequent allocation failure) so I really want to stop making the mess bloated. Yeah that would be great. Long time ago, when I saw Joonsoo's CMA agressive allocation patchset (ie, roundrobin allocation between CMA and normal movable pages) it was good to me at a first glance but it needs tweak of allocation path and doesn't solve reclaim path, either. Yes, reclaim path could be solved by another patch but I want to solve it altogether. At that time, I suggested big surgery to Joonsoo in offline that let's move CMA allocation with movable zone allocation. With it, we could make allocation/reclaim path simple but thing is we should I'm not sure I understand enough from this. You want to introduce a movable zone instead of CMA pageblocks? But how to size it, resize it, would it be possible? make VM be aware of overlapping MOVABLE zone which means some of pages in the zone could be part of another zones but I think we already have logics to handle it when I read comment in isolate_freepages so I think the design should work. Why would it overlap in the first place? Just because it wouldn't be sized on pageblock boundary? Or to make (re)sizing simpler? Yeah we could probably handle that, but it's not completely for free (you iterate over blocks/pages uselessly). A thing you guys might worry is bigger CMA latency because it makes CMA memory usage ratio higher than the approach you mentioned but anyone couldn't guarantee it once memory is fully utilized. In addition, we have used fair zone allocator policy so it makes round robin allocation automatically so I believe it should be way to go. Yeah maybe it could be simpler in the end. Although a new zone type could be a disturbing change, with some overhead to per-cpu structures etc. The allocations in that zone would be somewhat at disadvantage wrt LRU, as CMA allocation would mostly reclaim them instead of migrating away (assuming there wouldn't be so much spare space for migration as when CMA pageblocks are part of a much larger zone). But I guess the same could be said about the DMA zone... Thoughts? Vlastimil Thanks, Laura -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majord...@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: a href=mailto:d...@kvack.org; em...@kvack.org /a -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
Hello, On Wed, Oct 29, 2014 at 03:43:33PM +0100, Vlastimil Babka wrote: > On 10/16/2014 10:55 AM, Laura Abbott wrote: > >On 10/15/2014 8:35 PM, Hui Zhu wrote: > > > >It's good to see another proposal to fix CMA utilization. Do you have > >any data about the success rate of CMA contiguous allocation after > >this patch series? I played around with a similar approach of using > >CMA for MIGRATE_MOVABLE allocations and found that although utilization > >did increase, contiguous allocations failed at a higher rate and were > >much slower. I see what this series is trying to do with avoiding > >allocation from CMA pages when a contiguous allocation is progress. > >My concern is that there would still be problems with contiguous > >allocation after all the MIGRATE_MOVABLE fallback has happened. > > Hi, > > did anyone try/suggest the following idea? > > - keep CMA as fallback to MOVABLE as is is now, i.e. non-agressive > - when UNMOVABLE (RECLAIMABLE also?) allocation fails and CMA > pageblocks have space, don't OOM immediately, but first try to > migrate some MOVABLE pages to CMA pageblocks, to make space for the > UNMOVABLE allocation in non-CMA pageblocks > - this should keep CMA pageblocks free as long as possible and > useful for CMA allocations, but without restricting the non-MOVABLE > allocations even though there is free memory (but in CMA pageblocks) > - the fact that a MOVABLE page could be successfully migrated to CMA > pageblock, means it was not pinned or otherwise non-migratable, so > there's a good chance it can be migrated back again if CMA > pageblocks need to be used by CMA allocation I suggested exactly same idea long time ago. > - it's more complex, but I guess we have most of the necessary > infrastructure in compaction already :) I agree but still, it doesn't solve reclaim problem(ie, VM doesn't need to reclaim CMA pages when memory pressure of unmovable pages happens). Of course, we could make VM be aware of that via introducing new flag of __isolate_lru_page. However, I'd like to think CMA design from the beginning. It made page allocation logic complicated, even very fragile as we had recently and now we need to add new logics to migrate like you said. As well, we need to fix reclaim path, too. It makes mm complicated day by day even though it doesn't do the role enough well(ie, big latency and frequent allocation failure) so I really want to stop making the mess bloated. Long time ago, when I saw Joonsoo's CMA agressive allocation patchset (ie, roundrobin allocation between CMA and normal movable pages) it was good to me at a first glance but it needs tweak of allocation path and doesn't solve reclaim path, either. Yes, reclaim path could be solved by another patch but I want to solve it altogether. At that time, I suggested big surgery to Joonsoo in offline that let's move CMA allocation with movable zone allocation. With it, we could make allocation/reclaim path simple but thing is we should make VM be aware of overlapping MOVABLE zone which means some of pages in the zone could be part of another zones but I think we already have logics to handle it when I read comment in isolate_freepages so I think the design should work. A thing you guys might worry is bigger CMA latency because it makes CMA memory usage ratio higher than the approach you mentioned but anyone couldn't guarantee it once memory is fully utilized. In addition, we have used fair zone allocator policy so it makes round robin allocation automatically so I believe it should be way to go. > > Thoughts? > Vlastimil > > >Thanks, > >Laura > > > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majord...@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: mailto:"d...@kvack.org;> em...@kvack.org -- Kind regards, Minchan Kim -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On Mon, Nov 03, 2014 at 05:05:46PM +0900, Joonsoo Kim wrote: > On Mon, Nov 03, 2014 at 03:28:38PM +0800, Hui Zhu wrote: > > On Fri, Oct 24, 2014 at 1:25 PM, Joonsoo Kim wrote: > > > On Thu, Oct 16, 2014 at 11:35:47AM +0800, Hui Zhu wrote: > > >> In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of > > >> MIGRATE_MOVABLE. > > >> MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in > > >> order that Linux kernel want. > > >> > > >> If a system that has a lot of user space program is running, for > > >> instance, an Android board, most of memory is in MIGRATE_MOVABLE and > > >> allocated. Before function __rmqueue_fallback get memory from > > >> MIGRATE_CMA, the oom_killer will kill a task to release memory when > > >> kernel want get MIGRATE_UNMOVABLE memory because fallbacks of > > >> MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. > > >> This status is odd. The MIGRATE_CMA has a lot free memory but Linux > > >> kernel kill some tasks to release memory. > > >> > > >> This patch series adds a new function CMA_AGGRESSIVE to make CMA memory > > >> be more aggressive about allocation. > > >> If function CMA_AGGRESSIVE is available, when Linux kernel call function > > >> __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, > > >> MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA > > >> doesn't have enough pages for allocation, go back to allocate memory from > > >> MIGRATE_MOVABLE. > > >> Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and > > >> MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. > > > > > > Hello, > > > > > > I did some work similar to this. > > > Please reference following links. > > > > > > https://lkml.org/lkml/2014/5/28/64 > > > https://lkml.org/lkml/2014/5/28/57 > > > > > I tested #1 approach and found the problem. Although free memory on > > > meminfo can move around low watermark, there is large fluctuation on free > > > memory, because too many pages are reclaimed when kswapd is invoked. > > > Reason for this behaviour is that successive allocated CMA pages are > > > on the LRU list in that order and kswapd reclaim them in same order. > > > These memory doesn't help watermark checking from kwapd, so too many > > > pages are reclaimed, I guess. > > > > This issue can be handle with some change around shrink code. I am > > trying to integrate a patch for them. > > But I am not sure we met the same issue. Do you mind give me more > > info about this part? > > I forgot the issue because there is so big time-gap. I need sometime > to bring issue back to my brain. I will answer it soon after some thinking. Hello, Yes, the issue I mentioned before can be handled by modifying shrink code. I didn't dive into the problem so I also didn't know the detail. What I know is that there is large fluctuation on memory statistics and my guess is that it is caused by order of reclaimable pages. If we use #1 approach, the bulk of cma pages used for page cache or something are linked together and will be reclaimed all at once, because reclaiming cma pages are not counted and watermark check still fails until normal pages are reclaimed. I think that round-robin approach is better. Reasons are on the following: 1) Want to spread CMA freepages to whole users, not specific one user. We can modify shirnk code not to reclaim pages on CMA, because it doesn't help watermark checking in some cases. In this case, if we don't use round-robin, one specific user whose mapping with CMA pages can get all the benefit. Others would take all the overhead. I think that spreading will make all users fair. 2) Using CMA freepages first needlessly imposes overhead to CMA user. If the system has enough normal freepages, it is better not to use it as much as possible. Thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On Wed, Oct 29, 2014 at 10:43 PM, Vlastimil Babka wrote: > On 10/16/2014 10:55 AM, Laura Abbott wrote: >> >> On 10/15/2014 8:35 PM, Hui Zhu wrote: >> >> It's good to see another proposal to fix CMA utilization. Do you have >> any data about the success rate of CMA contiguous allocation after >> this patch series? I played around with a similar approach of using >> CMA for MIGRATE_MOVABLE allocations and found that although utilization >> did increase, contiguous allocations failed at a higher rate and were >> much slower. I see what this series is trying to do with avoiding >> allocation from CMA pages when a contiguous allocation is progress. >> My concern is that there would still be problems with contiguous >> allocation after all the MIGRATE_MOVABLE fallback has happened. > > > Hi, > > did anyone try/suggest the following idea? > > - keep CMA as fallback to MOVABLE as is is now, i.e. non-agressive > - when UNMOVABLE (RECLAIMABLE also?) allocation fails and CMA pageblocks > have space, don't OOM immediately, but first try to migrate some MOVABLE > pages to CMA pageblocks, to make space for the UNMOVABLE allocation in > non-CMA pageblocks > - this should keep CMA pageblocks free as long as possible and useful for > CMA allocations, but without restricting the non-MOVABLE allocations even > though there is free memory (but in CMA pageblocks) > - the fact that a MOVABLE page could be successfully migrated to CMA > pageblock, means it was not pinned or otherwise non-migratable, so there's a > good chance it can be migrated back again if CMA pageblocks need to be used > by CMA allocation > - it's more complex, but I guess we have most of the necessary > infrastructure in compaction already :) I think this idea make CMA allocation part become complex but make balance and shrink code become easy because it make CMA become real memory. I just worry about the speed of migrate memory with this idea. :) Thanks, Hui > > Thoughts? > Vlastimil > >> Thanks, >> Laura >> > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majord...@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: mailto:"d...@kvack.org;> em...@kvack.org -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On Mon, Nov 03, 2014 at 03:28:38PM +0800, Hui Zhu wrote: > On Fri, Oct 24, 2014 at 1:25 PM, Joonsoo Kim wrote: > > On Thu, Oct 16, 2014 at 11:35:47AM +0800, Hui Zhu wrote: > >> In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of > >> MIGRATE_MOVABLE. > >> MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in > >> order that Linux kernel want. > >> > >> If a system that has a lot of user space program is running, for > >> instance, an Android board, most of memory is in MIGRATE_MOVABLE and > >> allocated. Before function __rmqueue_fallback get memory from > >> MIGRATE_CMA, the oom_killer will kill a task to release memory when > >> kernel want get MIGRATE_UNMOVABLE memory because fallbacks of > >> MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. > >> This status is odd. The MIGRATE_CMA has a lot free memory but Linux > >> kernel kill some tasks to release memory. > >> > >> This patch series adds a new function CMA_AGGRESSIVE to make CMA memory > >> be more aggressive about allocation. > >> If function CMA_AGGRESSIVE is available, when Linux kernel call function > >> __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, > >> MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA > >> doesn't have enough pages for allocation, go back to allocate memory from > >> MIGRATE_MOVABLE. > >> Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and > >> MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. > > > > Hello, > > > > I did some work similar to this. > > Please reference following links. > > > > https://lkml.org/lkml/2014/5/28/64 > > https://lkml.org/lkml/2014/5/28/57 > > > I tested #1 approach and found the problem. Although free memory on > > meminfo can move around low watermark, there is large fluctuation on free > > memory, because too many pages are reclaimed when kswapd is invoked. > > Reason for this behaviour is that successive allocated CMA pages are > > on the LRU list in that order and kswapd reclaim them in same order. > > These memory doesn't help watermark checking from kwapd, so too many > > pages are reclaimed, I guess. > > This issue can be handle with some change around shrink code. I am > trying to integrate a patch for them. > But I am not sure we met the same issue. Do you mind give me more > info about this part? I forgot the issue because there is so big time-gap. I need sometime to bring issue back to my brain. I will answer it soon after some thinking. > > > > > And, aggressive allocation should be postponed until freepage counting > > bug is fixed, because aggressive allocation enlarge the possiblity > > of problem occurence. I tried to fix that bug, too. See following link. > > > > https://lkml.org/lkml/2014/10/23/90 > > I am following these patches. They are great! Thanks for your work. Thanks. :) -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On Mon, Nov 03, 2014 at 03:28:38PM +0800, Hui Zhu wrote: On Fri, Oct 24, 2014 at 1:25 PM, Joonsoo Kim iamjoonsoo@lge.com wrote: On Thu, Oct 16, 2014 at 11:35:47AM +0800, Hui Zhu wrote: In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of MIGRATE_MOVABLE. MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in order that Linux kernel want. If a system that has a lot of user space program is running, for instance, an Android board, most of memory is in MIGRATE_MOVABLE and allocated. Before function __rmqueue_fallback get memory from MIGRATE_CMA, the oom_killer will kill a task to release memory when kernel want get MIGRATE_UNMOVABLE memory because fallbacks of MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. This status is odd. The MIGRATE_CMA has a lot free memory but Linux kernel kill some tasks to release memory. This patch series adds a new function CMA_AGGRESSIVE to make CMA memory be more aggressive about allocation. If function CMA_AGGRESSIVE is available, when Linux kernel call function __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA doesn't have enough pages for allocation, go back to allocate memory from MIGRATE_MOVABLE. Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. Hello, I did some work similar to this. Please reference following links. https://lkml.org/lkml/2014/5/28/64 https://lkml.org/lkml/2014/5/28/57 I tested #1 approach and found the problem. Although free memory on meminfo can move around low watermark, there is large fluctuation on free memory, because too many pages are reclaimed when kswapd is invoked. Reason for this behaviour is that successive allocated CMA pages are on the LRU list in that order and kswapd reclaim them in same order. These memory doesn't help watermark checking from kwapd, so too many pages are reclaimed, I guess. This issue can be handle with some change around shrink code. I am trying to integrate a patch for them. But I am not sure we met the same issue. Do you mind give me more info about this part? I forgot the issue because there is so big time-gap. I need sometime to bring issue back to my brain. I will answer it soon after some thinking. And, aggressive allocation should be postponed until freepage counting bug is fixed, because aggressive allocation enlarge the possiblity of problem occurence. I tried to fix that bug, too. See following link. https://lkml.org/lkml/2014/10/23/90 I am following these patches. They are great! Thanks for your work. Thanks. :) -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On Wed, Oct 29, 2014 at 10:43 PM, Vlastimil Babka vba...@suse.cz wrote: On 10/16/2014 10:55 AM, Laura Abbott wrote: On 10/15/2014 8:35 PM, Hui Zhu wrote: It's good to see another proposal to fix CMA utilization. Do you have any data about the success rate of CMA contiguous allocation after this patch series? I played around with a similar approach of using CMA for MIGRATE_MOVABLE allocations and found that although utilization did increase, contiguous allocations failed at a higher rate and were much slower. I see what this series is trying to do with avoiding allocation from CMA pages when a contiguous allocation is progress. My concern is that there would still be problems with contiguous allocation after all the MIGRATE_MOVABLE fallback has happened. Hi, did anyone try/suggest the following idea? - keep CMA as fallback to MOVABLE as is is now, i.e. non-agressive - when UNMOVABLE (RECLAIMABLE also?) allocation fails and CMA pageblocks have space, don't OOM immediately, but first try to migrate some MOVABLE pages to CMA pageblocks, to make space for the UNMOVABLE allocation in non-CMA pageblocks - this should keep CMA pageblocks free as long as possible and useful for CMA allocations, but without restricting the non-MOVABLE allocations even though there is free memory (but in CMA pageblocks) - the fact that a MOVABLE page could be successfully migrated to CMA pageblock, means it was not pinned or otherwise non-migratable, so there's a good chance it can be migrated back again if CMA pageblocks need to be used by CMA allocation - it's more complex, but I guess we have most of the necessary infrastructure in compaction already :) I think this idea make CMA allocation part become complex but make balance and shrink code become easy because it make CMA become real memory. I just worry about the speed of migrate memory with this idea. :) Thanks, Hui Thoughts? Vlastimil Thanks, Laura -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majord...@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: a href=mailto:d...@kvack.org; em...@kvack.org /a -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On Mon, Nov 03, 2014 at 05:05:46PM +0900, Joonsoo Kim wrote: On Mon, Nov 03, 2014 at 03:28:38PM +0800, Hui Zhu wrote: On Fri, Oct 24, 2014 at 1:25 PM, Joonsoo Kim iamjoonsoo@lge.com wrote: On Thu, Oct 16, 2014 at 11:35:47AM +0800, Hui Zhu wrote: In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of MIGRATE_MOVABLE. MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in order that Linux kernel want. If a system that has a lot of user space program is running, for instance, an Android board, most of memory is in MIGRATE_MOVABLE and allocated. Before function __rmqueue_fallback get memory from MIGRATE_CMA, the oom_killer will kill a task to release memory when kernel want get MIGRATE_UNMOVABLE memory because fallbacks of MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. This status is odd. The MIGRATE_CMA has a lot free memory but Linux kernel kill some tasks to release memory. This patch series adds a new function CMA_AGGRESSIVE to make CMA memory be more aggressive about allocation. If function CMA_AGGRESSIVE is available, when Linux kernel call function __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA doesn't have enough pages for allocation, go back to allocate memory from MIGRATE_MOVABLE. Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. Hello, I did some work similar to this. Please reference following links. https://lkml.org/lkml/2014/5/28/64 https://lkml.org/lkml/2014/5/28/57 I tested #1 approach and found the problem. Although free memory on meminfo can move around low watermark, there is large fluctuation on free memory, because too many pages are reclaimed when kswapd is invoked. Reason for this behaviour is that successive allocated CMA pages are on the LRU list in that order and kswapd reclaim them in same order. These memory doesn't help watermark checking from kwapd, so too many pages are reclaimed, I guess. This issue can be handle with some change around shrink code. I am trying to integrate a patch for them. But I am not sure we met the same issue. Do you mind give me more info about this part? I forgot the issue because there is so big time-gap. I need sometime to bring issue back to my brain. I will answer it soon after some thinking. Hello, Yes, the issue I mentioned before can be handled by modifying shrink code. I didn't dive into the problem so I also didn't know the detail. What I know is that there is large fluctuation on memory statistics and my guess is that it is caused by order of reclaimable pages. If we use #1 approach, the bulk of cma pages used for page cache or something are linked together and will be reclaimed all at once, because reclaiming cma pages are not counted and watermark check still fails until normal pages are reclaimed. I think that round-robin approach is better. Reasons are on the following: 1) Want to spread CMA freepages to whole users, not specific one user. We can modify shirnk code not to reclaim pages on CMA, because it doesn't help watermark checking in some cases. In this case, if we don't use round-robin, one specific user whose mapping with CMA pages can get all the benefit. Others would take all the overhead. I think that spreading will make all users fair. 2) Using CMA freepages first needlessly imposes overhead to CMA user. If the system has enough normal freepages, it is better not to use it as much as possible. Thanks. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
Hello, On Wed, Oct 29, 2014 at 03:43:33PM +0100, Vlastimil Babka wrote: On 10/16/2014 10:55 AM, Laura Abbott wrote: On 10/15/2014 8:35 PM, Hui Zhu wrote: It's good to see another proposal to fix CMA utilization. Do you have any data about the success rate of CMA contiguous allocation after this patch series? I played around with a similar approach of using CMA for MIGRATE_MOVABLE allocations and found that although utilization did increase, contiguous allocations failed at a higher rate and were much slower. I see what this series is trying to do with avoiding allocation from CMA pages when a contiguous allocation is progress. My concern is that there would still be problems with contiguous allocation after all the MIGRATE_MOVABLE fallback has happened. Hi, did anyone try/suggest the following idea? - keep CMA as fallback to MOVABLE as is is now, i.e. non-agressive - when UNMOVABLE (RECLAIMABLE also?) allocation fails and CMA pageblocks have space, don't OOM immediately, but first try to migrate some MOVABLE pages to CMA pageblocks, to make space for the UNMOVABLE allocation in non-CMA pageblocks - this should keep CMA pageblocks free as long as possible and useful for CMA allocations, but without restricting the non-MOVABLE allocations even though there is free memory (but in CMA pageblocks) - the fact that a MOVABLE page could be successfully migrated to CMA pageblock, means it was not pinned or otherwise non-migratable, so there's a good chance it can be migrated back again if CMA pageblocks need to be used by CMA allocation I suggested exactly same idea long time ago. - it's more complex, but I guess we have most of the necessary infrastructure in compaction already :) I agree but still, it doesn't solve reclaim problem(ie, VM doesn't need to reclaim CMA pages when memory pressure of unmovable pages happens). Of course, we could make VM be aware of that via introducing new flag of __isolate_lru_page. However, I'd like to think CMA design from the beginning. It made page allocation logic complicated, even very fragile as we had recently and now we need to add new logics to migrate like you said. As well, we need to fix reclaim path, too. It makes mm complicated day by day even though it doesn't do the role enough well(ie, big latency and frequent allocation failure) so I really want to stop making the mess bloated. Long time ago, when I saw Joonsoo's CMA agressive allocation patchset (ie, roundrobin allocation between CMA and normal movable pages) it was good to me at a first glance but it needs tweak of allocation path and doesn't solve reclaim path, either. Yes, reclaim path could be solved by another patch but I want to solve it altogether. At that time, I suggested big surgery to Joonsoo in offline that let's move CMA allocation with movable zone allocation. With it, we could make allocation/reclaim path simple but thing is we should make VM be aware of overlapping MOVABLE zone which means some of pages in the zone could be part of another zones but I think we already have logics to handle it when I read comment in isolate_freepages so I think the design should work. A thing you guys might worry is bigger CMA latency because it makes CMA memory usage ratio higher than the approach you mentioned but anyone couldn't guarantee it once memory is fully utilized. In addition, we have used fair zone allocator policy so it makes round robin allocation automatically so I believe it should be way to go. Thoughts? Vlastimil Thanks, Laura -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majord...@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: a href=mailto:d...@kvack.org; em...@kvack.org /a -- Kind regards, Minchan Kim -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On Fri, Oct 24, 2014 at 1:25 PM, Joonsoo Kim wrote: > On Thu, Oct 16, 2014 at 11:35:47AM +0800, Hui Zhu wrote: >> In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of >> MIGRATE_MOVABLE. >> MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in >> order that Linux kernel want. >> >> If a system that has a lot of user space program is running, for >> instance, an Android board, most of memory is in MIGRATE_MOVABLE and >> allocated. Before function __rmqueue_fallback get memory from >> MIGRATE_CMA, the oom_killer will kill a task to release memory when >> kernel want get MIGRATE_UNMOVABLE memory because fallbacks of >> MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. >> This status is odd. The MIGRATE_CMA has a lot free memory but Linux >> kernel kill some tasks to release memory. >> >> This patch series adds a new function CMA_AGGRESSIVE to make CMA memory >> be more aggressive about allocation. >> If function CMA_AGGRESSIVE is available, when Linux kernel call function >> __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, >> MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA >> doesn't have enough pages for allocation, go back to allocate memory from >> MIGRATE_MOVABLE. >> Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and >> MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. > > Hello, > > I did some work similar to this. > Please reference following links. > > https://lkml.org/lkml/2014/5/28/64 > https://lkml.org/lkml/2014/5/28/57 > I tested #1 approach and found the problem. Although free memory on > meminfo can move around low watermark, there is large fluctuation on free > memory, because too many pages are reclaimed when kswapd is invoked. > Reason for this behaviour is that successive allocated CMA pages are > on the LRU list in that order and kswapd reclaim them in same order. > These memory doesn't help watermark checking from kwapd, so too many > pages are reclaimed, I guess. This issue can be handle with some change around shrink code. I am trying to integrate a patch for them. But I am not sure we met the same issue. Do you mind give me more info about this part? > > And, aggressive allocation should be postponed until freepage counting > bug is fixed, because aggressive allocation enlarge the possiblity > of problem occurence. I tried to fix that bug, too. See following link. > > https://lkml.org/lkml/2014/10/23/90 I am following these patches. They are great! Thanks for your work. Best, Hui > > Thanks. > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majord...@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: mailto:"d...@kvack.org;> em...@kvack.org -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On Fri, Oct 24, 2014 at 1:25 PM, Joonsoo Kim iamjoonsoo@lge.com wrote: On Thu, Oct 16, 2014 at 11:35:47AM +0800, Hui Zhu wrote: In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of MIGRATE_MOVABLE. MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in order that Linux kernel want. If a system that has a lot of user space program is running, for instance, an Android board, most of memory is in MIGRATE_MOVABLE and allocated. Before function __rmqueue_fallback get memory from MIGRATE_CMA, the oom_killer will kill a task to release memory when kernel want get MIGRATE_UNMOVABLE memory because fallbacks of MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. This status is odd. The MIGRATE_CMA has a lot free memory but Linux kernel kill some tasks to release memory. This patch series adds a new function CMA_AGGRESSIVE to make CMA memory be more aggressive about allocation. If function CMA_AGGRESSIVE is available, when Linux kernel call function __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA doesn't have enough pages for allocation, go back to allocate memory from MIGRATE_MOVABLE. Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. Hello, I did some work similar to this. Please reference following links. https://lkml.org/lkml/2014/5/28/64 https://lkml.org/lkml/2014/5/28/57 I tested #1 approach and found the problem. Although free memory on meminfo can move around low watermark, there is large fluctuation on free memory, because too many pages are reclaimed when kswapd is invoked. Reason for this behaviour is that successive allocated CMA pages are on the LRU list in that order and kswapd reclaim them in same order. These memory doesn't help watermark checking from kwapd, so too many pages are reclaimed, I guess. This issue can be handle with some change around shrink code. I am trying to integrate a patch for them. But I am not sure we met the same issue. Do you mind give me more info about this part? And, aggressive allocation should be postponed until freepage counting bug is fixed, because aggressive allocation enlarge the possiblity of problem occurence. I tried to fix that bug, too. See following link. https://lkml.org/lkml/2014/10/23/90 I am following these patches. They are great! Thanks for your work. Best, Hui Thanks. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majord...@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: a href=mailto:d...@kvack.org; em...@kvack.org /a -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On 10/16/2014 10:55 AM, Laura Abbott wrote: On 10/15/2014 8:35 PM, Hui Zhu wrote: It's good to see another proposal to fix CMA utilization. Do you have any data about the success rate of CMA contiguous allocation after this patch series? I played around with a similar approach of using CMA for MIGRATE_MOVABLE allocations and found that although utilization did increase, contiguous allocations failed at a higher rate and were much slower. I see what this series is trying to do with avoiding allocation from CMA pages when a contiguous allocation is progress. My concern is that there would still be problems with contiguous allocation after all the MIGRATE_MOVABLE fallback has happened. Hi, did anyone try/suggest the following idea? - keep CMA as fallback to MOVABLE as is is now, i.e. non-agressive - when UNMOVABLE (RECLAIMABLE also?) allocation fails and CMA pageblocks have space, don't OOM immediately, but first try to migrate some MOVABLE pages to CMA pageblocks, to make space for the UNMOVABLE allocation in non-CMA pageblocks - this should keep CMA pageblocks free as long as possible and useful for CMA allocations, but without restricting the non-MOVABLE allocations even though there is free memory (but in CMA pageblocks) - the fact that a MOVABLE page could be successfully migrated to CMA pageblock, means it was not pinned or otherwise non-migratable, so there's a good chance it can be migrated back again if CMA pageblocks need to be used by CMA allocation - it's more complex, but I guess we have most of the necessary infrastructure in compaction already :) Thoughts? Vlastimil Thanks, Laura -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On 10/16/2014 10:55 AM, Laura Abbott wrote: On 10/15/2014 8:35 PM, Hui Zhu wrote: It's good to see another proposal to fix CMA utilization. Do you have any data about the success rate of CMA contiguous allocation after this patch series? I played around with a similar approach of using CMA for MIGRATE_MOVABLE allocations and found that although utilization did increase, contiguous allocations failed at a higher rate and were much slower. I see what this series is trying to do with avoiding allocation from CMA pages when a contiguous allocation is progress. My concern is that there would still be problems with contiguous allocation after all the MIGRATE_MOVABLE fallback has happened. Hi, did anyone try/suggest the following idea? - keep CMA as fallback to MOVABLE as is is now, i.e. non-agressive - when UNMOVABLE (RECLAIMABLE also?) allocation fails and CMA pageblocks have space, don't OOM immediately, but first try to migrate some MOVABLE pages to CMA pageblocks, to make space for the UNMOVABLE allocation in non-CMA pageblocks - this should keep CMA pageblocks free as long as possible and useful for CMA allocations, but without restricting the non-MOVABLE allocations even though there is free memory (but in CMA pageblocks) - the fact that a MOVABLE page could be successfully migrated to CMA pageblock, means it was not pinned or otherwise non-migratable, so there's a good chance it can be migrated back again if CMA pageblocks need to be used by CMA allocation - it's more complex, but I guess we have most of the necessary infrastructure in compaction already :) Thoughts? Vlastimil Thanks, Laura -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On Thu, Oct 16, 2014 at 11:35:47AM +0800, Hui Zhu wrote: > In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of > MIGRATE_MOVABLE. > MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in > order that Linux kernel want. > > If a system that has a lot of user space program is running, for > instance, an Android board, most of memory is in MIGRATE_MOVABLE and > allocated. Before function __rmqueue_fallback get memory from > MIGRATE_CMA, the oom_killer will kill a task to release memory when > kernel want get MIGRATE_UNMOVABLE memory because fallbacks of > MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. > This status is odd. The MIGRATE_CMA has a lot free memory but Linux > kernel kill some tasks to release memory. > > This patch series adds a new function CMA_AGGRESSIVE to make CMA memory > be more aggressive about allocation. > If function CMA_AGGRESSIVE is available, when Linux kernel call function > __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, > MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA > doesn't have enough pages for allocation, go back to allocate memory from > MIGRATE_MOVABLE. > Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and > MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. Hello, I did some work similar to this. Please reference following links. https://lkml.org/lkml/2014/5/28/64 https://lkml.org/lkml/2014/5/28/57 And, aggressive allocation should be postponed until freepage counting bug is fixed, because aggressive allocation enlarge the possiblity of problem occurence. I tried to fix that bug, too. See following link. https://lkml.org/lkml/2014/10/23/90 Thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On Thu, Oct 16, 2014 at 11:35:47AM +0800, Hui Zhu wrote: In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of MIGRATE_MOVABLE. MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in order that Linux kernel want. If a system that has a lot of user space program is running, for instance, an Android board, most of memory is in MIGRATE_MOVABLE and allocated. Before function __rmqueue_fallback get memory from MIGRATE_CMA, the oom_killer will kill a task to release memory when kernel want get MIGRATE_UNMOVABLE memory because fallbacks of MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. This status is odd. The MIGRATE_CMA has a lot free memory but Linux kernel kill some tasks to release memory. This patch series adds a new function CMA_AGGRESSIVE to make CMA memory be more aggressive about allocation. If function CMA_AGGRESSIVE is available, when Linux kernel call function __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA doesn't have enough pages for allocation, go back to allocate memory from MIGRATE_MOVABLE. Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. Hello, I did some work similar to this. Please reference following links. https://lkml.org/lkml/2014/5/28/64 https://lkml.org/lkml/2014/5/28/57 And, aggressive allocation should be postponed until freepage counting bug is fixed, because aggressive allocation enlarge the possiblity of problem occurence. I tried to fix that bug, too. See following link. https://lkml.org/lkml/2014/10/23/90 Thanks. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On 10/22/14 20:02, Peter Hurley wrote: > On 10/16/2014 04:55 AM, Laura Abbott wrote: >> On 10/15/2014 8:35 PM, Hui Zhu wrote: >>> In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of >>> MIGRATE_MOVABLE. >>> MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in >>> order that Linux kernel want. >>> >>> If a system that has a lot of user space program is running, for >>> instance, an Android board, most of memory is in MIGRATE_MOVABLE and >>> allocated. Before function __rmqueue_fallback get memory from >>> MIGRATE_CMA, the oom_killer will kill a task to release memory when >>> kernel want get MIGRATE_UNMOVABLE memory because fallbacks of >>> MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. >>> This status is odd. The MIGRATE_CMA has a lot free memory but Linux >>> kernel kill some tasks to release memory. >>> >>> This patch series adds a new function CMA_AGGRESSIVE to make CMA memory >>> be more aggressive about allocation. >>> If function CMA_AGGRESSIVE is available, when Linux kernel call function >>> __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, >>> MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA >>> doesn't have enough pages for allocation, go back to allocate memory from >>> MIGRATE_MOVABLE. >>> Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and >>> MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. >>> >> >> It's good to see another proposal to fix CMA utilization. Do you have >> any data about the success rate of CMA contiguous allocation after >> this patch series? I played around with a similar approach of using >> CMA for MIGRATE_MOVABLE allocations and found that although utilization >> did increase, contiguous allocations failed at a higher rate and were >> much slower. I see what this series is trying to do with avoiding >> allocation from CMA pages when a contiguous allocation is progress. >> My concern is that there would still be problems with contiguous >> allocation after all the MIGRATE_MOVABLE fallback has happened. > > What impact does this series have on x86 platforms now that CMA is the > backup allocator for all iommu dma allocations? They will not affect driver CMA memory allocation. Thanks, Hui > > Regards, > Peter Hurley >
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On 10/16/2014 04:55 AM, Laura Abbott wrote: > On 10/15/2014 8:35 PM, Hui Zhu wrote: >> In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of >> MIGRATE_MOVABLE. >> MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in >> order that Linux kernel want. >> >> If a system that has a lot of user space program is running, for >> instance, an Android board, most of memory is in MIGRATE_MOVABLE and >> allocated. Before function __rmqueue_fallback get memory from >> MIGRATE_CMA, the oom_killer will kill a task to release memory when >> kernel want get MIGRATE_UNMOVABLE memory because fallbacks of >> MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. >> This status is odd. The MIGRATE_CMA has a lot free memory but Linux >> kernel kill some tasks to release memory. >> >> This patch series adds a new function CMA_AGGRESSIVE to make CMA memory >> be more aggressive about allocation. >> If function CMA_AGGRESSIVE is available, when Linux kernel call function >> __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, >> MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA >> doesn't have enough pages for allocation, go back to allocate memory from >> MIGRATE_MOVABLE. >> Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and >> MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. >> > > It's good to see another proposal to fix CMA utilization. Do you have > any data about the success rate of CMA contiguous allocation after > this patch series? I played around with a similar approach of using > CMA for MIGRATE_MOVABLE allocations and found that although utilization > did increase, contiguous allocations failed at a higher rate and were > much slower. I see what this series is trying to do with avoiding > allocation from CMA pages when a contiguous allocation is progress. > My concern is that there would still be problems with contiguous > allocation after all the MIGRATE_MOVABLE fallback has happened. What impact does this series have on x86 platforms now that CMA is the backup allocator for all iommu dma allocations? Regards, Peter Hurley -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On 10/22/14 20:02, Peter Hurley wrote: On 10/16/2014 04:55 AM, Laura Abbott wrote: On 10/15/2014 8:35 PM, Hui Zhu wrote: In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of MIGRATE_MOVABLE. MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in order that Linux kernel want. If a system that has a lot of user space program is running, for instance, an Android board, most of memory is in MIGRATE_MOVABLE and allocated. Before function __rmqueue_fallback get memory from MIGRATE_CMA, the oom_killer will kill a task to release memory when kernel want get MIGRATE_UNMOVABLE memory because fallbacks of MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. This status is odd. The MIGRATE_CMA has a lot free memory but Linux kernel kill some tasks to release memory. This patch series adds a new function CMA_AGGRESSIVE to make CMA memory be more aggressive about allocation. If function CMA_AGGRESSIVE is available, when Linux kernel call function __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA doesn't have enough pages for allocation, go back to allocate memory from MIGRATE_MOVABLE. Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. It's good to see another proposal to fix CMA utilization. Do you have any data about the success rate of CMA contiguous allocation after this patch series? I played around with a similar approach of using CMA for MIGRATE_MOVABLE allocations and found that although utilization did increase, contiguous allocations failed at a higher rate and were much slower. I see what this series is trying to do with avoiding allocation from CMA pages when a contiguous allocation is progress. My concern is that there would still be problems with contiguous allocation after all the MIGRATE_MOVABLE fallback has happened. What impact does this series have on x86 platforms now that CMA is the backup allocator for all iommu dma allocations? They will not affect driver CMA memory allocation. Thanks, Hui Regards, Peter Hurley
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On 10/16/2014 04:55 AM, Laura Abbott wrote: On 10/15/2014 8:35 PM, Hui Zhu wrote: In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of MIGRATE_MOVABLE. MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in order that Linux kernel want. If a system that has a lot of user space program is running, for instance, an Android board, most of memory is in MIGRATE_MOVABLE and allocated. Before function __rmqueue_fallback get memory from MIGRATE_CMA, the oom_killer will kill a task to release memory when kernel want get MIGRATE_UNMOVABLE memory because fallbacks of MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. This status is odd. The MIGRATE_CMA has a lot free memory but Linux kernel kill some tasks to release memory. This patch series adds a new function CMA_AGGRESSIVE to make CMA memory be more aggressive about allocation. If function CMA_AGGRESSIVE is available, when Linux kernel call function __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA doesn't have enough pages for allocation, go back to allocate memory from MIGRATE_MOVABLE. Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. It's good to see another proposal to fix CMA utilization. Do you have any data about the success rate of CMA contiguous allocation after this patch series? I played around with a similar approach of using CMA for MIGRATE_MOVABLE allocations and found that although utilization did increase, contiguous allocations failed at a higher rate and were much slower. I see what this series is trying to do with avoiding allocation from CMA pages when a contiguous allocation is progress. My concern is that there would still be problems with contiguous allocation after all the MIGRATE_MOVABLE fallback has happened. What impact does this series have on x86 platforms now that CMA is the backup allocator for all iommu dma allocations? Regards, Peter Hurley -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On 10/16/14 16:56, Laura Abbott wrote: > On 10/15/2014 8:35 PM, Hui Zhu wrote: >> In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of >> MIGRATE_MOVABLE. >> MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in >> order that Linux kernel want. >> >> If a system that has a lot of user space program is running, for >> instance, an Android board, most of memory is in MIGRATE_MOVABLE and >> allocated. Before function __rmqueue_fallback get memory from >> MIGRATE_CMA, the oom_killer will kill a task to release memory when >> kernel want get MIGRATE_UNMOVABLE memory because fallbacks of >> MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. >> This status is odd. The MIGRATE_CMA has a lot free memory but Linux >> kernel kill some tasks to release memory. >> >> This patch series adds a new function CMA_AGGRESSIVE to make CMA memory >> be more aggressive about allocation. >> If function CMA_AGGRESSIVE is available, when Linux kernel call function >> __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, >> MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA >> doesn't have enough pages for allocation, go back to allocate memory from >> MIGRATE_MOVABLE. >> Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and >> MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. >> > > It's good to see another proposal to fix CMA utilization. Thanks Laura. Do you have > any data about the success rate of CMA contiguous allocation after > this patch series? I played around with a similar approach of using > CMA for MIGRATE_MOVABLE allocations and found that although utilization > did increase, contiguous allocations failed at a higher rate and were > much slower. I see what this series is trying to do with avoiding > allocation from CMA pages when a contiguous allocation is progress. > My concern is that there would still be problems with contiguous > allocation after all the MIGRATE_MOVABLE fallback has happened. I did some test with the cma_alloc_counter and cma-aggressive-shrink in a android board that has 1g memory. Run some apps to make free CMA close to the value of cma_aggressive_free_min(500 pages). A driver Begin to request CMA more than 10 times. Each time, it will request more than 3000 pages. I don't have established number for that because it is really hard to get a fail. I think the success rate is over 95% at least. And I think maybe the isolate fail has relation with page alloc and free code. Maybe let zone->lock protect more code can handle this issue. Thanks, Hui > > Thanks, > Laura >
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On 10/16/14 16:56, Laura Abbott wrote: On 10/15/2014 8:35 PM, Hui Zhu wrote: In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of MIGRATE_MOVABLE. MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in order that Linux kernel want. If a system that has a lot of user space program is running, for instance, an Android board, most of memory is in MIGRATE_MOVABLE and allocated. Before function __rmqueue_fallback get memory from MIGRATE_CMA, the oom_killer will kill a task to release memory when kernel want get MIGRATE_UNMOVABLE memory because fallbacks of MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. This status is odd. The MIGRATE_CMA has a lot free memory but Linux kernel kill some tasks to release memory. This patch series adds a new function CMA_AGGRESSIVE to make CMA memory be more aggressive about allocation. If function CMA_AGGRESSIVE is available, when Linux kernel call function __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA doesn't have enough pages for allocation, go back to allocate memory from MIGRATE_MOVABLE. Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. It's good to see another proposal to fix CMA utilization. Thanks Laura. Do you have any data about the success rate of CMA contiguous allocation after this patch series? I played around with a similar approach of using CMA for MIGRATE_MOVABLE allocations and found that although utilization did increase, contiguous allocations failed at a higher rate and were much slower. I see what this series is trying to do with avoiding allocation from CMA pages when a contiguous allocation is progress. My concern is that there would still be problems with contiguous allocation after all the MIGRATE_MOVABLE fallback has happened. I did some test with the cma_alloc_counter and cma-aggressive-shrink in a android board that has 1g memory. Run some apps to make free CMA close to the value of cma_aggressive_free_min(500 pages). A driver Begin to request CMA more than 10 times. Each time, it will request more than 3000 pages. I don't have established number for that because it is really hard to get a fail. I think the success rate is over 95% at least. And I think maybe the isolate fail has relation with page alloc and free code. Maybe let zone-lock protect more code can handle this issue. Thanks, Hui Thanks, Laura
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On 10/15/2014 8:35 PM, Hui Zhu wrote: In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of MIGRATE_MOVABLE. MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in order that Linux kernel want. If a system that has a lot of user space program is running, for instance, an Android board, most of memory is in MIGRATE_MOVABLE and allocated. Before function __rmqueue_fallback get memory from MIGRATE_CMA, the oom_killer will kill a task to release memory when kernel want get MIGRATE_UNMOVABLE memory because fallbacks of MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. This status is odd. The MIGRATE_CMA has a lot free memory but Linux kernel kill some tasks to release memory. This patch series adds a new function CMA_AGGRESSIVE to make CMA memory be more aggressive about allocation. If function CMA_AGGRESSIVE is available, when Linux kernel call function __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA doesn't have enough pages for allocation, go back to allocate memory from MIGRATE_MOVABLE. Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. It's good to see another proposal to fix CMA utilization. Do you have any data about the success rate of CMA contiguous allocation after this patch series? I played around with a similar approach of using CMA for MIGRATE_MOVABLE allocations and found that although utilization did increase, contiguous allocations failed at a higher rate and were much slower. I see what this series is trying to do with avoiding allocation from CMA pages when a contiguous allocation is progress. My concern is that there would still be problems with contiguous allocation after all the MIGRATE_MOVABLE fallback has happened. Thanks, Laura -- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On 10/15/2014 8:35 PM, Hui Zhu wrote: In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of MIGRATE_MOVABLE. MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in order that Linux kernel want. If a system that has a lot of user space program is running, for instance, an Android board, most of memory is in MIGRATE_MOVABLE and allocated. Before function __rmqueue_fallback get memory from MIGRATE_CMA, the oom_killer will kill a task to release memory when kernel want get MIGRATE_UNMOVABLE memory because fallbacks of MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. This status is odd. The MIGRATE_CMA has a lot free memory but Linux kernel kill some tasks to release memory. This patch series adds a new function CMA_AGGRESSIVE to make CMA memory be more aggressive about allocation. If function CMA_AGGRESSIVE is available, when Linux kernel call function __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA doesn't have enough pages for allocation, go back to allocate memory from MIGRATE_MOVABLE. Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. It's good to see another proposal to fix CMA utilization. Do you have any data about the success rate of CMA contiguous allocation after this patch series? I played around with a similar approach of using CMA for MIGRATE_MOVABLE allocations and found that although utilization did increase, contiguous allocations failed at a higher rate and were much slower. I see what this series is trying to do with avoiding allocation from CMA pages when a contiguous allocation is progress. My concern is that there would still be problems with contiguous allocation after all the MIGRATE_MOVABLE fallback has happened. Thanks, Laura -- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On Thu, Oct 16, 2014 at 11:35 AM, Hui Zhu wrote: > In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of > MIGRATE_MOVABLE. > MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in > order that Linux kernel want. > > If a system that has a lot of user space program is running, for > instance, an Android board, most of memory is in MIGRATE_MOVABLE and > allocated. Before function __rmqueue_fallback get memory from > MIGRATE_CMA, the oom_killer will kill a task to release memory when > kernel want get MIGRATE_UNMOVABLE memory because fallbacks of > MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. > This status is odd. The MIGRATE_CMA has a lot free memory but Linux > kernel kill some tasks to release memory. I'm not very clear to this description, what issue do you try to solve? Make MIGRATE_CMA be the fallback of desired MIGRATE_UNMOVABLE? > This patch series adds a new function CMA_AGGRESSIVE to make CMA memory > be more aggressive about allocation. > If function CMA_AGGRESSIVE is available, when Linux kernel call function > __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, > MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA > doesn't have enough pages for allocation, go back to allocate memory from > MIGRATE_MOVABLE. I don't think so. That will cause MIGRATE_CMA depleted prematurely, and when a user(such as camera) wants CMA memory, he will not get the wanted memory. > Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and > MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. I don't think this is the root cause of oom. But I am interested in the CMA shrinker idea, I will follow this mail. Thanks for your work, add some test data will be better. > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of MIGRATE_MOVABLE. MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in order that Linux kernel want. If a system that has a lot of user space program is running, for instance, an Android board, most of memory is in MIGRATE_MOVABLE and allocated. Before function __rmqueue_fallback get memory from MIGRATE_CMA, the oom_killer will kill a task to release memory when kernel want get MIGRATE_UNMOVABLE memory because fallbacks of MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. This status is odd. The MIGRATE_CMA has a lot free memory but Linux kernel kill some tasks to release memory. This patch series adds a new function CMA_AGGRESSIVE to make CMA memory be more aggressive about allocation. If function CMA_AGGRESSIVE is available, when Linux kernel call function __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA doesn't have enough pages for allocation, go back to allocate memory from MIGRATE_MOVABLE. Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of MIGRATE_MOVABLE. MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in order that Linux kernel want. If a system that has a lot of user space program is running, for instance, an Android board, most of memory is in MIGRATE_MOVABLE and allocated. Before function __rmqueue_fallback get memory from MIGRATE_CMA, the oom_killer will kill a task to release memory when kernel want get MIGRATE_UNMOVABLE memory because fallbacks of MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. This status is odd. The MIGRATE_CMA has a lot free memory but Linux kernel kill some tasks to release memory. This patch series adds a new function CMA_AGGRESSIVE to make CMA memory be more aggressive about allocation. If function CMA_AGGRESSIVE is available, when Linux kernel call function __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA doesn't have enough pages for allocation, go back to allocate memory from MIGRATE_MOVABLE. Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] (CMA_AGGRESSIVE) Make CMA memory be more aggressive about allocation
On Thu, Oct 16, 2014 at 11:35 AM, Hui Zhu zhu...@xiaomi.com wrote: In fallbacks of page_alloc.c, MIGRATE_CMA is the fallback of MIGRATE_MOVABLE. MIGRATE_MOVABLE will use MIGRATE_CMA when it doesn't have a page in order that Linux kernel want. If a system that has a lot of user space program is running, for instance, an Android board, most of memory is in MIGRATE_MOVABLE and allocated. Before function __rmqueue_fallback get memory from MIGRATE_CMA, the oom_killer will kill a task to release memory when kernel want get MIGRATE_UNMOVABLE memory because fallbacks of MIGRATE_UNMOVABLE are MIGRATE_RECLAIMABLE and MIGRATE_MOVABLE. This status is odd. The MIGRATE_CMA has a lot free memory but Linux kernel kill some tasks to release memory. I'm not very clear to this description, what issue do you try to solve? Make MIGRATE_CMA be the fallback of desired MIGRATE_UNMOVABLE? This patch series adds a new function CMA_AGGRESSIVE to make CMA memory be more aggressive about allocation. If function CMA_AGGRESSIVE is available, when Linux kernel call function __rmqueue try to get pages from MIGRATE_MOVABLE and conditions allow, MIGRATE_CMA will be allocated as MIGRATE_MOVABLE first. If MIGRATE_CMA doesn't have enough pages for allocation, go back to allocate memory from MIGRATE_MOVABLE. I don't think so. That will cause MIGRATE_CMA depleted prematurely, and when a user(such as camera) wants CMA memory, he will not get the wanted memory. Then the memory of MIGRATE_MOVABLE can be kept for MIGRATE_UNMOVABLE and MIGRATE_RECLAIMABLE which doesn't have fallback MIGRATE_CMA. I don't think this is the root cause of oom. But I am interested in the CMA shrinker idea, I will follow this mail. Thanks for your work, add some test data will be better. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
