RE: [PATCHv19 00/15] Contiguous Memory Allocator
Hi Andrew, On Saturday, January 28, 2012 1:26 AM Andrew Morton wrote: These patches don't seem to have as many acked-bys and reviewed-bys as I'd expect. Given the scope and duration of this, it would be useful to gather these up. But please ensure they are real ones - people sometimes like to ack things without showing much sign of having actually read them. Also there is the supreme tag: Tested-by:.. Ohad (at least) has been testing the code. Let's mention that. The patches do seem to have been going round in ever-decreasing circles lately and I think we have decided to merge them (yes?) so we may as well get on and do that and sort out remaining issues in-tree. It looks that the CMA patch series reached the final version - I've just posted version 21 a few minutes ago. Most of the patches got acks from either Mel or Arnd and the remaining few needs only minor tweaking, but they affect only CMA users, which we hope to fix once the series is merged. That's why I would like to ask You to merge these patches to Your tree and finally give them a try in linux-next kernel. Best regards -- Marek Szyprowski Samsung Poland RD Center -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCHv19 00/15] Contiguous Memory Allocator
Hello, On Tuesday, January 31, 2012 6:17 PM Benjamin Gaignard wrote: I have rebase Linaro CMA test driver to be compatible with CMA v19, it now use dma-mapping API instead of v17 CMA API. A kernel for snowball with CMA v19 and test driver is available here: http://git.linaro.org/gitweb?p=people/bgaignard/linux-snowball-test-cma-v19.git;a=summary From this kernel build, I have execute CMA lava (the linaro automatic test tool) test, the same than we are running since v16, the test is OK. With previous versions of CMA some the test has found issues when the memory was filled with reclaimables pages, but with v19 this issue is no more present. Test logs are here: https://validation.linaro.org/lava-server/scheduler/job/10841 so you can add: Tested-by: Benjamin Gaignard benjamin.gaign...@linaro.org Thanks for Your contribution! Best regards -- Marek Szyprowski Samsung Poland RD Center -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv19 00/15] Contiguous Memory Allocator
On Fri, Jan 27, 2012 at 04:26:24PM -0800, Andrew Morton wrote: On Thu, 26 Jan 2012 15:31:40 + Arnd Bergmann a...@arndb.de wrote: On Thursday 26 January 2012, Marek Szyprowski wrote: Welcome everyone! Yes, that's true. This is yet another release of the Contiguous Memory Allocator patches. This version mainly includes code cleanups requested by Mel Gorman and a few minor bug fixes. Hi Marek, Thanks for keeping up this work! I really hope it works out for the next merge window. Someone please tell me when it's time to start paying attention again ;) These patches don't seem to have as many acked-bys and reviewed-bys as I'd expect. I reviewed the core MM changes and I've acked most of them so the next release should have a few acks where you expect them. I did not add a reviewed-by because I did not build and test the thing. For me, Patch 2 is the only one that must be fixed prior to merging as it can interfere with pages on a remote per-cpu list which is dangerous. I know your suggestion will be to delete the per-cpu lists and be done with it but I am a bit away from doing that just yet. Patch 8 could do with a bit more care too but it is not a potential hand grenade like patch 2 and could be fixed as part of a follow-up. Even if you don't see an ack from me there, it should not be treated as a show stopper. I highlighted some issues on how CMA interacts with reclaim but I think this is a problem specific to CMA and should not prevent it being merged. I just wanted to be sure that the CMA people were aware of the potential issues so they will recognise the class of bug if it occurs. Given the scope and duration of this, it would be useful to gather these up. But please ensure they are real ones - people sometimes like to ack things without showing much sign of having actually read them. FWIW, the acks I put on the core MM changes are real acks :) The patches do seem to have been going round in ever-decreasing circles lately and I think we have decided to merge them (yes?) so we may as well get on and do that and sort out remaining issues in-tree. I'm a lot happier with the core MM patches than I was when I reviewed this first around last September or October. -- Mel Gorman SUSE Labs -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv19 00/15] Contiguous Memory Allocator
On Mon, 30 Jan 2012 14:25:12 +0100, Mel Gorman m...@csn.ul.ie wrote: I reviewed the core MM changes and I've acked most of them so the next release should have a few acks where you expect them. I did not add a reviewed-by because I did not build and test the thing. Thanks! I've either replied to your comments or applied suggested changes. If anyone cares, not-tested changes are available at git://github.com/mina86/linux-2.6.git cma -- Best regards, _ _ .o. | Liege of Serenely Enlightened Majesty of o' \,=./ `o ..o | Computer Science, Michał “mina86” Nazarewicz(o o) ooo +email/xmpp: m...@google.com--ooO--(_)--Ooo-- -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv19 00/15] Contiguous Memory Allocator
On Fri, Jan 27, 2012 at 6:26 PM, Andrew Morton a...@linux-foundation.org wrote: On Thu, 26 Jan 2012 15:31:40 + Arnd Bergmann a...@arndb.de wrote: On Thursday 26 January 2012, Marek Szyprowski wrote: Welcome everyone! Yes, that's true. This is yet another release of the Contiguous Memory Allocator patches. This version mainly includes code cleanups requested by Mel Gorman and a few minor bug fixes. Hi Marek, Thanks for keeping up this work! I really hope it works out for the next merge window. Someone please tell me when it's time to start paying attention again ;) These patches don't seem to have as many acked-bys and reviewed-bys as I'd expect. Given the scope and duration of this, it would be useful to gather these up. But please ensure they are real ones - people sometimes like to ack things without showing much sign of having actually read them. Also there is the supreme tag: Tested-by:.. Ohad (at least) has been testing the code. Let's mention that. fyi Marek, I've been testing CMA as well, both in context of Ohad's rpmsg driver and my omapdrm driver (and combination of the two).. so you can add: Tested-by: Rob Clark rob.cl...@linaro.org And there are some others from linaro that have written a test driver, and various stress test scripts using the test driver. I guess that could also count for some additional Tested-by's. BR, -R The patches do seem to have been going round in ever-decreasing circles lately and I think we have decided to merge them (yes?) so we may as well get on and do that and sort out remaining issues in-tree. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv19 00/15] Contiguous Memory Allocator
Also there is the supreme tag: Tested-by:.. Ohad (at least) has been testing the code. Let's mention that. fyi Marek, I've been testing CMA as well, both in context of Ohad's rpmsg driver and my omapdrm driver (and combination of the two).. so you can add: Tested-by: Rob Clark rob.cl...@linaro.org And there are some others from linaro that have written a test driver, and various stress test scripts using the test driver. I guess that could also count for some additional Tested-by's. Convince them to report with Tested-by tag. This is a first step for them to face the open source. -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv19 00/15] Contiguous Memory Allocator
On Saturday 28 January 2012, Andrew Morton wrote: These patches don't seem to have as many acked-bys and reviewed-bys as I'd expect. Given the scope and duration of this, it would be useful to gather these up. But please ensure they are real ones - people sometimes like to ack things without showing much sign of having actually read them. I reviewed early versions of this patch set and had a lot of comments on the interfaces that were exposed to device drivers and platform maintainers. All of the comments were addressed back then and I gave an Acked-by. I assume that it was dropped in subsequent versions because the implementation changed significantly since, but I'm still happy with the way this looks to the user, in particular that it is practically invisible because all users just go through the dma mapping API instead of the horrors that were used in the original patches. From an ARM architecture perspective, we have come to the point (some versions ago) where we actually require the CMA patchset for correctness, even on IOMMU based systems because it avoids some nasty corner cases with pages that are both in the linear kernel mapping and in an uncached mapping for DMA: We know that the code we are using in mainline is broken on ARMv6 and later and that CMA fixes that problem. I'm not the right person to judge the memory management code changes, others need to comment on that. Aside from that: Acked-by: Arnd Bergmann a...@arndb.de Arnd -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv19 00/15] Contiguous Memory Allocator
On Thu, 26 Jan 2012 15:31:40 + Arnd Bergmann a...@arndb.de wrote: On Thursday 26 January 2012, Marek Szyprowski wrote: Welcome everyone! Yes, that's true. This is yet another release of the Contiguous Memory Allocator patches. This version mainly includes code cleanups requested by Mel Gorman and a few minor bug fixes. Hi Marek, Thanks for keeping up this work! I really hope it works out for the next merge window. Someone please tell me when it's time to start paying attention again ;) These patches don't seem to have as many acked-bys and reviewed-bys as I'd expect. Given the scope and duration of this, it would be useful to gather these up. But please ensure they are real ones - people sometimes like to ack things without showing much sign of having actually read them. Also there is the supreme tag: Tested-by:.. Ohad (at least) has been testing the code. Let's mention that. The patches do seem to have been going round in ever-decreasing circles lately and I think we have decided to merge them (yes?) so we may as well get on and do that and sort out remaining issues in-tree. -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCHv19 00/15] Contiguous Memory Allocator
Welcome everyone! Yes, that's true. This is yet another release of the Contiguous Memory Allocator patches. This version mainly includes code cleanups requested by Mel Gorman and a few minor bug fixes. ARM integration code has not been changed since v16. It provides implementation of the ideas that has been discussed during Linaro Sprint meeting in Cambourne, August 2011. Here are the details: This version provides a solution for complete integration of CMA to DMA mapping subsystem on ARM architecture. The issue caused by double dma pages mapping and possible aliasing in coherent memory mapping has been finally resolved, both for GFP_ATOMIC case (allocations comes from coherent memory pool) and non-GFP_ATOMIC case (allocations comes from CMA managed areas). For coherent, nommu, ARMv4 and ARMv5 systems the current DMA-mapping implementation has been kept. For ARMv6+ systems, CMA has been enabled and a special pool of coherent memory for atomic allocations has been created. The size of this pool defaults to DEFAULT_CONSISTEN_DMA_SIZE/8, but can be changed with coherent_pool kernel parameter (if really required). All atomic allocations are served from this pool. I've did a little simplification here, because there is no separate pool for writecombine memory - such requests are also served from coherent pool. I don't think that such simplification is a problem here - I found no driver that use dma_alloc_writecombine with GFP_ATOMIC flags. All non-atomic allocation are served from CMA area. Kernel mappings are updated to reflect required memory attributes changes. This is possible because during early boot, all CMA area are remapped with 4KiB pages in kernel low-memory. This version have been tested on Samsung S5PC110 based Goni machine and Exynos4 UniversalC210 board with various V4L2 multimedia drivers. Coherent atomic allocations has been tested by manually enabling the dma bounce for the s3c-sdhci device. All patches are prepared for Linux Kernel v3.3-rc1. A few words for these who see CMA for the first time: The Contiguous Memory Allocator (CMA) makes it possible for device drivers to allocate big contiguous chunks of memory after the system has booted. The main difference from the similar frameworks is the fact that CMA allows to transparently reuse memory region reserved for the big chunk allocation as a system memory, so no memory is wasted when no big chunk is allocated. Once the alloc request is issued, the framework will migrate system pages to create a required big chunk of physically contiguous memory. For more information you can refer to nice LWN articles: http://lwn.net/Articles/447405/ and http://lwn.net/Articles/450286/ as well as links to previous versions of the CMA framework. The CMA framework has been initially developed by Michal Nazarewicz at Samsung Poland RD Center. Since version 9, I've taken over the development, because Michal has left the company. Since version v17 Michal is working again on CMA patches and the current version is the result of our joint open-source effort. TODO (optional): - implement support for contiguous memory areas placed in HIGHMEM zone - resolve issue with movable pages with pending io operations Best regards Marek Szyprowski Samsung Poland RD Center Links to previous versions of the patchset: v18: http://www.spinics.net/lists/linux-mm/msg28125.html v17: http://www.spinics.net/lists/arm-kernel/msg148499.html v16: http://www.spinics.net/lists/linux-mm/msg25066.html v15: http://www.spinics.net/lists/linux-mm/msg23365.html v14: http://www.spinics.net/lists/linux-media/msg36536.html v13: (internal, intentionally not released) v12: http://www.spinics.net/lists/linux-media/msg35674.html v11: http://www.spinics.net/lists/linux-mm/msg21868.html v10: http://www.spinics.net/lists/linux-mm/msg20761.html v9: http://article.gmane.org/gmane.linux.kernel.mm/60787 v8: http://article.gmane.org/gmane.linux.kernel.mm/56855 v7: http://article.gmane.org/gmane.linux.kernel.mm/55626 v6: http://article.gmane.org/gmane.linux.kernel.mm/55626 v5: (intentionally left out as CMA v5 was identical to CMA v4) v4: http://article.gmane.org/gmane.linux.kernel.mm/52010 v3: http://article.gmane.org/gmane.linux.kernel.mm/51573 v2: http://article.gmane.org/gmane.linux.kernel.mm/50986 v1: http://article.gmane.org/gmane.linux.kernel.mm/50669 Changelog: v19: 1. Addressed another set of comments and suggestions from Mel Gorman, mainly related to breaking patches into smaller, single-feature related chunks and rewriting already existing functions in memory compaction code. 2. Reworked completely page reclaim code, removed it from split_free_page() and introduce direct call from alloc_contig_range(). 3. Merged a fix from Mans Rullgard for correct cma area limit alignment. 4. Replaced broken mm: page_alloc: set_migratetype_isolate:
Re: [PATCHv19 00/15] Contiguous Memory Allocator
On Thursday 26 January 2012, Marek Szyprowski wrote: Welcome everyone! Yes, that's true. This is yet another release of the Contiguous Memory Allocator patches. This version mainly includes code cleanups requested by Mel Gorman and a few minor bug fixes. Hi Marek, Thanks for keeping up this work! I really hope it works out for the next merge window. TODO (optional): - implement support for contiguous memory areas placed in HIGHMEM zone - resolve issue with movable pages with pending io operations Can you clarify these? I believe the contiguous memory areas in highmem is something that should really be after the existing code is merged into the upstream kernel and should better not be listed as TODO here. I haven't followed the last two releases so closely. It seems that in v17 the movable pages with pending i/o was still a major problem but in v18 you added a solution. Is that right? What is still left to be done here then? Arnd -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv19 00/15] Contiguous Memory Allocator
On Thu, 26 Jan 2012 16:31:40 +0100, Arnd Bergmann a...@arndb.de wrote: I haven't followed the last two releases so closely. It seems that in v17 the movable pages with pending i/o was still a major problem but in v18 you added a solution. Is that right? What is still left to be done here then? In the current version, when allocation fails because of a page with pending I/O, CMA automatically tries allocation in another region. -- Best regards, _ _ .o. | Liege of Serenely Enlightened Majesty of o' \,=./ `o ..o | Computer Science, Michał “mina86” Nazarewicz(o o) ooo +email/xmpp: m...@google.com--ooO--(_)--Ooo-- -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCHv19 00/15] Contiguous Memory Allocator
Hello, On Thursday, January 26, 2012 4:32 PM Arnd Bergmann wrote: On Thursday 26 January 2012, Marek Szyprowski wrote: Welcome everyone! Yes, that's true. This is yet another release of the Contiguous Memory Allocator patches. This version mainly includes code cleanups requested by Mel Gorman and a few minor bug fixes. Hi Marek, Thanks for keeping up this work! I really hope it works out for the next merge window. TODO (optional): - implement support for contiguous memory areas placed in HIGHMEM zone - resolve issue with movable pages with pending io operations Can you clarify these? I believe the contiguous memory areas in highmem is something that should really be after the existing code is merged into the upstream kernel and should better not be listed as TODO here. Ok, I will remove it from the TODO list. Core memory management is very little dependence on HIGHMEM, it is more about DMA-mapping framework to be aware that there might be no lowmem mappings for the allocated pages. This can be easily added once the initial version got merged. I haven't followed the last two releases so closely. It seems that in v17 the movable pages with pending i/o was still a major problem but in v18 you added a solution. Is that right? What is still left to be done here then? Since v18 the failed allocation is retried in a bit different place in the contiguous memory area what heavily increased overall reliability. This can be improved by making cma a bit more aware about pending io operations, but I want to leave this after the initial merge. I think that there are no major issues left to be resolved now. Best regards -- Marek Szyprowski Samsung Poland RD Center -- To unsubscribe from this list: send the line unsubscribe linux-media in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
