Re: [PATCH v7] RISC-V: enable XIP
On Mon, Apr 12, 2021 at 7:12 AM Alex Ghiti wrote: > > Le 4/9/21 à 10:42 AM, Vitaly Wool a écrit : > > On Fri, Apr 9, 2021 at 3:59 PM Mike Rapoport wrote: > >> > >> On Fri, Apr 09, 2021 at 02:46:17PM +0200, David Hildenbrand wrote: > >> Also, will that memory properly be exposed in the resource tree as > >> System RAM (e.g., /proc/iomem) ? Otherwise some things (/proc/kcore) > >> won't work as expected - the kernel won't be included in a dump. > Do we really need a XIP kernel to included in kdump? > And does not it sound weird to expose flash as System RAM in > /proc/iomem? ;-) > >>> > >>> See my other mail, maybe we actually want something different. > >>> > > > I have just checked and it does not appear in /proc/iomem. > > > > Ok your conclusion would be to have struct page, I'm going to implement > > this > > version then using memblock as you described. > > I'm not sure this is required. With XIP kernel text never gets into RAM, > so > it does not seem to require struct page. > > XIP by definition has some limitations relatively to "normal" operation, > so lack of kdump could be one of them. > >>> > >>> I agree. > >>> > > I might be wrong, but IMHO, artificially creating a memory map for part > of > flash would cause more problems in the long run. > >>> > >>> Can you elaborate? > >> > >> Nothing particular, just a gut feeling. Usually, when you force something > >> it comes out the wrong way later. > > > > It's possible still that MTD_XIP is implemented allowing to write to > > the flash used for XIP. While flash is being written, memory map > > doesn't make sense at all. I can't come up with a real life example > > when it can actually lead to problems but it is indeed weird when > > System RAM suddenly becomes unreadable. I really don't think exposing > > it in /proc/iomem is a good idea. > > > BTW, how does XIP account the kernel text on other architectures that > implement it? > >>> > >>> Interesting point, I thought XIP would be something new on RISC-V (well, > >>> at > >>> least to me :) ). If that concept exists already, we better mimic what > >>> existing implementations do. > >> > >> I had quick glance at ARM, it seems that kernel text does not have memory > >> map and does not show up in System RAM. > > > > Exactly, and I believe ARM64 won't do that too when it gets its own > > XIP support (which is underway). > > > > > memmap does not seem necessary and ARM/ARM64 do not use it. > > But if someone tries to get a struct page from a physical address that > lies in flash, as mentioned by David, that could lead to silent > corruptions if something exists at the address where the struct page > should be. And it is hard to know which features in the kernel depends > on that. > > Regarding SPARSEMEM, the vmemmap lies in its own region so that's > unlikely to happen, so we will catch those invalid accesses (and that's > what I observed on riscv). > > But for FLATMEM, memmap is in the linear mapping, then that could very > likely happen silently. > > Could a simple solution be to force SPARSEMEM for those XIP kernels ? > Then wrong things could happen, but we would see those and avoid > spending hours to debug :) > > I will at least send a v8 to remove the pfn_valid modifications for > FLATMEM that now returns true to pfn in flash. That sounds good to me. I am not very keen on spending 200K on struct pages for flash (we can think of this as of an option but I would definitely like to have the option to compile it out in the end), so let's remove pfn_valid and fix things that will eventually break, if some. Best regards, Vitaly
Re: [PATCH v7] RISC-V: enable XIP
Le 4/9/21 à 10:42 AM, Vitaly Wool a écrit : On Fri, Apr 9, 2021 at 3:59 PM Mike Rapoport wrote: On Fri, Apr 09, 2021 at 02:46:17PM +0200, David Hildenbrand wrote: Also, will that memory properly be exposed in the resource tree as System RAM (e.g., /proc/iomem) ? Otherwise some things (/proc/kcore) won't work as expected - the kernel won't be included in a dump. Do we really need a XIP kernel to included in kdump? And does not it sound weird to expose flash as System RAM in /proc/iomem? ;-) See my other mail, maybe we actually want something different. I have just checked and it does not appear in /proc/iomem. Ok your conclusion would be to have struct page, I'm going to implement this version then using memblock as you described. I'm not sure this is required. With XIP kernel text never gets into RAM, so it does not seem to require struct page. XIP by definition has some limitations relatively to "normal" operation, so lack of kdump could be one of them. I agree. I might be wrong, but IMHO, artificially creating a memory map for part of flash would cause more problems in the long run. Can you elaborate? Nothing particular, just a gut feeling. Usually, when you force something it comes out the wrong way later. It's possible still that MTD_XIP is implemented allowing to write to the flash used for XIP. While flash is being written, memory map doesn't make sense at all. I can't come up with a real life example when it can actually lead to problems but it is indeed weird when System RAM suddenly becomes unreadable. I really don't think exposing it in /proc/iomem is a good idea. BTW, how does XIP account the kernel text on other architectures that implement it? Interesting point, I thought XIP would be something new on RISC-V (well, at least to me :) ). If that concept exists already, we better mimic what existing implementations do. I had quick glance at ARM, it seems that kernel text does not have memory map and does not show up in System RAM. Exactly, and I believe ARM64 won't do that too when it gets its own XIP support (which is underway). memmap does not seem necessary and ARM/ARM64 do not use it. But if someone tries to get a struct page from a physical address that lies in flash, as mentioned by David, that could lead to silent corruptions if something exists at the address where the struct page should be. And it is hard to know which features in the kernel depends on that. Regarding SPARSEMEM, the vmemmap lies in its own region so that's unlikely to happen, so we will catch those invalid accesses (and that's what I observed on riscv). But for FLATMEM, memmap is in the linear mapping, then that could very likely happen silently. Could a simple solution be to force SPARSEMEM for those XIP kernels ? Then wrong things could happen, but we would see those and avoid spending hours to debug :) I will at least send a v8 to remove the pfn_valid modifications for FLATMEM that now returns true to pfn in flash. Thanks, Best regards, Vitaly ___ linux-riscv mailing list linux-ri...@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [PATCH v7] RISC-V: enable XIP
On Fri, Apr 9, 2021 at 3:59 PM Mike Rapoport wrote: > > On Fri, Apr 09, 2021 at 02:46:17PM +0200, David Hildenbrand wrote: > > > > > Also, will that memory properly be exposed in the resource tree as > > > > > System RAM (e.g., /proc/iomem) ? Otherwise some things (/proc/kcore) > > > > > won't work as expected - the kernel won't be included in a dump. > > > Do we really need a XIP kernel to included in kdump? > > > And does not it sound weird to expose flash as System RAM in /proc/iomem? > > > ;-) > > > > See my other mail, maybe we actually want something different. > > > > > > > > > I have just checked and it does not appear in /proc/iomem. > > > > > > > > Ok your conclusion would be to have struct page, I'm going to implement > > > > this > > > > version then using memblock as you described. > > > > > > I'm not sure this is required. With XIP kernel text never gets into RAM, > > > so > > > it does not seem to require struct page. > > > > > > XIP by definition has some limitations relatively to "normal" operation, > > > so lack of kdump could be one of them. > > > > I agree. > > > > > > > > I might be wrong, but IMHO, artificially creating a memory map for part of > > > flash would cause more problems in the long run. > > > > Can you elaborate? > > Nothing particular, just a gut feeling. Usually, when you force something > it comes out the wrong way later. It's possible still that MTD_XIP is implemented allowing to write to the flash used for XIP. While flash is being written, memory map doesn't make sense at all. I can't come up with a real life example when it can actually lead to problems but it is indeed weird when System RAM suddenly becomes unreadable. I really don't think exposing it in /proc/iomem is a good idea. > > > BTW, how does XIP account the kernel text on other architectures that > > > implement it? > > > > Interesting point, I thought XIP would be something new on RISC-V (well, at > > least to me :) ). If that concept exists already, we better mimic what > > existing implementations do. > > I had quick glance at ARM, it seems that kernel text does not have memory > map and does not show up in System RAM. Exactly, and I believe ARM64 won't do that too when it gets its own XIP support (which is underway). Best regards, Vitaly
Re: [PATCH v7] RISC-V: enable XIP
On 09.04.21 16:10, Mike Rapoport wrote: On Fri, Apr 09, 2021 at 02:07:24PM +0200, David Hildenbrand wrote: On 09.04.21 13:39, Alex Ghiti wrote: Hi David, I assume you still somehow create the direct mapping for the kernel, right? So it's really some memory region with a direct mapping but without a memmap (and right now, without a resource), correct? XIP kernel text is not a region in memory to begin with ;-) I think that's the part that confused me. I thought it would be mapped somehow into physical address space and would be addressed like other memory -- just that reads would be rewired to go to flash. It resides in a flash and it is executed directly from there without being relocated to RAM. That's why it does not need neither direct mapping, nor struct pages. Thanks for clarifying! :) -- Thanks, David / dhildenb
Re: [PATCH v7] RISC-V: enable XIP
On Fri, Apr 09, 2021 at 02:07:24PM +0200, David Hildenbrand wrote: > On 09.04.21 13:39, Alex Ghiti wrote: > > Hi David, > > I assume you still somehow create the direct mapping for the kernel, right? > So it's really some memory region with a direct mapping but without a memmap > (and right now, without a resource), correct? XIP kernel text is not a region in memory to begin with ;-) It resides in a flash and it is executed directly from there without being relocated to RAM. That's why it does not need neither direct mapping, nor struct pages. -- Sincerely yours, Mike.
Re: [PATCH v7] RISC-V: enable XIP
> Am 09.04.2021 um 15:59 schrieb Mike Rapoport : > > On Fri, Apr 09, 2021 at 02:46:17PM +0200, David Hildenbrand wrote: > Also, will that memory properly be exposed in the resource tree as > System RAM (e.g., /proc/iomem) ? Otherwise some things (/proc/kcore) > won't work as expected - the kernel won't be included in a dump. >>> Do we really need a XIP kernel to included in kdump? >>> And does not it sound weird to expose flash as System RAM in /proc/iomem? >>> ;-) >> >> See my other mail, maybe we actually want something different. >> >>> I have just checked and it does not appear in /proc/iomem. Ok your conclusion would be to have struct page, I'm going to implement this version then using memblock as you described. >>> >>> I'm not sure this is required. With XIP kernel text never gets into RAM, so >>> it does not seem to require struct page. >>> >>> XIP by definition has some limitations relatively to "normal" operation, >>> so lack of kdump could be one of them. >> >> I agree. >> >>> >>> I might be wrong, but IMHO, artificially creating a memory map for part of >>> flash would cause more problems in the long run. >> >> Can you elaborate? > > Nothing particular, just a gut feeling. Usually, when you force something > it comes out the wrong way later. > >>> >>> BTW, how does XIP account the kernel text on other architectures that >>> implement it? >> >> Interesting point, I thought XIP would be something new on RISC-V (well, at >> least to me :) ). If that concept exists already, we better mimic what >> existing implementations do. > > I had quick glance at ARM, it seems that kernel text does not have memory > map and does not show up in System RAM. > Does it show up in a different way or not at all? > -- > Sincerely yours, > Mike. >
Re: [PATCH v7] RISC-V: enable XIP
On Fri, Apr 09, 2021 at 02:46:17PM +0200, David Hildenbrand wrote: > > > > Also, will that memory properly be exposed in the resource tree as > > > > System RAM (e.g., /proc/iomem) ? Otherwise some things (/proc/kcore) > > > > won't work as expected - the kernel won't be included in a dump. > > Do we really need a XIP kernel to included in kdump? > > And does not it sound weird to expose flash as System RAM in /proc/iomem? > > ;-) > > See my other mail, maybe we actually want something different. > > > > > > I have just checked and it does not appear in /proc/iomem. > > > > > > Ok your conclusion would be to have struct page, I'm going to implement > > > this > > > version then using memblock as you described. > > > > I'm not sure this is required. With XIP kernel text never gets into RAM, so > > it does not seem to require struct page. > > > > XIP by definition has some limitations relatively to "normal" operation, > > so lack of kdump could be one of them. > > I agree. > > > > > I might be wrong, but IMHO, artificially creating a memory map for part of > > flash would cause more problems in the long run. > > Can you elaborate? Nothing particular, just a gut feeling. Usually, when you force something it comes out the wrong way later. > > > > BTW, how does XIP account the kernel text on other architectures that > > implement it? > > Interesting point, I thought XIP would be something new on RISC-V (well, at > least to me :) ). If that concept exists already, we better mimic what > existing implementations do. I had quick glance at ARM, it seems that kernel text does not have memory map and does not show up in System RAM. -- Sincerely yours, Mike.
Re: [PATCH v7] RISC-V: enable XIP
Le 4/9/21 à 8:07 AM, David Hildenbrand a écrit : On 09.04.21 13:39, Alex Ghiti wrote: Hi David, Le 4/9/21 à 4:23 AM, David Hildenbrand a écrit : On 09.04.21 09:14, Alex Ghiti wrote: Le 4/9/21 à 2:51 AM, Alexandre Ghiti a écrit : From: Vitaly Wool Introduce XIP (eXecute In Place) support for RISC-V platforms. It allows code to be executed directly from non-volatile storage directly addressable by the CPU, such as QSPI NOR flash which can be found on many RISC-V platforms. This makes way for significant optimization of RAM footprint. The XIP kernel is not compressed since it has to run directly from flash, so it will occupy more space on the non-volatile storage. The physical flash address used to link the kernel object files and for storing it has to be known at compile time and is represented by a Kconfig option. XIP on RISC-V will for the time being only work on MMU-enabled kernels. I added linux-mm and linux-arch to get feedbacks because I noticed that DEBUG_VM_PGTABLE fails for SPARSEMEM (it works for FLATMEM but I think it does not do what is expected): the fact that we don't have any struct page to back the text and rodata in flash is the problem but to which extent ? Just wondering, why can't we create a memmap for that memory -- or is it even desireable to not do that explicity? There might be some nasty side effects when not having a memmap for text and rodata. Do you have examples of such effects ? Any feature that will not work without that ? At least if it's not part of /proc/iomem in any way (maybe "System RAM" is not what we want without a memmap, TBD), kexec-tools won't be able to handle it properly e.g., for kdump. But not sure if that is really relevant in your setup. Regarding other features, anything that does a pfn_valid(), pfn_to_page() or pfn_to_online_page() would behave differently now -- assuming the kernel doesn't fall into a section with other System RAM (whereby we would still allocate the memmap for the whole section). I guess you might stumble over some surprises in some code paths, but nothing really comes to mind. Not sure if your zeropage is part of the kernel image on RISC-V (I remember that we sometimes need a memmap there, but I might be wrong)? It is in the kernel image and is located in bss which will be in RAM and then be backed by a memmap. I assume you still somehow create the direct mapping for the kernel, right? So it's really some memory region with a direct mapping but without a memmap (and right now, without a resource), correct? No I don't create any direct mapping for the text and the rodata. [...] Also, will that memory properly be exposed in the resource tree as System RAM (e.g., /proc/iomem) ? Otherwise some things (/proc/kcore) won't work as expected - the kernel won't be included in a dump. I have just checked and it does not appear in /proc/iomem. Ok your conclusion would be to have struct page, I'm going to implement this version then using memblock as you described. Let's first evaluate what the harm could be. You could (and should?) create the kernel resource manually - IIRC, that's independent of the memmap/memblock thing. @Mike, what's your take on not having a memmap for kernel text and ro data?
Re: [PATCH v7] RISC-V: enable XIP
Also, will that memory properly be exposed in the resource tree as System RAM (e.g., /proc/iomem) ? Otherwise some things (/proc/kcore) won't work as expected - the kernel won't be included in a dump. Do we really need a XIP kernel to included in kdump? And does not it sound weird to expose flash as System RAM in /proc/iomem? ;-) See my other mail, maybe we actually want something different. I have just checked and it does not appear in /proc/iomem. Ok your conclusion would be to have struct page, I'm going to implement this version then using memblock as you described. I'm not sure this is required. With XIP kernel text never gets into RAM, so it does not seem to require struct page. XIP by definition has some limitations relatively to "normal" operation, so lack of kdump could be one of them. I agree. I might be wrong, but IMHO, artificially creating a memory map for part of flash would cause more problems in the long run. Can you elaborate? BTW, how does XIP account the kernel text on other architectures that implement it? Interesting point, I thought XIP would be something new on RISC-V (well, at least to me :) ). If that concept exists already, we better mimic what existing implementations do. -- Thanks, David / dhildenb
Re: [PATCH v7] RISC-V: enable XIP
On Fri, Apr 09, 2021 at 07:39:59AM -0400, Alex Ghiti wrote: > Hi David, > > Le 4/9/21 à 4:23 AM, David Hildenbrand a écrit : > > On 09.04.21 09:14, Alex Ghiti wrote: > > > Le 4/9/21 à 2:51 AM, Alexandre Ghiti a écrit : > > > > From: Vitaly Wool > > > > > > > > Introduce XIP (eXecute In Place) support for RISC-V platforms. > > > > It allows code to be executed directly from non-volatile storage > > > > directly addressable by the CPU, such as QSPI NOR flash which can > > > > be found on many RISC-V platforms. This makes way for significant > > > > optimization of RAM footprint. The XIP kernel is not compressed > > > > since it has to run directly from flash, so it will occupy more > > > > space on the non-volatile storage. The physical flash address used > > > > to link the kernel object files and for storing it has to be known > > > > at compile time and is represented by a Kconfig option. > > > > > > > > XIP on RISC-V will for the time being only work on MMU-enabled > > > > kernels. > > > > > > > I added linux-mm and linux-arch to get feedbacks because I noticed that > > > DEBUG_VM_PGTABLE fails for SPARSEMEM (it works for FLATMEM but I think > > > it does not do what is expected): the fact that we don't have any struct > > > page to back the text and rodata in flash is the problem but to which > > > extent ? > > > > Just wondering, why can't we create a memmap for that memory -- or is it > > even desireable to not do that explicity? There might be some nasty side > > effects when not having a memmap for text and rodata. > > > Do you have examples of such effects ? Any feature that will not work > without that ? > > > > > > I would assume stimply exposing the physical memory range to memblock as > > RAM and marking it reserved would create a memmap that's fully > > initialized like any bootmem (PG_reserved). > > > > Or is there a reason why we cannot do that? > > > I did not want to do that if it was not needed as the overall goal of XIP > kernel is to save RAM (I may be cheap but 16MB backed by struct page > represents ~220KB). > > > > > > > Also, will that memory properly be exposed in the resource tree as > > System RAM (e.g., /proc/iomem) ? Otherwise some things (/proc/kcore) > > won't work as expected - the kernel won't be included in a dump. Do we really need a XIP kernel to included in kdump? And does not it sound weird to expose flash as System RAM in /proc/iomem? ;-) > I have just checked and it does not appear in /proc/iomem. > > Ok your conclusion would be to have struct page, I'm going to implement this > version then using memblock as you described. I'm not sure this is required. With XIP kernel text never gets into RAM, so it does not seem to require struct page. XIP by definition has some limitations relatively to "normal" operation, so lack of kdump could be one of them. I might be wrong, but IMHO, artificially creating a memory map for part of flash would cause more problems in the long run. BTW, how does XIP account the kernel text on other architectures that implement it? > Thanks David, > > Alex > > > > > > > > > > > Thanks, > > > > > > Alex > > > > > > > Signed-off-by: Alexandre Ghiti [ Rebase on top of "Move > > > > kernel mapping outside the linear mapping ] > > > > Signed-off-by: Vitaly Wool > > > > --- > > > > > > > > Changes in v2: > > > > - dedicated macro for XIP address fixup when MMU is not enabled yet > > > > o both for 32-bit and 64-bit RISC-V > > > > - SP is explicitly set to a safe place in RAM before __copy_data call > > > > - removed redundant alignment requirements in vmlinux-xip.lds.S > > > > - changed long -> uintptr_t typecast in __XIP_FIXUP macro. > > > > Changes in v3: > > > > - rebased against latest for-next > > > > - XIP address fixup macro now takes an argument > > > > - SMP related fixes > > > > Changes in v4: > > > > - rebased against the current for-next > > > > - less #ifdef's in C/ASM code > > > > - dedicated XIP_FIXUP_OFFSET assembler macro in head.S > > > > - C-specific definitions moved into #ifndef __ASSEMBLY__ > > > > - Fixed multi-core boot > > > > Changes in v5: > > > > - fixed build error for non-XIP kernels > > > > Changes in v6: > > > > - XIP_PHYS_RAM_BASE config option renamed to PHYS_RAM_BASE > > > > - added PHYS_RAM_BASE_FIXED config flag to allow usage of > > > > PHYS_RAM_BASE in non-XIP configurations if needed > > > > - XIP_FIXUP macro rewritten with a tempoarary variable to avoid side > > > > effects > > > > - fixed crash for non-XIP kernels that don't use built-in DTB > > > > Changes in v7: > > > > - Fix pfn_base that required FIXUP > > > > - Fix copy_data which lacked + 1 in size to copy > > > > - Fix pfn_valid for FLATMEM > > > > - Rebased on top of "Move kernel mapping outside the linear mapping": > > > > this is the biggest change and affected mm/init.c, > > > > kernel/vmlinux-xip.lds.S and include/asm/pgtable.h: XIP > > > > kernel is now > > > > mapped like 'normal'
Re: [PATCH v7] RISC-V: enable XIP
On 09.04.21 13:39, Alex Ghiti wrote: Hi David, Le 4/9/21 à 4:23 AM, David Hildenbrand a écrit : On 09.04.21 09:14, Alex Ghiti wrote: Le 4/9/21 à 2:51 AM, Alexandre Ghiti a écrit : From: Vitaly Wool Introduce XIP (eXecute In Place) support for RISC-V platforms. It allows code to be executed directly from non-volatile storage directly addressable by the CPU, such as QSPI NOR flash which can be found on many RISC-V platforms. This makes way for significant optimization of RAM footprint. The XIP kernel is not compressed since it has to run directly from flash, so it will occupy more space on the non-volatile storage. The physical flash address used to link the kernel object files and for storing it has to be known at compile time and is represented by a Kconfig option. XIP on RISC-V will for the time being only work on MMU-enabled kernels. I added linux-mm and linux-arch to get feedbacks because I noticed that DEBUG_VM_PGTABLE fails for SPARSEMEM (it works for FLATMEM but I think it does not do what is expected): the fact that we don't have any struct page to back the text and rodata in flash is the problem but to which extent ? Just wondering, why can't we create a memmap for that memory -- or is it even desireable to not do that explicity? There might be some nasty side effects when not having a memmap for text and rodata. Do you have examples of such effects ? Any feature that will not work without that ? At least if it's not part of /proc/iomem in any way (maybe "System RAM" is not what we want without a memmap, TBD), kexec-tools won't be able to handle it properly e.g., for kdump. But not sure if that is really relevant in your setup. Regarding other features, anything that does a pfn_valid(), pfn_to_page() or pfn_to_online_page() would behave differently now -- assuming the kernel doesn't fall into a section with other System RAM (whereby we would still allocate the memmap for the whole section). I guess you might stumble over some surprises in some code paths, but nothing really comes to mind. Not sure if your zeropage is part of the kernel image on RISC-V (I remember that we sometimes need a memmap there, but I might be wrong)? I assume you still somehow create the direct mapping for the kernel, right? So it's really some memory region with a direct mapping but without a memmap (and right now, without a resource), correct? [...] Also, will that memory properly be exposed in the resource tree as System RAM (e.g., /proc/iomem) ? Otherwise some things (/proc/kcore) won't work as expected - the kernel won't be included in a dump. I have just checked and it does not appear in /proc/iomem. Ok your conclusion would be to have struct page, I'm going to implement this version then using memblock as you described. Let's first evaluate what the harm could be. You could (and should?) create the kernel resource manually - IIRC, that's independent of the memmap/memblock thing. @Mike, what's your take on not having a memmap for kernel text and ro data? -- Thanks, David / dhildenb
Re: [PATCH v7] RISC-V: enable XIP
Hi David, Le 4/9/21 à 4:23 AM, David Hildenbrand a écrit : On 09.04.21 09:14, Alex Ghiti wrote: Le 4/9/21 à 2:51 AM, Alexandre Ghiti a écrit : From: Vitaly Wool Introduce XIP (eXecute In Place) support for RISC-V platforms. It allows code to be executed directly from non-volatile storage directly addressable by the CPU, such as QSPI NOR flash which can be found on many RISC-V platforms. This makes way for significant optimization of RAM footprint. The XIP kernel is not compressed since it has to run directly from flash, so it will occupy more space on the non-volatile storage. The physical flash address used to link the kernel object files and for storing it has to be known at compile time and is represented by a Kconfig option. XIP on RISC-V will for the time being only work on MMU-enabled kernels. I added linux-mm and linux-arch to get feedbacks because I noticed that DEBUG_VM_PGTABLE fails for SPARSEMEM (it works for FLATMEM but I think it does not do what is expected): the fact that we don't have any struct page to back the text and rodata in flash is the problem but to which extent ? Just wondering, why can't we create a memmap for that memory -- or is it even desireable to not do that explicity? There might be some nasty side effects when not having a memmap for text and rodata. Do you have examples of such effects ? Any feature that will not work without that ? I would assume stimply exposing the physical memory range to memblock as RAM and marking it reserved would create a memmap that's fully initialized like any bootmem (PG_reserved). Or is there a reason why we cannot do that? I did not want to do that if it was not needed as the overall goal of XIP kernel is to save RAM (I may be cheap but 16MB backed by struct page represents ~220KB). Also, will that memory properly be exposed in the resource tree as System RAM (e.g., /proc/iomem) ? Otherwise some things (/proc/kcore) won't work as expected - the kernel won't be included in a dump. I have just checked and it does not appear in /proc/iomem. Ok your conclusion would be to have struct page, I'm going to implement this version then using memblock as you described. Thanks David, Alex Thanks, Alex Signed-off-by: Alexandre Ghiti [ Rebase on top of "Move kernel mapping outside the linear mapping ] Signed-off-by: Vitaly Wool --- Changes in v2: - dedicated macro for XIP address fixup when MMU is not enabled yet o both for 32-bit and 64-bit RISC-V - SP is explicitly set to a safe place in RAM before __copy_data call - removed redundant alignment requirements in vmlinux-xip.lds.S - changed long -> uintptr_t typecast in __XIP_FIXUP macro. Changes in v3: - rebased against latest for-next - XIP address fixup macro now takes an argument - SMP related fixes Changes in v4: - rebased against the current for-next - less #ifdef's in C/ASM code - dedicated XIP_FIXUP_OFFSET assembler macro in head.S - C-specific definitions moved into #ifndef __ASSEMBLY__ - Fixed multi-core boot Changes in v5: - fixed build error for non-XIP kernels Changes in v6: - XIP_PHYS_RAM_BASE config option renamed to PHYS_RAM_BASE - added PHYS_RAM_BASE_FIXED config flag to allow usage of PHYS_RAM_BASE in non-XIP configurations if needed - XIP_FIXUP macro rewritten with a tempoarary variable to avoid side effects - fixed crash for non-XIP kernels that don't use built-in DTB Changes in v7: - Fix pfn_base that required FIXUP - Fix copy_data which lacked + 1 in size to copy - Fix pfn_valid for FLATMEM - Rebased on top of "Move kernel mapping outside the linear mapping": this is the biggest change and affected mm/init.c, kernel/vmlinux-xip.lds.S and include/asm/pgtable.h: XIP kernel is now mapped like 'normal' kernel at the end of the address space. arch/riscv/Kconfig | 51 ++- arch/riscv/Makefile | 8 +- arch/riscv/boot/Makefile | 13 +++ arch/riscv/include/asm/page.h | 28 ++ arch/riscv/include/asm/pgtable.h | 25 +- arch/riscv/kernel/head.S | 46 +- arch/riscv/kernel/head.h | 3 + arch/riscv/kernel/setup.c | 10 ++- arch/riscv/kernel/vmlinux-xip.lds.S | 133 arch/riscv/kernel/vmlinux.lds.S | 6 ++ arch/riscv/mm/init.c | 118 ++-- 11 files changed, 424 insertions(+), 17 deletions(-) create mode 100644 arch/riscv/kernel/vmlinux-xip.lds.S diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 8ea60a0a19ae..4d0153805927 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -28,7 +28,7 @@ config RISCV select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_SET_DIRECT_MAP select ARCH_HAS_SET_MEMORY - select ARCH_HAS_STRICT_KERNEL_RWX if MMU + select ARCH_HAS_STRICT_KERNEL_RWX if MMU && !XIP_KERNEL select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST
Re: [PATCH v7] RISC-V: enable XIP
On 09.04.21 09:14, Alex Ghiti wrote: Le 4/9/21 à 2:51 AM, Alexandre Ghiti a écrit : From: Vitaly Wool Introduce XIP (eXecute In Place) support for RISC-V platforms. It allows code to be executed directly from non-volatile storage directly addressable by the CPU, such as QSPI NOR flash which can be found on many RISC-V platforms. This makes way for significant optimization of RAM footprint. The XIP kernel is not compressed since it has to run directly from flash, so it will occupy more space on the non-volatile storage. The physical flash address used to link the kernel object files and for storing it has to be known at compile time and is represented by a Kconfig option. XIP on RISC-V will for the time being only work on MMU-enabled kernels. I added linux-mm and linux-arch to get feedbacks because I noticed that DEBUG_VM_PGTABLE fails for SPARSEMEM (it works for FLATMEM but I think it does not do what is expected): the fact that we don't have any struct page to back the text and rodata in flash is the problem but to which extent ? Just wondering, why can't we create a memmap for that memory -- or is it even desireable to not do that explicity? There might be some nasty side effects when not having a memmap for text and rodata. I would assume stimply exposing the physical memory range to memblock as RAM and marking it reserved would create a memmap that's fully initialized like any bootmem (PG_reserved). Or is there a reason why we cannot do that? Also, will that memory properly be exposed in the resource tree as System RAM (e.g., /proc/iomem) ? Otherwise some things (/proc/kcore) won't work as expected - the kernel won't be included in a dump. Thanks, Alex Signed-off-by: Alexandre Ghiti [ Rebase on top of "Move kernel mapping outside the linear mapping ] Signed-off-by: Vitaly Wool --- Changes in v2: - dedicated macro for XIP address fixup when MMU is not enabled yet o both for 32-bit and 64-bit RISC-V - SP is explicitly set to a safe place in RAM before __copy_data call - removed redundant alignment requirements in vmlinux-xip.lds.S - changed long -> uintptr_t typecast in __XIP_FIXUP macro. Changes in v3: - rebased against latest for-next - XIP address fixup macro now takes an argument - SMP related fixes Changes in v4: - rebased against the current for-next - less #ifdef's in C/ASM code - dedicated XIP_FIXUP_OFFSET assembler macro in head.S - C-specific definitions moved into #ifndef __ASSEMBLY__ - Fixed multi-core boot Changes in v5: - fixed build error for non-XIP kernels Changes in v6: - XIP_PHYS_RAM_BASE config option renamed to PHYS_RAM_BASE - added PHYS_RAM_BASE_FIXED config flag to allow usage of PHYS_RAM_BASE in non-XIP configurations if needed - XIP_FIXUP macro rewritten with a tempoarary variable to avoid side effects - fixed crash for non-XIP kernels that don't use built-in DTB Changes in v7: - Fix pfn_base that required FIXUP - Fix copy_data which lacked + 1 in size to copy - Fix pfn_valid for FLATMEM - Rebased on top of "Move kernel mapping outside the linear mapping": this is the biggest change and affected mm/init.c, kernel/vmlinux-xip.lds.S and include/asm/pgtable.h: XIP kernel is now mapped like 'normal' kernel at the end of the address space. arch/riscv/Kconfig | 51 ++- arch/riscv/Makefile | 8 +- arch/riscv/boot/Makefile| 13 +++ arch/riscv/include/asm/page.h | 28 ++ arch/riscv/include/asm/pgtable.h| 25 +- arch/riscv/kernel/head.S| 46 +- arch/riscv/kernel/head.h| 3 + arch/riscv/kernel/setup.c | 10 ++- arch/riscv/kernel/vmlinux-xip.lds.S | 133 arch/riscv/kernel/vmlinux.lds.S | 6 ++ arch/riscv/mm/init.c| 118 ++-- 11 files changed, 424 insertions(+), 17 deletions(-) create mode 100644 arch/riscv/kernel/vmlinux-xip.lds.S diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 8ea60a0a19ae..4d0153805927 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -28,7 +28,7 @@ config RISCV select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_SET_DIRECT_MAP select ARCH_HAS_SET_MEMORY - select ARCH_HAS_STRICT_KERNEL_RWX if MMU + select ARCH_HAS_STRICT_KERNEL_RWX if MMU && !XIP_KERNEL select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST select ARCH_OPTIONAL_KERNEL_RWX if ARCH_HAS_STRICT_KERNEL_RWX select ARCH_OPTIONAL_KERNEL_RWX_DEFAULT @@ -441,7 +441,7 @@ config EFI_STUB config EFI bool "UEFI runtime support" - depends on OF + depends on OF && !XIP_KERNEL select LIBFDT select UCS2_STRING select EFI_PARAMS_FROM_FDT @@ -465,11 +465,56 @@ config STACKPROTECTOR_PER_TASK def_bool y depends on STACKPROTECTOR && CC_HAVE_STACKPROTECTOR_TLS +config PHYS_RAM_BASE_FIXED + bool
Re: [PATCH v7] RISC-V: enable XIP
Le 4/9/21 à 2:51 AM, Alexandre Ghiti a écrit : From: Vitaly Wool Introduce XIP (eXecute In Place) support for RISC-V platforms. It allows code to be executed directly from non-volatile storage directly addressable by the CPU, such as QSPI NOR flash which can be found on many RISC-V platforms. This makes way for significant optimization of RAM footprint. The XIP kernel is not compressed since it has to run directly from flash, so it will occupy more space on the non-volatile storage. The physical flash address used to link the kernel object files and for storing it has to be known at compile time and is represented by a Kconfig option. XIP on RISC-V will for the time being only work on MMU-enabled kernels. I added linux-mm and linux-arch to get feedbacks because I noticed that DEBUG_VM_PGTABLE fails for SPARSEMEM (it works for FLATMEM but I think it does not do what is expected): the fact that we don't have any struct page to back the text and rodata in flash is the problem but to which extent ? Thanks, Alex Signed-off-by: Alexandre Ghiti [ Rebase on top of "Move kernel mapping outside the linear mapping ] Signed-off-by: Vitaly Wool --- Changes in v2: - dedicated macro for XIP address fixup when MMU is not enabled yet o both for 32-bit and 64-bit RISC-V - SP is explicitly set to a safe place in RAM before __copy_data call - removed redundant alignment requirements in vmlinux-xip.lds.S - changed long -> uintptr_t typecast in __XIP_FIXUP macro. Changes in v3: - rebased against latest for-next - XIP address fixup macro now takes an argument - SMP related fixes Changes in v4: - rebased against the current for-next - less #ifdef's in C/ASM code - dedicated XIP_FIXUP_OFFSET assembler macro in head.S - C-specific definitions moved into #ifndef __ASSEMBLY__ - Fixed multi-core boot Changes in v5: - fixed build error for non-XIP kernels Changes in v6: - XIP_PHYS_RAM_BASE config option renamed to PHYS_RAM_BASE - added PHYS_RAM_BASE_FIXED config flag to allow usage of PHYS_RAM_BASE in non-XIP configurations if needed - XIP_FIXUP macro rewritten with a tempoarary variable to avoid side effects - fixed crash for non-XIP kernels that don't use built-in DTB Changes in v7: - Fix pfn_base that required FIXUP - Fix copy_data which lacked + 1 in size to copy - Fix pfn_valid for FLATMEM - Rebased on top of "Move kernel mapping outside the linear mapping": this is the biggest change and affected mm/init.c, kernel/vmlinux-xip.lds.S and include/asm/pgtable.h: XIP kernel is now mapped like 'normal' kernel at the end of the address space. arch/riscv/Kconfig | 51 ++- arch/riscv/Makefile | 8 +- arch/riscv/boot/Makefile| 13 +++ arch/riscv/include/asm/page.h | 28 ++ arch/riscv/include/asm/pgtable.h| 25 +- arch/riscv/kernel/head.S| 46 +- arch/riscv/kernel/head.h| 3 + arch/riscv/kernel/setup.c | 10 ++- arch/riscv/kernel/vmlinux-xip.lds.S | 133 arch/riscv/kernel/vmlinux.lds.S | 6 ++ arch/riscv/mm/init.c| 118 ++-- 11 files changed, 424 insertions(+), 17 deletions(-) create mode 100644 arch/riscv/kernel/vmlinux-xip.lds.S diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 8ea60a0a19ae..4d0153805927 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -28,7 +28,7 @@ config RISCV select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_SET_DIRECT_MAP select ARCH_HAS_SET_MEMORY - select ARCH_HAS_STRICT_KERNEL_RWX if MMU + select ARCH_HAS_STRICT_KERNEL_RWX if MMU && !XIP_KERNEL select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST select ARCH_OPTIONAL_KERNEL_RWX if ARCH_HAS_STRICT_KERNEL_RWX select ARCH_OPTIONAL_KERNEL_RWX_DEFAULT @@ -441,7 +441,7 @@ config EFI_STUB config EFI bool "UEFI runtime support" - depends on OF + depends on OF && !XIP_KERNEL select LIBFDT select UCS2_STRING select EFI_PARAMS_FROM_FDT @@ -465,11 +465,56 @@ config STACKPROTECTOR_PER_TASK def_bool y depends on STACKPROTECTOR && CC_HAVE_STACKPROTECTOR_TLS +config PHYS_RAM_BASE_FIXED + bool "Explicitly specified physical RAM address" + default n + +config PHYS_RAM_BASE + hex "Platform Physical RAM address" + depends on PHYS_RAM_BASE_FIXED + default "0x8000" + help + This is the physical address of RAM in the system. It has to be + explicitly specified to run early relocations of read-write data + from flash to RAM. + +config XIP_KERNEL + bool "Kernel Execute-In-Place from ROM" + depends on MMU + select PHYS_RAM_BASE_FIXED + help + Execute-In-Place allows the kernel to run from non-volatile storage + directly addressable by the CPU, such as NOR flash. This saves RAM + space
[PATCH v7] RISC-V: enable XIP
From: Vitaly Wool Introduce XIP (eXecute In Place) support for RISC-V platforms. It allows code to be executed directly from non-volatile storage directly addressable by the CPU, such as QSPI NOR flash which can be found on many RISC-V platforms. This makes way for significant optimization of RAM footprint. The XIP kernel is not compressed since it has to run directly from flash, so it will occupy more space on the non-volatile storage. The physical flash address used to link the kernel object files and for storing it has to be known at compile time and is represented by a Kconfig option. XIP on RISC-V will for the time being only work on MMU-enabled kernels. Signed-off-by: Alexandre Ghiti [ Rebase on top of "Move kernel mapping outside the linear mapping ] Signed-off-by: Vitaly Wool --- Changes in v2: - dedicated macro for XIP address fixup when MMU is not enabled yet o both for 32-bit and 64-bit RISC-V - SP is explicitly set to a safe place in RAM before __copy_data call - removed redundant alignment requirements in vmlinux-xip.lds.S - changed long -> uintptr_t typecast in __XIP_FIXUP macro. Changes in v3: - rebased against latest for-next - XIP address fixup macro now takes an argument - SMP related fixes Changes in v4: - rebased against the current for-next - less #ifdef's in C/ASM code - dedicated XIP_FIXUP_OFFSET assembler macro in head.S - C-specific definitions moved into #ifndef __ASSEMBLY__ - Fixed multi-core boot Changes in v5: - fixed build error for non-XIP kernels Changes in v6: - XIP_PHYS_RAM_BASE config option renamed to PHYS_RAM_BASE - added PHYS_RAM_BASE_FIXED config flag to allow usage of PHYS_RAM_BASE in non-XIP configurations if needed - XIP_FIXUP macro rewritten with a tempoarary variable to avoid side effects - fixed crash for non-XIP kernels that don't use built-in DTB Changes in v7: - Fix pfn_base that required FIXUP - Fix copy_data which lacked + 1 in size to copy - Fix pfn_valid for FLATMEM - Rebased on top of "Move kernel mapping outside the linear mapping": this is the biggest change and affected mm/init.c, kernel/vmlinux-xip.lds.S and include/asm/pgtable.h: XIP kernel is now mapped like 'normal' kernel at the end of the address space. arch/riscv/Kconfig | 51 ++- arch/riscv/Makefile | 8 +- arch/riscv/boot/Makefile| 13 +++ arch/riscv/include/asm/page.h | 28 ++ arch/riscv/include/asm/pgtable.h| 25 +- arch/riscv/kernel/head.S| 46 +- arch/riscv/kernel/head.h| 3 + arch/riscv/kernel/setup.c | 10 ++- arch/riscv/kernel/vmlinux-xip.lds.S | 133 arch/riscv/kernel/vmlinux.lds.S | 6 ++ arch/riscv/mm/init.c| 118 ++-- 11 files changed, 424 insertions(+), 17 deletions(-) create mode 100644 arch/riscv/kernel/vmlinux-xip.lds.S diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 8ea60a0a19ae..4d0153805927 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -28,7 +28,7 @@ config RISCV select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_SET_DIRECT_MAP select ARCH_HAS_SET_MEMORY - select ARCH_HAS_STRICT_KERNEL_RWX if MMU + select ARCH_HAS_STRICT_KERNEL_RWX if MMU && !XIP_KERNEL select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST select ARCH_OPTIONAL_KERNEL_RWX if ARCH_HAS_STRICT_KERNEL_RWX select ARCH_OPTIONAL_KERNEL_RWX_DEFAULT @@ -441,7 +441,7 @@ config EFI_STUB config EFI bool "UEFI runtime support" - depends on OF + depends on OF && !XIP_KERNEL select LIBFDT select UCS2_STRING select EFI_PARAMS_FROM_FDT @@ -465,11 +465,56 @@ config STACKPROTECTOR_PER_TASK def_bool y depends on STACKPROTECTOR && CC_HAVE_STACKPROTECTOR_TLS +config PHYS_RAM_BASE_FIXED + bool "Explicitly specified physical RAM address" + default n + +config PHYS_RAM_BASE + hex "Platform Physical RAM address" + depends on PHYS_RAM_BASE_FIXED + default "0x8000" + help + This is the physical address of RAM in the system. It has to be + explicitly specified to run early relocations of read-write data + from flash to RAM. + +config XIP_KERNEL + bool "Kernel Execute-In-Place from ROM" + depends on MMU + select PHYS_RAM_BASE_FIXED + help + Execute-In-Place allows the kernel to run from non-volatile storage + directly addressable by the CPU, such as NOR flash. This saves RAM + space since the text section of the kernel is not loaded from flash + to RAM. Read-write sections, such as the data section and stack, + are still copied to RAM. The XIP kernel is not compressed since + it has to run directly from flash, so it will take more space to + store it. The flash address used to link the kernel object files, + and for
