Re: [PATCH v7 01/15] swiotlb: Refactor swiotlb init functions
On Fri, May 28, 2021 at 12:32 AM Tom Lendacky wrote: > > On 5/27/21 9:41 AM, Tom Lendacky wrote: > > On 5/27/21 8:02 AM, Christoph Hellwig wrote: > >> On Wed, May 19, 2021 at 11:50:07AM -0700, Florian Fainelli wrote: > >>> You convert this call site with swiotlb_init_io_tlb_mem() which did not > >>> do the set_memory_decrypted()+memset(). Is this okay or should > >>> swiotlb_init_io_tlb_mem() add an additional argument to do this > >>> conditionally? > >> > >> The zeroing is useful and was missing before. I think having a clean > >> state here is the right thing. > >> > >> Not sure about the set_memory_decrypted, swiotlb_update_mem_attributes > >> kinda suggests it is too early to set the memory decrupted. > >> > >> Adding Tom who should now about all this. > > > > The reason for adding swiotlb_update_mem_attributes() was because having > > the call to set_memory_decrypted() in swiotlb_init_with_tbl() triggered a > > BUG_ON() related to interrupts not being enabled yet during boot. So that > > call had to be delayed until interrupts were enabled. > > I pulled down and tested the patch set and booted with SME enabled. The > following was seen during the boot: > > [0.134184] BUG: Bad page state in process swapper pfn:108002 > [0.134196] page:(ptrval) refcount:0 mapcount:-128 > mapping: index:0x0 pfn:0x108002 > [0.134201] flags: 0x17c000(node=0|zone=2|lastcpupid=0x1f) > [0.134208] raw: 0017c000 88847f355e28 88847f355e28 > > [0.134210] raw: 0001 ff7f > > [0.134212] page dumped because: nonzero mapcount > [0.134213] Modules linked in: > [0.134218] CPU: 0 PID: 0 Comm: swapper Not tainted 5.13.0-rc2-sos-custom > #3 > [0.134221] Hardware name: ... > [0.134224] Call Trace: > [0.134233] dump_stack+0x76/0x94 > [0.134244] bad_page+0xa6/0xf0 > [0.134252] __free_pages_ok+0x331/0x360 > [0.134256] memblock_free_all+0x158/0x1c1 > [0.134267] mem_init+0x1f/0x14c > [0.134273] start_kernel+0x290/0x574 > [0.134279] secondary_startup_64_no_verify+0xb0/0xbb > > I see this about 40 times during the boot, each with a different PFN. The > system boots (which seemed odd), but I don't know if there will be side > effects to this (I didn't stress the system). > > I modified the code to add a flag to not do the set_memory_decrypted(), as > suggested by Florian, when invoked from swiotlb_init_with_tbl(), and that > eliminated the bad page state BUG. Thanks. Will add a flag to skip set_memory_decrypted() in v9. > > Thanks, > Tom > > > > > Thanks, > > Tom > > > >> ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v7 01/15] swiotlb: Refactor swiotlb init functions
On 5/27/21 9:41 AM, Tom Lendacky wrote: > On 5/27/21 8:02 AM, Christoph Hellwig wrote: >> On Wed, May 19, 2021 at 11:50:07AM -0700, Florian Fainelli wrote: >>> You convert this call site with swiotlb_init_io_tlb_mem() which did not >>> do the set_memory_decrypted()+memset(). Is this okay or should >>> swiotlb_init_io_tlb_mem() add an additional argument to do this >>> conditionally? >> >> The zeroing is useful and was missing before. I think having a clean >> state here is the right thing. >> >> Not sure about the set_memory_decrypted, swiotlb_update_mem_attributes >> kinda suggests it is too early to set the memory decrupted. >> >> Adding Tom who should now about all this. > > The reason for adding swiotlb_update_mem_attributes() was because having > the call to set_memory_decrypted() in swiotlb_init_with_tbl() triggered a > BUG_ON() related to interrupts not being enabled yet during boot. So that > call had to be delayed until interrupts were enabled. I pulled down and tested the patch set and booted with SME enabled. The following was seen during the boot: [0.134184] BUG: Bad page state in process swapper pfn:108002 [0.134196] page:(ptrval) refcount:0 mapcount:-128 mapping: index:0x0 pfn:0x108002 [0.134201] flags: 0x17c000(node=0|zone=2|lastcpupid=0x1f) [0.134208] raw: 0017c000 88847f355e28 88847f355e28 [0.134210] raw: 0001 ff7f [0.134212] page dumped because: nonzero mapcount [0.134213] Modules linked in: [0.134218] CPU: 0 PID: 0 Comm: swapper Not tainted 5.13.0-rc2-sos-custom #3 [0.134221] Hardware name: ... [0.134224] Call Trace: [0.134233] dump_stack+0x76/0x94 [0.134244] bad_page+0xa6/0xf0 [0.134252] __free_pages_ok+0x331/0x360 [0.134256] memblock_free_all+0x158/0x1c1 [0.134267] mem_init+0x1f/0x14c [0.134273] start_kernel+0x290/0x574 [0.134279] secondary_startup_64_no_verify+0xb0/0xbb I see this about 40 times during the boot, each with a different PFN. The system boots (which seemed odd), but I don't know if there will be side effects to this (I didn't stress the system). I modified the code to add a flag to not do the set_memory_decrypted(), as suggested by Florian, when invoked from swiotlb_init_with_tbl(), and that eliminated the bad page state BUG. Thanks, Tom > > Thanks, > Tom > >> ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v7 01/15] swiotlb: Refactor swiotlb init functions
On 5/27/21 8:02 AM, Christoph Hellwig wrote: > On Wed, May 19, 2021 at 11:50:07AM -0700, Florian Fainelli wrote: >> You convert this call site with swiotlb_init_io_tlb_mem() which did not >> do the set_memory_decrypted()+memset(). Is this okay or should >> swiotlb_init_io_tlb_mem() add an additional argument to do this >> conditionally? > > The zeroing is useful and was missing before. I think having a clean > state here is the right thing. > > Not sure about the set_memory_decrypted, swiotlb_update_mem_attributes > kinda suggests it is too early to set the memory decrupted. > > Adding Tom who should now about all this. The reason for adding swiotlb_update_mem_attributes() was because having the call to set_memory_decrypted() in swiotlb_init_with_tbl() triggered a BUG_ON() related to interrupts not being enabled yet during boot. So that call had to be delayed until interrupts were enabled. Thanks, Tom > ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v7 01/15] swiotlb: Refactor swiotlb init functions
On Wed, May 19, 2021 at 11:50:07AM -0700, Florian Fainelli wrote: > You convert this call site with swiotlb_init_io_tlb_mem() which did not > do the set_memory_decrypted()+memset(). Is this okay or should > swiotlb_init_io_tlb_mem() add an additional argument to do this > conditionally? The zeroing is useful and was missing before. I think having a clean state here is the right thing. Not sure about the set_memory_decrypted, swiotlb_update_mem_attributes kinda suggests it is too early to set the memory decrupted. Adding Tom who should now about all this. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v7 01/15] swiotlb: Refactor swiotlb init functions
On Mon, May 24, 2021 at 11:53 PM Konrad Rzeszutek Wilk wrote: > > > > do the set_memory_decrypted()+memset(). Is this okay or should > > > swiotlb_init_io_tlb_mem() add an additional argument to do this > > > conditionally? > > > > I'm actually not sure if this it okay. If not, will add an additional > > argument for it. > > Any observations discovered? (Want to make sure my memory-cache has the > correct semantics for set_memory_decrypted in mind). It works fine on my arm64 device. > > > > > -- > > > Florian ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v7 01/15] swiotlb: Refactor swiotlb init functions
> > do the set_memory_decrypted()+memset(). Is this okay or should > > swiotlb_init_io_tlb_mem() add an additional argument to do this > > conditionally? > > I'm actually not sure if this it okay. If not, will add an additional > argument for it. Any observations discovered? (Want to make sure my memory-cache has the correct semantics for set_memory_decrypted in mind). > > > -- > > Florian ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v7 01/15] swiotlb: Refactor swiotlb init functions
On Thu, May 20, 2021 at 2:50 AM Florian Fainelli wrote:
>
>
>
> On 5/17/2021 11:42 PM, Claire Chang wrote:
> > Add a new function, swiotlb_init_io_tlb_mem, for the io_tlb_mem struct
> > initialization to make the code reusable.
> >
> > Note that we now also call set_memory_decrypted in swiotlb_init_with_tbl.
> >
> > Signed-off-by: Claire Chang
> > ---
> > kernel/dma/swiotlb.c | 51 ++--
> > 1 file changed, 25 insertions(+), 26 deletions(-)
> >
> > diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
> > index 8ca7d505d61c..d3232fc19385 100644
> > --- a/kernel/dma/swiotlb.c
> > +++ b/kernel/dma/swiotlb.c
> > @@ -168,9 +168,30 @@ void __init swiotlb_update_mem_attributes(void)
> > memset(vaddr, 0, bytes);
> > }
> >
> > -int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int
> > verbose)
> > +static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t
> > start,
> > + unsigned long nslabs, bool late_alloc)
> > {
> > + void *vaddr = phys_to_virt(start);
> > unsigned long bytes = nslabs << IO_TLB_SHIFT, i;
> > +
> > + mem->nslabs = nslabs;
> > + mem->start = start;
> > + mem->end = mem->start + bytes;
> > + mem->index = 0;
> > + mem->late_alloc = late_alloc;
> > + spin_lock_init(>lock);
> > + for (i = 0; i < mem->nslabs; i++) {
> > + mem->slots[i].list = IO_TLB_SEGSIZE - io_tlb_offset(i);
> > + mem->slots[i].orig_addr = INVALID_PHYS_ADDR;
> > + mem->slots[i].alloc_size = 0;
> > + }
> > +
> > + set_memory_decrypted((unsigned long)vaddr, bytes >> PAGE_SHIFT);
> > + memset(vaddr, 0, bytes);
>
> You are doing an unconditional set_memory_decrypted() followed by a
> memset here, and then:
>
> > +}
> > +
> > +int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int
> > verbose)
> > +{
> > struct io_tlb_mem *mem;
> > size_t alloc_size;
> >
> > @@ -186,16 +207,8 @@ int __init swiotlb_init_with_tbl(char *tlb, unsigned
> > long nslabs, int verbose)
> > if (!mem)
> > panic("%s: Failed to allocate %zu bytes align=0x%lx\n",
> > __func__, alloc_size, PAGE_SIZE);
> > - mem->nslabs = nslabs;
> > - mem->start = __pa(tlb);
> > - mem->end = mem->start + bytes;
> > - mem->index = 0;
> > - spin_lock_init(>lock);
> > - for (i = 0; i < mem->nslabs; i++) {
> > - mem->slots[i].list = IO_TLB_SEGSIZE - io_tlb_offset(i);
> > - mem->slots[i].orig_addr = INVALID_PHYS_ADDR;
> > - mem->slots[i].alloc_size = 0;
> > - }
> > +
> > + swiotlb_init_io_tlb_mem(mem, __pa(tlb), nslabs, false);
>
> You convert this call site with swiotlb_init_io_tlb_mem() which did not
> do the set_memory_decrypted()+memset(). Is this okay or should
> swiotlb_init_io_tlb_mem() add an additional argument to do this
> conditionally?
I'm actually not sure if this it okay. If not, will add an additional
argument for it.
> --
> Florian
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v7 01/15] swiotlb: Refactor swiotlb init functions
On 5/17/2021 11:42 PM, Claire Chang wrote:
> Add a new function, swiotlb_init_io_tlb_mem, for the io_tlb_mem struct
> initialization to make the code reusable.
>
> Note that we now also call set_memory_decrypted in swiotlb_init_with_tbl.
>
> Signed-off-by: Claire Chang
> ---
> kernel/dma/swiotlb.c | 51 ++--
> 1 file changed, 25 insertions(+), 26 deletions(-)
>
> diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
> index 8ca7d505d61c..d3232fc19385 100644
> --- a/kernel/dma/swiotlb.c
> +++ b/kernel/dma/swiotlb.c
> @@ -168,9 +168,30 @@ void __init swiotlb_update_mem_attributes(void)
> memset(vaddr, 0, bytes);
> }
>
> -int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int
> verbose)
> +static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t
> start,
> + unsigned long nslabs, bool late_alloc)
> {
> + void *vaddr = phys_to_virt(start);
> unsigned long bytes = nslabs << IO_TLB_SHIFT, i;
> +
> + mem->nslabs = nslabs;
> + mem->start = start;
> + mem->end = mem->start + bytes;
> + mem->index = 0;
> + mem->late_alloc = late_alloc;
> + spin_lock_init(>lock);
> + for (i = 0; i < mem->nslabs; i++) {
> + mem->slots[i].list = IO_TLB_SEGSIZE - io_tlb_offset(i);
> + mem->slots[i].orig_addr = INVALID_PHYS_ADDR;
> + mem->slots[i].alloc_size = 0;
> + }
> +
> + set_memory_decrypted((unsigned long)vaddr, bytes >> PAGE_SHIFT);
> + memset(vaddr, 0, bytes);
You are doing an unconditional set_memory_decrypted() followed by a
memset here, and then:
> +}
> +
> +int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int
> verbose)
> +{
> struct io_tlb_mem *mem;
> size_t alloc_size;
>
> @@ -186,16 +207,8 @@ int __init swiotlb_init_with_tbl(char *tlb, unsigned
> long nslabs, int verbose)
> if (!mem)
> panic("%s: Failed to allocate %zu bytes align=0x%lx\n",
> __func__, alloc_size, PAGE_SIZE);
> - mem->nslabs = nslabs;
> - mem->start = __pa(tlb);
> - mem->end = mem->start + bytes;
> - mem->index = 0;
> - spin_lock_init(>lock);
> - for (i = 0; i < mem->nslabs; i++) {
> - mem->slots[i].list = IO_TLB_SEGSIZE - io_tlb_offset(i);
> - mem->slots[i].orig_addr = INVALID_PHYS_ADDR;
> - mem->slots[i].alloc_size = 0;
> - }
> +
> + swiotlb_init_io_tlb_mem(mem, __pa(tlb), nslabs, false);
You convert this call site with swiotlb_init_io_tlb_mem() which did not
do the set_memory_decrypted()+memset(). Is this okay or should
swiotlb_init_io_tlb_mem() add an additional argument to do this
conditionally?
--
Florian
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
[PATCH v7 01/15] swiotlb: Refactor swiotlb init functions
Add a new function, swiotlb_init_io_tlb_mem, for the io_tlb_mem struct
initialization to make the code reusable.
Note that we now also call set_memory_decrypted in swiotlb_init_with_tbl.
Signed-off-by: Claire Chang
---
kernel/dma/swiotlb.c | 51 ++--
1 file changed, 25 insertions(+), 26 deletions(-)
diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
index 8ca7d505d61c..d3232fc19385 100644
--- a/kernel/dma/swiotlb.c
+++ b/kernel/dma/swiotlb.c
@@ -168,9 +168,30 @@ void __init swiotlb_update_mem_attributes(void)
memset(vaddr, 0, bytes);
}
-int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose)
+static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t start,
+ unsigned long nslabs, bool late_alloc)
{
+ void *vaddr = phys_to_virt(start);
unsigned long bytes = nslabs << IO_TLB_SHIFT, i;
+
+ mem->nslabs = nslabs;
+ mem->start = start;
+ mem->end = mem->start + bytes;
+ mem->index = 0;
+ mem->late_alloc = late_alloc;
+ spin_lock_init(>lock);
+ for (i = 0; i < mem->nslabs; i++) {
+ mem->slots[i].list = IO_TLB_SEGSIZE - io_tlb_offset(i);
+ mem->slots[i].orig_addr = INVALID_PHYS_ADDR;
+ mem->slots[i].alloc_size = 0;
+ }
+
+ set_memory_decrypted((unsigned long)vaddr, bytes >> PAGE_SHIFT);
+ memset(vaddr, 0, bytes);
+}
+
+int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose)
+{
struct io_tlb_mem *mem;
size_t alloc_size;
@@ -186,16 +207,8 @@ int __init swiotlb_init_with_tbl(char *tlb, unsigned long
nslabs, int verbose)
if (!mem)
panic("%s: Failed to allocate %zu bytes align=0x%lx\n",
__func__, alloc_size, PAGE_SIZE);
- mem->nslabs = nslabs;
- mem->start = __pa(tlb);
- mem->end = mem->start + bytes;
- mem->index = 0;
- spin_lock_init(>lock);
- for (i = 0; i < mem->nslabs; i++) {
- mem->slots[i].list = IO_TLB_SEGSIZE - io_tlb_offset(i);
- mem->slots[i].orig_addr = INVALID_PHYS_ADDR;
- mem->slots[i].alloc_size = 0;
- }
+
+ swiotlb_init_io_tlb_mem(mem, __pa(tlb), nslabs, false);
io_tlb_default_mem = mem;
if (verbose)
@@ -282,7 +295,6 @@ swiotlb_late_init_with_default_size(size_t default_size)
int
swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs)
{
- unsigned long bytes = nslabs << IO_TLB_SHIFT, i;
struct io_tlb_mem *mem;
if (swiotlb_force == SWIOTLB_NO_FORCE)
@@ -297,20 +309,7 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs)
if (!mem)
return -ENOMEM;
- mem->nslabs = nslabs;
- mem->start = virt_to_phys(tlb);
- mem->end = mem->start + bytes;
- mem->index = 0;
- mem->late_alloc = 1;
- spin_lock_init(>lock);
- for (i = 0; i < mem->nslabs; i++) {
- mem->slots[i].list = IO_TLB_SEGSIZE - io_tlb_offset(i);
- mem->slots[i].orig_addr = INVALID_PHYS_ADDR;
- mem->slots[i].alloc_size = 0;
- }
-
- set_memory_decrypted((unsigned long)tlb, bytes >> PAGE_SHIFT);
- memset(tlb, 0, bytes);
+ swiotlb_init_io_tlb_mem(mem, virt_to_phys(tlb), nslabs, true);
io_tlb_default_mem = mem;
swiotlb_print_info();
--
2.31.1.751.gd2f1c929bd-goog
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
