Re: [RFC PATCH 11/11] powerpc/svm: Increase SWIOTLB buffer size
On Fri, Aug 24, 2018 at 01:25:35PM -0300, Thiago Jung Bauermann wrote:
> From: Anshuman Khandual
>
> SWIOTLB buffer default size (64MB) is not enough for large sequential write
> operations which eventually leads to kernel crash like here.
>
> virtio-pci :00:05.0: swiotlb buffer is full (sz: 327680 bytes)
> virtio-pci :00:05.0: DMA: Out of SW-IOMMU space for 327680 bytes
> Kernel panic - not syncing: DMA: Random memory could be DMA read
> CPU: 12 PID: 3985 Comm: mkfs.ext4 Not tainted 4.18.0-rc4+ #285
> Call Trace:
> [c007d2a27020] [c0cfdffc] dump_stack+0xb0/0xf4 (unreliable)
> [c007d2a27060] [c0112a98] panic+0x140/0x328
> [c007d2a270f0] [c01b4f88] swiotlb_full+0x108/0x130
> [c007d2a27180] [c01b5f6c] swiotlb_map_page+0x25c/0x2c0
> [c007d2a271e0] [c07bfaf8] vring_map_one_sg.isra.0+0x58/0x70
> [c007d2a27200] [c07c08dc] virtqueue_add_sgs+0x1bc/0x690
> [c007d2a272f0] [d42a1280] virtio_queue_rq+0x358/0x4a0 [virtio_blk]
> [c007d2a273d0] [c06b5d68] blk_mq_dispatch_rq_list+0x1f8/0x6d0
> ..
>
> Increase the SWIOTLB size to 1GB on Ultravisor based secure guests.
Gosh, that is huge.
What about making the SWIOTLB be more dynamic? That is expand it's size
dynamically if it can?
>
> Signed-off-by: Anshuman Khandual
> Signed-off-by: Thiago Jung Bauermann
> ---
> arch/powerpc/Kconfig | 5 +
> kernel/dma/swiotlb.c | 5 +
> 2 files changed, 10 insertions(+)
>
> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
> index 1466d1234723..fee7194ce9e4 100644
> --- a/arch/powerpc/Kconfig
> +++ b/arch/powerpc/Kconfig
> @@ -457,6 +457,11 @@ config PPC_SVM
>
>If unsure, say "N".
>
> +config SWIOTLB_DEFAULT_SIZE
> + int "Size of Software I/O TLB buffer (in MiB)"
> + default "1024"
> + depends on PPC_SVM
> +
> config PPC_TRANSACTIONAL_MEM
> bool "Transactional Memory support for POWERPC"
> depends on PPC_BOOK3S_64
> diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
> index 04b68d9dffac..32dc67422d8a 100644
> --- a/kernel/dma/swiotlb.c
> +++ b/kernel/dma/swiotlb.c
> @@ -146,8 +146,13 @@ void swiotlb_set_max_segment(unsigned int val)
> max_segment = rounddown(val, PAGE_SIZE);
> }
>
> +#ifdef CONFIG_SWIOTLB_DEFAULT_SIZE
> +#define IO_TLB_DEFAULT_SIZE ((unsigned long) CONFIG_SWIOTLB_DEFAULT_SIZE <<
> 20)
> +#else
> /* default to 64MB */
> #define IO_TLB_DEFAULT_SIZE (64UL<<20)
> +#endif
> +
> unsigned long swiotlb_size_or_default(void)
> {
> unsigned long size;
>
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 11/11] powerpc/svm: Increase SWIOTLB buffer size
> +config SWIOTLB_DEFAULT_SIZE > + int "Size of Software I/O TLB buffer (in MiB)" > + default "1024" > + depends on PPC_SVM > + This belongs into kernel/dma/Kconfig. That being said I really don't see why this should be a user visible Kconfig. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 11/11] powerpc/svm: Increase SWIOTLB buffer size
Hello Randy, Thanks for your quick review. Randy Dunlap writes: > On 08/24/2018 09:25 AM, Thiago Jung Bauermann wrote: >> From: Anshuman Khandual >> >> SWIOTLB buffer default size (64MB) is not enough for large sequential write >> operations which eventually leads to kernel crash like here. >> >> virtio-pci :00:05.0: swiotlb buffer is full (sz: 327680 bytes) >> virtio-pci :00:05.0: DMA: Out of SW-IOMMU space for 327680 bytes >> Kernel panic - not syncing: DMA: Random memory could be DMA read >> CPU: 12 PID: 3985 Comm: mkfs.ext4 Not tainted 4.18.0-rc4+ #285 >> Call Trace: >> [c007d2a27020] [c0cfdffc] dump_stack+0xb0/0xf4 (unreliable) >> [c007d2a27060] [c0112a98] panic+0x140/0x328 >> [c007d2a270f0] [c01b4f88] swiotlb_full+0x108/0x130 >> [c007d2a27180] [c01b5f6c] swiotlb_map_page+0x25c/0x2c0 >> [c007d2a271e0] [c07bfaf8] vring_map_one_sg.isra.0+0x58/0x70 >> [c007d2a27200] [c07c08dc] virtqueue_add_sgs+0x1bc/0x690 >> [c007d2a272f0] [d42a1280] virtio_queue_rq+0x358/0x4a0 >> [virtio_blk] >> [c007d2a273d0] [c06b5d68] blk_mq_dispatch_rq_list+0x1f8/0x6d0 >> .. >> >> Increase the SWIOTLB size to 1GB on Ultravisor based secure guests. >> >> Signed-off-by: Anshuman Khandual >> Signed-off-by: Thiago Jung Bauermann >> --- >> arch/powerpc/Kconfig | 5 + >> kernel/dma/swiotlb.c | 5 + >> 2 files changed, 10 insertions(+) >> >> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig >> index 1466d1234723..fee7194ce9e4 100644 >> --- a/arch/powerpc/Kconfig >> +++ b/arch/powerpc/Kconfig >> @@ -457,6 +457,11 @@ config PPC_SVM >> >> If unsure, say "N". >> >> +config SWIOTLB_DEFAULT_SIZE >> + int "Size of Software I/O TLB buffer (in MiB)" >> + default "1024" > > I would add a "range" to limit (restrict) how small or large that can be. > E.g.: > > range 16 102400 > > or even smaller for the maximum value... That's an interesting idea. Since this config option is restricted to PPC_SVM it may be possible to find out what reasonable limits would be. I'll have to experiment a bit to find out. Though I can say that 16 MB is too small since even with 64 MB the kernel panics with the simple task of formatting a filesystem... -- Thiago Jung Bauermann IBM Linux Technology Center ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 11/11] powerpc/svm: Increase SWIOTLB buffer size
On 08/24/2018 09:25 AM, Thiago Jung Bauermann wrote:
> From: Anshuman Khandual
>
> SWIOTLB buffer default size (64MB) is not enough for large sequential write
> operations which eventually leads to kernel crash like here.
>
> virtio-pci :00:05.0: swiotlb buffer is full (sz: 327680 bytes)
> virtio-pci :00:05.0: DMA: Out of SW-IOMMU space for 327680 bytes
> Kernel panic - not syncing: DMA: Random memory could be DMA read
> CPU: 12 PID: 3985 Comm: mkfs.ext4 Not tainted 4.18.0-rc4+ #285
> Call Trace:
> [c007d2a27020] [c0cfdffc] dump_stack+0xb0/0xf4 (unreliable)
> [c007d2a27060] [c0112a98] panic+0x140/0x328
> [c007d2a270f0] [c01b4f88] swiotlb_full+0x108/0x130
> [c007d2a27180] [c01b5f6c] swiotlb_map_page+0x25c/0x2c0
> [c007d2a271e0] [c07bfaf8] vring_map_one_sg.isra.0+0x58/0x70
> [c007d2a27200] [c07c08dc] virtqueue_add_sgs+0x1bc/0x690
> [c007d2a272f0] [d42a1280] virtio_queue_rq+0x358/0x4a0 [virtio_blk]
> [c007d2a273d0] [c06b5d68] blk_mq_dispatch_rq_list+0x1f8/0x6d0
> ..
>
> Increase the SWIOTLB size to 1GB on Ultravisor based secure guests.
>
> Signed-off-by: Anshuman Khandual
> Signed-off-by: Thiago Jung Bauermann
> ---
> arch/powerpc/Kconfig | 5 +
> kernel/dma/swiotlb.c | 5 +
> 2 files changed, 10 insertions(+)
>
> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
> index 1466d1234723..fee7194ce9e4 100644
> --- a/arch/powerpc/Kconfig
> +++ b/arch/powerpc/Kconfig
> @@ -457,6 +457,11 @@ config PPC_SVM
>
>If unsure, say "N".
>
> +config SWIOTLB_DEFAULT_SIZE
> + int "Size of Software I/O TLB buffer (in MiB)"
> + default "1024"
I would add a "range" to limit (restrict) how small or large that can be. E.g.:
range 16 102400
or even smaller for the maximum value...
> + depends on PPC_SVM
> +
> config PPC_TRANSACTIONAL_MEM
> bool "Transactional Memory support for POWERPC"
> depends on PPC_BOOK3S_64
> diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
> index 04b68d9dffac..32dc67422d8a 100644
> --- a/kernel/dma/swiotlb.c
> +++ b/kernel/dma/swiotlb.c
> @@ -146,8 +146,13 @@ void swiotlb_set_max_segment(unsigned int val)
> max_segment = rounddown(val, PAGE_SIZE);
> }
>
> +#ifdef CONFIG_SWIOTLB_DEFAULT_SIZE
> +#define IO_TLB_DEFAULT_SIZE ((unsigned long) CONFIG_SWIOTLB_DEFAULT_SIZE <<
> 20)
> +#else
> /* default to 64MB */
> #define IO_TLB_DEFAULT_SIZE (64UL<<20)
> +#endif
> +
> unsigned long swiotlb_size_or_default(void)
> {
> unsigned long size;
>
--
~Randy
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
[RFC PATCH 11/11] powerpc/svm: Increase SWIOTLB buffer size
From: Anshuman Khandual
SWIOTLB buffer default size (64MB) is not enough for large sequential write
operations which eventually leads to kernel crash like here.
virtio-pci :00:05.0: swiotlb buffer is full (sz: 327680 bytes)
virtio-pci :00:05.0: DMA: Out of SW-IOMMU space for 327680 bytes
Kernel panic - not syncing: DMA: Random memory could be DMA read
CPU: 12 PID: 3985 Comm: mkfs.ext4 Not tainted 4.18.0-rc4+ #285
Call Trace:
[c007d2a27020] [c0cfdffc] dump_stack+0xb0/0xf4 (unreliable)
[c007d2a27060] [c0112a98] panic+0x140/0x328
[c007d2a270f0] [c01b4f88] swiotlb_full+0x108/0x130
[c007d2a27180] [c01b5f6c] swiotlb_map_page+0x25c/0x2c0
[c007d2a271e0] [c07bfaf8] vring_map_one_sg.isra.0+0x58/0x70
[c007d2a27200] [c07c08dc] virtqueue_add_sgs+0x1bc/0x690
[c007d2a272f0] [d42a1280] virtio_queue_rq+0x358/0x4a0 [virtio_blk]
[c007d2a273d0] [c06b5d68] blk_mq_dispatch_rq_list+0x1f8/0x6d0
..
Increase the SWIOTLB size to 1GB on Ultravisor based secure guests.
Signed-off-by: Anshuman Khandual
Signed-off-by: Thiago Jung Bauermann
---
arch/powerpc/Kconfig | 5 +
kernel/dma/swiotlb.c | 5 +
2 files changed, 10 insertions(+)
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 1466d1234723..fee7194ce9e4 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -457,6 +457,11 @@ config PPC_SVM
If unsure, say "N".
+config SWIOTLB_DEFAULT_SIZE
+ int "Size of Software I/O TLB buffer (in MiB)"
+ default "1024"
+ depends on PPC_SVM
+
config PPC_TRANSACTIONAL_MEM
bool "Transactional Memory support for POWERPC"
depends on PPC_BOOK3S_64
diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
index 04b68d9dffac..32dc67422d8a 100644
--- a/kernel/dma/swiotlb.c
+++ b/kernel/dma/swiotlb.c
@@ -146,8 +146,13 @@ void swiotlb_set_max_segment(unsigned int val)
max_segment = rounddown(val, PAGE_SIZE);
}
+#ifdef CONFIG_SWIOTLB_DEFAULT_SIZE
+#define IO_TLB_DEFAULT_SIZE ((unsigned long) CONFIG_SWIOTLB_DEFAULT_SIZE << 20)
+#else
/* default to 64MB */
#define IO_TLB_DEFAULT_SIZE (64UL<<20)
+#endif
+
unsigned long swiotlb_size_or_default(void)
{
unsigned long size;
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
[RFC PATCH 11/11] powerpc/svm: Increase SWIOTLB buffer size
From: Anshuman Khandual
SWIOTLB buffer default size (64MB) is not enough for large sequential write
operations which eventually leads to kernel crash like here.
virtio-pci :00:05.0: swiotlb buffer is full (sz: 327680 bytes)
virtio-pci :00:05.0: DMA: Out of SW-IOMMU space for 327680 bytes
Kernel panic - not syncing: DMA: Random memory could be DMA read
CPU: 12 PID: 3985 Comm: mkfs.ext4 Not tainted 4.18.0-rc4+ #285
Call Trace:
[c007d2a27020] [c0cfdffc] dump_stack+0xb0/0xf4 (unreliable)
[c007d2a27060] [c0112a98] panic+0x140/0x328
[c007d2a270f0] [c01b4f88] swiotlb_full+0x108/0x130
[c007d2a27180] [c01b5f6c] swiotlb_map_page+0x25c/0x2c0
[c007d2a271e0] [c07bfaf8] vring_map_one_sg.isra.0+0x58/0x70
[c007d2a27200] [c07c08dc] virtqueue_add_sgs+0x1bc/0x690
[c007d2a272f0] [d42a1280] virtio_queue_rq+0x358/0x4a0 [virtio_blk]
[c007d2a273d0] [c06b5d68] blk_mq_dispatch_rq_list+0x1f8/0x6d0
..
Increase the SWIOTLB size to 1GB on Ultravisor based secure guests.
Signed-off-by: Anshuman Khandual
Signed-off-by: Thiago Jung Bauermann
---
arch/powerpc/Kconfig | 5 +
kernel/dma/swiotlb.c | 5 +
2 files changed, 10 insertions(+)
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 1466d1234723..fee7194ce9e4 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -457,6 +457,11 @@ config PPC_SVM
If unsure, say "N".
+config SWIOTLB_DEFAULT_SIZE
+ int "Size of Software I/O TLB buffer (in MiB)"
+ default "1024"
+ depends on PPC_SVM
+
config PPC_TRANSACTIONAL_MEM
bool "Transactional Memory support for POWERPC"
depends on PPC_BOOK3S_64
diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
index 04b68d9dffac..32dc67422d8a 100644
--- a/kernel/dma/swiotlb.c
+++ b/kernel/dma/swiotlb.c
@@ -146,8 +146,13 @@ void swiotlb_set_max_segment(unsigned int val)
max_segment = rounddown(val, PAGE_SIZE);
}
+#ifdef CONFIG_SWIOTLB_DEFAULT_SIZE
+#define IO_TLB_DEFAULT_SIZE ((unsigned long) CONFIG_SWIOTLB_DEFAULT_SIZE << 20)
+#else
/* default to 64MB */
#define IO_TLB_DEFAULT_SIZE (64UL<<20)
+#endif
+
unsigned long swiotlb_size_or_default(void)
{
unsigned long size;
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
