Public bug reported:
BugLink: https://bugs.launchpad.net/bugs/2064999
[Impact]
On systems with IOMMU enabled, every streaming DMA mapping involves an
IOVA to be allocated and freed. For small mappings, IOVA sizes are
normally cached, so IOVA allocations complete in a reasonable time. For
larger mappings, things can be significantly slower to the point where
softlockups occur due to lock contention on iova_rbtree_lock.
commit 9257b4a206fc ("iommu/iova: introduce per-cpu caching to iova allocation")
introduced a scalable IOVA cache mechanism, that helps increase performance up
to 128kb mappings.
On systems that do larger streaming DMA mappings, e.g. a NVMe device
with:
/sys/block/nvme0n1/queue/max_hw_sectors_kb
2048
The 2048kb mapping takes significantly longer, causing lock contention
on the iova_rbtree_lock, as other resources such as ethernet NICs are
also trying to acquire the lock.
We hit the following soft lockup:
watchdog: BUG: soft lockup - CPU#60 stuck for 24s!
CPU: 60 PID: 608304 Comm: segment-merger- Tainted: P W EL
5.15.0-76-generic #83~20.04.1-Ubuntu
RIP: 0010:_raw_spin_unlock_irqrestore+0x25/0x30
Call Trace:
<IRQ>
fq_flush_timeout+0x82/0xc0
? fq_ring_free+0x170/0x170
call_timer_fn+0x2e/0x120
run_timer_softirq+0x433/0x4c0
? lapic_next_event+0x21/0x30
? clockevents_program_event+0xab/0x130
__do_softirq+0xdd/0x2ee
irq_exit_rcu+0x7d/0xa0
sysvec_apic_timer_interrupt+0x80/0x90
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1b/0x20
RIP: 0010:_raw_spin_unlock_irqrestore+0x25/0x30
...
alloc_iova+0x1d8/0x1f0
alloc_iova_fast+0x5c/0x3a0
iommu_dma_alloc_iova.isra.0+0x128/0x170
? __kmalloc+0x1ab/0x4b0
iommu_dma_map_sg+0x1a4/0x4c0
__dma_map_sg_attrs+0x72/0x80
dma_map_sg_attrs+0xe/0x20
nvme_map_data+0xde/0x800 [nvme]
? recalibrate_cpu_khz+0x10/0x10
? ktime_get+0x46/0xc0
nvme_queue_rq+0xaf/0x1f0 [nvme]
? __update_load_avg_se+0x2a2/0x2c0
__blk_mq_try_issue_directly+0x15b/0x200
blk_mq_request_issue_directly+0x51/0xa0
blk_mq_try_issue_list_directly+0x7f/0xf0
blk_mq_sched_insert_requests+0xa4/0xf0
blk_mq_flush_plug_list+0x103/0x1c0
blk_flush_plug_list+0xe3/0x110
blk_mq_submit_bio+0x29d/0x600
__submit_bio+0x1e5/0x220
? ext4_inode_block_valid+0x9f/0xc0
submit_bio_noacct+0xac/0x2c0
? xa_load+0x61/0xa0
submit_bio+0x50/0x140
ext4_mpage_readpages+0x6a2/0xe20
? __mod_lruvec_page_state+0x6b/0xb0
ext4_readahead+0x37/0x40
read_pages+0x95/0x280
page_cache_ra_unbounded+0x161/0x220
do_page_cache_ra+0x3d/0x50
ondemand_readahead+0x137/0x330
page_cache_async_ra+0xa6/0xd0
filemap_get_pages+0x224/0x660
? filemap_get_pages+0x9e/0x660
filemap_read+0xbe/0x410
generic_file_read_iter+0xe5/0x150
ext4_file_read_iter+0x5b/0x190
new_sync_read+0x110/0x1a0
vfs_read+0x102/0x1a0
ksys_pread64+0x71/0xa0
__x64_sys_pread64+0x1e/0x30
unload_network_ops_symbols+0xc4de/0xf750 [falcon_lsm_pinned_15907]
do_syscall_64+0x5c/0xc0
? do_syscall_64+0x69/0xc0
? do_syscall_64+0x69/0xc0
entry_SYSCALL_64_after_hwframe+0x61/0xcb
A workaround is to disable IOMMU with "iommu=off amd_iommu=off" on the
kernel command line.
[Fix]
The fix is to clamp max_hw_sectors to the optimised IOVA size that still
fits in the cache, so allocation and freeing of the IOVA is faster,
during streaming DMA mapping.
The fix requires two dependency commits, which introduces a function to
find the optimal value, dma_opt_mapping_size().
commit a229cc14f3395311b899e5e582b71efa8dd01df0
Author: John Garry <[email protected]>
Date: Thu Jul 14 19:15:24 2022 +0800
Subject: dma-mapping: add dma_opt_mapping_size()
Link:
https://github.com/torvalds/linux/commit/a229cc14f3395311b899e5e582b71efa8dd01df0
commit 6d9870b7e5def2450e21316515b9efc0529204dd
Author: John Garry <[email protected]>
Date: Thu Jul 14 19:15:25 2022 +0800
Subject: dma-iommu: add iommu_dma_opt_mapping_size()
Link:
https://github.com/torvalds/linux/commit/6d9870b7e5def2450e21316515b9efc0529204dd
The dependencies are present in 6.0-rc1 and later.
The fix itself simply changes max_hw_sectors from dma_max_mapping_size()
to dma_opt_mapping_size(). The fix needs a backport, as setting
dev->ctrl.max_hw_sectors moved from nvme_reset_work() in 5.15 to
nvme_pci_alloc_dev() in later releases.
commit 3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd
Author: Adrian Huang <[email protected]>
Date: Fri Apr 21 16:08:00 2023 +0800
Subject: nvme-pci: clamp max_hw_sectors based on DMA optimized limitation
Link:
https://github.com/torvalds/linux/commit/3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd
The fix is present in 6.4-rc3 and later.
[Testcase]
The system needs to be extremely busy. So busy infact, that we cannot
reproduce it in lab environments, and only in production.
The systems that hit this issue have 64 cores, ~90%+ sustained CPU
usage, ~90%+ sustained memory usage, high disk I/O, and nearly saturated
network throughput with 100Gb NICs.
The NVMe disk MUST have /sys/block/nvme0n1/queue/max_hw_sectors_kb greater than
128kb, in this case, 2048kb.
Leave the system at sustained load until IOVA allocations slow to a halt
and soft or hardlockups occur, waiting for iova_rbtree_lock.
A test kernel is available in the following ppa:
https://launchpad.net/~mruffell/+archive/ubuntu/sf374805-test
If you install the kernel and leave it running, the soft lockups will no
longer occur.
[Where problems could occur]
We are changing the value of max_hw_sectors_kb for NVMe devices, for
systems with IOMMU enabled. For those without IOMMU or IOMMU disabled,
it will remain the same as it is now.
The value is the minimum between the maximum supported by hardware, and
the largest that fits into cache. For some workloads, this might have a
small impact on performance, due to the need to split up larger IOVA
allocations into multiple smaller ones, but there should be a larger net
gain due to IOVA allocations now fitting into the cache, and completing
much faster than a single large one.
If a regression were to occur, users could disable the IOMMU as a
workaround.
** Affects: linux (Ubuntu)
Importance: Undecided
Status: Fix Released
** Affects: linux (Ubuntu Jammy)
Importance: Medium
Assignee: Matthew Ruffell (mruffell)
Status: In Progress
** Tags: jammy sts
** Also affects: linux (Ubuntu Jammy)
Importance: Undecided
Status: New
** Changed in: linux (Ubuntu)
Status: New => Fix Released
** Changed in: linux (Ubuntu Jammy)
Status: New => In Progress
** Changed in: linux (Ubuntu Jammy)
Importance: Undecided => Medium
** Changed in: linux (Ubuntu Jammy)
Assignee: (unassigned) => Matthew Ruffell (mruffell)
** Tags added: jammy sts
** Description changed:
- BugLink: https://bugs.launchpad.net/bugs/
+ BugLink: https://bugs.launchpad.net/bugs/2064999
[Impact]
On systems with IOMMU enabled, every streaming DMA mapping involves an IOVA
to be allocated and freed. For small mappings, IOVA sizes are normally cached,
so IOVA allocations complete in a reasonable time. For larger mappings, things
can be significantly slower to the point where softlockups occur due to lock
contention
on iova_rbtree_lock.
- commit 9257b4a206fc ("iommu/iova: introduce per-cpu caching to iova
allocation")
+ commit 9257b4a206fc ("iommu/iova: introduce per-cpu caching to iova
allocation")
introduced a scalable IOVA cache mechanism, that helps increase performance up
- to 128kb mappings.
+ to 128kb mappings.
On systems that do larger streaming DMA mappings, e.g. a NVMe device
with:
/sys/block/nvme0n1/queue/max_hw_sectors_kb
2048
The 2048kb mapping takes significantly longer, causing lock contention on the
iova_rbtree_lock, as other resources such as ethernet NICs are also trying to
acquire the lock.
We hit the following soft lockup:
watchdog: BUG: soft lockup - CPU#60 stuck for 24s!
CPU: 60 PID: 608304 Comm: segment-merger- Tainted: P W EL
5.15.0-76-generic #83~20.04.1-Ubuntu
RIP: 0010:_raw_spin_unlock_irqrestore+0x25/0x30
Call Trace:
- <IRQ>
- fq_flush_timeout+0x82/0xc0
- ? fq_ring_free+0x170/0x170
- call_timer_fn+0x2e/0x120
- run_timer_softirq+0x433/0x4c0
- ? lapic_next_event+0x21/0x30
- ? clockevents_program_event+0xab/0x130
- __do_softirq+0xdd/0x2ee
- irq_exit_rcu+0x7d/0xa0
- sysvec_apic_timer_interrupt+0x80/0x90
- </IRQ>
- <TASK>
- asm_sysvec_apic_timer_interrupt+0x1b/0x20
+ <IRQ>
+ fq_flush_timeout+0x82/0xc0
+ ? fq_ring_free+0x170/0x170
+ call_timer_fn+0x2e/0x120
+ run_timer_softirq+0x433/0x4c0
+ ? lapic_next_event+0x21/0x30
+ ? clockevents_program_event+0xab/0x130
+ __do_softirq+0xdd/0x2ee
+ irq_exit_rcu+0x7d/0xa0
+ sysvec_apic_timer_interrupt+0x80/0x90
+ </IRQ>
+ <TASK>
+ asm_sysvec_apic_timer_interrupt+0x1b/0x20
RIP: 0010:_raw_spin_unlock_irqrestore+0x25/0x30
...
- alloc_iova+0x1d8/0x1f0
- alloc_iova_fast+0x5c/0x3a0
- iommu_dma_alloc_iova.isra.0+0x128/0x170
- ? __kmalloc+0x1ab/0x4b0
- iommu_dma_map_sg+0x1a4/0x4c0
- __dma_map_sg_attrs+0x72/0x80
- dma_map_sg_attrs+0xe/0x20
- nvme_map_data+0xde/0x800 [nvme]
- ? recalibrate_cpu_khz+0x10/0x10
- ? ktime_get+0x46/0xc0
- nvme_queue_rq+0xaf/0x1f0 [nvme]
- ? __update_load_avg_se+0x2a2/0x2c0
- __blk_mq_try_issue_directly+0x15b/0x200
- blk_mq_request_issue_directly+0x51/0xa0
- blk_mq_try_issue_list_directly+0x7f/0xf0
- blk_mq_sched_insert_requests+0xa4/0xf0
- blk_mq_flush_plug_list+0x103/0x1c0
- blk_flush_plug_list+0xe3/0x110
- blk_mq_submit_bio+0x29d/0x600
- __submit_bio+0x1e5/0x220
- ? ext4_inode_block_valid+0x9f/0xc0
- submit_bio_noacct+0xac/0x2c0
- ? xa_load+0x61/0xa0
- submit_bio+0x50/0x140
- ext4_mpage_readpages+0x6a2/0xe20
- ? __mod_lruvec_page_state+0x6b/0xb0
- ext4_readahead+0x37/0x40
- read_pages+0x95/0x280
- page_cache_ra_unbounded+0x161/0x220
- do_page_cache_ra+0x3d/0x50
- ondemand_readahead+0x137/0x330
- page_cache_async_ra+0xa6/0xd0
- filemap_get_pages+0x224/0x660
- ? filemap_get_pages+0x9e/0x660
- filemap_read+0xbe/0x410
- generic_file_read_iter+0xe5/0x150
- ext4_file_read_iter+0x5b/0x190
- new_sync_read+0x110/0x1a0
- vfs_read+0x102/0x1a0
- ksys_pread64+0x71/0xa0
- __x64_sys_pread64+0x1e/0x30
- unload_network_ops_symbols+0xc4de/0xf750 [falcon_lsm_pinned_15907]
- do_syscall_64+0x5c/0xc0
- ? do_syscall_64+0x69/0xc0
- ? do_syscall_64+0x69/0xc0
- entry_SYSCALL_64_after_hwframe+0x61/0xcb
+ alloc_iova+0x1d8/0x1f0
+ alloc_iova_fast+0x5c/0x3a0
+ iommu_dma_alloc_iova.isra.0+0x128/0x170
+ ? __kmalloc+0x1ab/0x4b0
+ iommu_dma_map_sg+0x1a4/0x4c0
+ __dma_map_sg_attrs+0x72/0x80
+ dma_map_sg_attrs+0xe/0x20
+ nvme_map_data+0xde/0x800 [nvme]
+ ? recalibrate_cpu_khz+0x10/0x10
+ ? ktime_get+0x46/0xc0
+ nvme_queue_rq+0xaf/0x1f0 [nvme]
+ ? __update_load_avg_se+0x2a2/0x2c0
+ __blk_mq_try_issue_directly+0x15b/0x200
+ blk_mq_request_issue_directly+0x51/0xa0
+ blk_mq_try_issue_list_directly+0x7f/0xf0
+ blk_mq_sched_insert_requests+0xa4/0xf0
+ blk_mq_flush_plug_list+0x103/0x1c0
+ blk_flush_plug_list+0xe3/0x110
+ blk_mq_submit_bio+0x29d/0x600
+ __submit_bio+0x1e5/0x220
+ ? ext4_inode_block_valid+0x9f/0xc0
+ submit_bio_noacct+0xac/0x2c0
+ ? xa_load+0x61/0xa0
+ submit_bio+0x50/0x140
+ ext4_mpage_readpages+0x6a2/0xe20
+ ? __mod_lruvec_page_state+0x6b/0xb0
+ ext4_readahead+0x37/0x40
+ read_pages+0x95/0x280
+ page_cache_ra_unbounded+0x161/0x220
+ do_page_cache_ra+0x3d/0x50
+ ondemand_readahead+0x137/0x330
+ page_cache_async_ra+0xa6/0xd0
+ filemap_get_pages+0x224/0x660
+ ? filemap_get_pages+0x9e/0x660
+ filemap_read+0xbe/0x410
+ generic_file_read_iter+0xe5/0x150
+ ext4_file_read_iter+0x5b/0x190
+ new_sync_read+0x110/0x1a0
+ vfs_read+0x102/0x1a0
+ ksys_pread64+0x71/0xa0
+ __x64_sys_pread64+0x1e/0x30
+ unload_network_ops_symbols+0xc4de/0xf750 [falcon_lsm_pinned_15907]
+ do_syscall_64+0x5c/0xc0
+ ? do_syscall_64+0x69/0xc0
+ ? do_syscall_64+0x69/0xc0
+ entry_SYSCALL_64_after_hwframe+0x61/0xcb
A workaround is to disable IOMMU with "iommu=off amd_iommu=off" on the
kernel command line.
[Fix]
The fix is to clamp max_hw_sectors to the optimised IOVA size that still
fits in the cache, so allocation and freeing of the IOVA is faster,
during streaming DMA mapping.
- The fix requires two dependency commits, which introduces a function to find
the optimal value, dma_opt_mapping_size().
-
+ The fix requires two dependency commits, which introduces a function to
+ find the optimal value, dma_opt_mapping_size().
+
commit a229cc14f3395311b899e5e582b71efa8dd01df0
Author: John Garry <[email protected]>
Date: Thu Jul 14 19:15:24 2022 +0800
Subject: dma-mapping: add dma_opt_mapping_size()
Link:
https://github.com/torvalds/linux/commit/a229cc14f3395311b899e5e582b71efa8dd01df0
commit 6d9870b7e5def2450e21316515b9efc0529204dd
Author: John Garry <[email protected]>
Date: Thu Jul 14 19:15:25 2022 +0800
Subject: dma-iommu: add iommu_dma_opt_mapping_size()
Link:
https://github.com/torvalds/linux/commit/6d9870b7e5def2450e21316515b9efc0529204dd
The dependencies are present in 6.0-rc1 and later.
The fix itself simply changes max_hw_sectors from dma_max_mapping_size()
to dma_opt_mapping_size(). The fix needs a backport, as setting
dev->ctrl.max_hw_sectors moved from nvme_reset_work() in 5.15 to
nvme_pci_alloc_dev() in later releases.
commit 3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd
Author: Adrian Huang <[email protected]>
Date: Fri Apr 21 16:08:00 2023 +0800
Subject: nvme-pci: clamp max_hw_sectors based on DMA optimized limitation
Link:
https://github.com/torvalds/linux/commit/3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd
The fix is present in 6.4-rc3 and later.
[Testcase]
The system needs to be extremely busy. So busy infact, that we cannot
reproduce it in lab environments, and only in production.
The systems that hit this issue have 64 cores, ~90%+ sustained CPU
usage, ~90%+ sustained memory usage, high disk I/O, and nearly saturated
network throughput with 100Gb NICs.
The NVMe disk MUST have /sys/block/nvme0n1/queue/max_hw_sectors_kb greater
than
128kb, in this case, 2048kb.
Leave the system at sustained load until IOVA allocations slow to a halt
and soft or hardlockups occur, waiting for iova_rbtree_lock.
A test kernel is available in the following ppa:
https://launchpad.net/~mruffell/+archive/ubuntu/sf374805-test
If you install the kernel and leave it running, the soft lockups will no
longer occur.
[Where problems could occur]
We are changing the value of max_hw_sectors_kb for NVMe devices, for
systems with IOMMU enabled. For those without IOMMU or IOMMU disabled,
it will remain the same as it is now.
The value is the minimum between the maximum supported by hardware, and
the largest that fits into cache. For some workloads, this might have a
small impact on performance, due to the need to split up larger IOVA
allocations into multiple smaller ones, but there should be a larger net
gain due to IOVA allocations now fitting into the cache, and completing
much faster than a single large one.
If a regression were to occur, users could disable the IOMMU as a
workaround.
** Description changed:
BugLink: https://bugs.launchpad.net/bugs/2064999
[Impact]
- On systems with IOMMU enabled, every streaming DMA mapping involves an IOVA
to be allocated and freed. For small mappings, IOVA sizes are normally cached,
so IOVA allocations complete in a reasonable time. For larger mappings, things
can be significantly slower to the point where softlockups occur due to lock
contention
- on iova_rbtree_lock.
+ On systems with IOMMU enabled, every streaming DMA mapping involves an
+ IOVA to be allocated and freed. For small mappings, IOVA sizes are
+ normally cached, so IOVA allocations complete in a reasonable time. For
+ larger mappings, things can be significantly slower to the point where
+ softlockups occur due to lock contention on iova_rbtree_lock.
commit 9257b4a206fc ("iommu/iova: introduce per-cpu caching to iova
allocation")
introduced a scalable IOVA cache mechanism, that helps increase performance up
to 128kb mappings.
On systems that do larger streaming DMA mappings, e.g. a NVMe device
with:
/sys/block/nvme0n1/queue/max_hw_sectors_kb
2048
The 2048kb mapping takes significantly longer, causing lock contention on the
iova_rbtree_lock, as other resources such as ethernet NICs are also trying to
acquire the lock.
We hit the following soft lockup:
watchdog: BUG: soft lockup - CPU#60 stuck for 24s!
CPU: 60 PID: 608304 Comm: segment-merger- Tainted: P W EL
5.15.0-76-generic #83~20.04.1-Ubuntu
RIP: 0010:_raw_spin_unlock_irqrestore+0x25/0x30
Call Trace:
<IRQ>
fq_flush_timeout+0x82/0xc0
? fq_ring_free+0x170/0x170
call_timer_fn+0x2e/0x120
run_timer_softirq+0x433/0x4c0
? lapic_next_event+0x21/0x30
? clockevents_program_event+0xab/0x130
__do_softirq+0xdd/0x2ee
irq_exit_rcu+0x7d/0xa0
sysvec_apic_timer_interrupt+0x80/0x90
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1b/0x20
RIP: 0010:_raw_spin_unlock_irqrestore+0x25/0x30
...
alloc_iova+0x1d8/0x1f0
alloc_iova_fast+0x5c/0x3a0
iommu_dma_alloc_iova.isra.0+0x128/0x170
? __kmalloc+0x1ab/0x4b0
iommu_dma_map_sg+0x1a4/0x4c0
__dma_map_sg_attrs+0x72/0x80
dma_map_sg_attrs+0xe/0x20
nvme_map_data+0xde/0x800 [nvme]
? recalibrate_cpu_khz+0x10/0x10
? ktime_get+0x46/0xc0
nvme_queue_rq+0xaf/0x1f0 [nvme]
? __update_load_avg_se+0x2a2/0x2c0
__blk_mq_try_issue_directly+0x15b/0x200
blk_mq_request_issue_directly+0x51/0xa0
blk_mq_try_issue_list_directly+0x7f/0xf0
blk_mq_sched_insert_requests+0xa4/0xf0
blk_mq_flush_plug_list+0x103/0x1c0
blk_flush_plug_list+0xe3/0x110
blk_mq_submit_bio+0x29d/0x600
__submit_bio+0x1e5/0x220
? ext4_inode_block_valid+0x9f/0xc0
submit_bio_noacct+0xac/0x2c0
? xa_load+0x61/0xa0
submit_bio+0x50/0x140
ext4_mpage_readpages+0x6a2/0xe20
? __mod_lruvec_page_state+0x6b/0xb0
ext4_readahead+0x37/0x40
read_pages+0x95/0x280
page_cache_ra_unbounded+0x161/0x220
do_page_cache_ra+0x3d/0x50
ondemand_readahead+0x137/0x330
page_cache_async_ra+0xa6/0xd0
filemap_get_pages+0x224/0x660
? filemap_get_pages+0x9e/0x660
filemap_read+0xbe/0x410
generic_file_read_iter+0xe5/0x150
ext4_file_read_iter+0x5b/0x190
new_sync_read+0x110/0x1a0
vfs_read+0x102/0x1a0
ksys_pread64+0x71/0xa0
__x64_sys_pread64+0x1e/0x30
unload_network_ops_symbols+0xc4de/0xf750 [falcon_lsm_pinned_15907]
do_syscall_64+0x5c/0xc0
? do_syscall_64+0x69/0xc0
? do_syscall_64+0x69/0xc0
entry_SYSCALL_64_after_hwframe+0x61/0xcb
A workaround is to disable IOMMU with "iommu=off amd_iommu=off" on the
kernel command line.
[Fix]
The fix is to clamp max_hw_sectors to the optimised IOVA size that still
fits in the cache, so allocation and freeing of the IOVA is faster,
during streaming DMA mapping.
The fix requires two dependency commits, which introduces a function to
find the optimal value, dma_opt_mapping_size().
commit a229cc14f3395311b899e5e582b71efa8dd01df0
Author: John Garry <[email protected]>
Date: Thu Jul 14 19:15:24 2022 +0800
Subject: dma-mapping: add dma_opt_mapping_size()
Link:
https://github.com/torvalds/linux/commit/a229cc14f3395311b899e5e582b71efa8dd01df0
commit 6d9870b7e5def2450e21316515b9efc0529204dd
Author: John Garry <[email protected]>
Date: Thu Jul 14 19:15:25 2022 +0800
Subject: dma-iommu: add iommu_dma_opt_mapping_size()
Link:
https://github.com/torvalds/linux/commit/6d9870b7e5def2450e21316515b9efc0529204dd
The dependencies are present in 6.0-rc1 and later.
The fix itself simply changes max_hw_sectors from dma_max_mapping_size()
to dma_opt_mapping_size(). The fix needs a backport, as setting
dev->ctrl.max_hw_sectors moved from nvme_reset_work() in 5.15 to
nvme_pci_alloc_dev() in later releases.
commit 3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd
Author: Adrian Huang <[email protected]>
Date: Fri Apr 21 16:08:00 2023 +0800
Subject: nvme-pci: clamp max_hw_sectors based on DMA optimized limitation
Link:
https://github.com/torvalds/linux/commit/3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd
The fix is present in 6.4-rc3 and later.
[Testcase]
The system needs to be extremely busy. So busy infact, that we cannot
reproduce it in lab environments, and only in production.
The systems that hit this issue have 64 cores, ~90%+ sustained CPU
usage, ~90%+ sustained memory usage, high disk I/O, and nearly saturated
network throughput with 100Gb NICs.
The NVMe disk MUST have /sys/block/nvme0n1/queue/max_hw_sectors_kb greater
than
128kb, in this case, 2048kb.
Leave the system at sustained load until IOVA allocations slow to a halt
and soft or hardlockups occur, waiting for iova_rbtree_lock.
A test kernel is available in the following ppa:
https://launchpad.net/~mruffell/+archive/ubuntu/sf374805-test
If you install the kernel and leave it running, the soft lockups will no
longer occur.
[Where problems could occur]
We are changing the value of max_hw_sectors_kb for NVMe devices, for
systems with IOMMU enabled. For those without IOMMU or IOMMU disabled,
it will remain the same as it is now.
The value is the minimum between the maximum supported by hardware, and
the largest that fits into cache. For some workloads, this might have a
small impact on performance, due to the need to split up larger IOVA
allocations into multiple smaller ones, but there should be a larger net
gain due to IOVA allocations now fitting into the cache, and completing
much faster than a single large one.
If a regression were to occur, users could disable the IOMMU as a
workaround.
** Description changed:
BugLink: https://bugs.launchpad.net/bugs/2064999
[Impact]
On systems with IOMMU enabled, every streaming DMA mapping involves an
IOVA to be allocated and freed. For small mappings, IOVA sizes are
normally cached, so IOVA allocations complete in a reasonable time. For
larger mappings, things can be significantly slower to the point where
softlockups occur due to lock contention on iova_rbtree_lock.
commit 9257b4a206fc ("iommu/iova: introduce per-cpu caching to iova
allocation")
introduced a scalable IOVA cache mechanism, that helps increase performance up
to 128kb mappings.
On systems that do larger streaming DMA mappings, e.g. a NVMe device
with:
/sys/block/nvme0n1/queue/max_hw_sectors_kb
2048
- The 2048kb mapping takes significantly longer, causing lock contention on the
iova_rbtree_lock, as other resources such as ethernet NICs are also trying to
- acquire the lock.
+ The 2048kb mapping takes significantly longer, causing lock contention
+ on the iova_rbtree_lock, as other resources such as ethernet NICs are
+ also trying to acquire the lock.
We hit the following soft lockup:
watchdog: BUG: soft lockup - CPU#60 stuck for 24s!
CPU: 60 PID: 608304 Comm: segment-merger- Tainted: P W EL
5.15.0-76-generic #83~20.04.1-Ubuntu
RIP: 0010:_raw_spin_unlock_irqrestore+0x25/0x30
Call Trace:
<IRQ>
fq_flush_timeout+0x82/0xc0
? fq_ring_free+0x170/0x170
call_timer_fn+0x2e/0x120
run_timer_softirq+0x433/0x4c0
? lapic_next_event+0x21/0x30
? clockevents_program_event+0xab/0x130
__do_softirq+0xdd/0x2ee
irq_exit_rcu+0x7d/0xa0
sysvec_apic_timer_interrupt+0x80/0x90
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1b/0x20
RIP: 0010:_raw_spin_unlock_irqrestore+0x25/0x30
...
alloc_iova+0x1d8/0x1f0
alloc_iova_fast+0x5c/0x3a0
iommu_dma_alloc_iova.isra.0+0x128/0x170
? __kmalloc+0x1ab/0x4b0
iommu_dma_map_sg+0x1a4/0x4c0
__dma_map_sg_attrs+0x72/0x80
dma_map_sg_attrs+0xe/0x20
nvme_map_data+0xde/0x800 [nvme]
? recalibrate_cpu_khz+0x10/0x10
? ktime_get+0x46/0xc0
nvme_queue_rq+0xaf/0x1f0 [nvme]
? __update_load_avg_se+0x2a2/0x2c0
__blk_mq_try_issue_directly+0x15b/0x200
blk_mq_request_issue_directly+0x51/0xa0
blk_mq_try_issue_list_directly+0x7f/0xf0
blk_mq_sched_insert_requests+0xa4/0xf0
blk_mq_flush_plug_list+0x103/0x1c0
blk_flush_plug_list+0xe3/0x110
blk_mq_submit_bio+0x29d/0x600
__submit_bio+0x1e5/0x220
? ext4_inode_block_valid+0x9f/0xc0
submit_bio_noacct+0xac/0x2c0
? xa_load+0x61/0xa0
submit_bio+0x50/0x140
ext4_mpage_readpages+0x6a2/0xe20
? __mod_lruvec_page_state+0x6b/0xb0
ext4_readahead+0x37/0x40
read_pages+0x95/0x280
page_cache_ra_unbounded+0x161/0x220
do_page_cache_ra+0x3d/0x50
ondemand_readahead+0x137/0x330
page_cache_async_ra+0xa6/0xd0
filemap_get_pages+0x224/0x660
? filemap_get_pages+0x9e/0x660
filemap_read+0xbe/0x410
generic_file_read_iter+0xe5/0x150
ext4_file_read_iter+0x5b/0x190
new_sync_read+0x110/0x1a0
vfs_read+0x102/0x1a0
ksys_pread64+0x71/0xa0
__x64_sys_pread64+0x1e/0x30
unload_network_ops_symbols+0xc4de/0xf750 [falcon_lsm_pinned_15907]
do_syscall_64+0x5c/0xc0
? do_syscall_64+0x69/0xc0
? do_syscall_64+0x69/0xc0
entry_SYSCALL_64_after_hwframe+0x61/0xcb
A workaround is to disable IOMMU with "iommu=off amd_iommu=off" on the
kernel command line.
[Fix]
The fix is to clamp max_hw_sectors to the optimised IOVA size that still
fits in the cache, so allocation and freeing of the IOVA is faster,
during streaming DMA mapping.
The fix requires two dependency commits, which introduces a function to
find the optimal value, dma_opt_mapping_size().
commit a229cc14f3395311b899e5e582b71efa8dd01df0
Author: John Garry <[email protected]>
Date: Thu Jul 14 19:15:24 2022 +0800
Subject: dma-mapping: add dma_opt_mapping_size()
Link:
https://github.com/torvalds/linux/commit/a229cc14f3395311b899e5e582b71efa8dd01df0
commit 6d9870b7e5def2450e21316515b9efc0529204dd
Author: John Garry <[email protected]>
Date: Thu Jul 14 19:15:25 2022 +0800
Subject: dma-iommu: add iommu_dma_opt_mapping_size()
Link:
https://github.com/torvalds/linux/commit/6d9870b7e5def2450e21316515b9efc0529204dd
The dependencies are present in 6.0-rc1 and later.
The fix itself simply changes max_hw_sectors from dma_max_mapping_size()
to dma_opt_mapping_size(). The fix needs a backport, as setting
dev->ctrl.max_hw_sectors moved from nvme_reset_work() in 5.15 to
nvme_pci_alloc_dev() in later releases.
commit 3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd
Author: Adrian Huang <[email protected]>
Date: Fri Apr 21 16:08:00 2023 +0800
Subject: nvme-pci: clamp max_hw_sectors based on DMA optimized limitation
Link:
https://github.com/torvalds/linux/commit/3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd
The fix is present in 6.4-rc3 and later.
[Testcase]
The system needs to be extremely busy. So busy infact, that we cannot
reproduce it in lab environments, and only in production.
The systems that hit this issue have 64 cores, ~90%+ sustained CPU
usage, ~90%+ sustained memory usage, high disk I/O, and nearly saturated
network throughput with 100Gb NICs.
The NVMe disk MUST have /sys/block/nvme0n1/queue/max_hw_sectors_kb greater
than
128kb, in this case, 2048kb.
Leave the system at sustained load until IOVA allocations slow to a halt
and soft or hardlockups occur, waiting for iova_rbtree_lock.
A test kernel is available in the following ppa:
https://launchpad.net/~mruffell/+archive/ubuntu/sf374805-test
If you install the kernel and leave it running, the soft lockups will no
longer occur.
[Where problems could occur]
We are changing the value of max_hw_sectors_kb for NVMe devices, for
systems with IOMMU enabled. For those without IOMMU or IOMMU disabled,
it will remain the same as it is now.
The value is the minimum between the maximum supported by hardware, and
the largest that fits into cache. For some workloads, this might have a
small impact on performance, due to the need to split up larger IOVA
allocations into multiple smaller ones, but there should be a larger net
gain due to IOVA allocations now fitting into the cache, and completing
much faster than a single large one.
If a regression were to occur, users could disable the IOMMU as a
workaround.
** Description changed:
BugLink: https://bugs.launchpad.net/bugs/2064999
[Impact]
On systems with IOMMU enabled, every streaming DMA mapping involves an
IOVA to be allocated and freed. For small mappings, IOVA sizes are
normally cached, so IOVA allocations complete in a reasonable time. For
larger mappings, things can be significantly slower to the point where
softlockups occur due to lock contention on iova_rbtree_lock.
commit 9257b4a206fc ("iommu/iova: introduce per-cpu caching to iova
allocation")
- introduced a scalable IOVA cache mechanism, that helps increase performance up
- to 128kb mappings.
+ introduced a scalable IOVA cache mechanism, that helps increase performance
up to 128kb mappings.
On systems that do larger streaming DMA mappings, e.g. a NVMe device
with:
/sys/block/nvme0n1/queue/max_hw_sectors_kb
2048
The 2048kb mapping takes significantly longer, causing lock contention
on the iova_rbtree_lock, as other resources such as ethernet NICs are
also trying to acquire the lock.
We hit the following soft lockup:
watchdog: BUG: soft lockup - CPU#60 stuck for 24s!
CPU: 60 PID: 608304 Comm: segment-merger- Tainted: P W EL
5.15.0-76-generic #83~20.04.1-Ubuntu
RIP: 0010:_raw_spin_unlock_irqrestore+0x25/0x30
Call Trace:
<IRQ>
fq_flush_timeout+0x82/0xc0
? fq_ring_free+0x170/0x170
call_timer_fn+0x2e/0x120
run_timer_softirq+0x433/0x4c0
? lapic_next_event+0x21/0x30
? clockevents_program_event+0xab/0x130
__do_softirq+0xdd/0x2ee
irq_exit_rcu+0x7d/0xa0
sysvec_apic_timer_interrupt+0x80/0x90
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1b/0x20
RIP: 0010:_raw_spin_unlock_irqrestore+0x25/0x30
...
alloc_iova+0x1d8/0x1f0
alloc_iova_fast+0x5c/0x3a0
iommu_dma_alloc_iova.isra.0+0x128/0x170
? __kmalloc+0x1ab/0x4b0
iommu_dma_map_sg+0x1a4/0x4c0
__dma_map_sg_attrs+0x72/0x80
dma_map_sg_attrs+0xe/0x20
nvme_map_data+0xde/0x800 [nvme]
? recalibrate_cpu_khz+0x10/0x10
? ktime_get+0x46/0xc0
nvme_queue_rq+0xaf/0x1f0 [nvme]
? __update_load_avg_se+0x2a2/0x2c0
__blk_mq_try_issue_directly+0x15b/0x200
blk_mq_request_issue_directly+0x51/0xa0
blk_mq_try_issue_list_directly+0x7f/0xf0
blk_mq_sched_insert_requests+0xa4/0xf0
blk_mq_flush_plug_list+0x103/0x1c0
blk_flush_plug_list+0xe3/0x110
blk_mq_submit_bio+0x29d/0x600
__submit_bio+0x1e5/0x220
? ext4_inode_block_valid+0x9f/0xc0
submit_bio_noacct+0xac/0x2c0
? xa_load+0x61/0xa0
submit_bio+0x50/0x140
ext4_mpage_readpages+0x6a2/0xe20
? __mod_lruvec_page_state+0x6b/0xb0
ext4_readahead+0x37/0x40
read_pages+0x95/0x280
page_cache_ra_unbounded+0x161/0x220
do_page_cache_ra+0x3d/0x50
ondemand_readahead+0x137/0x330
page_cache_async_ra+0xa6/0xd0
filemap_get_pages+0x224/0x660
? filemap_get_pages+0x9e/0x660
filemap_read+0xbe/0x410
generic_file_read_iter+0xe5/0x150
ext4_file_read_iter+0x5b/0x190
new_sync_read+0x110/0x1a0
vfs_read+0x102/0x1a0
ksys_pread64+0x71/0xa0
__x64_sys_pread64+0x1e/0x30
unload_network_ops_symbols+0xc4de/0xf750 [falcon_lsm_pinned_15907]
do_syscall_64+0x5c/0xc0
? do_syscall_64+0x69/0xc0
? do_syscall_64+0x69/0xc0
entry_SYSCALL_64_after_hwframe+0x61/0xcb
A workaround is to disable IOMMU with "iommu=off amd_iommu=off" on the
kernel command line.
[Fix]
The fix is to clamp max_hw_sectors to the optimised IOVA size that still
fits in the cache, so allocation and freeing of the IOVA is faster,
during streaming DMA mapping.
The fix requires two dependency commits, which introduces a function to
find the optimal value, dma_opt_mapping_size().
commit a229cc14f3395311b899e5e582b71efa8dd01df0
Author: John Garry <[email protected]>
Date: Thu Jul 14 19:15:24 2022 +0800
Subject: dma-mapping: add dma_opt_mapping_size()
Link:
https://github.com/torvalds/linux/commit/a229cc14f3395311b899e5e582b71efa8dd01df0
commit 6d9870b7e5def2450e21316515b9efc0529204dd
Author: John Garry <[email protected]>
Date: Thu Jul 14 19:15:25 2022 +0800
Subject: dma-iommu: add iommu_dma_opt_mapping_size()
Link:
https://github.com/torvalds/linux/commit/6d9870b7e5def2450e21316515b9efc0529204dd
The dependencies are present in 6.0-rc1 and later.
The fix itself simply changes max_hw_sectors from dma_max_mapping_size()
to dma_opt_mapping_size(). The fix needs a backport, as setting
dev->ctrl.max_hw_sectors moved from nvme_reset_work() in 5.15 to
nvme_pci_alloc_dev() in later releases.
commit 3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd
Author: Adrian Huang <[email protected]>
Date: Fri Apr 21 16:08:00 2023 +0800
Subject: nvme-pci: clamp max_hw_sectors based on DMA optimized limitation
Link:
https://github.com/torvalds/linux/commit/3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd
The fix is present in 6.4-rc3 and later.
[Testcase]
The system needs to be extremely busy. So busy infact, that we cannot
reproduce it in lab environments, and only in production.
The systems that hit this issue have 64 cores, ~90%+ sustained CPU
usage, ~90%+ sustained memory usage, high disk I/O, and nearly saturated
network throughput with 100Gb NICs.
The NVMe disk MUST have /sys/block/nvme0n1/queue/max_hw_sectors_kb greater
than
128kb, in this case, 2048kb.
Leave the system at sustained load until IOVA allocations slow to a halt
and soft or hardlockups occur, waiting for iova_rbtree_lock.
A test kernel is available in the following ppa:
https://launchpad.net/~mruffell/+archive/ubuntu/sf374805-test
If you install the kernel and leave it running, the soft lockups will no
longer occur.
[Where problems could occur]
We are changing the value of max_hw_sectors_kb for NVMe devices, for
systems with IOMMU enabled. For those without IOMMU or IOMMU disabled,
it will remain the same as it is now.
The value is the minimum between the maximum supported by hardware, and
the largest that fits into cache. For some workloads, this might have a
small impact on performance, due to the need to split up larger IOVA
allocations into multiple smaller ones, but there should be a larger net
gain due to IOVA allocations now fitting into the cache, and completing
much faster than a single large one.
If a regression were to occur, users could disable the IOMMU as a
workaround.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2064999
Title:
Prevent soft lockups during IOMMU streaming DMA mapping by limiting
nvme max_hw_sectors_kb to cache optimised size
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2064999/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
