On Wed, 7 Jan 2026, Uladzislau Rezki wrote:
> On Tue, Jan 06, 2026 at 05:59:16PM +0100, Mikulas Patocka wrote:
> >
> >
> > On Tue, 6 Jan 2026, Uladzislau Rezki wrote:
> >
> > > On Tue, Jan 06, 2026 at 04:56:07PM +0100, Mikulas Patocka wrote:
> > > > On the kernel 6.19-rc, I am experiencing 15-second boot stall in a
> > > > virtual machine when probing a virtio-scsi disk:
> > > > [ 1.011641] SCSI subsystem initialized
> > > > [ 1.013972] virtio_scsi virtio6: 16/0/0 default/read/poll queues
> > > > [ 1.015983] scsi host0: Virtio SCSI HBA
> > > > [ 1.019578] ACPI: \_SB_.GSIA: Enabled at IRQ 16
> > > > [ 1.020225] ahci 0000:00:1f.2: AHCI vers 0001.0000, 32 command
> > > > slots, 1.5 Gbps, SATA mode
> > > > [ 1.020228] ahci 0000:00:1f.2: 6/6 ports implemented (port mask 0x3f)
> > > > [ 1.020230] ahci 0000:00:1f.2: flags: 64bit ncq only
> > > > [ 1.024688] scsi host1: ahci
> > > > [ 1.025432] scsi host2: ahci
> > > > [ 1.025966] scsi host3: ahci
> > > > [ 1.026511] scsi host4: ahci
> > > > [ 1.028371] scsi host5: ahci
> > > > [ 1.028918] scsi host6: ahci
> > > > [ 1.029266] ata1: SATA max UDMA/133 abar m4096@0xfea23000 port
> > > > 0xfea23100 irq 16 lpm-pol 1
> > > > [ 1.029305] ata2: SATA max UDMA/133 abar m4096@0xfea23000 port
> > > > 0xfea23180 irq 16 lpm-pol 1
> > > > [ 1.029316] ata3: SATA max UDMA/133 abar m4096@0xfea23000 port
> > > > 0xfea23200 irq 16 lpm-pol 1
> > > > [ 1.029327] ata4: SATA max UDMA/133 abar m4096@0xfea23000 port
> > > > 0xfea23280 irq 16 lpm-pol 1
> > > > [ 1.029341] ata5: SATA max UDMA/133 abar m4096@0xfea23000 port
> > > > 0xfea23300 irq 16 lpm-pol 1
> > > > [ 1.029356] ata6: SATA max UDMA/133 abar m4096@0xfea23000 port
> > > > 0xfea23380 irq 16 lpm-pol 1
> > > > [ 1.118111] scsi 0:0:0:0: Direct-Access QEMU QEMU HARDDISK
> > > > 2.5+ PQ: 0 ANSI: 5
> > > > [ 1.348916] ata1: SATA link down (SStatus 0 SControl 300)
> > > > [ 1.350713] ata2: SATA link down (SStatus 0 SControl 300)
> > > > [ 1.351025] ata6: SATA link down (SStatus 0 SControl 300)
> > > > [ 1.351160] ata5: SATA link down (SStatus 0 SControl 300)
> > > > [ 1.351326] ata3: SATA link down (SStatus 0 SControl 300)
> > > > [ 1.351536] ata4: SATA link down (SStatus 0 SControl 300)
> > > > [ 1.449153] input: ImExPS/2 Generic Explorer Mouse as
> > > > /devices/platform/i8042/serio1/input/input2
> > > > [ 16.483477] sd 0:0:0:0: Power-on or device reset occurred
> > > > [ 16.483691] sd 0:0:0:0: [sda] 2097152 512-byte logical blocks: (1.07
> > > > GB/1.00 GiB)
> > > > [ 16.483762] sd 0:0:0:0: [sda] Write Protect is off
> > > > [ 16.483877] sd 0:0:0:0: [sda] Write cache: enabled, read cache:
> > > > enabled, doesn't support DPO or FUA
> > > > [ 16.569225] sd 0:0:0:0: [sda] Attached SCSI disk
> > > >
> > > > I bisected it and it is caused by the commit 89e1fb7ceffd which
> > > > introduces calls to synchronize_rcu_expedited.
> > > >
> > > > This commit replaces synchronize_rcu_expedited and kfree with a call to
> > > > kfree_rcu_mightsleep, avoiding the 15-second delay.
> > > >
> > > > Signed-off-by: Mikulas Patocka <[email protected]>
> > > > Fixes: 89e1fb7ceffd ("blk-mq: fix potential uaf for 'queue_hw_ctx'")
> > > >
> > > > ---
> > > > block/blk-mq.c | 3 +--
> > > > 1 file changed, 1 insertion(+), 2 deletions(-)
> > > >
> > > > Index: linux-2.6/block/blk-mq.c
> > > > ===================================================================
> > > > --- linux-2.6.orig/block/blk-mq.c 2026-01-06 16:45:11.000000000
> > > > +0100
> > > > +++ linux-2.6/block/blk-mq.c 2026-01-06 16:48:00.000000000 +0100
> > > > @@ -4553,8 +4553,7 @@ static void __blk_mq_realloc_hw_ctxs(str
> > > > * Make sure reading the old queue_hw_ctx from other
> > > > * context concurrently won't trigger uaf.
> > > > */
> > > > - synchronize_rcu_expedited();
> > > > - kfree(hctxs);
> > > > + kfree_rcu_mightsleep(hctxs);
> > > >
> > > I agree, doing freeing that way is not optimal. But kfree_rcu_mightsleep()
> > > also might not work. It has a fallback, if we can not place an object into
> > > "page" due to memory allocation failure, it inlines freeing:
> > >
> > > <snip>
> > > synchronize_rcu();
> > > free().
> > > <snip>
> > >
> > > Please note, synchronize_rcu() can easily be converted into expedited
> > > version. See rcu_gp_is_expedited().
> > >
> > > --
> > > Uladzislau Rezki
> >
> > Would this patch be better? It does GFP_KERNEL allocation which dones't
> > fail in practice.
> >
> > > Inlining is a corner case but it can happen. The best way is to add
> > > rcu_head to the blk_mq_hw_ctx structure and use kfree_rcu(). It never
> > > blocks.
> >
> > We are not protecting the blk_mq_hw_ctx structure with RCU, we are
> > protecting the q->queue_hw_ctx array. So, rcu_head cannot be added to an
> > array. We could cast the array to rcu_head (and make sure that the initial
> > allocation is at least sizeof(struct rcu_head)), but that is hacky.
> >
> > Mikulas
> >
> > ---
> > block/blk-mq.c | 23 +++++++++++++++++++++--
> > 1 file changed, 21 insertions(+), 2 deletions(-)
> >
> > Index: linux-2.6/block/blk-mq.c
> > ===================================================================
> > --- linux-2.6.orig/block/blk-mq.c 2026-01-06 15:55:41.000000000 +0100
> > +++ linux-2.6/block/blk-mq.c 2026-01-06 16:22:40.000000000 +0100
> > @@ -4531,6 +4531,18 @@ static struct blk_mq_hw_ctx *blk_mq_allo
> > return NULL;
> > }
> >
> > +struct rcu_free_hctxs {
> > + struct rcu_head head;
> > + struct blk_mq_hw_ctx **hctxs;
> > +};
> > +
> > +static void rcu_free_hctxs(struct rcu_head *head)
> > +{
> > + struct rcu_free_hctxs *r = container_of(head, struct rcu_free_hctxs,
> > head);
> > + kfree(r->hctxs);
> > + kfree(r);
> > +}
> > +
> > static void __blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
> > struct request_queue *q)
> > {
> > @@ -4539,6 +4551,7 @@ static void __blk_mq_realloc_hw_ctxs(str
> >
> > if (q->nr_hw_queues < set->nr_hw_queues) {
> > struct blk_mq_hw_ctx **new_hctxs;
> > + struct rcu_free_hctxs *r;
> >
> > new_hctxs = kcalloc_node(set->nr_hw_queues,
> > sizeof(*new_hctxs), GFP_KERNEL,
> > @@ -4553,8 +4566,14 @@ static void __blk_mq_realloc_hw_ctxs(str
> > * Make sure reading the old queue_hw_ctx from other
> > * context concurrently won't trigger uaf.
> > */
> > - synchronize_rcu_expedited();
> > - kfree(hctxs);
> > + r = kmalloc(sizeof(struct rcu_free_hctxs), GFP_KERNEL);
> > + if (!r) {
> > + synchronize_rcu_expedited();
> > + kfree(hctxs);
> > + } else {
> > + r->hctxs = hctxs;
> > + call_rcu(&r->head, rcu_free_hctxs);
> > + }
> > hctxs = new_hctxs;
> > }
> >
> > >
> >
> I see. That will work but this looks like a temporary fix. It would be
> great to understand why synchronize_rcu_expedited() is blocked for so long.
> 16 seconds is a way too long.
synchronize_rcu_expedited is called 257 times from the block layer. One
call is approximately 50ms.
This is one of the stacktraces:
[ 3.087639] CPU: 12 UID: 0 PID: 260 Comm: (udev-worker) Not tainted
6.19.0-rc4 #26 PREEMPT_{RT,(full)}
[ 3.087642] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
1.16.2-debian-1.16.2-1 04/01/2014
[ 3.087643] Call Trace:
[ 3.087644] <TASK>
[ 3.087644] dump_stack_lvl+0x47/0x60
[ 3.087649] __blk_mq_realloc_hw_ctxs+0x14e/0x170
[ 3.087652] blk_mq_init_allocated_queue+0xf5/0x3b0
[ 3.087654] blk_mq_alloc_queue+0x62/0xa0
[ 3.087655] scsi_alloc_sdev+0x1e2/0x300 [scsi_mod]
[ 3.087658] scsi_probe_and_add_lun+0x1de/0x280 [scsi_mod]
[ 3.087660] ? preempt_count_add+0x48/0xc0
[ 3.087663] ? rt_spin_unlock+0x47/0xa0
[ 3.087665] ? rt_spin_unlock+0x2d/0xa0
[ 3.087666] __scsi_scan_target+0xd3/0x1d0 [scsi_mod]
[ 3.087667] scsi_scan_channel+0x4f/0x80 [scsi_mod]
[ 3.087668] scsi_scan_host_selected+0xc0/0xf0 [scsi_mod]
[ 3.087670] scsi_scan_host+0x181/0x1a0 [scsi_mod]
[ 3.087671] virtscsi_probe+0x333/0x341 [virtio_scsi]
[ 3.087674] virtio_dev_probe+0x1e5/0x300
[ 3.087676] really_probe+0xb9/0x240
[ 3.087678] __driver_probe_device+0x6e/0x100
[ 3.087679] driver_probe_device+0x1a/0x70
[ 3.087680] ? __device_attach_driver+0xa0/0xa0
[ 3.087681] __driver_attach+0x84/0x140
[ 3.087682] bus_for_each_dev+0x5e/0xa0
[ 3.087683] bus_add_driver+0xd8/0x1c0
[ 3.087684] ? libata_transport_exit+0x930/0x930 [libata]
[ 3.087686] driver_register+0x6c/0xd0
[ 3.087687] virtio_scsi_init+0xa1/0x1000 [virtio_scsi]
[ 3.087688] do_one_initcall+0x35/0x160
[ 3.087690] ? do_init_module+0x1f/0x250
[ 3.087691] ? __kmalloc_cache_noprof+0x183/0x330
[ 3.087694] do_init_module+0x5d/0x250
[ 3.087695] ? init_module_from_file+0x9e/0xc0
[ 3.087695] init_module_from_file+0x9e/0xc0
[ 3.087696] idempotent_init_module+0xee/0x2d0
[ 3.087697] __x64_sys_finit_module+0x56/0xa0
[ 3.087698] do_syscall_64+0x31e/0x370
[ 3.087701] entry_SYSCALL_64_after_hwframe+0x4b/0x53
> Is that easy to reproduce?
I uploaded my config here:
http://www.jikos.cz/~mikulas/testcases/config/.config-6.19-rc4
Mikulas
> --
> Uladzislau Rezki
>
