Hi Stefan,
On 2020/10/13 18:00, Stefan Hajnoczi wrote:
>
> Sorry, I lost track of this on-going email thread.
>
> Thanks for the backtrace. It shows the io_submit call is done while the
> AioContext lock is held. The monitor thread is waiting for the
> IOThread's AioContext lock. vcpus threads can get stuck waiting on the
> big QEMU lock (BQL) that is held by the monitor in the meantime.
>
> Please collect the kernel backtrace for io_submit so we can understand
> why multi-second io_submit latencies happen.
>
> I also suggest trying aio=io_uring to check if Linux io_uring avoids the
> latency problem.
>
The kernel backtrace for io_submit in GUEST is:
guest# ./offcputime -K -p `pgrep -nx fio`
b'finish_task_switch'
b'__schedule'
b'schedule'
b'io_schedule'
b'blk_mq_get_tag'
b'blk_mq_get_request'
b'blk_mq_make_request'
b'generic_make_request'
b'submit_bio'
b'blkdev_direct_IO'
b'generic_file_read_iter'
b'aio_read'
b'io_submit_one'
b'__x64_sys_io_submit'
b'do_syscall_64'
b'entry_SYSCALL_64_after_hwframe'
- fio (1464)
40031912
And Linux io_uring can avoid the latency problem.
Thanks,
Zhenyu
