Hi Patrick, Some thoughts about blk-mq:
*(virtio-blk)* - it's activated by default on kernels >= 3.13 on driver virtio-blk - *The blk-mq feature is currently implemented, and enabled by default, in the following drivers: virtio-blk, mtip32xx, nvme, and rbd*. ( https://access.redhat.com/documentation/en-us/red_hat_enter prise_linux/7/html/7.2_release_notes/storage <https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.2_release_notes/storage> ) - can be checked with "cat /sys/block/vda/queue/scheduler", it appears as none - https://serverfault.com/questions/693348/what-does-it-mean- when-linux-has-no-i-o-scheduler <https://serverfault.com/questions/693348/what-does-it-mean-when-linux-has-no-i-o-scheduler> *hosts local disk (scsi-mq)* - with disks "sda" (scsi), either rotational or ssd IT'S NOT activated by default on ubuntu (cat /sys/block/sda/queue/scheduler) - canonical deactivated in >= 3.18 https://bugs.launchpad. net/ubuntu/+source/linux/+bug/1397061 <https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1397061> - suse says it does not suit, for scsi rotational, but it's ok for SSD: https://doc.opensuse.org/documentation/leap/tuning/html /book.sle.tuning/cha.tuning.io.html#cha.tuning.io.scsimq <https://doc.opensuse.org/documentation/leap/tuning/html/book.sle.tuning/cha.tuning.io.html#cha.tuning.io.scsimq> - redhat says: "*The scsi-mq feature is provided as a Technology Preview in Red Hat Enterprise Linux 7.2. To enable scsi-mq, specify scsi_mod.use_blk_mq=y on the kernel command line. The default value is n (disabled).*" (https://access.redhat.com/doc umentation/en-us/red_hat_enterprise_linux/7/html/7.2_release _notes/storage <https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.2_release_notes/storage> ) - how to change it: vi /etc/default/grub: GRUB_CMDLINE_LINUX="scsi_mod.use_blk_mq=1"; update-grub; reboot *ceph (rbd)* - it's activated by default: *The blk-mq feature is currently implemented, and enabled by default, in the following drivers: virtio-blk, mtip32xx, nvme, and rbd*. (https://access.redhat.com/doc umentation/en-us/red_hat_enterprise_linux/7/html/7.2_release _notes/storage <https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.2_release_notes/storage> ) *multipath (device mapper; dm / dm-mpath)* - how to change it: dm_mod.use_blk_mq=y - deactivated by default, how to verify: *To determine whether DM multipath is using blk-mq on a system, cat the file /sys/block/dm-X/dm/use_blk_mq, where dm-X is replaced by the DM multipath device of interest. This file is read-only and reflects what the global value in /sys/module/dm_mod/parameters/use_blk_mq was at the time the request-based DM multipath device was created*. ( https://access.redhat.com/documentation/en-us/red_hat_enter prise_linux/7/html/7.2_release_notes/storage <https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.2_release_notes/storage> ) - I thought it would not make any sense, since iscsi is by definition (network) much slower than SSD/NVMe, which is what blk-mq was generated for, but...:* It may be beneficial to set dm_mod.use_blk_mq=y if the underlying SCSI devices are also using blk-mq, as doing so reduces locking overhead at the DM layer*. (redhat) observations - WARNING low performance https://www.redhat.com/archives/dm-devel/ 2016-February/msg00036.html - request-based device mapper targets planned for 4.1 - now with >= 4.12 linux come with BFQ, a scheduler based en blk-mq We try in our environment several schedulers but we did't notice a real important improvement, in order to justify a global change in the whole environment. But the best thing is to change/test/document an repeat again and again :) Hope it helps Best, *German* 2017-12-11 18:17 GMT-03:00 Patrick Fruh <p...@qloc.de>: > Hi, > > > > after reading a lot about I/O schedulers and performance gains with > blk-mq, I switched to a custom 4.14.5 kernel with CONFIG_SCSI_MQ_DEFAULT > enabled to have blk-mq for all devices on my cluster. > > > > This allows me to use the following schedulers for HDDs and SSDs: > > mq-deadline, kyber, bfq, none > > > > I’ve currently set the HDD scheduler to bfq and the SSD scheduler to none, > however I’m still not sure if this is the best solution performance-wise. > > Does anyone have more experience with this and can maybe give me a > recommendation? I’m not even sure if blk-mq is a good idea for ceph, since > I haven’t really found anything on the topic. > > > > Best, > > Patrick > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com