Re: [PATCH] vhost-blk: Add vhost-blk support v6
On Sat, Dec 1, 2012 at 5:33 PM, Asias He wrote: > vhost-blk is an in-kernel virito-blk device accelerator. > > Due to lack of proper in-kernel AIO interface, this version converts > guest's I/O request to bio and use submit_bio() to submit I/O directly. > So this version any supports raw block device as guest's disk image, > e.g. /dev/sda, /dev/ram0. We can add file based image support to > vhost-blk once we have in-kernel AIO interface. There are some work in > progress for in-kernel AIO interface from Dave Kleikamp and Zach Brown: > >http://marc.info/?l=linux-fsdevel=133312234313122 > > Performance evaluation: > - > LKVM: Fio with libaio ioengine on 1 Fusion IO device > IOPS(k)BeforeAfterImprovement > seq-read 107 121 +13.0% > seq-write 130 179 +37.6% > rnd-read 102 122 +19.6% > rnd-write 125 159 +27.0% > > QEMU: Fio with libaio ioengine on 1 Fusion IO device > IOPS(k)BeforeAfterImprovement > seq-read 76123 +61.8% > seq-write 139 173 +24.4% > rnd-read 73120 +64.3% > rnd-write 75156 +108.0% > > QEMU: Fio with libaio ioengine on 1 Ramdisk device > IOPS(k)BeforeAfterImprovement > seq-read 138 437 +216% > seq-write 191 436 +128% > rnd-read 137 426 +210% > rnd-write 140 415 +196% > > QEMU: Fio with libaio ioengine on 8 Ramdisk device > 50% read + 50% write > IOPS(k)BeforeAfterImprovement > randrw 64/64 189/189 +195%/+195% > > Userspace bits: > - > 1) LKVM > The latest vhost-blk userspace bits for kvm tool can be found here: > g...@github.com:asias/linux-kvm.git blk.vhost-blk > > 2) QEMU > The latest vhost-blk userspace prototype for QEMU can be found here: > g...@github.com:asias/qemu.git blk.vhost-blk > > Changes in v6: > - Use inline req_page_list to reduce kmalloc > - Switch to single thread model, thanks mst! > - Wait until requests fired before vhost_blk_flush to be finished > > Changes in v5: > - Do not assume the buffer layout > - Fix wakeup race > > Changes in v4: > - Mark req->status as userspace pointer > - Use __copy_to_user() instead of copy_to_user() in vhost_blk_set_status() > - Add if (need_resched()) schedule() in blk thread > - Kill vhost_blk_stop_vq() and move it into vhost_blk_stop() > - Use vq_err() instead of pr_warn() > - Fail un Unsupported request > - Add flush in vhost_blk_set_features() > > Changes in v3: > - Sending REQ_FLUSH bio instead of vfs_fsync, thanks Christoph! > - Check file passed by user is a raw block device file > > Acked-by: David S. Miller > Signed-off-by: Asias He Hi Asias, Is this still under development or stopped? -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] vhost-blk: Add vhost-blk support v6
On Sat, Dec 1, 2012 at 5:33 PM, Asias He as...@redhat.com wrote: vhost-blk is an in-kernel virito-blk device accelerator. Due to lack of proper in-kernel AIO interface, this version converts guest's I/O request to bio and use submit_bio() to submit I/O directly. So this version any supports raw block device as guest's disk image, e.g. /dev/sda, /dev/ram0. We can add file based image support to vhost-blk once we have in-kernel AIO interface. There are some work in progress for in-kernel AIO interface from Dave Kleikamp and Zach Brown: http://marc.info/?l=linux-fsdevelm=133312234313122 Performance evaluation: - LKVM: Fio with libaio ioengine on 1 Fusion IO device IOPS(k)BeforeAfterImprovement seq-read 107 121 +13.0% seq-write 130 179 +37.6% rnd-read 102 122 +19.6% rnd-write 125 159 +27.0% QEMU: Fio with libaio ioengine on 1 Fusion IO device IOPS(k)BeforeAfterImprovement seq-read 76123 +61.8% seq-write 139 173 +24.4% rnd-read 73120 +64.3% rnd-write 75156 +108.0% QEMU: Fio with libaio ioengine on 1 Ramdisk device IOPS(k)BeforeAfterImprovement seq-read 138 437 +216% seq-write 191 436 +128% rnd-read 137 426 +210% rnd-write 140 415 +196% QEMU: Fio with libaio ioengine on 8 Ramdisk device 50% read + 50% write IOPS(k)BeforeAfterImprovement randrw 64/64 189/189 +195%/+195% Userspace bits: - 1) LKVM The latest vhost-blk userspace bits for kvm tool can be found here: g...@github.com:asias/linux-kvm.git blk.vhost-blk 2) QEMU The latest vhost-blk userspace prototype for QEMU can be found here: g...@github.com:asias/qemu.git blk.vhost-blk Changes in v6: - Use inline req_page_list to reduce kmalloc - Switch to single thread model, thanks mst! - Wait until requests fired before vhost_blk_flush to be finished Changes in v5: - Do not assume the buffer layout - Fix wakeup race Changes in v4: - Mark req-status as userspace pointer - Use __copy_to_user() instead of copy_to_user() in vhost_blk_set_status() - Add if (need_resched()) schedule() in blk thread - Kill vhost_blk_stop_vq() and move it into vhost_blk_stop() - Use vq_err() instead of pr_warn() - Fail un Unsupported request - Add flush in vhost_blk_set_features() Changes in v3: - Sending REQ_FLUSH bio instead of vfs_fsync, thanks Christoph! - Check file passed by user is a raw block device file Acked-by: David S. Miller da...@davemloft.net Signed-off-by: Asias He as...@redhat.com Hi Asias, Is this still under development or stopped? -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] vhost-blk: Add vhost-blk support v6
On 12/06/2012 09:00 PM, Michael S. Tsirkin wrote: > On Sun, Dec 02, 2012 at 09:33:53AM +0800, Asias He wrote: >> diff --git a/drivers/vhost/Kconfig.blk b/drivers/vhost/Kconfig.blk >> new file mode 100644 >> index 000..ff8ab76 >> --- /dev/null >> +++ b/drivers/vhost/Kconfig.blk >> @@ -0,0 +1,10 @@ >> +config VHOST_BLK >> +tristate "Host kernel accelerator for virtio blk (EXPERIMENTAL)" >> +depends on BLOCK && EXPERIMENTAL && m > > > should depend on eventfd as well. Okay, added. -- Asias -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] vhost-blk: Add vhost-blk support v6
On Sun, Dec 02, 2012 at 09:33:53AM +0800, Asias He wrote: > diff --git a/drivers/vhost/Kconfig.blk b/drivers/vhost/Kconfig.blk > new file mode 100644 > index 000..ff8ab76 > --- /dev/null > +++ b/drivers/vhost/Kconfig.blk > @@ -0,0 +1,10 @@ > +config VHOST_BLK > + tristate "Host kernel accelerator for virtio blk (EXPERIMENTAL)" > + depends on BLOCK && EXPERIMENTAL && m should depend on eventfd as well. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] vhost-blk: Add vhost-blk support v6
On Sun, Dec 02, 2012 at 09:33:53AM +0800, Asias He wrote: diff --git a/drivers/vhost/Kconfig.blk b/drivers/vhost/Kconfig.blk new file mode 100644 index 000..ff8ab76 --- /dev/null +++ b/drivers/vhost/Kconfig.blk @@ -0,0 +1,10 @@ +config VHOST_BLK + tristate Host kernel accelerator for virtio blk (EXPERIMENTAL) + depends on BLOCK EXPERIMENTAL m should depend on eventfd as well. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] vhost-blk: Add vhost-blk support v6
On 12/06/2012 09:00 PM, Michael S. Tsirkin wrote: On Sun, Dec 02, 2012 at 09:33:53AM +0800, Asias He wrote: diff --git a/drivers/vhost/Kconfig.blk b/drivers/vhost/Kconfig.blk new file mode 100644 index 000..ff8ab76 --- /dev/null +++ b/drivers/vhost/Kconfig.blk @@ -0,0 +1,10 @@ +config VHOST_BLK +tristate Host kernel accelerator for virtio blk (EXPERIMENTAL) +depends on BLOCK EXPERIMENTAL m should depend on eventfd as well. Okay, added. -- Asias -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH] vhost-blk: Add vhost-blk support v6
vhost-blk is an in-kernel virito-blk device accelerator.
Due to lack of proper in-kernel AIO interface, this version converts
guest's I/O request to bio and use submit_bio() to submit I/O directly.
So this version any supports raw block device as guest's disk image,
e.g. /dev/sda, /dev/ram0. We can add file based image support to
vhost-blk once we have in-kernel AIO interface. There are some work in
progress for in-kernel AIO interface from Dave Kleikamp and Zach Brown:
http://marc.info/?l=linux-fsdevel=133312234313122
Performance evaluation:
-
LKVM: Fio with libaio ioengine on 1 Fusion IO device
IOPS(k)BeforeAfterImprovement
seq-read 107 121 +13.0%
seq-write 130 179 +37.6%
rnd-read 102 122 +19.6%
rnd-write 125 159 +27.0%
QEMU: Fio with libaio ioengine on 1 Fusion IO device
IOPS(k)BeforeAfterImprovement
seq-read 76123 +61.8%
seq-write 139 173 +24.4%
rnd-read 73120 +64.3%
rnd-write 75156 +108.0%
QEMU: Fio with libaio ioengine on 1 Ramdisk device
IOPS(k)BeforeAfterImprovement
seq-read 138 437 +216%
seq-write 191 436 +128%
rnd-read 137 426 +210%
rnd-write 140 415 +196%
QEMU: Fio with libaio ioengine on 8 Ramdisk device
50% read + 50% write
IOPS(k)BeforeAfterImprovement
randrw 64/64 189/189 +195%/+195%
Userspace bits:
-
1) LKVM
The latest vhost-blk userspace bits for kvm tool can be found here:
g...@github.com:asias/linux-kvm.git blk.vhost-blk
2) QEMU
The latest vhost-blk userspace prototype for QEMU can be found here:
g...@github.com:asias/qemu.git blk.vhost-blk
Changes in v6:
- Use inline req_page_list to reduce kmalloc
- Switch to single thread model, thanks mst!
- Wait until requests fired before vhost_blk_flush to be finished
Changes in v5:
- Do not assume the buffer layout
- Fix wakeup race
Changes in v4:
- Mark req->status as userspace pointer
- Use __copy_to_user() instead of copy_to_user() in vhost_blk_set_status()
- Add if (need_resched()) schedule() in blk thread
- Kill vhost_blk_stop_vq() and move it into vhost_blk_stop()
- Use vq_err() instead of pr_warn()
- Fail un Unsupported request
- Add flush in vhost_blk_set_features()
Changes in v3:
- Sending REQ_FLUSH bio instead of vfs_fsync, thanks Christoph!
- Check file passed by user is a raw block device file
Acked-by: David S. Miller
Signed-off-by: Asias He
---
drivers/vhost/Kconfig | 1 +
drivers/vhost/Kconfig.blk | 10 +
drivers/vhost/Makefile| 2 +
drivers/vhost/blk.c | 724 ++
drivers/vhost/blk.h | 8 +
5 files changed, 745 insertions(+)
create mode 100644 drivers/vhost/Kconfig.blk
create mode 100644 drivers/vhost/blk.c
create mode 100644 drivers/vhost/blk.h
diff --git a/drivers/vhost/Kconfig b/drivers/vhost/Kconfig
index 202bba6..acd8038 100644
--- a/drivers/vhost/Kconfig
+++ b/drivers/vhost/Kconfig
@@ -11,4 +11,5 @@ config VHOST_NET
if STAGING
source "drivers/vhost/Kconfig.tcm"
+source "drivers/vhost/Kconfig.blk"
endif
diff --git a/drivers/vhost/Kconfig.blk b/drivers/vhost/Kconfig.blk
new file mode 100644
index 000..ff8ab76
--- /dev/null
+++ b/drivers/vhost/Kconfig.blk
@@ -0,0 +1,10 @@
+config VHOST_BLK
+ tristate "Host kernel accelerator for virtio blk (EXPERIMENTAL)"
+ depends on BLOCK && EXPERIMENTAL && m
+ ---help---
+ This kernel module can be loaded in host kernel to accelerate
+ guest block with virtio_blk. Not to be confused with virtio_blk
+ module itself which needs to be loaded in guest kernel.
+
+ To compile this driver as a module, choose M here: the module will
+ be called vhost_blk.
diff --git a/drivers/vhost/Makefile b/drivers/vhost/Makefile
index a27b053..1a8a4a5 100644
--- a/drivers/vhost/Makefile
+++ b/drivers/vhost/Makefile
@@ -2,3 +2,5 @@ obj-$(CONFIG_VHOST_NET) += vhost_net.o
vhost_net-y := vhost.o net.o
obj-$(CONFIG_TCM_VHOST) += tcm_vhost.o
+obj-$(CONFIG_VHOST_BLK) += vhost_blk.o
+vhost_blk-y := blk.o
diff --git a/drivers/vhost/blk.c b/drivers/vhost/blk.c
new file mode 100644
index 000..e4ca4b6
--- /dev/null
+++ b/drivers/vhost/blk.c
@@ -0,0 +1,724 @@
+/*
+ * Copyright (C) 2011 Taobao, Inc.
+ * Author: Liu Yuan
+ *
+ * Copyright (C) 2012 Red Hat, Inc.
+ * Author: Asias He
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ *
+ * virtio-blk server in host kernel.
+ */
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "vhost.c"
+#include "vhost.h"
+#include "blk.h"
+
+static DEFINE_IDA(vhost_blk_index_ida);
+
+enum {
+ VHOST_BLK_VQ_REQ = 0,
+ VHOST_BLK_VQ_MAX = 1,
+};
+
+struct req_page_list {
+ struct page **pages;
+ int pages_nr;
+};
+
+#define NR_INLINE 16
+
+struct
[PATCH] vhost-blk: Add vhost-blk support v6
vhost-blk is an in-kernel virito-blk device accelerator.
Due to lack of proper in-kernel AIO interface, this version converts
guest's I/O request to bio and use submit_bio() to submit I/O directly.
So this version any supports raw block device as guest's disk image,
e.g. /dev/sda, /dev/ram0. We can add file based image support to
vhost-blk once we have in-kernel AIO interface. There are some work in
progress for in-kernel AIO interface from Dave Kleikamp and Zach Brown:
http://marc.info/?l=linux-fsdevelm=133312234313122
Performance evaluation:
-
LKVM: Fio with libaio ioengine on 1 Fusion IO device
IOPS(k)BeforeAfterImprovement
seq-read 107 121 +13.0%
seq-write 130 179 +37.6%
rnd-read 102 122 +19.6%
rnd-write 125 159 +27.0%
QEMU: Fio with libaio ioengine on 1 Fusion IO device
IOPS(k)BeforeAfterImprovement
seq-read 76123 +61.8%
seq-write 139 173 +24.4%
rnd-read 73120 +64.3%
rnd-write 75156 +108.0%
QEMU: Fio with libaio ioengine on 1 Ramdisk device
IOPS(k)BeforeAfterImprovement
seq-read 138 437 +216%
seq-write 191 436 +128%
rnd-read 137 426 +210%
rnd-write 140 415 +196%
QEMU: Fio with libaio ioengine on 8 Ramdisk device
50% read + 50% write
IOPS(k)BeforeAfterImprovement
randrw 64/64 189/189 +195%/+195%
Userspace bits:
-
1) LKVM
The latest vhost-blk userspace bits for kvm tool can be found here:
g...@github.com:asias/linux-kvm.git blk.vhost-blk
2) QEMU
The latest vhost-blk userspace prototype for QEMU can be found here:
g...@github.com:asias/qemu.git blk.vhost-blk
Changes in v6:
- Use inline req_page_list to reduce kmalloc
- Switch to single thread model, thanks mst!
- Wait until requests fired before vhost_blk_flush to be finished
Changes in v5:
- Do not assume the buffer layout
- Fix wakeup race
Changes in v4:
- Mark req-status as userspace pointer
- Use __copy_to_user() instead of copy_to_user() in vhost_blk_set_status()
- Add if (need_resched()) schedule() in blk thread
- Kill vhost_blk_stop_vq() and move it into vhost_blk_stop()
- Use vq_err() instead of pr_warn()
- Fail un Unsupported request
- Add flush in vhost_blk_set_features()
Changes in v3:
- Sending REQ_FLUSH bio instead of vfs_fsync, thanks Christoph!
- Check file passed by user is a raw block device file
Acked-by: David S. Miller da...@davemloft.net
Signed-off-by: Asias He as...@redhat.com
---
drivers/vhost/Kconfig | 1 +
drivers/vhost/Kconfig.blk | 10 +
drivers/vhost/Makefile| 2 +
drivers/vhost/blk.c | 724 ++
drivers/vhost/blk.h | 8 +
5 files changed, 745 insertions(+)
create mode 100644 drivers/vhost/Kconfig.blk
create mode 100644 drivers/vhost/blk.c
create mode 100644 drivers/vhost/blk.h
diff --git a/drivers/vhost/Kconfig b/drivers/vhost/Kconfig
index 202bba6..acd8038 100644
--- a/drivers/vhost/Kconfig
+++ b/drivers/vhost/Kconfig
@@ -11,4 +11,5 @@ config VHOST_NET
if STAGING
source drivers/vhost/Kconfig.tcm
+source drivers/vhost/Kconfig.blk
endif
diff --git a/drivers/vhost/Kconfig.blk b/drivers/vhost/Kconfig.blk
new file mode 100644
index 000..ff8ab76
--- /dev/null
+++ b/drivers/vhost/Kconfig.blk
@@ -0,0 +1,10 @@
+config VHOST_BLK
+ tristate Host kernel accelerator for virtio blk (EXPERIMENTAL)
+ depends on BLOCK EXPERIMENTAL m
+ ---help---
+ This kernel module can be loaded in host kernel to accelerate
+ guest block with virtio_blk. Not to be confused with virtio_blk
+ module itself which needs to be loaded in guest kernel.
+
+ To compile this driver as a module, choose M here: the module will
+ be called vhost_blk.
diff --git a/drivers/vhost/Makefile b/drivers/vhost/Makefile
index a27b053..1a8a4a5 100644
--- a/drivers/vhost/Makefile
+++ b/drivers/vhost/Makefile
@@ -2,3 +2,5 @@ obj-$(CONFIG_VHOST_NET) += vhost_net.o
vhost_net-y := vhost.o net.o
obj-$(CONFIG_TCM_VHOST) += tcm_vhost.o
+obj-$(CONFIG_VHOST_BLK) += vhost_blk.o
+vhost_blk-y := blk.o
diff --git a/drivers/vhost/blk.c b/drivers/vhost/blk.c
new file mode 100644
index 000..e4ca4b6
--- /dev/null
+++ b/drivers/vhost/blk.c
@@ -0,0 +1,724 @@
+/*
+ * Copyright (C) 2011 Taobao, Inc.
+ * Author: Liu Yuan tailai...@taobao.com
+ *
+ * Copyright (C) 2012 Red Hat, Inc.
+ * Author: Asias He as...@redhat.com
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ *
+ * virtio-blk server in host kernel.
+ */
+
+#include linux/miscdevice.h
+#include linux/module.h
+#include linux/vhost.h
+#include linux/virtio_blk.h
+#include linux/mutex.h
+#include linux/file.h
+#include linux/kthread.h
+#include linux/blkdev.h
+#include linux/llist.h
+
+#include vhost.c
+#include vhost.h
+#include blk.h
+
+static DEFINE_IDA(vhost_blk_index_ida);
+
+enum {
+
