[Differential] D13995: NVMe controller emulator for bhyve.
grehan abandoned this revision. grehan added a comment. Closing this revision - moving to https://reviews.freebsd.org/D14022 REVISION DETAIL https://reviews.freebsd.org/D13995 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: grehan, trasz, imp, sux2mfgj_gmail.com Cc: chuck, seanc, rgrimes, cem, freebsd-virtualization-list ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Differential] D13995: NVMe controller emulator for bhyve.
grehan commandeered this revision. grehan edited reviewers, added: sux2mfgj_gmail.com; removed: grehan. REVISION DETAIL https://reviews.freebsd.org/D13995 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: grehan, trasz, imp, sux2mfgj_gmail.com Cc: chuck, seanc, rgrimes, cem, freebsd-virtualization-list ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Differential] D13995: NVMe controller emulator for bhyve.
grehan added a comment. Yep, that's correct. The code in the new review is based on this code so it's really a continuation of the effort. REVISION DETAIL https://reviews.freebsd.org/D13995 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: sux2mfgj_gmail.com, grehan, trasz, imp Cc: chuck, seanc, rgrimes, cem, freebsd-virtualization-list ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Differential] D13995: NVMe controller emulator for bhyve.
chuck added a comment. @grehan OK. So is the thought that this review be closed and efforts should focus on https://reviews.freebsd.org/D14022? REVISION DETAIL https://reviews.freebsd.org/D13995 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: sux2mfgj_gmail.com, grehan, trasz, imp Cc: chuck, seanc, rgrimes, cem, freebsd-virtualization-list ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Differential] D13995: NVMe controller emulator for bhyve.
grehan requested changes to this revision. grehan added a comment. This revision now requires changes to proceed. Chuck - the review for this is now in https://reviews.freebsd.org/D14022, where I believe your comments have already been addressed. REVISION DETAIL https://reviews.freebsd.org/D13995 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: sux2mfgj_gmail.com, grehan, trasz, imp Cc: chuck, seanc, rgrimes, cem, freebsd-virtualization-list ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Differential] D13995: NVMe controller emulator for bhyve.
grehan added a comment. The updated version of this code is at https://reviews.freebsd.org/D14022 I've moved the reviewer/subscriber list over as well. REVISION DETAIL https://reviews.freebsd.org/D13995 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: sux2mfgj_gmail.com, grehan, trasz, imp Cc: seanc, rgrimes, cem, freebsd-virtualization-list, chuck ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Differential] D13995: NVMe controller emulator for bhyve.
grehan added a comment. Leon Dang has also been working on NVMe emulation, and his version works with Linux, Windows and UEFI boot. I'll post that code for review since it is a bit more recent and tested. In the meantime, that version can be seen at www.freebsd.org/~grehan/pci_nvme.c (There is also a version of UEFI with NVMe support compiled in at www.freebsd.org/~grehan/BHYVE_NVMe.fd.xz ) REVISION DETAIL https://reviews.freebsd.org/D13995 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: sux2mfgj_gmail.com, grehan, trasz, imp Cc: rgrimes, cem, freebsd-virtualization-list, chuck ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Differential] D13995: NVMe controller emulator for bhyve.
rgrimes added a comment. I did a very quick one time scan down the code just to see what was here, these are my comments about it. None of the functions have block start comments stating what that function does. INLINE COMMENTS > pci_nvme.c:1 > +#include > +#include File is missing a copyright, and a $FreeBSD$, where did this code come from? Who wrote it? > pci_nvme.c:850 > + > + > +int data_size = number_of_lb * logic_block_size; style nit, extra blank line > pci_nvme.c:985 > +/* pci_generate_msix(sc->pi, sq_info->completion_qid); */ > +/* } */ > + Why is this code here and all commented out? #if 0/#endif would make this more readable. REVISION DETAIL https://reviews.freebsd.org/D13995 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: sux2mfgj_gmail.com, grehan, trasz, imp Cc: rgrimes, cem, freebsd-virtualization-list, chuck ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Differential] D13995: NVMe controller emulator for bhyve.
cem added a reviewer: imp.
cem added a comment.
Awesome!
I can't speak to the NVMe emulation itself (I know almost nothing about the
protocol and nothing about Bhyve HW emu interfaces). Maybe Warner can.
INLINE COMMENTS
> nvme.h:127
> enum shn_value {
> +NVME_SHN_NOEFCT = 0x0,
> NVME_SHN_NORMAL = 0x1,
indentation looks off here. is it just phabricator?
> Makefile:65
>
> -CFLAGS+= -I${BHYVE_SYSDIR}/sys/dev/usb/controller
> +CFLAGS+= -I${BHYVE_SYSDIR}/sys/dev/usb/controller
> -I${BHYVE_SYSDIR}/sys/dev/nvme/
>
don't need trailing slash
> pci_nvme.c:19
> +#ifdef NVME_DEBUG
> +static FILE* dbg;
> +#define DPRINTF(format, arg...) \
style nit: `static FILE *dbg;` (asterisk goes on the right side of the space)
> pci_nvme.c:21
> +#define DPRINTF(format, arg...) \
> +do { \
> +fprintf(dbg, format, ##arg); \
style nit: Should be tab indents throughout (can't tell if that's just Phab
making it look like 4 spaces or actually 4 spaces).
REVISION DETAIL
https://reviews.freebsd.org/D13995
EMAIL PREFERENCES
https://reviews.freebsd.org/settings/panel/emailpreferences/
To: sux2mfgj_gmail.com, grehan, trasz, imp
Cc: cem, freebsd-virtualization-list, chuck
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
[Differential] D13995: NVMe controller emulator for bhyve.
sux2mfgj_gmail.com edited the summary of this revision. REVISION DETAIL https://reviews.freebsd.org/D13995 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: sux2mfgj_gmail.com, grehan, trasz Cc: freebsd-virtualization-list, chuck ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Differential] D13995: NVMe controller emulator for bhyve.
sux2mfgj_gmail.com edited the summary of this revision. REVISION DETAIL https://reviews.freebsd.org/D13995 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: sux2mfgj_gmail.com, grehan, trasz Cc: freebsd-virtualization-list, chuck ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Differential] D13995: NVMe controller emulator for bhyve.
sux2mfgj_gmail.com created this revision.
sux2mfgj_gmail.com added reviewers: grehan, trasz.
sux2mfgj_gmail.com added a project: bhyve.
REVISION SUMMARY
This emulator is implemented according to NVMe specification 1.0, and didn't
covered all. However, I tested using `newfs` and `nvmecontrol perf`; These was
cleared. The test was executed manually.
REVISION DETAIL
https://reviews.freebsd.org/D13995
AFFECTED FILES
sys/dev/nvme/nvme.h
usr.sbin/bhyve/Makefile
usr.sbin/bhyve/bhyve.8
usr.sbin/bhyve/pci_nvme.c
EMAIL PREFERENCES
https://reviews.freebsd.org/settings/panel/emailpreferences/
To: sux2mfgj_gmail.com, grehan, trasz
Cc: freebsd-virtualization-list, chuck
diff --git a/usr.sbin/bhyve/pci_nvme.c b/usr.sbin/bhyve/pci_nvme.c
new file mode 100644
--- /dev/null
+++ b/usr.sbin/bhyve/pci_nvme.c
@@ -0,0 +1,1257 @@
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include
+#include
+
+#include
+
+#include "pci_emul.h"
+#include "block_if.h"
+#include "bhyverun.h"
+
+#ifdef NVME_DEBUG
+static FILE* dbg;
+#define DPRINTF(format, arg...) \
+do { \
+fprintf(dbg, format, ##arg); \
+fflush(dbg); \
+} while (0)
+#else
+#define DPRINTF(format, arg...)
+#endif
+
+enum nvme_controller_register_offsets {
+NVME_CR_CAP_LOW = 0x00,
+NVME_CR_CAP_HI = 0x04,
+NVME_CR_VS = 0x08,
+NVME_CR_INTMS = 0x0c,
+NVME_CR_INTMC = 0x10,
+NVME_CR_CC = 0x14,
+NVME_CR_CSTS = 0x1c,
+NVME_CR_NSSR = 0x20,
+NVME_CR_AQA = 0x24,
+NVME_CR_ASQ_LOW = 0x28,
+NVME_CR_ASQ_HI = 0x2c,
+NVME_CR_ACQ_LOW = 0x30,
+NVME_CR_ACQ_HI = 0x34,
+// submission queue 0 tail doorbell (admin)
+NVME_CR_IO_QUEUE_BASE = 0x1000,
+/*
+ * 0x1000 ~ 0x1003 : submission queue 0 head doorbell(admin)
+ * 0x1004 ~ 0x1007 : completion queue 0 tail doorbell(admin)
+ * 0x1008 ~ 0x100b : submission queue 1 head doorbell
+ * 0x100c ~ 0x100f : completion queue 1 tail doorbell
+ */
+NVME_CR_ADMIN_SQ_TAIL = 0x1000,
+NVME_CR_ADMIN_CQ_HEAD = 0x1004,
+NVME_CR_SQ_1_TAIL = 0x1008,
+NVME_CR_CQ_1_HEAD = 0x100c,
+NVME_CR_SIZE = 0x1010,
+};
+
+#define NVME_IO_SQ_NUM 1
+#define NVME_IO_CQ_NUM 1
+
+#define NVME_IO_SQS_SIZE (NVME_IO_SQ_NUM + 1)
+#define NVME_IO_CQS_SIZE (NVME_IO_CQ_NUM + 1)
+
+enum nvme_pci_bar {
+NVME_BAR_CR = 0, // 0 and 1
+NVME_BAR_RSV = 3, // reserved
+NVME_BAR_MSIX = 4, // 4 and 5
+};
+
+/*
+ * NVME_CR_ADMIN_CQ_HEAD and NVMe I/O completion queues
+ */
+#define NVME_COMPLETION_QUEUE_NUM (NVME_IO_CQ_NUM + 1)
+
+enum nvme_cmd_identify_cdw10 {
+NVME_CMD_IDENTIFY_CDW10_CNTID = 0x,
+NVME_CMD_IDENTIFY_CDW10_RSV = 0xff00,
+NVME_CMD_IDENTIFY_CDW10_CNS = 0x00ff,
+};
+
+enum nvme_cmd_identify_data {
+NVME_CMD_IDENTIFY_CNS_NAMESPACE = 0x0,
+NVME_CMD_IDENTIFY_CNS_CONTROLLER = 0x1,
+};
+
+enum nvme_cc_bits {
+NVME_CC_EN = 0x0001,
+NVME_CC_RSV0 = 0x000e,
+NVME_CC_CCS = 0x0070,
+NVME_CC_MPS = 0x0780,
+NVME_CC_AMS = 0x3800,
+NVME_CC_SHN = 0xc000,
+NVME_CC_IOSQES = 0x000f,
+NVME_CC_IOCQES = 0x00f0,
+NVME_CC_RSV1 = 0xff00,
+};
+
+struct nvme_features {
+union {
+uint32_t raw;
+struct {
+uint8_t ab : 2;
+uint8_t reserved : 5;
+uint8_t lpw : 8;
+uint8_t mpw : 8;
+uint8_t hpw : 8;
+} __packed bits;
+} __packed arbitration;
+
+union {
+uint32_t raw;
+struct {
+uint8_t ps : 5;
+uint32_t reserved : 27;
+} __packed bits;
+} __packed power_management;
+
+union {
+uint32_t raw;
+struct {
+uint8_t num : 6;
+uint32_t reserved : 26;
+} __packed bits;
+} __packed lba_range_type;
+
+union {
+uint32_t raw;
+struct {
+uint16_t over;
+uint16_t under;
+} __packed bits;
+} __packed temparture_threshold;
+
+union {
+uint32_t raw;
+struct {
+uint16_t tler;
+uint16_t reserved;
+} __packed bits;
+} __packed error_recovery;
+
+union {
+uint32_t raw;
+struct {
+uint8_t wce : 1;
+uint32_t reserved : 31;
+} __packed bits;
+} __packed volatile_write_cache;
+
+union {
+uint32_t raw;
+struct {
+uint16_t ncqr : 16;
+uint16_t nsqr : 16;
+} __packed bits;
+} __packed num_of_queues;
+
+union {
+uint32_t raw;
+struct {
+uint8_t thr : 8;
+uint8_t time : 8;
+uint16_t reserved : 16;
+} __packed bits;
+} __packed interrupt_coalscing;
+
+union {
+uint32_t raw;
+struct {
+uint16_t iv;
+uint8_t cd : 1;
+uint16_t reserved : 15;
+} __packed bits;
+
+
