[PATCH v1 QEMU CXL modifications for openspdm 1/1] pcie/spdm: PCIe CMA implementation
From: hchkuo
The Data Object Exchange implementation of Component Measurement
and Authentication (CMA). This patch is basically based on Openspdm:
https://github.com/jyao1/openspdm.git.
Openspdm is an emulator composed of an SPDM requester and an SPDM
responder. The requester and responder communicate with each other via
a TCP socket. The Openspdm requester is merged to this patch as a DOE
capability in hw/mem/cxl_type3.c. The "-spdm=" is provided to turn
on/off the CMA capability. Once the option is turned on (-spdm=true) the
CXL device can communicate with Openspdm's responder to get the data
object of SPDM/secured SPDM.
Signed-off-by: hchkuo
Signed-off-by: Chris Browy
---
hw/mem/cxl_type3.c | 31 +++-
hw/pci/Kconfig | 4 +
hw/pci/SpdmEmuCommand.c | 319
hw/pci/meson.build | 1 +
include/hw/cxl/cxl_device.h | 2 +
include/hw/pci/SpdmEmuCommand.h | 21 +++
include/hw/pci/pcie_doe.h | 2 +
7 files changed, 377 insertions(+), 3 deletions(-)
create mode 100644 hw/pci/SpdmEmuCommand.c
create mode 100644 include/hw/pci/SpdmEmuCommand.h
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index 4b4097f..da38f3f 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -16,6 +16,8 @@
#include "hw/pci/msi.h"
#include "hw/pci/msix.h"
+#include "hw/pci/SpdmEmuCommand.h"
+
#define DWORD_BYTE 4
/* This function will be used when cdat file is not specified */
@@ -266,6 +268,9 @@ static uint32_t ct3d_config_read(PCIDevice *pci_dev,
uint32_t addr, int size)
if (pcie_doe_read_config(>doe_comp, addr, size, )) {
return val;
+} else if (ct3d->use_spdm &&
+ pcie_doe_read_config(>doe_spdm, addr, size, )) {
+return val;
} else if (pcie_doe_read_config(>doe_cdat, addr, size, )) {
return val;
}
@@ -278,6 +283,9 @@ static void ct3d_config_write(PCIDevice *pci_dev, uint32_t
addr, uint32_t val,
{
CXLType3Dev *ct3d = CT3(pci_dev);
+if (ct3d->use_spdm) {
+pcie_doe_write_config(>doe_spdm, addr, val, size);
+}
pcie_doe_write_config(>doe_comp, addr, val, size);
pcie_doe_write_config(>doe_cdat, addr, val, size);
pci_default_write_config(pci_dev, addr, val, size);
@@ -472,6 +480,12 @@ static MemoryRegion
*cxl_md_get_memory_region(MemoryDeviceState *md,
return ct3d->cxl_dstate.pmem;
}
+static DOEProtocol doe_spdm_prot[] = {
+{PCI_VENDOR_ID_PCI_SIG, PCI_SIG_DOE_CMA, pcie_doe_spdm_rsp},
+{PCI_VENDOR_ID_PCI_SIG, PCI_SIG_DOE_SECURED_CMA, pcie_doe_spdm_rsp},
+{},
+};
+
static DOEProtocol doe_comp_prot[] = {
{CXL_VENDOR_ID, CXL_DOE_COMPLIANCE, cxl_doe_compliance_rsp},
{},
@@ -489,7 +503,7 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)
ComponentRegisters *regs = _cstate->crb;
MemoryRegion *mr = >component_registers;
uint8_t *pci_conf = pci_dev->config;
-unsigned short msix_num = 2;
+unsigned short msix_num = 3;
int i;
if (!ct3d->cxl_dstate.pmem) {
@@ -528,13 +542,22 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)
}
/* DOE Initailization */
-pcie_doe_init(pci_dev, >doe_comp, 0x160, doe_comp_prot, true, 0);
-pcie_doe_init(pci_dev, >doe_cdat, 0x190, doe_cdat_prot, true, 1);
+if (ct3d->use_spdm) {
+spdm_sock_init(errp);
+pcie_doe_init(pci_dev, >doe_spdm, 0x160, doe_spdm_prot, true, 2);
+}
+pcie_doe_init(pci_dev, >doe_comp, 0x190, doe_comp_prot, true, 1);
+pcie_doe_init(pci_dev, >doe_cdat, 0x1b0, doe_cdat_prot, true, 0);
cxl_cstate->cdat.build_cdat_table = build_default_cdat_table;
cxl_doe_cdat_init(cxl_cstate, errp);
}
+static void ct3_exit(PCIDevice *pci_dev)
+{
+spdm_sock_fini();
+}
+
static uint64_t cxl_md_get_addr(const MemoryDeviceState *md)
{
CXLType3Dev *ct3d = CT3(md);
@@ -570,6 +593,7 @@ static Property ct3_props[] = {
DEFINE_PROP_LINK("lsa", CXLType3Dev, lsa, TYPE_MEMORY_BACKEND,
HostMemoryBackend *),
DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename),
+DEFINE_PROP_BOOL("spdm", CXLType3Dev, use_spdm, false),
DEFINE_PROP_END_OF_LIST(),
};
@@ -658,6 +682,7 @@ static void ct3_class_init(ObjectClass *oc, void *data)
CXLType3Class *cvc = CXL_TYPE3_DEV_CLASS(oc);
pc->realize = ct3_realize;
+pc->exit = ct3_exit;
pc->class_id = PCI_CLASS_STORAGE_EXPRESS;
pc->vendor_id = PCI_VENDOR_ID_INTEL;
pc->device_id = 0xd93; /* LVF for now */
diff --git a/hw/pci/Kconfig b/hw/pci/Kconfig
index 77f8b00..181495e 100644
--- a/hw/pci/Kconfig
+++ b/hw/pci/Kconfig
@@ -13,3 +13,7 @@ config MSI_NONBROKEN
# or support it and have a good implementation. See commit
# 47d2b0f33c664533b8dbd5cb17faa8e6a01afe1f.
boo
[PATCH v1 QEMU CXL modifications for openspdm 0/1] Testing PCIe DOE in QEMU CXL/PCIe Device using openspdm
This patch series provides an implementation of the the Data Object Exchange (DOE) for Component Measurement and Authentication (CMA) of the Security Protocol and Data Model (SPDM). This patch is based on [1] [PATCH v1 openspdm on QEMU CXL/PCIe Device 0/2] Testing PCIe DOE in QEMU CXL/PCIe Device using openspdm https://lore.kernel.org/qemu-devel/1624665280-3595-1-git-send-email-cbr...@avery-design.com/T/#u [2] QEMU DOE: [PATCH v6 cxl2.0-v6-doe 0/6] QEMU PCIe DOE for PCIe 4.0/5.0 and CXL 2.0 https://lore.kernel.org/qemu-devel/162332-15662-1-git-send-email-cbr...@avery-design.com/ Openspdm is an emulator composed of an SPDM requester and an SPDM responder. The default SpdmEmu usage have the requester and responder communicate with each other via a TCP socket. However to test PCIe DOE support in QEMU PCIe/CXL device directly, the openspdm requester is modified to use pwrite/pread for MMIO access to the QEMU CXL Device DOE capability (hw/mem/cxl_type3.c). The openspdm requester is run as user application targeting the CXL Device. Follow the readme.md under [1] to build this enhanced version of openspdm. The QEMU CXL device is extended usig the "-spdm=" option to turn on/off the DOE/CMA capability. Once the option is turned on (-spdm=true) the CXL device can communicate with Openspdm's responder to get the data object of SPDM/secured SPDM. QEMU and SPDM responder communicate over client-server method. Build the QEMU with patch series applied to [2]. Now run the system as follows: 1. Start Responder process: ./openspdm/Build/RELEASE_GCC/X64/SpdmResponderEmu --trans PCI_DOE 2. Start QEMU process: qemu-system-x86_64 \ -nic user,hostfwd=tcp::-:22 \ -machine type=pc-q35-4.0 \ -smp 8,sockets=2,cores=2,threads=2 \ -m 4G \ -boot order=d \ -k 'en-us' \ -vga virtio \ -drive file=,format=qcow2 \ -drive if=pflash,format=raw,readonly=on,file= \ -drive if=pflash,format=raw,file= \ -object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxl-mem2,size=1K \ -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxl-mem,size=512M \ -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=0,len-window-base=1,window-base[0]=0x4c000,memdev[0]=cxl-mem1 \ -device cxl-rp,id=rp0,bus=cxl.0,addr=0.0,chassis=0,slot=0 \ -device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M,lsa=cxl-mem2,spdm=true 3. Next copy the openspdm build into QEMU qcow scp -rP openspdm qemu@localhost:. 4. Next ssh to QEMU emulator and execute the requester user application sudo ./openspdm/Build/RELEASE_GCC/X64/SpdmRequesterEmu --trans PCI_DOE -s 35:00.0 hchkuo (1): pcie/spdm: PCIe CMA implementation hw/mem/cxl_type3.c | 31 +++- hw/pci/Kconfig | 4 + hw/pci/SpdmEmuCommand.c | 319 hw/pci/meson.build | 1 + include/hw/cxl/cxl_device.h | 2 + include/hw/pci/SpdmEmuCommand.h | 21 +++ include/hw/pci/pcie_doe.h | 2 + 7 files changed, 377 insertions(+), 3 deletions(-) create mode 100644 hw/pci/SpdmEmuCommand.c create mode 100644 include/hw/pci/SpdmEmuCommand.h -- 1.8.3.1
[PATCH v1 openspdm on QEMU CXL/PCIe Device 2/2] requester: Modified for QEMU emulation
From: hchkuo
The requester should be used as a PCIe app to access the SPDM object in
the PCEe device.
Signed-off-by: Chris Browy
---
Include/IndustryStandard/PciDoeBinding.h| 27 +++
SpdmEmu/SpdmEmuCommon/SpdmEmu.c | 85 ++
SpdmEmu/SpdmEmuCommon/SpdmEmu.h | 7 ++
SpdmEmu/SpdmRequesterEmu/SpdmRequester.c| 100 +++--
SpdmEmu/SpdmRequesterEmu/SpdmRequesterEmu.c | 109 +---
SpdmEmu/SpdmRequesterEmu/SpdmRequesterEmu.h | 5 ++
6 files changed, 300 insertions(+), 33 deletions(-)
diff --git a/Include/IndustryStandard/PciDoeBinding.h
b/Include/IndustryStandard/PciDoeBinding.h
index c7cd7d3..6d199f7 100644
--- a/Include/IndustryStandard/PciDoeBinding.h
+++ b/Include/IndustryStandard/PciDoeBinding.h
@@ -29,6 +29,33 @@ typedef struct {
//UINT32 DataObjectDW[Length];
} PCI_DOE_DATA_OBJECT_HEADER;
+/* Extended Capabilities (PCI-X 2.0 and Express) */
+#define PCI_EXT_CAP_ID(header) (header & 0x)
+#define PCI_EXT_CAP_VER(header) ((header >> 16) & 0xf)
+#define PCI_EXT_CAP_NEXT(header)((header >> 20) & 0xffc)
+
+#define PCIE_EXT_CAP_OFFSET 0x100
+#define PCI_EXT_CAP_ID_DOE 0x2e/* Data Object Exchange */
+
+/* DOE Capabilities Register */
+#define PCIE_DOE_CAP0x04
+#define PCIE_DOE_CAP_INTR_SUPP 0x0001
+/* DOE Control Register */
+#define PCIE_DOE_CTRL 0x08
+#define PCIE_DOE_CTRL_ABORT0x0001
+#define PCIE_DOE_CTRL_INTR_EN 0x0002
+#define PCIE_DOE_CTRL_GO 0x8000
+/* DOE Status Register */
+#define PCIE_DOE_STATUS 0x0c
+#define PCIE_DOE_STATUS_BUSY 0x0001
+#define PCIE_DOE_STATUS_INTR 0x0002
+#define PCIE_DOE_STATUS_ERR0x0004
+#define PCIE_DOE_STATUS_DO_RDY 0x8000
+/* DOE Write Data Mailbox Register */
+#define PCIE_DOE_WR_DATA_MBOX 0x10
+/* DOE Read Data Mailbox Register */
+#define PCIE_DOE_RD_DATA_MBOX 0x14
+
#define PCI_DOE_VENDOR_ID_PCISIG 0x0001
#define PCI_DOE_DATA_OBJECT_TYPE_DOE_DISCOVERY 0x00
diff --git a/SpdmEmu/SpdmEmuCommon/SpdmEmu.c b/SpdmEmu/SpdmEmuCommon/SpdmEmu.c
index b9a4311..768a8b0 100644
--- a/SpdmEmu/SpdmEmuCommon/SpdmEmu.c
+++ b/SpdmEmu/SpdmEmuCommon/SpdmEmu.c
@@ -32,6 +32,11 @@ UINT32 mExeSession = (0 |
EXE_SESSION_MEAS |
0);
+extern struct pcie_dev dev;
+extern char filename[41];
+struct pcie_dev dev = {0};
+char filename[41];
+
VOID
PrintUsage (
IN CHAR8* Name
@@ -364,6 +369,64 @@ Done:
return Ret;
}
+/* Ref: pciutils/lib/filter.c */
+/* Slot filter syntax: [[[domain]:][bus]:][slot][.[func]] */
+static char *pci_filter_parse_slot(struct pcie_dev *f, char *str)
+{
+char *colon = strrchr(str, ':');
+char *dot = strchr((colon ? colon + 1 : str), '.');
+char *mid = str;
+char *e, *bus, *colon2;
+
+if (colon) {
+*colon++ = 0;
+mid = colon;
+colon2 = strchr(str, ':');
+
+if (colon2) {
+*colon2++ = 0;
+bus = colon2;
+if (str[0] && strcmp(str, "*")) {
+long int x = strtol(str, , 16);
+if ((e && *e) || (x < 0 || x > 0x7fff)) {
+return "Invalid domain number";
+}
+f->domain = x;
+}
+} else
+bus = str;
+
+if (bus[0] && strcmp(bus, "*")) {
+long int x = strtol(bus, , 16);
+if ((e && *e) || (x < 0 || x > 0xff)) {
+return "Invalid bus number";
+}
+f->bus = x;
+}
+}
+
+if (dot) {
+*dot++ = 0;
+}
+
+if (mid[0] && strcmp(mid, "*")) {
+long int x = strtol(mid, , 16);
+if ((e && *e) || (x < 0 || x > 0x1f)) {
+return "Invalid slot number";
+}
+f->slot = x;
+}
+
+if (dot && dot[0] && strcmp(dot, "*")) {
+long int x = strtol(dot, , 16);
+if ((e && *e) || (x < 0 || x > 7)) {
+return "Invalid function number";
+}
+f->func = x;
+}
+return NULL;
+}
+
void
ProcessArgs (
char *ProgramName,
@@ -373,6 +436,7 @@ ProcessArgs (
{
UINT32 Data32;
CHAR8 *PcapFileName;
+ char *err;
PcapFileName = NULL;
@@ -854,6 +918,27 @@ ProcessArgs (
}
}
+if (strcmp (argv[0], "-s") == 0) {
+ if (argc >= 2) {
+err = pci_filter_parse_slot(, argv[1]);
+if (err) {
+ printf("%s\n", err);
+ PrintUsage (ProgramName);
+ exit (0);
+}
+sprintf(filename, "/sys/bus/pci/devices/%04x:%02x:%02x.%01x/config",
+dev.domain, dev.bus, dev.slot, dev.func);
+
+
[PATCH v1 openspdm on QEMU CXL/PCIe Device 1/2] build: gcc to CC in GNUMakefile
From: hchkuo
Modified gcc to CC in GNUMakefile, so that we can specify the gcc version
Signed-off-by: Chris Browy
---
GNUmakefile.Flags | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/GNUmakefile.Flags b/GNUmakefile.Flags
index 3586284..33baceb 100644
--- a/GNUmakefile.Flags
+++ b/GNUmakefile.Flags
@@ -114,9 +114,9 @@ else ifeq ("$(TOOLCHAIN)","GCC")
CMOCKA_FLAGS = -std=gnu99 -Wpedantic -Wall -Wshadow -Wmissing-prototypes
-Wcast-align -Werror=address -Wstrict-prototypes -Werror=strict-prototypes
-Wwrite-strings -Werror=write-strings -Werror-implicit-function-declaration
-Wpointer-arith -Werror=pointer-arith -Wdeclaration-after-statement
-Werror=declaration-after-statement -Wreturn-type -Werror=return-type
-Wuninitialized -Werror=uninitialized -Werror=strict-overflow
-Wstrict-overflow=2 -Wno-format-zero-length -Wmissing-field-initializers
-Wformat-security -Werror=format-security -fno-common -Wformat -fno-common
-fstack-protector-strong
CC_FLAGS += --coverage -fprofile-arcs -ftest-coverage
-SLINK = gcc-ar
+SLINK = $(CC)-ar
-DLINK = gcc
+DLINK = $(CC)
DLINK_OBJECT_FILES = -Wl,--start-group,@$(OUTPUT_DIR)/tmp.list,--end-group
DLINK_FLAGS = -o $(BIN_DIR)/$(BASE_NAME) -flto -L/usr/X11R6/lib
DLINK_FLAGS2 = -Wno-error -no-pie
--
1.8.3.1
[PATCH v1 openspdm on QEMU CXL/PCIe Device 0/2] Testing PCIe DOE in QEMU CXL/PCIe Device using openspdm
This patch series provides an implementation of the the Data Object Exchange (DOE) for Component Measurement and Authentication (CMA) of the Security Protocol and Data Model (SPDM). This patch is based on [1] Openspdm: https://github.com/jyao1/openspdm.git Openspdm is an emulator composed of an SPDM requester and an SPDM responder. The default SpdmEmu usage have the requester and responder communicate with each other via a TCP socket. However to test PCIe DOE support in QEMU PCIe/CXL device directly, the openspdm requester is modified to use pwrite/pread for MMIO access to the QEMU CXL Device DOE capability (hw/mem/cxl_type3.c). The openspdm requester is run as user application targeting the CXL Device. Follow the readme.md under [1] to build this enhanced version of openspdm. This patch series is to be used with a subsequent QEMU patch series to be concurrently with this patch series. Full instructions will be included there. hchkuo (2): build: gcc to CC in GNUMakefile requester: Modified for QEMU emulation GNUmakefile.Flags | 4 +- Include/IndustryStandard/PciDoeBinding.h| 27 +++ SpdmEmu/SpdmEmuCommon/SpdmEmu.c | 85 ++ SpdmEmu/SpdmEmuCommon/SpdmEmu.h | 7 ++ SpdmEmu/SpdmRequesterEmu/SpdmRequester.c| 100 +++-- SpdmEmu/SpdmRequesterEmu/SpdmRequesterEmu.c | 109 +--- SpdmEmu/SpdmRequesterEmu/SpdmRequesterEmu.h | 5 ++ 7 files changed, 302 insertions(+), 35 deletions(-) -- 1.8.3.1
[PATCH v6 cxl2.0-v6-doe 4/6] cxl/compliance: CXL Compliance Data Object Exchange implementation
From: hchkuo
The Data Object Exchange implementation of CXL Compliance Mode is
referring to "Compute Express Link (CXL) Specification, Rev. 2.0, Oct.
2020".
The data structure of CXL compliance request and response is added to
the header. Due to the scope limitation of QEMU, most of the compliance
response is limited to returning corresponding length.
A DOE capability of CXL Compliance is added to hw/mem/cxl_type3.c with
capability offset 0x160. The config read/write to this capability range
can be generated in the OS to request the Compliance info.
Signed-off-by: hchkuo
Signed-off-by: Chris Browy
---
hw/mem/cxl_type3.c | 147
include/hw/cxl/cxl_compliance.h | 293
include/hw/cxl/cxl_component.h | 3 +
include/hw/cxl/cxl_device.h | 3 +
include/hw/cxl/cxl_pci.h| 1 +
5 files changed, 447 insertions(+)
create mode 100644 include/hw/cxl/cxl_compliance.h
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index bf33ddb915..569872eb36 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -13,6 +13,134 @@
#include "qemu/rcu.h"
#include "sysemu/hostmem.h"
#include "hw/cxl/cxl.h"
+#include "hw/pci/msi.h"
+#include "hw/pci/msix.h"
+
+#define DWORD_BYTE 4
+
+bool cxl_doe_compliance_rsp(DOECap *doe_cap)
+{
+CompRsp *rsp = (doe_cap->pdev)->cxl_cstate.compliance.response;
+struct compliance_req_header *req = pcie_doe_get_write_mbox_ptr(doe_cap);
+uint32_t type, req_len = 0, rsp_len = 0;
+
+type = req->req_code;
+
+switch (type) {
+case CXL_COMP_MODE_CAP:
+req_len = sizeof(struct cxl_compliance_cap_req);
+rsp_len = sizeof(struct cxl_compliance_cap_rsp);
+rsp->cap_rsp.status = 0x0;
+rsp->cap_rsp.available_cap_bitmask = 0;
+rsp->cap_rsp.enabled_cap_bitmask = 0;
+break;
+case CXL_COMP_MODE_STATUS:
+req_len = sizeof(struct cxl_compliance_status_req);
+rsp_len = sizeof(struct cxl_compliance_status_rsp);
+rsp->status_rsp.cap_bitfield = 0;
+rsp->status_rsp.cache_size = 0;
+rsp->status_rsp.cache_size_units = 0;
+break;
+case CXL_COMP_MODE_HALT:
+req_len = sizeof(struct cxl_compliance_halt_req);
+rsp_len = sizeof(struct cxl_compliance_halt_rsp);
+break;
+case CXL_COMP_MODE_MULT_WR_STREAM:
+req_len = sizeof(struct cxl_compliance_multi_write_streaming_req);
+rsp_len = sizeof(struct cxl_compliance_multi_write_streaming_rsp);
+break;
+case CXL_COMP_MODE_PRO_CON:
+req_len = sizeof(struct cxl_compliance_producer_consumer_req);
+rsp_len = sizeof(struct cxl_compliance_producer_consumer_rsp);
+break;
+case CXL_COMP_MODE_BOGUS:
+req_len = sizeof(struct cxl_compliance_bogus_writes_req);
+rsp_len = sizeof(struct cxl_compliance_bogus_writes_rsp);
+break;
+case CXL_COMP_MODE_INJ_POISON:
+req_len = sizeof(struct cxl_compliance_inject_poison_req);
+rsp_len = sizeof(struct cxl_compliance_inject_poison_rsp);
+break;
+case CXL_COMP_MODE_INJ_CRC:
+req_len = sizeof(struct cxl_compliance_inject_crc_req);
+rsp_len = sizeof(struct cxl_compliance_inject_crc_rsp);
+break;
+case CXL_COMP_MODE_INJ_FC:
+req_len = sizeof(struct cxl_compliance_inject_flow_ctrl_req);
+rsp_len = sizeof(struct cxl_compliance_inject_flow_ctrl_rsp);
+break;
+case CXL_COMP_MODE_TOGGLE_CACHE:
+req_len = sizeof(struct cxl_compliance_toggle_cache_flush_req);
+rsp_len = sizeof(struct cxl_compliance_toggle_cache_flush_rsp);
+break;
+case CXL_COMP_MODE_INJ_MAC:
+req_len = sizeof(struct cxl_compliance_inject_mac_delay_req);
+rsp_len = sizeof(struct cxl_compliance_inject_mac_delay_rsp);
+break;
+case CXL_COMP_MODE_INS_UNEXP_MAC:
+req_len = sizeof(struct cxl_compliance_insert_unexp_mac_req);
+rsp_len = sizeof(struct cxl_compliance_insert_unexp_mac_rsp);
+break;
+case CXL_COMP_MODE_INJ_VIRAL:
+req_len = sizeof(struct cxl_compliance_inject_viral_req);
+rsp_len = sizeof(struct cxl_compliance_inject_viral_rsp);
+break;
+case CXL_COMP_MODE_INJ_ALMP:
+req_len = sizeof(struct cxl_compliance_inject_almp_req);
+rsp_len = sizeof(struct cxl_compliance_inject_almp_rsp);
+break;
+case CXL_COMP_MODE_IGN_ALMP:
+req_len = sizeof(struct cxl_compliance_ignore_almp_req);
+rsp_len = sizeof(struct cxl_compliance_ignore_almp_rsp);
+break;
+case CXL_COMP_MODE_INJ_BIT_ERR:
+req_len = sizeof(struct cxl_compliance_inject_bit_err_in_flit_req);
+rsp_len = sizeof(struct cxl_compliance_inject_bit_err_in_flit_rsp);
+break;
+default:
+break;
+}
+
+/* Discard if request length mismatched */
+if
[PATCH v6 cxl2.0-v6-doe 6/6] test/cdat: CXL CDAT test data
From: hchkuo
Pre-built CDAT table for testing, contains one CDAT header and six
CDAT entries: DSMAS, DSLBIS, DSMSCIS, DSIS, DSEMTS, and SSLBIS
respectively.
Signed-off-by: hchkuo
Signed-off-by: Chris Browy
---
tests/data/cdat/cdat.dat | Bin 0 -> 148 bytes
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 tests/data/cdat/cdat.dat
diff --git a/tests/data/cdat/cdat.dat b/tests/data/cdat/cdat.dat
new file mode 100644
index
..b66c5d5836bcce7490e698f9ab5071c623425c48
GIT binary patch
literal 148
ycmbQjz`($`14zJu1e^tBD1c~21`KhqG!ugem_{a;892aP794t585EF}W3U1CI069x
literal 0
HcmV?d1
--
2.17.1
[PATCH v6 cxl2.0-v6-doe 3/6] hw/pci: PCIe Data Object Exchange implementation
From: hchkuo
PCIe Data Object Exchange (DOE) implementation for QEMU referring to
"PCIe Data Object Exchange ECN, March 12, 2020".
The patch supports multiple DOE capabilities for a single PCIe device in
QEMU. For each capability, a static array of DOEProtocol should be
passed to pcie_doe_init(). The protocols in that array will be
registered under the DOE capability structure. For each protocol, vendor
ID, type, and corresponding callback function (handle_request()) should
be implemented. This callback function represents how the DOE request
for corresponding protocol will be handled.
pcie_doe_{read/write}_config() must be appended to corresponding PCI
device's config_read/write() handler to enable DOE access. In
pcie_doe_read_config(), false will be returned if pci_config_read()
offset is not within DOE capability range. In pcie_doe_write_config(),
the function will be early returned if not within the related DOE range.
Signed-off-by: hchkuo
Signed-off-by: Chris Browy
---
MAINTAINERS | 7 +
hw/pci/meson.build| 1 +
hw/pci/pcie_doe.c | 374 ++
include/hw/pci/pcie.h | 1 +
include/hw/pci/pcie_doe.h | 123 +
5 files changed, 506 insertions(+)
create mode 100644 hw/pci/pcie_doe.c
create mode 100644 include/hw/pci/pcie_doe.h
diff --git a/MAINTAINERS b/MAINTAINERS
index f9097ed9e7..e77e9892e3 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1681,6 +1681,13 @@ F: docs/pci*
F: docs/specs/*pci*
F: default-configs/pci.mak
+PCIE DOE
+M: Huai-Cheng Kuo
+M: Chris Browy
+S: Supported
+F: include/hw/pci/pcie_doe.h
+F: hw/pci/pcie_doe.c
+
ACPI/SMBIOS
M: Michael S. Tsirkin
M: Igor Mammedov
diff --git a/hw/pci/meson.build b/hw/pci/meson.build
index 5c4bbac817..115e50222f 100644
--- a/hw/pci/meson.build
+++ b/hw/pci/meson.build
@@ -12,6 +12,7 @@ pci_ss.add(files(
# allow plugging PCIe devices into PCI buses, include them even if
# CONFIG_PCI_EXPRESS=n.
pci_ss.add(files('pcie.c', 'pcie_aer.c'))
+pci_ss.add(files('pcie_doe.c'))
softmmu_ss.add(when: 'CONFIG_PCI_EXPRESS', if_true: files('pcie_port.c',
'pcie_host.c'))
softmmu_ss.add_all(when: 'CONFIG_PCI', if_true: pci_ss)
diff --git a/hw/pci/pcie_doe.c b/hw/pci/pcie_doe.c
new file mode 100644
index 00..ec36bda259
--- /dev/null
+++ b/hw/pci/pcie_doe.c
@@ -0,0 +1,374 @@
+/*
+ * PCIe Data Object Exchange
+ *
+ * Copyright (C) 2021 Avery Design Systems, Inc.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/log.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "qemu/range.h"
+#include "hw/pci/pci.h"
+#include "hw/pci/pcie.h"
+#include "hw/pci/pcie_doe.h"
+#include "hw/pci/msi.h"
+#include "hw/pci/msix.h"
+
+#define DWORD_BYTE 4
+#define BYTE_LSHIFT(b, pos) (b << ((pos) * 8))
+#define BYTE_RSHIFT(b, pos) (b >> ((pos) * 8))
+
+struct doe_discovery_req {
+DOEHeader header;
+uint8_t index;
+uint8_t reserved[3];
+} QEMU_PACKED;
+
+struct doe_discovery_rsp {
+DOEHeader header;
+uint16_t vendor_id;
+uint8_t data_obj_type;
+uint8_t next_index;
+} QEMU_PACKED;
+
+static bool pcie_doe_discovery(DOECap *doe_cap)
+{
+struct doe_discovery_req *req = pcie_doe_get_write_mbox_ptr(doe_cap);
+struct doe_discovery_rsp rsp;
+uint8_t index = req->index;
+DOEProtocol *prot;
+
+/* Discard request if length does not match doe_discovery */
+if (pcie_doe_get_obj_len(req) <
+DIV_ROUND_UP(sizeof(struct doe_discovery_req), DWORD_BYTE)) {
+return false;
+}
+
+rsp.header = (DOEHeader) {
+.vendor_id = PCI_VENDOR_ID_PCI_SIG,
+.data_obj_type = PCI_SIG_DOE_DISCOVERY,
+.length = DIV_ROUND_UP(sizeof(struct doe_discovery_rsp), DWORD_BYTE),
+};
+
+/* Point to the requested protocol, index 0 must be Discovery */
+if (index == 0) {
+rsp.vendor_id = PCI_VENDOR_ID_PCI_SIG;
+rsp.data_obj_type = PCI_SIG_DOE_DISCOVERY;
+} else {
+if (index < doe_cap->protocol_num) {
+prot = _cap->protocols[index - 1];
+rsp.vendor_id = prot->vendor_id;
+rsp.data_obj_type = prot->data_obj_type;
+} else {
+rsp.vendor_id = 0x;
+rsp.data_obj_type = 0xFF;
+}
+}
+
+if (index + 1 == doe_cap->protocol_num) {
+rsp.next_index = 0;
+} else {
+rsp.next_index = index + 1;
+}
+
+pcie_doe_set_rsp(doe_cap, );
+
+return true;
+}
+
+static void pcie_doe_reset_mbox(DOECap *st)
+{
+st->read_mbox_idx = 0;
+st->read_mbox_len = 0;
+st->write_mbox_len = 0;
+
+memset(st->read_mbox, 0, PCI_DOE_DW_SIZE_MAX * DWORD_BYTE);
+memset(st->write_mbox, 0, PCI_DOE_DW_SIZE_
[PATCH v6 cxl2.0-v6-doe 2/6] include/hw/pci: headers for PCIe DOE
From: hchkuo
Macros for the vender ID of PCI-SIG mentioned in "PCIe Data Object
Exchange ECN, March 12, 2020" and the size of PCIe Data Object
Exchange.
Signed-off-by: hchkuo
Signed-off-by: Chris Browy
---
include/hw/pci/pci_ids.h | 3 +++
include/hw/pci/pcie_regs.h | 4
2 files changed, 7 insertions(+)
diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h
index 95f92d98e9..2656009cfe 100644
--- a/include/hw/pci/pci_ids.h
+++ b/include/hw/pci/pci_ids.h
@@ -157,6 +157,9 @@
/* Vendors and devices. Sort key: vendor first, device next. */
+/* Ref: PCIe Data Object Exchange ECN, March 12, 2020, Table 7-x2 */
+#define PCI_VENDOR_ID_PCI_SIG0x0001
+
#define PCI_VENDOR_ID_LSI_LOGIC 0x1000
#define PCI_DEVICE_ID_LSI_53C810 0x0001
#define PCI_DEVICE_ID_LSI_53C895A0x0012
diff --git a/include/hw/pci/pcie_regs.h b/include/hw/pci/pcie_regs.h
index 1db86b0ec4..963dc2e170 100644
--- a/include/hw/pci/pcie_regs.h
+++ b/include/hw/pci/pcie_regs.h
@@ -179,4 +179,8 @@ typedef enum PCIExpLinkWidth {
#define PCI_ACS_VER 0x1
#define PCI_ACS_SIZEOF 8
+/* DOE Capability Register Fields */
+#define PCI_DOE_VER 0x1
+#define PCI_DOE_SIZEOF 24
+
#endif /* QEMU_PCIE_REGS_H */
--
2.17.1
[PATCH v6 cxl2.0-v6-doe 5/6] cxl/cdat: CXL CDAT Data Object Exchange implementation
From: hchkuo
The Data Object Exchange implementation of CXL Coherent Device Attribute
Table (CDAT). This implementation is referring to "Coherent Device
Attribute Table Specification, Rev. 1.02, Oct. 2020" and "Compute
Express Link Specification, Rev. 2.0, Oct. 2020"
The CDAT can be specified in two ways. One is to add ",cdat="
in "-device cxl-type3"'s command option. The file is required to provide
the whole CDAT table in binary mode. The other is to use the default
CDAT value created by build_cdat_table in hw/cxl/cxl-cdat.c.
A DOE capability of CDAT is added to hw/mem/cxl_type3.c with capability
offset 0x190. The config read/write to this capability range can be
generated in the OS to request the CDAT data.
Signed-off-by: hchkuo
Signed-off-by: Chris Browy
---
hw/cxl/cxl-cdat.c | 139 ++
hw/cxl/meson.build | 1 +
hw/mem/cxl_type3.c | 153 -
include/hw/cxl/cxl_cdat.h | 150
include/hw/cxl/cxl_component.h | 4 +
include/hw/cxl/cxl_device.h| 1 +
include/hw/cxl/cxl_pci.h | 1 +
7 files changed, 448 insertions(+), 1 deletion(-)
create mode 100644 hw/cxl/cxl-cdat.c
create mode 100644 include/hw/cxl/cxl_cdat.h
diff --git a/hw/cxl/cxl-cdat.c b/hw/cxl/cxl-cdat.c
new file mode 100644
index 00..9cce701f84
--- /dev/null
+++ b/hw/cxl/cxl-cdat.c
@@ -0,0 +1,139 @@
+/*
+ * CXL CDAT Structure
+ *
+ * Copyright (C) 2021 Avery Design Systems, Inc.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "hw/pci/pci.h"
+#include "hw/cxl/cxl.h"
+#include "qapi/error.h"
+#include "qemu/error-report.h"
+
+static void cdat_len_check(struct cdat_sub_header *hdr, Error **errp)
+{
+assert(hdr->length);
+assert(hdr->reserved == 0);
+
+switch (hdr->type) {
+case CDAT_TYPE_DSMAS:
+assert(hdr->length == sizeof(struct cdat_dsmas));
+break;
+case CDAT_TYPE_DSLBIS:
+assert(hdr->length == sizeof(struct cdat_dslbis));
+break;
+case CDAT_TYPE_DSMSCIS:
+assert(hdr->length == sizeof(struct cdat_dsmscis));
+break;
+case CDAT_TYPE_DSIS:
+assert(hdr->length == sizeof(struct cdat_dsis));
+break;
+case CDAT_TYPE_DSEMTS:
+assert(hdr->length == sizeof(struct cdat_dsemts));
+break;
+case CDAT_TYPE_SSLBIS:
+assert(hdr->length >= sizeof(struct cdat_sslbis_header));
+assert((hdr->length - sizeof(struct cdat_sslbis_header)) %
+ sizeof(struct cdat_sslbe) == 0);
+break;
+default:
+error_setg(errp, "Type %d is reserved", hdr->type);
+}
+}
+
+void cxl_doe_cdat_init(CXLComponentState *cxl_cstate, Error **errp)
+{
+CDATObject *cdat = _cstate->cdat;
+CDATEntry cdat_st[1024];
+uint8_t sum = 0, *buf;
+int i = 0, ent = 1, file_size = 0, cdat_table_len = 0;
+struct cdat_sub_header *hdr;
+struct cdat_table_header *cdat_header;
+FILE *fp;
+void **cdat_table = NULL;
+
+fp = fopen(cdat->filename, "r");
+
+if (fp) {
+/* Read CDAT file and create its cache */
+fseek(fp, 0, SEEK_END);
+file_size = ftell(fp);
+fseek(fp, 0, SEEK_SET);
+cdat->buf = g_malloc0(file_size);
+
+if (fread(cdat->buf, file_size, 1, fp) == 0) {
+error_setg(errp, "File read failed");
+}
+
+fclose(fp);
+
+/* Set CDAT header, Entry = 0 */
+cdat_st[0].base = cdat->buf;
+cdat_st[0].length = sizeof(struct cdat_table_header);
+while (i < cdat_st[0].length) {
+sum += cdat->buf[i++];
+}
+
+/* Read CDAT structures */
+while (i < file_size) {
+hdr = (struct cdat_sub_header *)(cdat->buf + i);
+cdat_len_check(hdr, errp);
+
+cdat_st[ent].base = hdr;
+cdat_st[ent].length = hdr->length;
+
+while (cdat->buf + i < (char *)cdat_st[ent].base +
cdat_st[ent].length) {
+assert(i < file_size);
+sum += cdat->buf[i++];
+}
+
+ent++;
+}
+
+if (sum != 0) {
+warn_report("Found checksum mismatch in %s\n", cdat->filename);
+}
+} else {
+/* Use default table if fopen == NULL */
+assert(cdat->build_cdat_table);
+cdat_header = g_malloc0(sizeof(struct cdat_table_header));
+cdat->build_cdat_table(_table, _table_len);
+
+/* Entry 0 for CDAT header, starts with Entry 1 */
+for (; ent < cdat_table_len + 1; ent++) {
+hdr = cdat_table[ent - 1];
+buf = cdat_tabl
[PATCH v6 cxl2.0-v6-doe 1/6] standard-headers/linux/pci_regs: PCI header from Linux kernel
From: hchkuo Linux standard header for the registers of PCI Data Object Exchange (DOE). This header might be generated via script. The DOE feature should be added in the future Linux release so this patch can be removed then. Signed-off-by: hchkuo Signed-off-by: Chris Browy --- include/standard-headers/linux/pci_regs.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/standard-headers/linux/pci_regs.h b/include/standard-headers/linux/pci_regs.h index e709ae8235..2a8df63e11 100644 --- a/include/standard-headers/linux/pci_regs.h +++ b/include/standard-headers/linux/pci_regs.h @@ -730,7 +730,8 @@ #define PCI_EXT_CAP_ID_DVSEC 0x23/* Designated Vendor-Specific */ #define PCI_EXT_CAP_ID_DLF 0x25/* Data Link Feature */ #define PCI_EXT_CAP_ID_PL_16GT 0x26/* Physical Layer 16.0 GT/s */ -#define PCI_EXT_CAP_ID_MAX PCI_EXT_CAP_ID_PL_16GT +#define PCI_EXT_CAP_ID_DOE 0x2E/* Data Object Exchange */ +#define PCI_EXT_CAP_ID_MAX PCI_EXT_CAP_ID_DOE #define PCI_EXT_CAP_DSN_SIZEOF 12 #define PCI_EXT_CAP_MCAST_ENDPOINT_SIZEOF 40 -- 2.17.1
[PATCH v6 cxl2.0-v6-doe 0/6] QEMU PCIe DOE for PCIe 4.0/5.0 and CXL 2.0
This patch implements the PCIe Data Object Exchange (DOE) for PCIe 4.0/5.0 and later and CXL 2.0 "type-3" memory devices supporting the following protocols: 1: PCIe DOE Discovery protocol 2: CXL DOE Compliance Mode protocol 3: CXL DOE CDAT protocol Implementation is based on QEMU version which added CXL 2.0 "type-3" support https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0v4 6882c0453eea74d639ac75ec0f362d0cf9f1c744 PCIe Data Object Exchange (DOE) implementation for QEMU refers to "Data Object Exchange ECN, March 12, 2020" [1] The Data Object Exchange implementation of CXL Compliance Mode is refers to "Compute Express Link (CXL) Specification, Rev. 2.0, Oct. 2020" [2] The Data Object Exchange implementation of CXL Coherent Device Attribute Table (CDAT). This implementation is referring to "Coherent Device Attribute Table Specification, Rev. 1.02, Oct. 2020" [3] and "Compute Express Link Specification, Rev. 2.0, Oct. 2020" [2] The CDAT can be specified in two ways. One is to add ",cdat=" in "-device cxl-type3"'s command option. The file is required to provide the whole CDAT table in binary mode. The other is to use the default CDAT value created by build_cdat_table in hw/cxl/cxl-cdat.c. Pre-built CDAT table for testing, contains one CDAT header and six CDAT entries: DSMAS, DSLBIS, DSMSCIS, DSIS, DSEMTS, and SSLBIS respectively. Changes since PATCH v5: 1-3: PCIe DOE linux header and macros and PCIe Discovery protocol 4: Clean up CXL compliance mode DOE protocol including default responses 5-6: Clean up CXL CDAT DOE protocol including tesing built-in and external CDAT tables [1]: https://members.pcisig.com/wg/PCI-SIG/document/14143 [2]: https://www.computeexpresslink.org/ [3]: https://uefi.org/sites/default/files/resources/Coherent%20Device%20Attribute%20Table_1.02.pdf hchkuo (6): standard-headers/linux/pci_regs: PCI header from Linux kernel include/hw/pci: headers for PCIe DOE hw/pci: PCIe Data Object Exchange implementation cxl/compliance: CXL Compliance Data Object Exchange implementation cxl/cdat: CXL CDAT Data Object Exchange implementation test/cdat: CXL CDAT test data MAINTAINERS | 7 + hw/cxl/cxl-cdat.c | 139 hw/cxl/meson.build| 1 + hw/mem/cxl_type3.c| 298 + hw/pci/meson.build| 1 + hw/pci/pcie_doe.c | 374 ++ include/hw/cxl/cxl_cdat.h | 150 + include/hw/cxl/cxl_compliance.h | 293 + include/hw/cxl/cxl_component.h| 7 + include/hw/cxl/cxl_device.h | 4 + include/hw/cxl/cxl_pci.h | 2 + include/hw/pci/pci_ids.h | 3 + include/hw/pci/pcie.h | 1 + include/hw/pci/pcie_doe.h | 123 +++ include/hw/pci/pcie_regs.h| 4 + include/standard-headers/linux/pci_regs.h | 3 +- tests/data/cdat/cdat.dat | Bin 0 -> 148 bytes 17 files changed, 1409 insertions(+), 1 deletion(-) create mode 100644 hw/cxl/cxl-cdat.c create mode 100644 hw/pci/pcie_doe.c create mode 100644 include/hw/cxl/cxl_cdat.h create mode 100644 include/hw/cxl/cxl_compliance.h create mode 100644 include/hw/pci/pcie_doe.h create mode 100644 tests/data/cdat/cdat.dat -- 2.17.1
[PATCH v5 cxl2.0-v3-doe 6/6] test/cdat: CXL CDAT test data
From: hchkuo
Pre-built CDAT table for testing, contains one CDAT header and six
CDAT entries: DSMAS, DSLBIS, DSMSCIS, DSIS, DSEMTS, and SSLBIS
respectively.
Signed-off-by: Chris Browy
---
tests/data/cdat/cdat.dat | Bin 0 -> 148 bytes
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 tests/data/cdat/cdat.dat
diff --git a/tests/data/cdat/cdat.dat b/tests/data/cdat/cdat.dat
new file mode 100644
index
..b66c5d5836bcce7490e698f9ab5071c623425c48
GIT binary patch
literal 148
ycmbQjz`($`14zJu1e^tBD1c~21`KhqG!ugem_{a;892aP794t585EF}W3U1CI069x
literal 0
HcmV?d1
--
2.17.1
[PATCH v5 cxl2.0-v3-doe 5/6] cxl/cdat: CXL CDAT Data Object Exchange implementation
From: hchkuo
The Data Object Exchange implementation of CXL Coherent Device Attribute
Table (CDAT). This implementation is referring to "Coherent Device
Attribute Table Specification, Rev. 1.02, Oct. 2020" and "Compute
Express Link Specification, Rev. 2.0, Oct. 2020"
The CDAT can be specified in two ways. One is to add ",cdat="
in "-device cxl-type3"'s command option. The file is required to provide
the whole CDAT table in binary mode. The other is to use the default
CDAT value created by build_cdat_table in hw/cxl/cxl-cdat.c.
A DOE capability of CDAT is added to hw/mem/cxl_type3.c with capability
offset 0x190. The config read/write to this capability range can be
generated in the OS to request the CDAT data.
Signed-off-by: Chris Browy
---
hw/cxl/cxl-cdat.c | 228 +
hw/cxl/meson.build | 1 +
hw/mem/cxl_type3.c | 57 -
include/hw/cxl/cxl_cdat.h | 149 +
include/hw/cxl/cxl_component.h | 4 +
include/hw/cxl/cxl_device.h| 1 +
include/hw/cxl/cxl_pci.h | 1 +
7 files changed, 440 insertions(+), 1 deletion(-)
create mode 100644 hw/cxl/cxl-cdat.c
create mode 100644 include/hw/cxl/cxl_cdat.h
diff --git a/hw/cxl/cxl-cdat.c b/hw/cxl/cxl-cdat.c
new file mode 100644
index 00..3b86ecaddf
--- /dev/null
+++ b/hw/cxl/cxl-cdat.c
@@ -0,0 +1,228 @@
+/*
+ * CXL CDAT Structure
+ *
+ * Copyright (C) 2021 Avery Design Systems, Inc.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "hw/pci/pci.h"
+#include "hw/cxl/cxl.h"
+#include "qapi/error.h"
+#include "qemu/error-report.h"
+
+static void build_cdat_table(void ***cdat_table, int *len) {
+struct cdat_dsmas *dsmas = g_malloc(sizeof(struct cdat_dsmas));
+struct cdat_dslbis *dslbis = g_malloc(sizeof(struct cdat_dslbis));
+struct cdat_dsmscis *dsmscis = g_malloc(sizeof(struct cdat_dsmscis));
+struct cdat_dsis *dsis = g_malloc(sizeof(struct cdat_dsis));
+struct cdat_dsemts *dsemts = g_malloc(sizeof(struct cdat_dsemts));
+struct cdat_sslbis {
+struct cdat_sslbis_header sslbis_header;
+struct cdat_sslbe sslbe[2];
+};
+struct cdat_sslbis *sslbis = g_malloc(sizeof(struct cdat_sslbis));
+void *__cdat_table[] = {
+dsmas,
+dslbis,
+dsmscis,
+dsis,
+dsemts,
+sslbis,
+};
+
+*dsmas = (struct cdat_dsmas){
+.header = {
+.type = CDAT_TYPE_DSMAS,
+.length = sizeof(struct cdat_dsmas),
+},
+.DSMADhandle = 0,
+.flags = 0,
+.DPA_base = 0,
+.DPA_length = 0,
+};
+*dslbis = (struct cdat_dslbis){
+.header = {
+.type = CDAT_TYPE_DSLBIS,
+.length = sizeof(struct cdat_dslbis),
+},
+.handle = 0,
+.flags = 0,
+.data_type = 0,
+.entry_base_unit = 0,
+};
+*dsmscis = (struct cdat_dsmscis){
+.header = {
+.type = CDAT_TYPE_DSMSCIS,
+.length = sizeof(struct cdat_dsmscis),
+},
+.DSMAS_handle = 0,
+.memory_side_cache_size = 0,
+.cache_attributes = 0,
+};
+*dsis = (struct cdat_dsis){
+.header = {
+.type = CDAT_TYPE_DSIS,
+.length = sizeof(struct cdat_dsis),
+},
+.flags = 0,
+.handle = 0,
+};
+*dsemts = (struct cdat_dsemts){
+.header = {
+.type = CDAT_TYPE_DSEMTS,
+.length = sizeof(struct cdat_dsemts),
+},
+.DSMAS_handle = 0,
+.EFI_memory_type_attr = 0,
+.DPA_offset = 0,
+.DPA_length = 0,
+};
+*sslbis = (struct cdat_sslbis){
+.sslbis_header = {
+.header = {
+.type = CDAT_TYPE_SSLBIS,
+.length = sizeof(sslbis->sslbis_header) +
+ sizeof(struct cdat_sslbe) * 2,
+},
+.data_type = 0,
+.entry_base_unit = 0,
+},
+.sslbe[0] = {
+.port_x_id = 0,
+.port_y_id = 0,
+.latency_bandwidth = 0,
+},
+.sslbe[1] = {
+.port_x_id = 0,
+.port_y_id = 0,
+.latency_bandwidth = 0,
+},
+};
+
+*len = ARRAY_SIZE(__cdat_table);
+*cdat_table = g_malloc0((*len) * sizeof(void *));
+memcpy(*cdat_table, __cdat_table, (*len) * sizeof(void *));
+}
+
+static void cdat_len_check(struct cdat_sub_header *hdr, Error **errp)
+{
+assert(hdr->length);
+assert(hdr->reserved == 0);
+
+switch (hdr->type) {
+case CDAT_TYPE_DSMAS:
+assert(hdr->length == sizeof(struct cdat_dsmas));
+break;
+case CDAT_TYPE_DSLBIS:
+assert(hdr->length == sizeof(str
[PATCH v5 cxl2.0-v3-doe 4/6] cxl/compliance: CXL Compliance Data Object Exchange implementation
From: hchkuo
The Data Object Exchange implementation of CXL Compliance Mode is
referring to "Compute Express Link (CXL) Specification, Rev. 2.0, Oct.
2020".
The data structure of CXL compliance request and response is added to
the header. Due to the scope limitation of QEMU, most of the compliance
response is limited to returning corresponding length.
A DOE capability of CXL Compliance is added to hw/mem/cxl_type3.c with
capability offset 0x160. The config read/write to this capability range
can be generated in the OS to request the Compliance info.
Signed-off-by: Chris Browy
---
hw/mem/cxl_type3.c | 147
include/hw/cxl/cxl_compliance.h | 293
include/hw/cxl/cxl_component.h | 3 +
include/hw/cxl/cxl_device.h | 3 +
include/hw/cxl/cxl_pci.h| 1 +
5 files changed, 447 insertions(+)
create mode 100644 include/hw/cxl/cxl_compliance.h
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index bf33ddb915..569872eb36 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -13,6 +13,134 @@
#include "qemu/rcu.h"
#include "sysemu/hostmem.h"
#include "hw/cxl/cxl.h"
+#include "hw/pci/msi.h"
+#include "hw/pci/msix.h"
+
+#define DWORD_BYTE 4
+
+bool cxl_doe_compliance_rsp(DOECap *doe_cap)
+{
+CompRsp *rsp = (doe_cap->pdev)->cxl_cstate.compliance.response;
+struct compliance_req_header *req = pcie_doe_get_write_mbox_ptr(doe_cap);
+uint32_t type, req_len = 0, rsp_len = 0;
+
+type = req->req_code;
+
+switch (type) {
+case CXL_COMP_MODE_CAP:
+req_len = sizeof(struct cxl_compliance_cap_req);
+rsp_len = sizeof(struct cxl_compliance_cap_rsp);
+rsp->cap_rsp.status = 0x0;
+rsp->cap_rsp.available_cap_bitmask = 0;
+rsp->cap_rsp.enabled_cap_bitmask = 0;
+break;
+case CXL_COMP_MODE_STATUS:
+req_len = sizeof(struct cxl_compliance_status_req);
+rsp_len = sizeof(struct cxl_compliance_status_rsp);
+rsp->status_rsp.cap_bitfield = 0;
+rsp->status_rsp.cache_size = 0;
+rsp->status_rsp.cache_size_units = 0;
+break;
+case CXL_COMP_MODE_HALT:
+req_len = sizeof(struct cxl_compliance_halt_req);
+rsp_len = sizeof(struct cxl_compliance_halt_rsp);
+break;
+case CXL_COMP_MODE_MULT_WR_STREAM:
+req_len = sizeof(struct cxl_compliance_multi_write_streaming_req);
+rsp_len = sizeof(struct cxl_compliance_multi_write_streaming_rsp);
+break;
+case CXL_COMP_MODE_PRO_CON:
+req_len = sizeof(struct cxl_compliance_producer_consumer_req);
+rsp_len = sizeof(struct cxl_compliance_producer_consumer_rsp);
+break;
+case CXL_COMP_MODE_BOGUS:
+req_len = sizeof(struct cxl_compliance_bogus_writes_req);
+rsp_len = sizeof(struct cxl_compliance_bogus_writes_rsp);
+break;
+case CXL_COMP_MODE_INJ_POISON:
+req_len = sizeof(struct cxl_compliance_inject_poison_req);
+rsp_len = sizeof(struct cxl_compliance_inject_poison_rsp);
+break;
+case CXL_COMP_MODE_INJ_CRC:
+req_len = sizeof(struct cxl_compliance_inject_crc_req);
+rsp_len = sizeof(struct cxl_compliance_inject_crc_rsp);
+break;
+case CXL_COMP_MODE_INJ_FC:
+req_len = sizeof(struct cxl_compliance_inject_flow_ctrl_req);
+rsp_len = sizeof(struct cxl_compliance_inject_flow_ctrl_rsp);
+break;
+case CXL_COMP_MODE_TOGGLE_CACHE:
+req_len = sizeof(struct cxl_compliance_toggle_cache_flush_req);
+rsp_len = sizeof(struct cxl_compliance_toggle_cache_flush_rsp);
+break;
+case CXL_COMP_MODE_INJ_MAC:
+req_len = sizeof(struct cxl_compliance_inject_mac_delay_req);
+rsp_len = sizeof(struct cxl_compliance_inject_mac_delay_rsp);
+break;
+case CXL_COMP_MODE_INS_UNEXP_MAC:
+req_len = sizeof(struct cxl_compliance_insert_unexp_mac_req);
+rsp_len = sizeof(struct cxl_compliance_insert_unexp_mac_rsp);
+break;
+case CXL_COMP_MODE_INJ_VIRAL:
+req_len = sizeof(struct cxl_compliance_inject_viral_req);
+rsp_len = sizeof(struct cxl_compliance_inject_viral_rsp);
+break;
+case CXL_COMP_MODE_INJ_ALMP:
+req_len = sizeof(struct cxl_compliance_inject_almp_req);
+rsp_len = sizeof(struct cxl_compliance_inject_almp_rsp);
+break;
+case CXL_COMP_MODE_IGN_ALMP:
+req_len = sizeof(struct cxl_compliance_ignore_almp_req);
+rsp_len = sizeof(struct cxl_compliance_ignore_almp_rsp);
+break;
+case CXL_COMP_MODE_INJ_BIT_ERR:
+req_len = sizeof(struct cxl_compliance_inject_bit_err_in_flit_req);
+rsp_len = sizeof(struct cxl_compliance_inject_bit_err_in_flit_rsp);
+break;
+default:
+break;
+}
+
+/* Discard if request length mismatched */
+if (pcie_doe_get_obj_len(req) <
[PATCH v5 cxl2.0-v3-doe 3/6] hw/pci: PCIe Data Object Exchange implementation
From: hchkuo
PCIe Data Object Exchange (DOE) implementation for QEMU referring to
"PCIe Data Object Exchange ECN, March 12, 2020".
The patch supports multiple DOE capabilities for a single PCIe device in
QEMU. For each capability, a static array of DOEProtocol should be
passed to pcie_doe_init(). The protocols in that array will be
registered under the DOE capability structure. For each protocol, vendor
ID, type, and corresponding callback function (handle_request()) should
be implemented. This callback function represents how the DOE request
for corresponding protocol will be handled.
pcie_doe_{read/write}_config() must be appended to corresponding PCI
device's config_read/write() handler to enable DOE access. In
pcie_doe_read_config(), false will be returned if pci_config_read()
offset is not within DOE capability range. In pcie_doe_write_config(),
the function will be early returned if not within the related DOE range.
Signed-off-by: Chris Browy
---
MAINTAINERS | 7 +
hw/pci/meson.build| 1 +
hw/pci/pcie_doe.c | 374 ++
include/hw/pci/pcie.h | 1 +
include/hw/pci/pcie_doe.h | 123 +
5 files changed, 506 insertions(+)
create mode 100644 hw/pci/pcie_doe.c
create mode 100644 include/hw/pci/pcie_doe.h
diff --git a/MAINTAINERS b/MAINTAINERS
index f9097ed9e7..e77e9892e3 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1681,6 +1681,13 @@ F: docs/pci*
F: docs/specs/*pci*
F: default-configs/pci.mak
+PCIE DOE
+M: Huai-Cheng Kuo
+M: Chris Browy
+S: Supported
+F: include/hw/pci/pcie_doe.h
+F: hw/pci/pcie_doe.c
+
ACPI/SMBIOS
M: Michael S. Tsirkin
M: Igor Mammedov
diff --git a/hw/pci/meson.build b/hw/pci/meson.build
index 5c4bbac817..115e50222f 100644
--- a/hw/pci/meson.build
+++ b/hw/pci/meson.build
@@ -12,6 +12,7 @@ pci_ss.add(files(
# allow plugging PCIe devices into PCI buses, include them even if
# CONFIG_PCI_EXPRESS=n.
pci_ss.add(files('pcie.c', 'pcie_aer.c'))
+pci_ss.add(files('pcie_doe.c'))
softmmu_ss.add(when: 'CONFIG_PCI_EXPRESS', if_true: files('pcie_port.c',
'pcie_host.c'))
softmmu_ss.add_all(when: 'CONFIG_PCI', if_true: pci_ss)
diff --git a/hw/pci/pcie_doe.c b/hw/pci/pcie_doe.c
new file mode 100644
index 00..b2a933
--- /dev/null
+++ b/hw/pci/pcie_doe.c
@@ -0,0 +1,374 @@
+/*
+ * PCIe Data Object Exchange
+ *
+ * Copyright (C) 2021 Avery Design Systems, Inc.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/log.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "qemu/range.h"
+#include "hw/pci/pci.h"
+#include "hw/pci/pcie.h"
+#include "hw/pci/pcie_doe.h"
+#include "hw/pci/msi.h"
+#include "hw/pci/msix.h"
+
+#define DWORD_BYTE 4
+#define BYTE_LSHIFT(b, pos) (b << ((pos) * 8))
+#define BYTE_RSHIFT(b, pos) (b >> ((pos) * 8))
+
+struct doe_discovery_req {
+DOEHeader header;
+uint8_t index;
+uint8_t reserved[3];
+} QEMU_PACKED;
+
+struct doe_discovery_rsp {
+DOEHeader header;
+uint16_t vendor_id;
+uint8_t data_obj_type;
+uint8_t next_index;
+} QEMU_PACKED;
+
+static bool pcie_doe_discovery(DOECap *doe_cap)
+{
+struct doe_discovery_req *req = pcie_doe_get_write_mbox_ptr(doe_cap);
+struct doe_discovery_rsp rsp;
+uint8_t index = req->index;
+DOEProtocol *prot;
+
+/* Discard request if length does not match doe_discovery */
+if (pcie_doe_get_obj_len(req) <
+DIV_ROUND_UP(sizeof(struct doe_discovery_req), DWORD_BYTE)) {
+return false;
+}
+
+rsp.header = (DOEHeader) {
+.vendor_id = PCI_VENDOR_ID_PCI_SIG,
+.data_obj_type = PCI_SIG_DOE_DISCOVERY,
+.length = DIV_ROUND_UP(sizeof(struct doe_discovery_rsp), DWORD_BYTE),
+};
+
+/* Point to the requested protocol, index 0 must be Discovery */
+if (index == 0) {
+rsp.vendor_id = PCI_VENDOR_ID_PCI_SIG;
+rsp.data_obj_type = PCI_SIG_DOE_DISCOVERY;
+} else {
+if (index < doe_cap->protocol_num) {
+prot = _cap->protocols[index - 1];
+rsp.vendor_id = prot->vendor_id;
+rsp.data_obj_type = prot->data_obj_type;
+} else {
+rsp.vendor_id = 0x;
+rsp.data_obj_type = 0xFF;
+}
+}
+
+if (index + 1 == doe_cap->protocol_num) {
+rsp.next_index = 0;
+} else {
+rsp.next_index = index + 1;
+}
+
+pcie_doe_set_rsp(doe_cap, );
+
+return true;
+}
+
+static void pcie_doe_reset_mbox(DOECap *st)
+{
+st->read_mbox_idx = 0;
+st->read_mbox_len = 0;
+st->write_mbox_len = 0;
+
+memset(st->read_mbox, 0, PCI_DOE_DW_SIZE_MAX * DWORD_BYTE);
+memset(st->write_mbox, 0, PCI_DOE_DW_SIZE_
[PATCH v5 cxl2.0-v3-doe 2/6] include/hw/pci: headers for PCIe DOE
From: hchkuo
Macros for the vender ID of PCI-SIG and the size of PCIe Data Object
Exchange.
Signed-off-by: Chris Browy
---
include/hw/pci/pci_ids.h | 2 ++
include/hw/pci/pcie_regs.h | 3 +++
2 files changed, 5 insertions(+)
diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h
index 95f92d98e9..471c915395 100644
--- a/include/hw/pci/pci_ids.h
+++ b/include/hw/pci/pci_ids.h
@@ -157,6 +157,8 @@
/* Vendors and devices. Sort key: vendor first, device next. */
+#define PCI_VENDOR_ID_PCI_SIG0x0001
+
#define PCI_VENDOR_ID_LSI_LOGIC 0x1000
#define PCI_DEVICE_ID_LSI_53C810 0x0001
#define PCI_DEVICE_ID_LSI_53C895A0x0012
diff --git a/include/hw/pci/pcie_regs.h b/include/hw/pci/pcie_regs.h
index 1db86b0ec4..5ec7014211 100644
--- a/include/hw/pci/pcie_regs.h
+++ b/include/hw/pci/pcie_regs.h
@@ -179,4 +179,7 @@ typedef enum PCIExpLinkWidth {
#define PCI_ACS_VER 0x1
#define PCI_ACS_SIZEOF 8
+/* DOE Capability Register Fields */
+#define PCI_DOE_SIZEOF 24
+
#endif /* QEMU_PCIE_REGS_H */
--
2.17.1
[PATCH v5 cxl2.0-v3-doe 1/6] standard-headers/linux/pci_regs: PCI header from Linux kernel
From: hchkuo Linux standard header for the registers of PCI Data Object Exchange (DOE). This header might be generated via script. The DOE feature should be added in the future Linux release so this patch can be removed then. Signed-off-by: Chris Browy --- include/standard-headers/linux/pci_regs.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/standard-headers/linux/pci_regs.h b/include/standard-headers/linux/pci_regs.h index e709ae8235..2a8df63e11 100644 --- a/include/standard-headers/linux/pci_regs.h +++ b/include/standard-headers/linux/pci_regs.h @@ -730,7 +730,8 @@ #define PCI_EXT_CAP_ID_DVSEC 0x23/* Designated Vendor-Specific */ #define PCI_EXT_CAP_ID_DLF 0x25/* Data Link Feature */ #define PCI_EXT_CAP_ID_PL_16GT 0x26/* Physical Layer 16.0 GT/s */ -#define PCI_EXT_CAP_ID_MAX PCI_EXT_CAP_ID_PL_16GT +#define PCI_EXT_CAP_ID_DOE 0x2E/* Data Object Exchange */ +#define PCI_EXT_CAP_ID_MAX PCI_EXT_CAP_ID_DOE #define PCI_EXT_CAP_DSN_SIZEOF 12 #define PCI_EXT_CAP_MCAST_ENDPOINT_SIZEOF 40 -- 2.17.1
[PATCH v5 cxl2.0-v3-doe 0/6] QEMU PCIe DOE for PCIe 4.0/5.0 and CXL 2.0
This patch implements the PCIe Data Object Exchange (DOE) for PCIe 4.0/5.0 and later and CXL 2.0 "type-3" memory devices supporting the following protocols: 1: PCIe DOE Discovery protocol 2: CXL DOE Compliance Mode protocol 3: CXL DOE CDAT protocol Implementation is based on QEMU version which added CXL 2.0 "type-3" support https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0v4 6882c0453eea74d639ac75ec0f362d0cf9f1c744 PCIe Data Object Exchange (DOE) implementation for QEMU refers to "Data Object Exchange ECN, March 12, 2020" [1] The Data Object Exchange implementation of CXL Compliance Mode is refers to "Compute Express Link (CXL) Specification, Rev. 2.0, Oct. 2020" [2] The Data Object Exchange implementation of CXL Coherent Device Attribute Table (CDAT). This implementation is referring to "Coherent Device Attribute Table Specification, Rev. 1.02, Oct. 2020" [3] and "Compute Express Link Specification, Rev. 2.0, Oct. 2020" [2] The CDAT can be specified in two ways. One is to add ",cdat=" in "-device cxl-type3"'s command option. The file is required to provide the whole CDAT table in binary mode. The other is to use the default CDAT value created by build_cdat_table in hw/cxl/cxl-cdat.c. Pre-built CDAT table for testing, contains one CDAT header and six CDAT entries: DSMAS, DSLBIS, DSMSCIS, DSIS, DSEMTS, and SSLBIS respectively. Changes since PATCH v4: 1-3: PCIe DOE linux header and macros and PCIe Discovery protocol 4: Clean up CXL compliance mode DOE protocol including default responses 5-6: Clean up CXL CDAT DOE protocol including tesing built-in and external CDAT tables [1]: https://members.pcisig.com/wg/PCI-SIG/document/14143 [2]: https://www.computeexpresslink.org/ [3]: https://uefi.org/sites/default/files/resources/Coherent%20Device%20Attribute%20Table_1.02.pdf hchkuo (6): standard-headers/linux/pci_regs: PCI header from Linux kernel include/hw/pci: headers for PCIe DOE hw/pci: PCIe Data Object Exchange implementation cxl/compliance: CXL Compliance Data Object Exchange implementation cxl/cdat: CXL CDAT Data Object Exchange implementation test/cdat: CXL CDAT test data MAINTAINERS | 7 + hw/cxl/cxl-cdat.c | 228 + hw/cxl/meson.build| 1 + hw/mem/cxl_type3.c| 202 hw/pci/meson.build| 1 + hw/pci/pcie_doe.c | 374 ++ include/hw/cxl/cxl_cdat.h | 149 + include/hw/cxl/cxl_compliance.h | 293 + include/hw/cxl/cxl_component.h| 7 + include/hw/cxl/cxl_device.h | 4 + include/hw/cxl/cxl_pci.h | 2 + include/hw/pci/pci_ids.h | 2 + include/hw/pci/pcie.h | 1 + include/hw/pci/pcie_doe.h | 123 +++ include/hw/pci/pcie_regs.h| 3 + include/standard-headers/linux/pci_regs.h | 3 +- tests/data/cdat/cdat.dat | Bin 0 -> 148 bytes 17 files changed, 1399 insertions(+), 1 deletion(-) create mode 100644 hw/cxl/cxl-cdat.c create mode 100644 hw/pci/pcie_doe.c create mode 100644 include/hw/cxl/cxl_cdat.h create mode 100644 include/hw/cxl/cxl_compliance.h create mode 100644 include/hw/pci/pcie_doe.h create mode 100644 tests/data/cdat/cdat.dat -- 2.17.1
[PATCH v4 cxl-2.0-doe 3/3] PCIe standard header for DOE
From: hchkuo
Signed-off-by: hchkuo
---
include/hw/pci/pci_ids.h | 2 ++
include/hw/pci/pcie_regs.h| 3 +++
include/standard-headers/linux/pci_regs.h | 3 ++-
3 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h
index 95f92d9..471c915 100644
--- a/include/hw/pci/pci_ids.h
+++ b/include/hw/pci/pci_ids.h
@@ -157,6 +157,8 @@
/* Vendors and devices. Sort key: vendor first, device next. */
+#define PCI_VENDOR_ID_PCI_SIG0x0001
+
#define PCI_VENDOR_ID_LSI_LOGIC 0x1000
#define PCI_DEVICE_ID_LSI_53C810 0x0001
#define PCI_DEVICE_ID_LSI_53C895A0x0012
diff --git a/include/hw/pci/pcie_regs.h b/include/hw/pci/pcie_regs.h
index 1db86b0..5ec7014 100644
--- a/include/hw/pci/pcie_regs.h
+++ b/include/hw/pci/pcie_regs.h
@@ -179,4 +179,7 @@ typedef enum PCIExpLinkWidth {
#define PCI_ACS_VER 0x1
#define PCI_ACS_SIZEOF 8
+/* DOE Capability Register Fields */
+#define PCI_DOE_SIZEOF 24
+
#endif /* QEMU_PCIE_REGS_H */
diff --git a/include/standard-headers/linux/pci_regs.h
b/include/standard-headers/linux/pci_regs.h
index e709ae8..2a8df63 100644
--- a/include/standard-headers/linux/pci_regs.h
+++ b/include/standard-headers/linux/pci_regs.h
@@ -730,7 +730,8 @@
#define PCI_EXT_CAP_ID_DVSEC 0x23/* Designated Vendor-Specific */
#define PCI_EXT_CAP_ID_DLF 0x25/* Data Link Feature */
#define PCI_EXT_CAP_ID_PL_16GT 0x26/* Physical Layer 16.0 GT/s */
-#define PCI_EXT_CAP_ID_MAX PCI_EXT_CAP_ID_PL_16GT
+#define PCI_EXT_CAP_ID_DOE 0x2E/* Data Object Exchange */
+#define PCI_EXT_CAP_ID_MAX PCI_EXT_CAP_ID_DOE
#define PCI_EXT_CAP_DSN_SIZEOF 12
#define PCI_EXT_CAP_MCAST_ENDPOINT_SIZEOF 40
--
1.8.3.1
[PATCH v4 cxl-2.0-doe 2/3] CXL Data Object Exchange implementation
From: hchkuo
Signed-off-by: hchkuo
---
hw/cxl/cxl-cdat.c | 220 +
hw/cxl/meson.build | 1 +
hw/mem/cxl_type3.c | 200 +++
include/hw/cxl/cxl_cdat.h | 149
include/hw/cxl/cxl_compliance.h | 297
include/hw/cxl/cxl_component.h | 7 +
include/hw/cxl/cxl_device.h | 4 +
include/hw/cxl/cxl_pci.h| 2 +
tests/data/cdat/cdat.dat| Bin 0 -> 148 bytes
9 files changed, 880 insertions(+)
create mode 100644 hw/cxl/cxl-cdat.c
create mode 100644 include/hw/cxl/cxl_cdat.h
create mode 100644 include/hw/cxl/cxl_compliance.h
create mode 100644 tests/data/cdat/cdat.dat
diff --git a/hw/cxl/cxl-cdat.c b/hw/cxl/cxl-cdat.c
new file mode 100644
index 000..fa54506
--- /dev/null
+++ b/hw/cxl/cxl-cdat.c
@@ -0,0 +1,220 @@
+/*
+ * CXL CDAT Structure
+ *
+ * Copyright (C) 2021 Avery Design Systems, Inc.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "hw/pci/pci.h"
+#include "hw/cxl/cxl.h"
+#include "qapi/error.h"
+
+static struct cdat_dsmas dsmas = {
+.header = {
+.type = CDAT_TYPE_DSMAS,
+.length = sizeof(dsmas),
+},
+.DSMADhandle = 0,
+.flags = 0,
+.DPA_base = 0,
+.DPA_length = 0,
+};
+
+static struct cdat_dslbis dslbis = {
+.header = {
+.type = CDAT_TYPE_DSLBIS,
+.length = sizeof(dslbis),
+},
+.handle = 0,
+.flags = 0,
+.data_type = 0,
+.entry_base_unit = 0,
+};
+
+static struct cdat_dsmscis dsmscis = {
+.header = {
+.type = CDAT_TYPE_DSMSCIS,
+.length = sizeof(dsmscis),
+},
+.DSMAS_handle = 0,
+.memory_side_cache_size = 0,
+.cache_attributes = 0,
+};
+
+static struct cdat_dsis dsis = {
+.header = {
+.type = CDAT_TYPE_DSIS,
+.length = sizeof(dsis),
+},
+.flags = 0,
+.handle = 0,
+};
+
+static struct cdat_dsemts dsemts = {
+.header = {
+.type = CDAT_TYPE_DSEMTS,
+.length = sizeof(dsemts),
+},
+.DSMAS_handle = 0,
+.EFI_memory_type_attr = 0,
+.DPA_offset = 0,
+.DPA_length = 0,
+};
+
+struct cdat_sslbis {
+struct cdat_sslbis_header sslbis_header;
+struct cdat_sslbe sslbe[];
+};
+
+static struct cdat_sslbis sslbis = {
+.sslbis_header = {
+.header = {
+.type = CDAT_TYPE_SSLBIS,
+.length = sizeof(sslbis.sslbis_header) +
+ sizeof(struct cdat_sslbe) * 2,
+},
+.data_type = 0,
+.entry_base_unit = 0,
+},
+.sslbe[0] = {
+.port_x_id = 0,
+.port_y_id = 0,
+.latency_bandwidth = 0,
+},
+.sslbe[1] = {
+.port_x_id = 0,
+.port_y_id = 0,
+.latency_bandwidth = 0,
+},
+};
+
+static void *cdat_table[] = {
+(void *) ,
+(void *) ,
+(void *) ,
+(void *) ,
+(void *) ,
+(void *) ,
+};
+
+static void cdat_len_check(struct cdat_sub_header *hdr, Error **errp)
+{
+assert(hdr->length);
+assert(hdr->reserved == 0);
+
+switch (hdr->type) {
+case CDAT_TYPE_DSMAS:
+assert(hdr->length == sizeof(struct cdat_dsmas));
+break;
+case CDAT_TYPE_DSLBIS:
+assert(hdr->length == sizeof(struct cdat_dslbis));
+break;
+case CDAT_TYPE_DSMSCIS:
+assert(hdr->length == sizeof(struct cdat_dsmscis));
+break;
+case CDAT_TYPE_DSIS:
+assert(hdr->length == sizeof(struct cdat_dsis));
+break;
+case CDAT_TYPE_DSEMTS:
+assert(hdr->length == sizeof(struct cdat_dsemts));
+break;
+case CDAT_TYPE_SSLBIS:
+assert(hdr->length >= sizeof(struct cdat_sslbis_header));
+assert((hdr->length - sizeof(struct cdat_sslbis_header)) %
+ sizeof(struct cdat_sslbe) == 0);
+break;
+default:
+error_setg(errp, "Type %d is reserved", hdr->type);
+}
+}
+
+void cxl_doe_cdat_init(CXLComponentState *cxl_cstate, Error **errp)
+{
+CDATObject *cdat = _cstate->cdat;
+CDATEntry cdat_st[1024];
+uint8_t sum = 0, *buf;
+int i = 0, ent = 0, file_size = 0;
+struct cdat_sub_header *hdr;
+struct cdat_table_header *cdat_header;
+FILE *fp;
+
+fp = fopen(cdat->filename, "r");
+
+if (fp) {
+/* Read CDAT file and create its cache */
+fseek(fp, 0, SEEK_END);
+file_size = ftell(fp);
+fseek(fp, 0, SEEK_SET);
+cdat->buf = g_malloc0(file_size);
+
+if (fread(cdat->buf, file_size, 1, fp) == 0) {
+error_setg(errp, "File read failed");
+}
+
+fclose(fp);
+
+/* Set CDAT header, ent = 0 */
+cdat_st[ent].base = cdat->buf;
+cdat_st[ent].length = sizeof(struct cdat_table_header);
+ent++;
+while (i < cdat_st[0].length) {
+sum +=
[PATCH v4 cxl-2.0-doe 1/3] PCIe Data Object Exchange implementation
From: hchkuo
Signed-off-by: hchkuo
---
MAINTAINERS | 7 +
hw/pci/meson.build| 1 +
hw/pci/pcie_doe.c | 356 ++
include/hw/pci/pcie.h | 1 +
include/hw/pci/pcie_doe.h | 123
5 files changed, 488 insertions(+)
create mode 100644 hw/pci/pcie_doe.c
create mode 100644 include/hw/pci/pcie_doe.h
diff --git a/MAINTAINERS b/MAINTAINERS
index f9097ed..e77e989 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1681,6 +1681,13 @@ F: docs/pci*
F: docs/specs/*pci*
F: default-configs/pci.mak
+PCIE DOE
+M: Huai-Cheng Kuo
+M: Chris Browy
+S: Supported
+F: include/hw/pci/pcie_doe.h
+F: hw/pci/pcie_doe.c
+
ACPI/SMBIOS
M: Michael S. Tsirkin
M: Igor Mammedov
diff --git a/hw/pci/meson.build b/hw/pci/meson.build
index 5c4bbac..115e502 100644
--- a/hw/pci/meson.build
+++ b/hw/pci/meson.build
@@ -12,6 +12,7 @@ pci_ss.add(files(
# allow plugging PCIe devices into PCI buses, include them even if
# CONFIG_PCI_EXPRESS=n.
pci_ss.add(files('pcie.c', 'pcie_aer.c'))
+pci_ss.add(files('pcie_doe.c'))
softmmu_ss.add(when: 'CONFIG_PCI_EXPRESS', if_true: files('pcie_port.c',
'pcie_host.c'))
softmmu_ss.add_all(when: 'CONFIG_PCI', if_true: pci_ss)
diff --git a/hw/pci/pcie_doe.c b/hw/pci/pcie_doe.c
new file mode 100644
index 000..ec937ac
--- /dev/null
+++ b/hw/pci/pcie_doe.c
@@ -0,0 +1,356 @@
+/*
+ * PCIe Data Object Exchange
+ *
+ * Copyright (C) 2021 Avery Design Systems, Inc.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/log.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "qemu/range.h"
+#include "hw/pci/pci.h"
+#include "hw/pci/pcie.h"
+#include "hw/pci/pcie_doe.h"
+#include "hw/pci/msi.h"
+#include "hw/pci/msix.h"
+
+/* Discovery Request Object */
+struct doe_discovery {
+DOEHeader header;
+uint8_t index;
+uint8_t reserved[3];
+} QEMU_PACKED;
+
+/* Discovery Response Object */
+struct doe_discovery_rsp {
+DOEHeader header;
+uint16_t vendor_id;
+uint8_t data_obj_type;
+uint8_t next_index;
+} QEMU_PACKED;
+
+static bool pcie_doe_discovery_rsp(DOECap *doe_cap)
+{
+struct doe_discovery *req = pcie_doe_get_req(doe_cap);
+struct doe_discovery_rsp rsp;
+uint8_t index = req->index;
+DOEProtocol *prot;
+
+/* Request length mismatch, discard */
+if (pcie_doe_get_obj_len(req) <
+DIV_ROUND_UP(sizeof(struct doe_discovery), 4)) {
+return false;
+}
+
+rsp.header = (DOEHeader) {
+.vendor_id = PCI_VENDOR_ID_PCI_SIG,
+.data_obj_type = PCI_SIG_DOE_DISCOVERY,
+.length = DIV_ROUND_UP(sizeof(struct doe_discovery_rsp), 4),
+};
+
+/* Point to the requested protocol, index 0 must be Discovery */
+if (index == 0) {
+rsp.vendor_id = PCI_VENDOR_ID_PCI_SIG;
+rsp.data_obj_type = PCI_SIG_DOE_DISCOVERY;
+} else {
+if (index < doe_cap->protocol_num) {
+prot = _cap->protocols[index - 1];
+} else {
+prot = NULL;
+}
+
+rsp.vendor_id = (prot) ? prot->vendor_id : 0x;
+rsp.data_obj_type = (prot) ? prot->data_obj_type : 0xFF;
+}
+
+rsp.next_index = (index + 1) % doe_cap->protocol_num,
+
+pcie_doe_set_rsp(doe_cap, );
+
+return true;
+}
+
+static void pcie_doe_reset_mbox(DOECap *st)
+{
+st->read_mbox_idx = 0;
+st->read_mbox_len = 0;
+st->write_mbox_len = 0;
+
+memset(st->read_mbox, 0, PCI_DOE_MAX_DW_SIZE * sizeof(uint32_t));
+memset(st->write_mbox, 0, PCI_DOE_MAX_DW_SIZE * sizeof(uint32_t));
+}
+
+/*
+ * Add DOE cap to a device
+ * Initialize its protocol.
+ */
+void pcie_doe_init(PCIDevice *dev, DOECap *doe_cap, uint16_t offset,
+ DOEProtocol *protocols, bool intr, uint16_t vec)
+{
+pcie_add_capability(dev, PCI_EXT_CAP_ID_DOE, 0x1, offset,
+PCI_DOE_SIZEOF);
+
+doe_cap->pdev = dev;
+doe_cap->offset = offset;
+
+/* Configure MSI/MSI-X */
+if (intr && (msi_present(dev) || msix_present(dev))) {
+doe_cap->cap.intr = intr;
+doe_cap->cap.vec = vec;
+}
+
+doe_cap->write_mbox = g_malloc0(PCI_DOE_MAX_DW_SIZE * sizeof(uint32_t));
+doe_cap->read_mbox = g_malloc0(PCI_DOE_MAX_DW_SIZE * sizeof(uint32_t));
+
+pcie_doe_reset_mbox(doe_cap);
+
+/* Register default discovery protocol */
+doe_cap->protocols = protocols;
+for (; protocols->vendor_id; protocols++) {
+doe_cap->protocol_num++;
+}
+assert(doe_cap->protocol_num < PCI_DOE_PROTOCOL_MAX);
+
+doe_cap->protocol_num++;
+}
+
+void pcie_doe_fini(DOECap *doe_cap)
+{
+g_free(doe_cap->read_mbox);
+g_free(doe_cap->write_
[PATCH v4 cxl-2.0-doe 0/3] QEMU PCIe DOE for PCIe and CXL2.0
Version 4 patch series for PCIe DOE for PCIe and CXL 2.0 completes all planned functionality. Based on QEMU version: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0v4 Summary of changes: 1: PCIe DOE support for Discovery - Fix the alignment error in DOE config write - Fix the interrupt disabling issue in DOE config write - Naming and comments clean up - Removed CMA support 2: CXL DOE for CDAT and Compliance Mode. - Add default values for the absence of CDAT file - Add cdatCDAT file example for use with cdat= - Refactor DOE CXL compliance mode - Naming and comments clean up hchkuo (3): PCIe Data Object Exchange implementation CXL Data Object Exchange implementation PCIe standard header for DOE MAINTAINERS | 7 + hw/cxl/cxl-cdat.c | 220 ++ hw/cxl/meson.build| 1 + hw/mem/cxl_type3.c| 200 + hw/pci/meson.build| 1 + hw/pci/pcie_doe.c | 356 ++ include/hw/cxl/cxl_cdat.h | 149 + include/hw/cxl/cxl_compliance.h | 297 + include/hw/cxl/cxl_component.h| 7 + include/hw/cxl/cxl_device.h | 4 + include/hw/cxl/cxl_pci.h | 2 + include/hw/pci/pci_ids.h | 2 + include/hw/pci/pcie.h | 1 + include/hw/pci/pcie_doe.h | 123 +++ include/hw/pci/pcie_regs.h| 3 + include/standard-headers/linux/pci_regs.h | 3 +- tests/data/cdat/cdat.dat | Bin 0 -> 148 bytes 17 files changed, 1375 insertions(+), 1 deletion(-) create mode 100644 hw/cxl/cxl-cdat.c create mode 100644 hw/pci/pcie_doe.c create mode 100644 include/hw/cxl/cxl_cdat.h create mode 100644 include/hw/cxl/cxl_compliance.h create mode 100644 include/hw/pci/pcie_doe.h create mode 100644 tests/data/cdat/cdat.dat -- 1.8.3.1
[RFC PATCH v3 cxl-2.0-doe 2/2] CXL DOE support for CDAT and Compliance Mode
---
hw/cxl/cxl-component-utils.c | 93
hw/mem/cxl_type3.c | 184
include/hw/cxl/cxl_cdat.h | 127 +
include/hw/cxl/cxl_compl.h | 252 +
include/hw/cxl/cxl_component.h | 74 ++
include/hw/cxl/cxl_device.h| 4 +
include/hw/cxl/cxl_pci.h | 2 +
7 files changed, 736 insertions(+)
create mode 100644 include/hw/cxl/cxl_cdat.h
create mode 100644 include/hw/cxl/cxl_compl.h
diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
index 41d36f802a..1a2408dc76 100644
--- a/hw/cxl/cxl-component-utils.c
+++ b/hw/cxl/cxl-component-utils.c
@@ -11,6 +11,7 @@
#include "qemu/log.h"
#include "hw/pci/pci.h"
#include "hw/cxl/cxl.h"
+#include "qapi/error.h"
static uint64_t cxl_cache_mem_read_reg(void *opaque, hwaddr offset,
unsigned size)
@@ -206,3 +207,95 @@ void cxl_component_create_dvsec(CXLComponentState *cxl,
uint16_t length,
range_init_nofail(>dvsecs[type], cxl->dvsec_offset, length);
cxl->dvsec_offset += length;
}
+
+static void cdat_len_check(struct cdat_sub_header *hdr, Error **errp)
+{
+assert(hdr->length);
+assert(hdr->reserved == 0);
+
+switch (hdr->type) {
+case CDAT_TYPE_DSMAS:
+assert(hdr->length == sizeof(struct cdat_dsmas));
+break;
+case CDAT_TYPE_DSLBIS:
+assert(hdr->length == sizeof(struct cdat_dslbis));
+break;
+case CDAT_TYPE_DSMSCIS:
+assert(hdr->length == sizeof(struct cdat_dsmscis));
+break;
+case CDAT_TYPE_DSIS:
+assert(hdr->length == sizeof(struct cdat_dsis));
+break;
+case CDAT_TYPE_DSEMTS:
+assert(hdr->length == sizeof(struct cdat_dsemts));
+break;
+case CDAT_TYPE_SSLBIS:
+assert(hdr->length >= sizeof(struct cdat_sslbis_header));
+assert((hdr->length - sizeof(struct cdat_sslbis_header)) %
+ sizeof(struct cdat_sslbe) == 0);
+break;
+default:
+error_setg(errp, "Type %d is reserved", hdr->type);
+}
+}
+
+#define IASL_HEADER_LEN 0x24
+void cxl_doe_cdat_init(CXLComponentState *cxl_cstate, Error **errp)
+{
+uint8_t sum = 0;
+int i, curr = 0, ent = 0;
+CDATStruct cdat_st[1024];
+struct cdat_sub_header hdr;
+FILE *fp;
+uint8_t iasl_hdr[IASL_HEADER_LEN];
+size_t rcount;
+
+fp = fopen(cxl_cstate->cdat_filename, "r");
+
+if (fp) {
+/* Read iASL header */
+rcount= fread(_hdr, sizeof(iasl_hdr), 1, fp);
+ if (rcount == 0)
+error_setg(errp, "File read failed");
+
+
+for (i = 0; i < IASL_HEADER_LEN; i++) {
+sum += iasl_hdr[i];
+}
+sum = (~sum + 1) & 0xFF;
+
+curr = ftell(fp);
+
+/* Read CDAT structures */
+while (fread(, sizeof(hdr), 1, fp)) {
+cdat_len_check(, errp);
+
+cdat_st[ent].base2 = curr;
+cdat_st[ent].length = hdr.length;
+ent++;
+
+fseek(fp, curr + hdr.length, SEEK_SET);
+curr = ftell(fp);
+}
+/* Check the last structure */
+fseek(fp, 0, SEEK_END);
+assert(curr == ftell(fp));
+} else {
+error_setg(errp, "Please specify the CDAT file by using ',cdat='");
+}
+cxl_cstate->cdat_file = fp;
+
+cxl_cstate->cdat_ent_len = ent;
+cxl_cstate->cdat_ent = g_malloc0(sizeof(CDATStruct) * ent);
+memcpy(cxl_cstate->cdat_ent, cdat_st, sizeof(CDATStruct) * ent);
+
+/* Set CDAT header, ent = 0 */
+cxl_cstate->cdat_header.revision = CXL_CDAT_REV;
+cxl_cstate->cdat_header.sequence = 0;
+cxl_cstate->cdat_header.length += curr + sizeof(cxl_cstate->cdat_header);
+
+sum += cxl_cstate->cdat_header.revision +
+ cxl_cstate->cdat_header.sequence + cxl_cstate->cdat_header.length;
+
+cxl_cstate->cdat_header.checksum = ~sum + 1;
+}
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index bf33ddb915..33f571e1a5 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -13,6 +13,156 @@
#include "qemu/rcu.h"
#include "sysemu/hostmem.h"
#include "hw/cxl/cxl.h"
+#include "hw/pci/msi.h"
+#include "hw/pci/msix.h"
+
+uint32_t cxl_doe_compliance_init(DOECap *doe_cap)
+{
+CXLComponentState *cxl_cstate = (doe_cap->pdev)->cxl_cstate;
+uint32_t req;
+uint32_t byte_cnt = 0;
+
+DOE_DBG(">> %s\n", __func__);
+
+req = ((struct cxl_compliance_mode_cap *)pcie_doe_get_req(doe_cap))
+->req_header.req_code;
+switch (req) {
+case CXL_COMP_MODE_CAP:
+byte_cnt = sizeof(struct cxl_compliance_mode_cap_rsp);
+cxl_cstate->doe_resp.cap_rsp.header.vendor_id = CXL_VENDOR_ID;
+cxl_cstate->doe_resp.cap_rsp.header.doe_type = CXL_DOE_COMPLIANCE;
+cxl_cstate->doe_resp.cap_rsp.header.reserved = 0x0;
+cxl_cstate->doe_resp.cap_rsp.header.length =
+DIV_ROUND_UP(sizeof(struct
[RFC PATCH v3 cxl-2.0-doe 1/2] Basic PCIe DOE support
--- MAINTAINERS | 49 +-- hw/pci/meson.build| 1 + hw/pci/pci.c | 13 +- hw/pci/pcie_doe.c | 416 ++ include/hw/pci/pci_ids.h | 5 +- include/hw/pci/pcie.h | 1 + include/hw/pci/pcie_doe.h | 142 include/hw/pci/pcie_regs.h| 4 + include/standard-headers/linux/pci_regs.h | 3 +- 9 files changed, 591 insertions(+), 43 deletions(-) create mode 100644 hw/pci/pcie_doe.c create mode 100644 include/hw/pci/pcie_doe.h diff --git a/MAINTAINERS b/MAINTAINERS index f9097ed9e7..8c5a9690a5 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -87,7 +87,7 @@ S390 general architecture support M: Cornelia Huck M: Thomas Huth S: Supported -F: default-configs/*/s390x-softmmu.mak +F: default-configs/s390x-softmmu.mak F: gdb-xml/s390*.xml F: hw/char/sclp*.[hc] F: hw/char/terminal3270.c @@ -188,15 +188,6 @@ F: include/hw/cris/ F: tests/tcg/cris/ F: disas/cris.c -Hexagon TCG CPUs -M: Taylor Simpson -S: Supported -F: target/hexagon/ -F: linux-user/hexagon/ -F: tests/tcg/hexagon/ -F: disas/hexagon.c -F: default-configs/targets/hexagon-linux-user.mak - HPPA (PA-RISC) TCG CPUs M: Richard Henderson S: Maintained @@ -239,7 +230,7 @@ R: Jiaxun Yang R: Aleksandar Rikalo S: Odd Fixes F: target/mips/ -F: default-configs/*/*mips* +F: default-configs/*mips* F: disas/mips.c F: docs/system/cpu-models-mips.rst.inc F: hw/intc/mips_gic.c @@ -263,7 +254,7 @@ S: Maintained F: target/moxie/ F: disas/moxie.c F: hw/moxie/ -F: default-configs/*/moxie-softmmu.mak +F: default-configs/moxie-softmmu.mak NiosII TCG CPUs M: Chris Wulff @@ -272,7 +263,7 @@ S: Maintained F: target/nios2/ F: hw/nios2/ F: disas/nios2.c -F: default-configs/*/nios2-softmmu.mak +F: default-configs/nios2-softmmu.mak OpenRISC TCG CPUs M: Stafford Horne @@ -367,7 +358,7 @@ F: hw/xtensa/ F: tests/tcg/xtensa/ F: disas/xtensa.c F: include/hw/xtensa/xtensa-isa.h -F: default-configs/*/xtensa*.mak +F: default-configs/xtensa*.mak TriCore TCG CPUs M: Bastian Koppelmann @@ -1038,7 +1029,7 @@ AVR MCUs M: Michael Rolnik R: Sarah Harris S: Maintained -F: default-configs/*/avr-softmmu.mak +F: default-configs/avr-softmmu.mak F: hw/avr/ F: include/hw/char/avr_usart.h F: hw/char/avr_usart.c @@ -1067,7 +1058,7 @@ HP B160L M: Richard Henderson R: Helge Deller S: Odd Fixes -F: default-configs/*/hppa-softmmu.mak +F: default-configs/hppa-softmmu.mak F: hw/hppa/ F: pc-bios/hppa-firmware.img @@ -1183,7 +1174,6 @@ F: hw/intc/loongson_liointc.c F: hw/mips/loongson3_bootp.c F: hw/mips/loongson3_bootp.h F: hw/mips/loongson3_virt.c -F: tests/acceptance/machine_mips_loongson3v.py Boston M: Paul Burton @@ -1373,15 +1363,6 @@ F: include/hw/misc/mchp_pfsoc_dmc.h F: include/hw/misc/mchp_pfsoc_ioscb.h F: include/hw/misc/mchp_pfsoc_sysreg.h -SiFive Machines -M: Alistair Francis -M: Bin Meng -M: Palmer Dabbelt -L: qemu-ri...@nongnu.org -S: Supported -F: hw/*/*sifive*.c -F: include/hw/*/*sifive*.h - RX Machines --- rx-gdbsim @@ -1468,7 +1449,7 @@ F: hw/s390x/ F: include/hw/s390x/ F: hw/watchdog/wdt_diag288.c F: include/hw/watchdog/wdt_diag288.h -F: default-configs/*/s390x-softmmu.mak +F: default-configs/s390x-softmmu.mak F: tests/acceptance/machine_s390_ccw_virtio.py T: git https://gitlab.com/cohuck/qemu.git s390-next T: git https://github.com/borntraeger/qemu.git s390-next @@ -1681,6 +1662,13 @@ F: docs/pci* F: docs/specs/*pci* F: default-configs/pci.mak +PCIE DOE +M: Huai-Cheng Kuo +M: Chris Browy +S: Supported +F: include/hw/pci/pcie_doe.h +F: hw/pci/pcie_doe.c + ACPI/SMBIOS M: Michael S. Tsirkin M: Igor Mammedov @@ -1764,7 +1752,6 @@ F: hw/ssi/xilinx_* SD (Secure Card) M: Philippe Mathieu-Daudé -M: Bin Meng L: qemu-bl...@nongnu.org S: Odd Fixes F: include/hw/sd/sd* @@ -1859,7 +1846,6 @@ F: fsdev/ F: docs/interop/virtfs-proxy-helper.rst F: tests/qtest/virtio-9p-test.c T: git https://gitlab.com/gkurz/qemu.git 9p-next -T: git https://github.com/cschoenebeck/qemu.git 9p.next virtio-blk M: Stefan Hajnoczi @@ -2904,13 +2890,13 @@ F: accel/tcg/user-exec*.c BSD user S: Orphan F: bsd-user/ -F: default-configs/targets/*-bsd-user.mak +F: default-configs/*-bsd-user.mak Linux user M: Laurent Vivier S: Maintained F: linux-user/ -F: default-configs/targets/*linux-user.mak +F: default-configs/*-linux-user.mak F: scripts/qemu-binfmt-conf.sh F: scripts/update-syscalltbl.sh F: scripts/update-mips-syscall-args.sh @@ -2930,7 +2916,6 @@ S: Maintained F: docs/devel/tcg-plugins.rst F: plugins/ F: tests/plugin/ -F: tests/acceptance/tcg_plugins.py F: contrib/plugins/ AArch64 TCG target diff --git a/hw/pci/meson.build b/hw/pci/meson.build index 5c4bbac817..115e50222f 100644 --- a/hw/pci/meson.build +++ b/hw/pci/meson.build @@ -12,6 +12,7 @@ pci_ss.add(files( # allow plugging PCIe devices
[RFC PATCH v3 cxl-2.0-doe 0/2] Version 3 patch series for PCIe DOE for PCIe and CXL 2.0
Version 3 patch series for PCIe DOE for PCIe and CXL 2.0 implements all planned functionality. Based on QEMU version: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0v4 Summary: 1: PCIe DOE support for Discovery - Support multiple DOE instances for each own protocol set - MSI-X and polling supported - Update error and interrupt status in DOE Status register - Use static array to register callback function for DOE protocols - Deprecate DOE_SUCCESS and DOE_DISCARD - Add license headers 2: CXL DOE for CDAT and Compliance Mode. - Device supports pre-defined CDAT or user-provided CDAT. - Support on reading the iASL aml file via specifying "cdat=" property to -device cxl-type3 skips over the ACPI header and writes only CDAT table entries - Clean up CXL compliance structures - DOE CDAT response returns one CDAT Structure instance based on request EntryHandle value. Example cdat.dat file: (compile with iasl -G cdat.dat) CDAT file may contain any mix and number of supported CDAT Structure types -- /* Header */ Signature : "CDAT" Table Length : Revision : 01 Checksum : 00 Oem ID : AVERY Oem Table ID : 0 Oem Revision : 1 Asl Compiler ID : "INTL" Asl Compiler Revision : 20160527 /* CDAT structures */ Label : DSMAS // FieldByte Length UINT8 : 0 // Type 1 UINT8 : 00 // Reserved 1 UINT16 : 0018 // Length 2 UINT8 : 00 // DSMADHandle 1 UINT8 : 00 // Flags1 UINT16 : // Reserved 2 UINT64 : // DPA Base 8 UINT64 : // DPA Length 8 Label : DSLBIS // Field Byte Length UINT8 : 01 // Type 1 UINT8 : 00 // Reserved 1 UINT16 : 0018 // Length 2 UINT8 : 00 // Handle 1 UINT8 : 00 // Flags1 UINT8 : 00 // Data Type1 UINT8 : 00 // Reserved 1 UINT64 : // Entry Base Unit 8 UINT16 : // Entry[0] 2 UINT16 : // Entry[1] 2 UINT16 : // Entry[2] 2 UINT16 : // Reserved 2 Label: DSMSCIS // FieldByte Length UINT8 : 02 // Type 1 UINT8 : 00 // Reserved 1 UINT16 : 0014 // Length 2 UINT8 : 00 // DSMASHandle 1 UINT24 : 00 // Reserved 3 UINT64 : // Memory Side Cache Size8 UINT32 : // Cache Attributes 4 Label : DSIS// FieldByte Length UINT8 : 03 // Type 1 UINT8 : 00 // Reserved 1 UINT16 : 0008 // Length 2 UINT8 : 00 // Flags1 UINT8 : 00 // Handle 1 UINT16 : // Reserved 2 Label : DSEMTS // FieldByte Length UINT8 : 04 // Type 1 UINT8 : 00 // Reserved 1 UINT16 : 0018 // Length 2 UINT8 : 00 // DSMAS Handle 1 UINT8 : 00 // EFI Memory Type and Attribute1 UINT16 : // Reserved 2 UINT64 : // DPA Offset 8 UINT64 : // DPA Length 8 Label : SSLBIS // FieldByte Length UINT8 : 05 // Type 1 UINT8 : 00 // Reserved 1 UINT16 : 0020 // Length 2 UINT8 : 00 // Data Type1 UINT24 : 00 // Reserved 3 UINT64 : // Entry Base Unit 8 Label : SSLBE // SSLBE[0] UINT16 : // Port X ID2 UINT16 : // Port Y ID2 UINT16 : // Latency or Bandwidth2 UINT16 : // Reserved 2 Label : SSLBE // SSLBE[1] UINT16 : // Port X ID2 UINT16 : // Port Y ID2 UINT16 : // Latency or Bandwidth2 UINT16 : BBBC // Reserved 2 References: 1. CXL 2.0 specification https://www.computeexpresslink.org/download-the-specification 2. PCI-SIG ECN: Data Object Exchange (DOE) http://www.pcisig.com 3. Coherent Device Attribute Table CDAT 1.02 https://uefi.org/sites/default/files/resources/Coherent%20Device%20Attribute%20Table_1.02.pdf --- Chris Browy (2): Basic PCIe DOE support CXL DOE support for CDAT and Compliance Mode MAINTAINERS |
Re: [RFC PATCH v2 1/2] Basic PCIe DOE support
> On Mar 4, 2021, at 2:21 PM, Jonathan Cameron
> wrote:
>
> On Tue, 9 Feb 2021 15:35:49 -0500
> Chris Browy wrote:
>
> Hi Chris,
>
> One more thing hit whilst debugging linux side of this.
>
>> +static void pcie_doe_irq_assert(DOECap *doe_cap)
>> +{
>> +PCIDevice *dev = doe_cap->doe->pdev;
>> +
>> +if (doe_cap->cap.intr && doe_cap->ctrl.intr) {
>
>
> need something like
>
> doe_cap->status.intr = 1;
>
> I think or anyone checking the status register is going to think
> this interrupt is spurious.
You’re absolutely right, good catch!
>
> Otherwise all seems to work. I need to do a bit of tidying up on
> kernel code but should be able to send out early next week.
>
We’re putting out the v3 by end of this week. We’re spent a bit longer
tidying up on our end but sounds like coming together real soon in 5.12
release!
>> +/* Interrupt notify */
>> +if (msix_enabled(dev)) {
>> +msix_notify(dev, doe_cap->cap.vec);
>> +} else if (msi_enabled(dev)) {
>> +msi_notify(dev, doe_cap->cap.vec);
>> +}
>> +/* Not support legacy IRQ */
>> +}
>> +}
Re: [RFC v2 2/2] Basic CXL DOE for CDAT and Compliance Mode
> On Feb 18, 2021, at 2:15 PM, Jonathan Cameron
> wrote:
>
> On Fri, 12 Feb 2021 17:26:50 -0500
> Chris Browy wrote:
>
>>> On Feb 12, 2021, at 12:23 PM, Jonathan Cameron
>>> wrote:
>>>
>>> On Tue, 9 Feb 2021 15:36:03 -0500
>>> Chris Browy wrote:
>>>
>>> Split this into two patches for v3. CDAT in one, compliance mode in the
>>> other.
>>>
>>
>> Compliance mode is an optional feature. We’ll split it out.
>>
>>> I'd also move the actual elements out into the cxl components so that we
>>> can register only what makes sense for a given device. My guess
>>> is that for now that will be static const anyway.
>>>
>>> Coming together fine. Hopefully I'll start poking at the linux side of
>>> things
>>> next week. First job being simply providing a file to allow us to dump
>>> the whole CDAT table. Let me know if you get this loading an .aml file
>>> in the meantime as that'll make it easier to test (if not I'll hack it
>>> on top of these patches)
>>
>> We can get the .aml loading by Thurs next week. Holiday next few days for
>> some of our folks.
>>
>>>
>>> If needed I'll add it to iASL as well (may well be already in hand!)
>
> There is a potential problem doing this. CDAT doesn't have the table
> type ID that an ACPI table would have. That means raw CDAT tables
> are not identifiable and I think this makes it hard to use iASL with them
> without changing it's general means of functioning.
>
> We can probably do something with an extra parameter, but this lack of
> identifier is going to make it harder to persuade people that it's sensible to
> including CDAT in iASL.
This would be worth requesting the responsible ACPI or UEFI working group
of UEFI.org to weight in on the original intent since this must have been
considered
despite not being addressed in the specification.
The spec is clear:
Note: The data structures defined in this document are NOT ACPI tables.
I can’t find the author or working group designator in the spec although it is
Copyright 2020 Unified EFI, Inc. All Rights Reserved.
>
>>>
>>> I think my version of this stuff did a useful job in improving my
>>> understanding
>>> of what we were trying to do, but that done I'm assuming we'll just abandon
>>> it
>>> as the disposable prototype it was :)
>>>
>>
>> Thanks for focusing in on the area and uncovering problems with both our
>> versions!
>>
>> Still lots of pieces need to come together and get working to be able to
>> fully enumerate
>> and configure the device!
>>
>>> Jonathan
>>>
>>>
>>>> ---
>>>> hw/cxl/cxl-component-utils.c | 132 +++
>>>> hw/mem/cxl_type3.c | 172
>>>> include/hw/cxl/cxl_cdat.h | 120 +
>>>> include/hw/cxl/cxl_compl.h | 289
>>>> +
>>>> include/hw/cxl/cxl_component.h | 126 ++
>>>> include/hw/cxl/cxl_device.h| 3 +
>>>> include/hw/cxl/cxl_pci.h | 4 +
>>>> 7 files changed, 846 insertions(+)
>>>> create mode 100644 include/hw/cxl/cxl_cdat.h
>>>> create mode 100644 include/hw/cxl/cxl_compl.h
>>>>
>>>> diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
>>>> index e1bcee5..fc6c538 100644
>>>> --- a/hw/cxl/cxl-component-utils.c
>>>> +++ b/hw/cxl/cxl-component-utils.c
>>>> @@ -195,3 +195,135 @@ void cxl_component_create_dvsec(CXLComponentState
>>>> *cxl, uint16_t length,
>>>>range_init_nofail(>dvsecs[type], cxl->dvsec_offset, length);
>>>>cxl->dvsec_offset += length;
>>>> }
>>>> +
>>>> +/* Return the sum of bytes */
>>>> +static void cdat_ent_init(CDATStruct *cs, void *base, uint32_t len)
>>>> +{
>>>> +cs->base = base;
>>>> +cs->length = len;
>>>> +}
>>>> +
>>>> +void cxl_doe_cdat_init(CXLComponentState *cxl_cstate)
>>>> +{
>>>> +uint8_t sum = 0;
>>>> +uint32_t len = 0;
>>>> +int i, j;
>>>> +
>>>> +cxl_cstate->cdat_ent_len = 7;
>>>> +cxl_cstate->cdat_ent =
>>>> +g_malloc0(sizeof(CDATStruct) * cxl_cstate->cdat_ent_len);
>>>> +
>>>
Re: [RFC PATCH v2 1/2] Basic PCIe DOE support
> On Feb 18, 2021, at 2:11 PM, Jonathan Cameron > wrote: > > On Fri, 12 Feb 2021 16:58:21 -0500 > Chris Browy wrote: > >>> On Feb 12, 2021, at 11:24 AM, Jonathan Cameron >>> wrote: >>> >>> On Tue, 9 Feb 2021 15:35:49 -0500 >>> Chris Browy wrote: >>> >>> Run ./scripts/checkpatch.pl over the patches and fix all the warnings before >>> posting. It will save time by clearing out most of the minor formatting >>> issues >>> and similar that inevitably sneak in during development. >>> >> Excellent suggestion. We’re still newbies! >> >>> The biggest issue I'm seeing in here is that the abstraction of >>> multiple DOE capabiltiies accessing same protocols doesn't make sense. >>> >>> Each DOE ecap region and hence mailbox can have it's own set of >>> (possibly overlapping) protocols. >>> >>> From the ECN: >>> "It is permitted for a protocol using data object exchanges to require >>> that a Function implement a unique instance of DOE for that specific >>> protocol, and/or to allow sharing of a DOE instance to only a specific >>> set of protocols using data object exchange, and/or to allow a Function >>> to implement multiple instances of DOE supporting the specific protocol." >>> >>> Tightly couple the ECAP and DOE. If we are in the multiple instances >>> of DOE supporting a specific protocol case, then register it separately >>> for each one. The individual device emulation then needs to deal with >>> any possible clashes etc. >> >> Not sure how configurable we want to make the device. It is a simple type 3 >> device after all. > > Agreed, but what I (or someone else) really doesn't want to have to do > in the future is reimplement DOE because we made design decisions that make > this version hard to reuse. Unless it is particularly nasty to do we should > try to design something that is generally useful rather than targeted to > closely at the specific case we are dealing with. > > I'd argue the ECAP and the DOE mailbox are always tightly coupled 1-to-1. > Whether the device wants to implement multiple protocols on each DOE mailbox > or indeed run individual protocols on multiple DOE mailboxes is a design > decision, but the actual mechanics of DOE match up with the config > space structures anything else is impdef on the device. Yes I agree that there is 1-to-1 between DOE extended cap (ECAP) and DOE Mailbox. If we want to provide complete flexibility we should let the user pass device property arrays to QEMU command for how many DOE ECAP’s to build out and how to assign protocol(s) to each of them. Array index is the DOE instance #. Also we can provide a property for cdat binary (blob) filename to initialize the CDAT structure[entries]. This just reads in whatever mix of CDAT structure types are in the blob. -device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M \ doe-ecap-instances=2 \ doe-ecap[0]=5 // bitwise OR for protocols shared doe-ecap[1]=2 //bitwise OR for protocols shared doe-ecap-cdat[1]=mycdat.bin where let’s say protocols bitvector bit [0]=CMA bit [1]=CDAT bit [2}=Compliance Let me know if you some better alternatives and we’ll implement it. > >> >> The DOE spec does leave it pretty arbitrary regarding N DOE instances (DOE >> Extended Cap entry points) for M protocols, including where N>1 and M=1. >> Currently we implement N=2 DOE caps (instances), one for CDAT, one for >> Compliance Mode.[ >> >> Maybe a more complex MLD device might have one or more DOE instances >> for the CDAT protocol alone to define each HDM but currently we only have >> one pmem (SLD) so we can’t really do much more than what’s supported. >> >> Open to further suggestion though. Based on answer to above we’ll follow >> the suggestion lower in the code review regarding >> > ... >
Re: [RFC v2 2/2] Basic CXL DOE for CDAT and Compliance Mode
> On Feb 12, 2021, at 12:23 PM, Jonathan Cameron
> wrote:
>
> On Tue, 9 Feb 2021 15:36:03 -0500
> Chris Browy wrote:
>
> Split this into two patches for v3. CDAT in one, compliance mode in the
> other.
>
Compliance mode is an optional feature. We’ll split it out.
> I'd also move the actual elements out into the cxl components so that we
> can register only what makes sense for a given device. My guess
> is that for now that will be static const anyway.
>
> Coming together fine. Hopefully I'll start poking at the linux side of things
> next week. First job being simply providing a file to allow us to dump
> the whole CDAT table. Let me know if you get this loading an .aml file
> in the meantime as that'll make it easier to test (if not I'll hack it
> on top of these patches)
We can get the .aml loading by Thurs next week. Holiday next few days for
some of our folks.
>
> If needed I'll add it to iASL as well (may well be already in hand!)
>
> I think my version of this stuff did a useful job in improving my
> understanding
> of what we were trying to do, but that done I'm assuming we'll just abandon it
> as the disposable prototype it was :)
>
Thanks for focusing in on the area and uncovering problems with both our
versions!
Still lots of pieces need to come together and get working to be able to fully
enumerate
and configure the device!
> Jonathan
>
>
>> ---
>> hw/cxl/cxl-component-utils.c | 132 +++
>> hw/mem/cxl_type3.c | 172
>> include/hw/cxl/cxl_cdat.h | 120 +
>> include/hw/cxl/cxl_compl.h | 289
>> +
>> include/hw/cxl/cxl_component.h | 126 ++
>> include/hw/cxl/cxl_device.h| 3 +
>> include/hw/cxl/cxl_pci.h | 4 +
>> 7 files changed, 846 insertions(+)
>> create mode 100644 include/hw/cxl/cxl_cdat.h
>> create mode 100644 include/hw/cxl/cxl_compl.h
>>
>> diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
>> index e1bcee5..fc6c538 100644
>> --- a/hw/cxl/cxl-component-utils.c
>> +++ b/hw/cxl/cxl-component-utils.c
>> @@ -195,3 +195,135 @@ void cxl_component_create_dvsec(CXLComponentState
>> *cxl, uint16_t length,
>> range_init_nofail(>dvsecs[type], cxl->dvsec_offset, length);
>> cxl->dvsec_offset += length;
>> }
>> +
>> +/* Return the sum of bytes */
>> +static void cdat_ent_init(CDATStruct *cs, void *base, uint32_t len)
>> +{
>> +cs->base = base;
>> +cs->length = len;
>> +}
>> +
>> +void cxl_doe_cdat_init(CXLComponentState *cxl_cstate)
>> +{
>> +uint8_t sum = 0;
>> +uint32_t len = 0;
>> +int i, j;
>> +
>> +cxl_cstate->cdat_ent_len = 7;
>> +cxl_cstate->cdat_ent =
>> +g_malloc0(sizeof(CDATStruct) * cxl_cstate->cdat_ent_len);
>> +
>> +cdat_ent_init(_cstate->cdat_ent[0],
>> + _cstate->cdat_header,
>> sizeof(cxl_cstate->cdat_header));
>> +cdat_ent_init(_cstate->cdat_ent[1],
>> + _cstate->dsmas, sizeof(cxl_cstate->dsmas));
>> +cdat_ent_init(_cstate->cdat_ent[2],
>> + _cstate->dslbis, sizeof(cxl_cstate->dslbis));
>> +cdat_ent_init(_cstate->cdat_ent[3],
>> + _cstate->dsmscis, sizeof(cxl_cstate->dsmscis));
>> +cdat_ent_init(_cstate->cdat_ent[4],
>> + _cstate->dsis, sizeof(cxl_cstate->dsis));
>> +cdat_ent_init(_cstate->cdat_ent[5],
>> + _cstate->dsemts, sizeof(cxl_cstate->dsemts));
>> +cdat_ent_init(_cstate->cdat_ent[6],
>> + _cstate->sslbis, sizeof(cxl_cstate->sslbis));
>> +
>> +/* Set the DSMAS entry, ent = 1 */
>> +cxl_cstate->dsmas.header.type = CDAT_TYPE_DSMAS;
>> +cxl_cstate->dsmas.header.reserved = 0x0;
>> +cxl_cstate->dsmas.header.length = sizeof(cxl_cstate->dsmas);
>> +cxl_cstate->dsmas.DSMADhandle = 0x0;
>> +cxl_cstate->dsmas.flags = 0x0;
>> +cxl_cstate->dsmas.reserved2 = 0x0;
>> +cxl_cstate->dsmas.DPA_base = 0x0;
>> +cxl_cstate->dsmas.DPA_length = 0x4;
>
> Look to move the instances of these down into the memory device and expose
> cdat_ent_init() to there.
>
> That way, we can add whatever elements make sense for each type
> of component.
>
> Also have a cdat_ents_finalize() or similar to call at the end
> which calculates overall length + checksum.
Re: [RFC PATCH v2 1/2] Basic PCIe DOE support
> On Feb 12, 2021, at 11:24 AM, Jonathan Cameron
> wrote:
>
> On Tue, 9 Feb 2021 15:35:49 -0500
> Chris Browy wrote:
>
> Run ./scripts/checkpatch.pl over the patches and fix all the warnings before
> posting. It will save time by clearing out most of the minor formatting
> issues
> and similar that inevitably sneak in during development.
>
Excellent suggestion. We’re still newbies!
> The biggest issue I'm seeing in here is that the abstraction of
> multiple DOE capabiltiies accessing same protocols doesn't make sense.
>
> Each DOE ecap region and hence mailbox can have it's own set of
> (possibly overlapping) protocols.
>
> From the ECN:
> "It is permitted for a protocol using data object exchanges to require
> that a Function implement a unique instance of DOE for that specific
> protocol, and/or to allow sharing of a DOE instance to only a specific
> set of protocols using data object exchange, and/or to allow a Function
> to implement multiple instances of DOE supporting the specific protocol."
>
> Tightly couple the ECAP and DOE. If we are in the multiple instances
> of DOE supporting a specific protocol case, then register it separately
> for each one. The individual device emulation then needs to deal with
> any possible clashes etc.
Not sure how configurable we want to make the device. It is a simple type 3
device after all.
The DOE spec does leave it pretty arbitrary regarding N DOE instances (DOE
Extended Cap entry points) for M protocols, including where N>1 and M=1.
Currently we implement N=2 DOE caps (instances), one for CDAT, one for
Compliance Mode.
Maybe a more complex MLD device might have one or more DOE instances
for the CDAT protocol alone to define each HDM but currently we only have
one pmem (SLD) so we can’t really do much more than what’s supported.
Open to further suggestion though. Based on answer to above we’ll follow
the suggestion lower in the code review regarding
>
> Various comments inline, but mostly small stuff.
>
> Jonathan
>
>
>> ---
>> MAINTAINERS | 7 +
>> hw/pci/meson.build| 1 +
>> hw/pci/pcie.c | 2 +-
>> hw/pci/pcie_doe.c | 414
>> ++
>> include/hw/pci/pci_ids.h | 2 +
>> include/hw/pci/pcie.h | 1 +
>> include/hw/pci/pcie_doe.h | 166
>> include/hw/pci/pcie_regs.h| 4 +
>> include/standard-headers/linux/pci_regs.h | 3 +-
>> 9 files changed, 598 insertions(+), 2 deletions(-)
>> create mode 100644 hw/pci/pcie_doe.c
>> create mode 100644 include/hw/pci/pcie_doe.h
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 981dc92..4fb865e 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -1655,6 +1655,13 @@ F: docs/pci*
>> F: docs/specs/*pci*
>> F: default-configs/pci.mak
>>
>> +PCIE DOE
>> +M: Huai-Cheng Kuo
>> +M: Chris Browy
>> +S: Supported
>> +F: include/hw/pci/pcie_doe.h
>> +F: hw/pci/pcie_doe.c
>> +
>> ACPI/SMBIOS
>> M: Michael S. Tsirkin
>> M: Igor Mammedov
>> diff --git a/hw/pci/meson.build b/hw/pci/meson.build
>> index 5c4bbac..115e502 100644
>> --- a/hw/pci/meson.build
>> +++ b/hw/pci/meson.build
>> @@ -12,6 +12,7 @@ pci_ss.add(files(
>> # allow plugging PCIe devices into PCI buses, include them even if
>> # CONFIG_PCI_EXPRESS=n.
>> pci_ss.add(files('pcie.c', 'pcie_aer.c'))
>> +pci_ss.add(files('pcie_doe.c'))
>> softmmu_ss.add(when: 'CONFIG_PCI_EXPRESS', if_true: files('pcie_port.c',
>> 'pcie_host.c'))
>> softmmu_ss.add_all(when: 'CONFIG_PCI', if_true: pci_ss)
>>
>
> ...
>
>> diff --git a/hw/pci/pcie_doe.c b/hw/pci/pcie_doe.c
>> new file mode 100644
>> index 000..df8e92e
>> --- /dev/null
>> +++ b/hw/pci/pcie_doe.c
>> @@ -0,0 +1,414 @@
>
> Add a copyright header / license etc before v3.
>
>> +#include "qemu/osdep.h"
>> +#include "qemu/log.h"
>> +#include "qemu/error-report.h"
>> +#include "qapi/error.h"
>> +#include "qemu/range.h"
>> +#include "hw/pci/pci.h"
>> +#include "hw/pci/pcie.h"
>> +#include "hw/pci/pcie_doe.h"
>> +#include "hw/pci/msi.h"
>> +#include "hw/pci/msix.h"
>> +
>> +/*
>> + * DOE Default Protocols (Discovery, CMA)
>> + */
>> +/* Discovery Request Object */
>> +struct doe_discovery {
>> +DOEHeader header;
>>
Re: [RFC v2 2/2] Basic CXL DOE for CDAT and Compliance Mode
> On Feb 9, 2021, at 4:53 PM, Ben Widawsky wrote:
>
> A couple of high level comments below. Overall your approach was what I had
> imagined originally. The approach Jonathan took is likely more versatile (but
> harder to read, for sure).
>
> I'm fine with either and I hope you two can come to an agreement on what the
> best way forward is.
>
> My ultimate goal was to be able to take a CDAT from a real device and load it
> as
> a blob into the ct3d for regression testing. Not sure if that's actually
> possible or not.
I’d think so.
For CDAT/DOE method, you could setup CDAT as non-ACPI tables but compile
with ACPI iASL? UEFI owns the ACPI and CDAT specs and all the info is public.
For example using generic datatypes one can describe CDAT structure types and
create an arbitrary CDAT table with any mix of struct types and describe one or
more proximity
domains and their memory attributes. The ct3d device can read the “blob” or
.aml and setup
entry indexing as Jonathan mentioned previously. For example user could create
a
CDAT table and compile using iasl -G into a file.aml and disassemble
back
Into a file.dsl.
Here is example of CDAT header and DSMAS (with ACPI standard header as well):
Signature : "CDAT"
Table Length :
Revision : 01
Checksum : 00
Oem ID : "TEST"
Oem Table ID : "QEMU "
Oem Revision : 0001
Asl Compiler ID : "INTL"
Asl Compiler Revision : 0001
Label : CDATST
Label : CDAT_HDR
UINT32 : $CDATEND - $CDATST
UINT8 : 01 // Revision 1
UINT8 : 00 // Checksum 1
UINT24 : 00 // Reserved 6
UINT32 : // Sequence 4
Label : DSMAS // FieldByte Length
UINT8 : 00 // Type 1
UINT8 : 00 // Reserved 1
UINT16 : 0018 // Length 2
UINT8 : 00 // DSMADHandle 1
UINT8 : 00 // Flags1
UINT16 : // Reserved 2
UINT64 : // DPA Base 8
UINT64 : // DPA Length 8
For Device Option ROM method for CDAT, we could add a option rom to ct3d so
UEFI could
access CDAT through a EFI_ADAPTER_INFORMATION_PROTOCOL (CDAT type) entry.
>
> Thanks.
> Ben
>
> On 21-02-09 15:36:03, Chris Browy wrote:
>> ---
>> hw/cxl/cxl-component-utils.c | 132 +++
>> hw/mem/cxl_type3.c | 172
>> include/hw/cxl/cxl_cdat.h | 120 +
>> include/hw/cxl/cxl_compl.h | 289
>> +
>> include/hw/cxl/cxl_component.h | 126 ++
>> include/hw/cxl/cxl_device.h| 3 +
>> include/hw/cxl/cxl_pci.h | 4 +
>> 7 files changed, 846 insertions(+)
>> create mode 100644 include/hw/cxl/cxl_cdat.h
>> create mode 100644 include/hw/cxl/cxl_compl.h
>>
>> diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
>> index e1bcee5..fc6c538 100644
>> --- a/hw/cxl/cxl-component-utils.c
>> +++ b/hw/cxl/cxl-component-utils.c
>> @@ -195,3 +195,135 @@ void cxl_component_create_dvsec(CXLComponentState
>> *cxl, uint16_t length,
>> range_init_nofail(>dvsecs[type], cxl->dvsec_offset, length);
>> cxl->dvsec_offset += length;
>> }
>> +
>> +/* Return the sum of bytes */
>> +static void cdat_ent_init(CDATStruct *cs, void *base, uint32_t len)
>> +{
>> +cs->base = base;
>> +cs->length = len;
>> +}
>> +
>> +void cxl_doe_cdat_init(CXLComponentState *cxl_cstate)
>> +{
>> +uint8_t sum = 0;
>> +uint32_t len = 0;
>> +int i, j;
>> +
>> +cxl_cstate->cdat_ent_len = 7;
>> +cxl_cstate->cdat_ent =
>> +g_malloc0(sizeof(CDATStruct) * cxl_cstate->cdat_ent_len);
>> +
>> +cdat_ent_init(_cstate->cdat_ent[0],
>> + _cstate->cdat_header,
>> sizeof(cxl_cstate->cdat_header));
>> +cdat_ent_init(_cstate->cdat_ent[1],
>> + _cstate->dsmas, sizeof(cxl_cstate->dsmas));
>> +cdat_ent_init(_cstate->cdat_ent[2],
>> + _cstate->dslbis, sizeof(cxl_cstate->dslbis));
>> +cdat_ent_init(_cstate->cdat_ent[3],
>> + _cstate->dsmscis, sizeof(cxl_cstate->dsmscis));
>> +cdat_ent_init(_cstate->cdat_ent[4],
>> + _cstate->dsis, sizeof(cxl_cstate->dsis));
>> +cdat_ent_init(_cstate->cdat_ent[5],
>>
Re: [RFC PATCH v2 1/2] Basic PCIe DOE support
No consensus yet but I’d suggest that we’ll do the QEMU work and Jonathan
focuses
on the linux kernel and UEFI/edk2 and CXL SSWG efforts. Seems like
a way to maximize resources and everyone’s contribution and expertise. QEMU
part
requires the least expertise which is why we’re best suited for it compared to
other
areas ;)
Review comments will be folded into next patch.
> On Feb 9, 2021, at 4:42 PM, Ben Widawsky wrote:
>
> Have you/Jonathan come to consensus about which implementation is going
> forward?
> I'd rather not have to review two :D
>
> On 21-02-09 15:35:49, Chris Browy wrote:
>> ---
>> MAINTAINERS | 7 +
>> hw/pci/meson.build| 1 +
>> hw/pci/pcie.c | 2 +-
>> hw/pci/pcie_doe.c | 414
>> ++
>> include/hw/pci/pci_ids.h | 2 +
>> include/hw/pci/pcie.h | 1 +
>> include/hw/pci/pcie_doe.h | 166
>> include/hw/pci/pcie_regs.h| 4 +
>> include/standard-headers/linux/pci_regs.h | 3 +-
>> 9 files changed, 598 insertions(+), 2 deletions(-)
>> create mode 100644 hw/pci/pcie_doe.c
>> create mode 100644 include/hw/pci/pcie_doe.h
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 981dc92..4fb865e 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -1655,6 +1655,13 @@ F: docs/pci*
>> F: docs/specs/*pci*
>> F: default-configs/pci.mak
>>
>> +PCIE DOE
>> +M: Huai-Cheng Kuo
>> +M: Chris Browy
>> +S: Supported
>> +F: include/hw/pci/pcie_doe.h
>> +F: hw/pci/pcie_doe.c
>> +
>> ACPI/SMBIOS
>> M: Michael S. Tsirkin
>> M: Igor Mammedov
>> diff --git a/hw/pci/meson.build b/hw/pci/meson.build
>> index 5c4bbac..115e502 100644
>> --- a/hw/pci/meson.build
>> +++ b/hw/pci/meson.build
>> @@ -12,6 +12,7 @@ pci_ss.add(files(
>> # allow plugging PCIe devices into PCI buses, include them even if
>> # CONFIG_PCI_EXPRESS=n.
>> pci_ss.add(files('pcie.c', 'pcie_aer.c'))
>> +pci_ss.add(files('pcie_doe.c'))
>
> It looks like this should be like the below line:
> softmmu_ss.add(when: 'CONFIG_PCI_EXPRESS', if_true: pci_doe.c))
>
>> softmmu_ss.add(when: 'CONFIG_PCI_EXPRESS', if_true: files('pcie_port.c',
>> 'pcie_host.c'))
>> softmmu_ss.add_all(when: 'CONFIG_PCI', if_true: pci_ss)
>>
>> diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c
>> index 1ecf6f6..f7516c4 100644
>> --- a/hw/pci/pcie.c
>> +++ b/hw/pci/pcie.c
>> @@ -735,7 +735,7 @@ void pcie_cap_slot_write_config(PCIDevice *dev,
>>
>> hotplug_event_notify(dev);
>>
>> -/*
>> +/*
>
> Please drop this.
>
>> * 6.7.3.2 Command Completed Events
>> *
>> * Software issues a command to a hot-plug capable Downstream Port by
>> diff --git a/hw/pci/pcie_doe.c b/hw/pci/pcie_doe.c
>> new file mode 100644
>> index 000..df8e92e
>> --- /dev/null
>> +++ b/hw/pci/pcie_doe.c
>> @@ -0,0 +1,414 @@
>> +#include "qemu/osdep.h"
>> +#include "qemu/log.h"
>> +#include "qemu/error-report.h"
>> +#include "qapi/error.h"
>> +#include "qemu/range.h"
>> +#include "hw/pci/pci.h"
>> +#include "hw/pci/pcie.h"
>> +#include "hw/pci/pcie_doe.h"
>> +#include "hw/pci/msi.h"
>> +#include "hw/pci/msix.h"
>> +
>> +/*
>> + * DOE Default Protocols (Discovery, CMA)
>> + */
>> +/* Discovery Request Object */
>> +struct doe_discovery {
>> +DOEHeader header;
>> +uint8_t index;
>> +uint8_t reserved[3];
>> +} QEMU_PACKED;
>> +
>> +/* Discovery Response Object */
>> +struct doe_discovery_rsp {
>> +DOEHeader header;
>> +uint16_t vendor_id;
>> +uint8_t doe_type;
>> +uint8_t next_index;
>> +} QEMU_PACKED;
>> +
>> +/* Callback for Discovery */
>> +static bool pcie_doe_discovery_rsp(DOECap *doe_cap)
>> +{
>> +PCIEDOE *doe = doe_cap->doe;
>> +struct doe_discovery *req = pcie_doe_get_req(doe_cap);
>> +uint8_t index = req->index;
>> +DOEProtocol *prot = NULL;
>> +
>> +/* Request length mismatch, discard */
>> +if (req->header.length < dwsizeof(struct doe_discovery)) {
>
> Use DIV_ROUND_UP instead of rolling your own thing.
>
>> +return DOE_DISCARD;
>> +}
>> +
>> +/* Point to the r
[RFC PATCH v2 1/2] Basic PCIe DOE support
---
MAINTAINERS | 7 +
hw/pci/meson.build| 1 +
hw/pci/pcie.c | 2 +-
hw/pci/pcie_doe.c | 414 ++
include/hw/pci/pci_ids.h | 2 +
include/hw/pci/pcie.h | 1 +
include/hw/pci/pcie_doe.h | 166
include/hw/pci/pcie_regs.h| 4 +
include/standard-headers/linux/pci_regs.h | 3 +-
9 files changed, 598 insertions(+), 2 deletions(-)
create mode 100644 hw/pci/pcie_doe.c
create mode 100644 include/hw/pci/pcie_doe.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 981dc92..4fb865e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1655,6 +1655,13 @@ F: docs/pci*
F: docs/specs/*pci*
F: default-configs/pci.mak
+PCIE DOE
+M: Huai-Cheng Kuo
+M: Chris Browy
+S: Supported
+F: include/hw/pci/pcie_doe.h
+F: hw/pci/pcie_doe.c
+
ACPI/SMBIOS
M: Michael S. Tsirkin
M: Igor Mammedov
diff --git a/hw/pci/meson.build b/hw/pci/meson.build
index 5c4bbac..115e502 100644
--- a/hw/pci/meson.build
+++ b/hw/pci/meson.build
@@ -12,6 +12,7 @@ pci_ss.add(files(
# allow plugging PCIe devices into PCI buses, include them even if
# CONFIG_PCI_EXPRESS=n.
pci_ss.add(files('pcie.c', 'pcie_aer.c'))
+pci_ss.add(files('pcie_doe.c'))
softmmu_ss.add(when: 'CONFIG_PCI_EXPRESS', if_true: files('pcie_port.c',
'pcie_host.c'))
softmmu_ss.add_all(when: 'CONFIG_PCI', if_true: pci_ss)
diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c
index 1ecf6f6..f7516c4 100644
--- a/hw/pci/pcie.c
+++ b/hw/pci/pcie.c
@@ -735,7 +735,7 @@ void pcie_cap_slot_write_config(PCIDevice *dev,
hotplug_event_notify(dev);
-/*
+/*
* 6.7.3.2 Command Completed Events
*
* Software issues a command to a hot-plug capable Downstream Port by
diff --git a/hw/pci/pcie_doe.c b/hw/pci/pcie_doe.c
new file mode 100644
index 000..df8e92e
--- /dev/null
+++ b/hw/pci/pcie_doe.c
@@ -0,0 +1,414 @@
+#include "qemu/osdep.h"
+#include "qemu/log.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "qemu/range.h"
+#include "hw/pci/pci.h"
+#include "hw/pci/pcie.h"
+#include "hw/pci/pcie_doe.h"
+#include "hw/pci/msi.h"
+#include "hw/pci/msix.h"
+
+/*
+ * DOE Default Protocols (Discovery, CMA)
+ */
+/* Discovery Request Object */
+struct doe_discovery {
+DOEHeader header;
+uint8_t index;
+uint8_t reserved[3];
+} QEMU_PACKED;
+
+/* Discovery Response Object */
+struct doe_discovery_rsp {
+DOEHeader header;
+uint16_t vendor_id;
+uint8_t doe_type;
+uint8_t next_index;
+} QEMU_PACKED;
+
+/* Callback for Discovery */
+static bool pcie_doe_discovery_rsp(DOECap *doe_cap)
+{
+PCIEDOE *doe = doe_cap->doe;
+struct doe_discovery *req = pcie_doe_get_req(doe_cap);
+uint8_t index = req->index;
+DOEProtocol *prot = NULL;
+
+/* Request length mismatch, discard */
+if (req->header.length < dwsizeof(struct doe_discovery)) {
+return DOE_DISCARD;
+}
+
+/* Point to the requested protocol */
+if (index < doe->protocol_num) {
+prot = >protocols[index];
+}
+
+struct doe_discovery_rsp rsp = {
+.header = {
+.vendor_id = PCI_VENDOR_ID_PCI_SIG,
+.doe_type = PCI_SIG_DOE_DISCOVERY,
+.reserved = 0x0,
+.length = dwsizeof(struct doe_discovery_rsp),
+},
+.vendor_id = (prot) ? prot->vendor_id : 0x,
+.doe_type = (prot) ? prot->doe_type : 0xFF,
+.next_index = (index + 1) < doe->protocol_num ?
+ (index + 1) : 0,
+};
+
+pcie_doe_set_rsp(doe_cap, );
+
+return DOE_SUCCESS;
+}
+
+/* Callback for CMA */
+static bool pcie_doe_cma_rsp(DOECap *doe_cap)
+{
+doe_cap->status.error = 1;
+
+memset(doe_cap->read_mbox, 0,
+ PCI_DOE_MAX_DW_SIZE * sizeof(uint32_t));
+
+doe_cap->write_mbox_len = 0;
+
+return DOE_DISCARD;
+}
+
+/*
+ * DOE Utilities
+ */
+static void pcie_doe_reset_mbox(DOECap *st)
+{
+st->read_mbox_idx = 0;
+
+st->read_mbox_len = 0;
+st->write_mbox_len = 0;
+
+memset(st->read_mbox, 0, PCI_DOE_MAX_DW_SIZE * sizeof(uint32_t));
+memset(st->write_mbox, 0, PCI_DOE_MAX_DW_SIZE * sizeof(uint32_t));
+}
+
+/*
+ * Initialize the list and protocol for a device.
+ * This function won't add the DOE capabitity to your PCIe device.
+ */
+void pcie_doe_init(PCIDevice *dev, PCIEDOE *doe)
+{
+doe->pdev = dev;
+doe->head = NULL;
+doe->protocol_num = 0;
+
+/* Register two default protocol */
+//TODO : LINK LIST
+pcie_doe_register_protocol(doe, PCI_VENDOR_ID_PCI_SIG,
+PCI_SIG_DOE_DISCOVERY, pcie_doe_discovery_rsp);
+pcie_doe_register_protocol(doe, PCI_VENDOR_ID_PCI_SIG,
+PCI_SIG_DOE_CMA, pcie_doe_cma_rsp);
+}
+
+int p
[RFC v2 2/2] Basic CXL DOE for CDAT and Compliance Mode
---
hw/cxl/cxl-component-utils.c | 132 +++
hw/mem/cxl_type3.c | 172
include/hw/cxl/cxl_cdat.h | 120 +
include/hw/cxl/cxl_compl.h | 289 +
include/hw/cxl/cxl_component.h | 126 ++
include/hw/cxl/cxl_device.h| 3 +
include/hw/cxl/cxl_pci.h | 4 +
7 files changed, 846 insertions(+)
create mode 100644 include/hw/cxl/cxl_cdat.h
create mode 100644 include/hw/cxl/cxl_compl.h
diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
index e1bcee5..fc6c538 100644
--- a/hw/cxl/cxl-component-utils.c
+++ b/hw/cxl/cxl-component-utils.c
@@ -195,3 +195,135 @@ void cxl_component_create_dvsec(CXLComponentState *cxl,
uint16_t length,
range_init_nofail(>dvsecs[type], cxl->dvsec_offset, length);
cxl->dvsec_offset += length;
}
+
+/* Return the sum of bytes */
+static void cdat_ent_init(CDATStruct *cs, void *base, uint32_t len)
+{
+cs->base = base;
+cs->length = len;
+}
+
+void cxl_doe_cdat_init(CXLComponentState *cxl_cstate)
+{
+uint8_t sum = 0;
+uint32_t len = 0;
+int i, j;
+
+cxl_cstate->cdat_ent_len = 7;
+cxl_cstate->cdat_ent =
+g_malloc0(sizeof(CDATStruct) * cxl_cstate->cdat_ent_len);
+
+cdat_ent_init(_cstate->cdat_ent[0],
+ _cstate->cdat_header, sizeof(cxl_cstate->cdat_header));
+cdat_ent_init(_cstate->cdat_ent[1],
+ _cstate->dsmas, sizeof(cxl_cstate->dsmas));
+cdat_ent_init(_cstate->cdat_ent[2],
+ _cstate->dslbis, sizeof(cxl_cstate->dslbis));
+cdat_ent_init(_cstate->cdat_ent[3],
+ _cstate->dsmscis, sizeof(cxl_cstate->dsmscis));
+cdat_ent_init(_cstate->cdat_ent[4],
+ _cstate->dsis, sizeof(cxl_cstate->dsis));
+cdat_ent_init(_cstate->cdat_ent[5],
+ _cstate->dsemts, sizeof(cxl_cstate->dsemts));
+cdat_ent_init(_cstate->cdat_ent[6],
+ _cstate->sslbis, sizeof(cxl_cstate->sslbis));
+
+/* Set the DSMAS entry, ent = 1 */
+cxl_cstate->dsmas.header.type = CDAT_TYPE_DSMAS;
+cxl_cstate->dsmas.header.reserved = 0x0;
+cxl_cstate->dsmas.header.length = sizeof(cxl_cstate->dsmas);
+cxl_cstate->dsmas.DSMADhandle = 0x0;
+cxl_cstate->dsmas.flags = 0x0;
+cxl_cstate->dsmas.reserved2 = 0x0;
+cxl_cstate->dsmas.DPA_base = 0x0;
+cxl_cstate->dsmas.DPA_length = 0x4;
+
+/* Set the DSLBIS entry, ent = 2 */
+cxl_cstate->dslbis.header.type = CDAT_TYPE_DSLBIS;
+cxl_cstate->dslbis.header.reserved = 0;
+cxl_cstate->dslbis.header.length = sizeof(cxl_cstate->dslbis);
+cxl_cstate->dslbis.handle = 0;
+cxl_cstate->dslbis.flags = 0;
+cxl_cstate->dslbis.data_type = 0;
+cxl_cstate->dslbis.reserved2 = 0;
+cxl_cstate->dslbis.entry_base_unit = 0;
+cxl_cstate->dslbis.entry[0] = 0;
+cxl_cstate->dslbis.entry[1] = 0;
+cxl_cstate->dslbis.entry[2] = 0;
+cxl_cstate->dslbis.reserved3 = 0;
+
+/* Set the DSMSCIS entry, ent = 3 */
+cxl_cstate->dsmscis.header.type = CDAT_TYPE_DSMSCIS;
+cxl_cstate->dsmscis.header.reserved = 0;
+cxl_cstate->dsmscis.header.length = sizeof(cxl_cstate->dsmscis);
+cxl_cstate->dsmscis.DSMASH_handle = 0;
+cxl_cstate->dsmscis.reserved2[0] = 0;
+cxl_cstate->dsmscis.reserved2[1] = 0;
+cxl_cstate->dsmscis.reserved2[2] = 0;
+cxl_cstate->dsmscis.memory_side_cache_size = 0;
+cxl_cstate->dsmscis.cache_attributes = 0;
+
+/* Set the DSIS entry, ent = 4 */
+cxl_cstate->dsis.header.type = CDAT_TYPE_DSIS;
+cxl_cstate->dsis.header.reserved = 0;
+cxl_cstate->dsis.header.length = sizeof(cxl_cstate->dsis);
+cxl_cstate->dsis.flags = 0;
+cxl_cstate->dsis.handle = 0;
+cxl_cstate->dsis.reserved2 = 0;
+
+/* Set the DSEMTS entry, ent = 5 */
+cxl_cstate->dsemts.header.type = CDAT_TYPE_DSEMTS;
+cxl_cstate->dsemts.header.reserved = 0;
+cxl_cstate->dsemts.header.length = sizeof(cxl_cstate->dsemts);
+cxl_cstate->dsemts.DSMAS_handle = 0;
+cxl_cstate->dsemts.EFI_memory_type_attr = 0;
+cxl_cstate->dsemts.reserved2 = 0;
+cxl_cstate->dsemts.DPA_offset = 0;
+cxl_cstate->dsemts.DPA_length = 0;
+
+/* Set the SSLBIS entry, ent = 6 */
+cxl_cstate->sslbis.sslbis_h.header.type = CDAT_TYPE_SSLBIS;
+cxl_cstate->sslbis.sslbis_h.header.reserved = 0;
+cxl_cstate->sslbis.sslbis_h.header.length = sizeof(cxl_cstate->sslbis);
+cxl_cstate->sslbis.sslbis_h.data_type = 0;
+cxl_cstate->sslbis.sslbis_h.reserved2[0] = 0;
+cxl_cstate->sslbis.sslbis_h.reserved2[1] = 0;
+cxl_cstate->sslbis.sslbis_h.reserved2[2] = 0;
+/* Set the SSLBE entry */
+cxl_cstate->sslbis.sslbe[0].port_x_id = 0;
+cxl_cstate->sslbis.sslbe[0].port_y_id = 0;
+cxl_cstate->sslbis.sslbe[0].latency_bandwidth = 0;
+cxl_cstate->sslbis.sslbe[0].reserved = 0;
+/* Set the SSLBE entry */
+cxl_cstate->sslbis.sslbe[1].port_x_id = 1;
+
[RFC PATCH v2 0/2] PCIe DOE for PCIe and CXL 2.0 v2 release
Version 2 patch series for PCIe DOE for PCIe and CXL 2.0 Summary is 1: PCIe DOE support for Discovery and CMA. - MSI-X and polling supported 2: CXL DOE for CDAT and Compliance Mode. - DOE CDAT response returns one CDAT Structure instance based on request EntryHandle value. - One of each CDAT Structure types supported Based on QEMU version: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0v3 References: 1. CXL 2.0 specification https://www.computeexpresslink.org/download-the-specification 2. PCI-SIG ECN: Data Object Exchange (DOE) http://www.pcisig.com 3. Coherent Device Attribute Table CDAT 1.02 https://uefi.org/sites/default/files/resources/Coherent%20Device%20Attribute%20Table_1.02.pdf --- Chris Browy (2): Basic PCIe DOE support Basic CXL DOE for CDAT and Compliance Mode MAINTAINERS | 7 + hw/cxl/cxl-component-utils.c | 132 ++ hw/mem/cxl_type3.c| 172 + hw/pci/meson.build| 1 + hw/pci/pcie.c | 2 +- hw/pci/pcie_doe.c | 414 ++ include/hw/cxl/cxl_cdat.h | 120 + include/hw/cxl/cxl_compl.h| 289 + include/hw/cxl/cxl_component.h| 126 + include/hw/cxl/cxl_device.h | 3 + include/hw/cxl/cxl_pci.h | 4 + include/hw/pci/pci_ids.h | 2 + include/hw/pci/pcie.h | 1 + include/hw/pci/pcie_doe.h | 166 include/hw/pci/pcie_regs.h| 4 + include/standard-headers/linux/pci_regs.h | 3 +- 16 files changed, 1444 insertions(+), 2 deletions(-) create mode 100644 hw/pci/pcie_doe.c create mode 100644 include/hw/cxl/cxl_cdat.h create mode 100644 include/hw/cxl/cxl_compl.h create mode 100644 include/hw/pci/pcie_doe.h -- 1.8.3.1
Re: [RFC PATCH v1 01/01] PCIe DOE for PCIe and CXL 2.0
> On Feb 5, 2021, at 1:49 PM, Jonathan Cameron > wrote: > > On Fri, 5 Feb 2021 09:19:36 -0800 > Ben Widawsky wrote: > >> On 21-02-05 16:09:54, Jonathan Cameron wrote: >>> On Wed, 3 Feb 2021 23:53:53 -0500 >>> Chris Browy wrote: >>> >>>> Hi Jonathan, >>>> >>>> Thanks for the review comments and we'll put out a v2 patch series >>>> based on a genuine git send-email flow in a day or so and plan to include >>>> - functionally separate patches >>>> - new MSI-X support >>>> - few bugs found in CDAT table header + checksum generation >>>> - more fully respond to review comments (thanks again!) >>>> >>>> After the SSWG responds to your email on spec clarifications we'll work on >>>> adding user-defined CDAT entries. Thanks for raising the issues with SSWG! >>>> >>>> It would be good to collaborate on how best to specify external CDAT files. >>>> One idea is to provide -device command line property for filenames. Files >>>> could be ascii format specifying the CDAT struct instances with named >>>> fields and >>>> value pairs. Some checks could be adding when reading in the files. >>>> Users could >>>> specify the CDAT structure types in any order and have multiple instances. >>>> >>> >>> I'd keep away from ascii files for this. Whilst it is horrible in some ways >>> we should stick to command line ops. If we need a more structured format >>> then >>> similar to was proposed with hmat, via libvirt. >>> >>> Alternatively we could use compiled tables though we'd end up having to >>> parse >>> them to some degree. >>> >> >> Why parse? Initially (6 months ago), I was thinking CDAT could just be a >> blob. >> The thing I liked about that approach was that when real devices came along, >> we >> could dump their CDATs and use it directly. > > See the CXL SSWG thread. Need to break it up into entries. So trivial bit of > walking to find the starts of the different entries (not really parsing I > guess) Right we do need to extract each CDAT Structure[n] to generate the response. Is blob The same as using IASL/AML Data Table Compiler in a manner consistent with how the non-IASL/AML tables (like SRAT) are compiled? 1.1 Data Table Compiler The Data Table compiler is used to compile the “non-ASL/AML” ACPI tables such as the FADT, MADT, SRAT, etc. These tables are not compiled to AML byte code, but are compiled to simple binary data, usually with the standard ACPI table header (signature, length, checksum, etc.) Seems that they may have intended given the CDAT table header is consistent with ACPI table header. Jonathan will you clarify this topic in the ECN? > >> >>>> >>>> Just like you we feel what's most important is to have DOE supported so >>>> that >>>> UEFI and Linux kernel and drivers can progress. We're also contributing to >>>> writing compliance tests for the CXL Compliance Software Development WG. >>> >>> Great. >> >> Is anyone doing the kernel enabling for it? > > Planning to look at this but plenty of other things on my todo list if someone > else gets to it first. > > Generic DOE support should be straight forward (the infrastructure). > Parsing CDAT also straight forward. > Doing something with the results is hard unless we just provide an interface > for > userspace to query them for a given device - or dump the table > (I think we do want to be able to that). > > What I'm really not sure on is how to handle NUMA domains that are created > late > in the kernel boot sequence. The ACPI flow is set up with the assumption > that we can get them from SRAT very early in boot. Need to figure out how to > work around that. (e.g. preallocate a bunch of spare nodes for example though > that's > ugly). Note IIRC the kernel doesn't do runtime update of any of the ACPI > performance parameters yet (_SLI, _HMA) so there probably isn't any > infrastructure > that we can reuse. > > There is also the firmware based enumeration and description option (OS not > necessarily > aware of CXL) in which this is all up to EDK2 and the kernel gets it all > presented > as standard tables. Do we know who’s on this as part of the EDK2 development? It would be great if they could address the SRAT/HMAT generation from reading CDAT. EDK2 does address CXL 1.1 now. > > As you can perhaps tell from my half done reviews, this week disappeared
Re: [RFC PATCH v1 01/01] PCIe DOE for PCIe and CXL 2.0
Hi Jonathan,
Thanks for the review comments and we'll put out a v2 patch series
based on a genuine git send-email flow in a day or so and plan to include
- functionally separate patches
- new MSI-X support
- few bugs found in CDAT table header + checksum generation
- more fully respond to review comments (thanks again!)
After the SSWG responds to your email on spec clarifications we'll work on
adding user-defined CDAT entries. Thanks for raising the issues with SSWG!
It would be good to collaborate on how best to specify external CDAT files.
One idea is to provide -device command line property for filenames. Files
could be ascii format specifying the CDAT struct instances with named fields and
value pairs. Some checks could be adding when reading in the files. Users
could
specify the CDAT structure types in any order and have multiple instances.
Just like you we feel what's most important is to have DOE supported so that
UEFI and Linux kernel and drivers can progress. We're also contributing to
writing compliance tests for the CXL Compliance Software Development WG.
Note your email did not post to lore.kernel.org/qemu-devel despite being CC’d.
Maybe a --in-replies-to issue. I’ve restored that here in this email reply.
Best Regards,
Chris
On 2/3/21, 12:19 PM, "Jonathan Cameron" wrote:
On Tue, 2 Feb 2021 15:43:28 -0500
Chris Browy wrote:
Hi Chris,
Whilst I appreciate that this is very much an RFC and so not in the
form you would eventually aim to present it in, please look for
a v2 to break this into a series of functionally separate patches.
Probably.
1. Introduce DOE support with no users - probably including the
discovery protocol
2. CMA support
3. CDAT support for CXL
4. Compliance part.
It's also well worth jumping through the hoops needed to get a
git send-email workflow up and running as you seem to have had some
trouble with getting the thread to send in one go etc.
Clearly we now have two possible implementations for this functionality.
Personally I don't care which one we take forwards - if nothing else
the exercise has highlighted some disagreements in spec interpretation
that need clearing up. I've mailed one big one to the SSWG list today.
I found a few things I definitely got wrong as well whilst reading this :)
Always advantages in having multiple implementations given we don't have
hardware yet.
Jonathan
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 981dc92e25..4fb865e0b3 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1655,6 +1655,13 @@ F: docs/pci*
> F: docs/specs/*pci*
> F: default-configs/pci.mak
>
> +PCIE DOE
> +M: Huai-Cheng Kuo
> +M: Chris Browy
> +S: Supported
> +F: include/hw/pci/pcie_doe.h
> +F: hw/pci/pcie_doe.c
> +
> ACPI/SMBIOS
> M: Michael S. Tsirkin
> M: Igor Mammedov
> diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
> index e1bcee5bdb..c49d2aa896 100644
> --- a/hw/cxl/cxl-component-utils.c
> +++ b/hw/cxl/cxl-component-utils.c
> @@ -195,3 +195,154 @@ void cxl_component_create_dvsec(CXLComponentState
*cxl, uint16_t length,
> range_init_nofail(>dvsecs[type], cxl->dvsec_offset, length);
> cxl->dvsec_offset += length;
> }
> +
> +uint32_t cxl_doe_compliance_init(CXLComponentState *cxl_cstate)
> +{
> +PCIDevice *pci_dev = cxl_cstate->pdev;
> +uint32_t req;
> +uint32_t byte_cnt = 0;
> +
> +DOE_DBG(">> %s\n", __func__);
> +
> +req = ((struct cxl_compliance_mode_cap *)pcie_doe_get_req(pci_dev))
> +->req_code;
> +switch (req) {
> +case CXL_COMP_MODE_CAP:
> +byte_cnt = sizeof(struct cxl_compliance_mode_cap_rsp);
> +cxl_cstate->doe_resp.cap_rsp.header.vendor_id = CXL_VENDOR_ID;
> +cxl_cstate->doe_resp.cap_rsp.header.doe_type =
CXL_DOE_COMPLIANCE;
> +cxl_cstate->doe_resp.cap_rsp.header.reserved = 0x0;
> +cxl_cstate->doe_resp.cap_rsp.header.length =
> +dwsizeof(struct cxl_compliance_mode_cap_rsp);
> +cxl_cstate->doe_resp.cap_rsp.rsp_code = 0x0;
> +cxl_cstate->doe_resp.cap_rsp.version = 0x1;
> +cxl_cstate->doe_resp.cap_rsp.length = 0x1c;
> +cxl_cstate->doe_resp.cap_rsp.status = 0x0;
> +cxl_cstate->doe_resp.cap_rsp.available_cap_bitmask = 0x3;
> +cxl_cstate->doe_resp.cap_rsp.enabled_cap_bitmask = 0x3;
> +break;
> +case CXL_COMP_MODE_STATUS:
> +byte_cnt = sizeof(struct cxl_compliance_mode_status_rsp);
[RFC PATCH v1 00/01] PCIe DOE for PCIe and CXL 2.0
PCIe Data Object Exchange (DOE) protocol for PCIe and CXL is available
https://gitlab.com/avery-qemu/cxl2.0-v3-doe/
based on Ben Widawsky's CXL QEMU cxl2.0-v3 gitlab branch
https://lore.kernel.org/qemu-devel/20210202005948.241655-1-ben.widaw...@intel.com
which is located at
https://gitlab.com/bwidawsk/qemu
The changes from Ben’s latest cxl-2.0v3 are:
MAINTAINERS | 7 +
hw/cxl/cxl-component-utils.c | 151 +++
hw/mem/cxl_type3.c| 121 +
hw/pci/meson.build| 1 +
hw/pci/pcie.c | 4 +-
hw/pci/pcie_doe.c | 360 +
include/hw/cxl/cxl_component.h| 120 +
include/hw/cxl/cxl_pci.h | 428
++
include/hw/pci/pcie.h | 5 +
include/hw/pci/pcie_doe.h | 130 +
include/hw/pci/pcie_regs.h| 4 +
include/standard-headers/linux/pci_regs.h | 3 +-
12 files changed, 1332 insertions(+), 2 deletions(-)
The DOE protocol defines a mailbox method that allows either UEFI or OS
methods
to read the device and do further setup of ACPI tables, etc.
There are 2 PCIe DOE protocols (PCI-SIG ECN Data Object Exchange (DOE) March
2020)
- Discovery
- Component Measurement (CMA)
And 2 CXL specific ones:
- Compliance Mode (Compute Express Link Specification September 2, 2020
Revision: 2.0, Version 1.0)
- CDAT (Coherent Device Attribute Table (CDAT) Specification
October 2020 Revision 1.02)
For CXL, the CDAT table defines the memory device so that UEFI or OS can read
it out of device using DOE and then can configure the system’s ACPI SRAT/HMAT
tables for system memory, and DEVSEC, Component, and Device registers in CXL
device.
Current version provides fixed CDAT table defined in the CXL Type3 device
model. Updates are planned shortly to allow for user to provide CDAT tables
through -device option property to vary from run to run. The format will be
ASCII with structure/field-values pairs that are read by the device during
initialization as shown here:
-device
cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M,cdat_file=
For testing, a cxl_app.c user program is enhanced to test all supported DOE
protocols which are comprised of sequences of CFG RD/WR to various DOE cap
registers. The Linux kernel updates and CXL Type3 Device driver provide
sufficient ioctl() support to exercise the DOE protocol. See
https://lore.kernel.org/linux-cxl/20210130002438.1872527-1-ben.widaw...@intel.com
cxl_app.c
#include
#include
#include
#include
#include
#include "cxl_mem_wrapper.h"
const char* help= "\
-h help message\n\
-query IOCTL CXL_MEM_QUERY_COMMANDS\n\
-cfg_rd [0xoffset] IOCTL CXL_MEM_CONFIG_WR Read Hex\n\
-cfg_wr [0xoffset] [0xaddr] IOCTL CXL_MEM_CONFIG_WR Write Hex\n\
-doe_discovery [0xindex=0-3] IOCTL CXL_MEM_CONFIG_WR Write Hex\n\
-doe_cxl [0xprotocol=0 or 2] [0xreq_code=0,1 for protocol=0]\n\
-doe_cma [0xnum = 0] IOCTL CXL_MEM_CONFIG_WR Write Hex\n\
example:\n\
./cxl_app.exe -cfg_rd 0x00\n\
./cxl_app.exe -cfg_wr 0x10 0x00ff0004\n\
./cxl_app.exe -doe_discovery 0\n\
./cxl_app.exe -doe_cxl 2\n\
./cxl_app.exe -doe_cxl 0 1\n\
./cxl_app.exe -doe_cma 0\n\
";
#define READ 0 #define WRITE 1
int FD;
typedef struct cxl_pdev_config cxl_pdev_config;
int cxl_query() {
typedef struct cxl_mem_query_commands cxl_mem_query_commands;
typedef struct cxl_command_info cxl_command_info;
int n_cmds= 0;
// QUERY with n_commands == 0 to get command size
ioctl(FD, CXL_MEM_QUERY_COMMANDS, _cmds);
printf("Querying\n");
cxl_mem_query_commands* cmds= malloc(sizeof(cxl_mem_query_commands)
+ n_cmds * sizeof(cxl_command_info));
cmds->n_commands= n_cmds;
// QUERY with command size & pre-alloc memory
ioctl(FD, CXL_MEM_QUERY_COMMANDS, cmds);
for (int i= 0; i < (int)cmds->n_commands; i++) {
printf(" id %d", cmds->commands[i].id);
printf(" flags %d", cmds->commands[i].flags);
printf(" size_in %d", cmds->commands[i].size_in);
printf(" size_out %d\n", cmds->commands[i].size_out);
}
return 0;
};
int cxl_config(char* offset_s, char* data_s) {
int offset, data, is_write;
cxl_pdev_config* config_payload= malloc(sizeof(cxl_pdev_config));
if (data_s == NULL)
is_write= 0;
else {
is_write= 1;
data= strtol(data_s, NULL, 16);
}
offset= strtol(offset_s, NULL, 16);
config_payload->offset= offset;
config_payload->data= data;
config_payload->is_write= is_write;
ioctl(FD, CXL_MEM_CONFIG_WR, config_payload);
printf("CONFIG_WR %s [%0x] ", (is_write)? "write" : "read",
config_payload->offset);
for (int i= 0; i < 32; i += 8) printf("
Re: [RFC PATCH v1 01/01] PCIe DOE for PCIe and CXL 2.0
diff --git a/MAINTAINERS b/MAINTAINERS
index 981dc92e25..4fb865e0b3 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1655,6 +1655,13 @@ F: docs/pci*
F: docs/specs/*pci*
F: default-configs/pci.mak
+PCIE DOE
+M: Huai-Cheng Kuo
+M: Chris Browy
+S: Supported
+F: include/hw/pci/pcie_doe.h
+F: hw/pci/pcie_doe.c
+
ACPI/SMBIOS
M: Michael S. Tsirkin
M: Igor Mammedov
diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
index e1bcee5bdb..c49d2aa896 100644
--- a/hw/cxl/cxl-component-utils.c
+++ b/hw/cxl/cxl-component-utils.c
@@ -195,3 +195,154 @@ void cxl_component_create_dvsec(CXLComponentState *cxl,
uint16_t length,
range_init_nofail(>dvsecs[type], cxl->dvsec_offset, length);
cxl->dvsec_offset += length;
}
+
+uint32_t cxl_doe_compliance_init(CXLComponentState *cxl_cstate)
+{
+PCIDevice *pci_dev = cxl_cstate->pdev;
+uint32_t req;
+uint32_t byte_cnt = 0;
+
+DOE_DBG(">> %s\n", __func__);
+
+req = ((struct cxl_compliance_mode_cap *)pcie_doe_get_req(pci_dev))
+->req_code;
+switch (req) {
+case CXL_COMP_MODE_CAP:
+byte_cnt = sizeof(struct cxl_compliance_mode_cap_rsp);
+cxl_cstate->doe_resp.cap_rsp.header.vendor_id = CXL_VENDOR_ID;
+cxl_cstate->doe_resp.cap_rsp.header.doe_type = CXL_DOE_COMPLIANCE;
+cxl_cstate->doe_resp.cap_rsp.header.reserved = 0x0;
+cxl_cstate->doe_resp.cap_rsp.header.length =
+dwsizeof(struct cxl_compliance_mode_cap_rsp);
+cxl_cstate->doe_resp.cap_rsp.rsp_code = 0x0;
+cxl_cstate->doe_resp.cap_rsp.version = 0x1;
+cxl_cstate->doe_resp.cap_rsp.length = 0x1c;
+cxl_cstate->doe_resp.cap_rsp.status = 0x0;
+cxl_cstate->doe_resp.cap_rsp.available_cap_bitmask = 0x3;
+cxl_cstate->doe_resp.cap_rsp.enabled_cap_bitmask = 0x3;
+break;
+case CXL_COMP_MODE_STATUS:
+byte_cnt = sizeof(struct cxl_compliance_mode_status_rsp);
+cxl_cstate->doe_resp.status_rsp.header.vendor_id = CXL_VENDOR_ID;
+cxl_cstate->doe_resp.status_rsp.header.doe_type = CXL_DOE_COMPLIANCE;
+cxl_cstate->doe_resp.status_rsp.header.reserved = 0x0;
+cxl_cstate->doe_resp.status_rsp.header.length =
+dwsizeof(struct cxl_compliance_mode_status_rsp);
+cxl_cstate->doe_resp.status_rsp.rsp_code = 0x1;
+cxl_cstate->doe_resp.status_rsp.version = 0x1;
+cxl_cstate->doe_resp.status_rsp.length = 0x14;
+cxl_cstate->doe_resp.status_rsp.cap_bitfield = 0x3;
+cxl_cstate->doe_resp.status_rsp.cache_size = 0;
+cxl_cstate->doe_resp.status_rsp.cache_size_units = 0;
+break;
+default:
+break;
+}
+
+DOE_DBG(" REQ=%x, RSP BYTE_CNT=%d\n", req, byte_cnt);
+DOE_DBG("<< %s\n", __func__);
+return byte_cnt;
+}
+
+void cxl_doe_cdat_init(CXLComponentState *cxl_cstate)
+{
+
+DOE_DBG(">> %s\n", __func__);
+
+cxl_cstate->doe_resp.cdat_rsp.header.vendor_id = CXL_VENDOR_ID;
+cxl_cstate->doe_resp.cdat_rsp.header.doe_type = CXL_DOE_TABLE_ACCESS;
+cxl_cstate->doe_resp.cdat_rsp.header.reserved = 0x0;
+cxl_cstate->doe_resp.cdat_rsp.header.length = 0;
+cxl_cstate->doe_resp.cdat_rsp.rsp_code = 0x0;
+cxl_cstate->doe_resp.cdat_rsp.table_type = 0x0;
+cxl_cstate->doe_resp.cdat_rsp.next_entry_handle = 0x;
+
+/* copy the DSMAS entry */
+cxl_cstate->dsmas.type = CDAT_TYPE_DSMAS;
+cxl_cstate->dsmas.reserved = 0x0;
+cxl_cstate->dsmas.length = 0x0;
+cxl_cstate->dsmas.DSMADhandle = 0x0;
+cxl_cstate->dsmas.flags = 0x0;
+cxl_cstate->dsmas.reserved2 = 0x0;
+cxl_cstate->dsmas.DPA_base = 0x0;
+cxl_cstate->dsmas.DPA_length = 0x4;
+
+/* copy the DSLBIS entry */
+cxl_cstate->dslbis.type = CDAT_TYPE_DSLBIS;
+cxl_cstate->dslbis.reserved = 0;
+cxl_cstate->dslbis.length = 16;
+cxl_cstate->dslbis.handle = 0;
+cxl_cstate->dslbis.flags = 0;
+cxl_cstate->dslbis.data_type = 0;
+cxl_cstate->dslbis.reserved2 = 0;
+cxl_cstate->dslbis.entry_base_unit = 0;
+cxl_cstate->dslbis.entry[0] = 0;
+cxl_cstate->dslbis.entry[1] = 0;
+cxl_cstate->dslbis.entry[2] = 0;
+cxl_cstate->dslbis.reserved3 = 0;
+
+/* copy the DSMSCIS entry */
+cxl_cstate->dsmscis.type = CDAT_TYPE_DSMSCIS;
+cxl_cstate->dsmscis.reserved = 0;
+cxl_cstate->dsmscis.length = 20;
+cxl_cstate->dsmscis.DSMASH_handle = 0;
+cxl_cstate->dsmscis.reserved2[0] = 0;
+cxl_cstate->dsmscis.reserved2[1] = 0;
+cxl_cstate->dsmscis.reserved2[2] = 0;
+cxl_cstate->dsmscis.memory_side_cache_size = 0;
+cxl_cstate->dsmscis.cache_attributes = 0;
+
+/* copy the DSIS entry */
+cxl_cstate->dsis.type = CDAT_TYPE_DSIS;
+cxl_cstate->dsis.reserved = 0
[RFC PATCH v1 00/01] PCIe DOE for PCIe and CXL 2.0
PCIe Data Object Exchange (DOE) protocol for PCIe and CXL is available
https://gitlab.com/avery-qemu/cxl2.0-v3-doe/
based on Ben Widawsky's CXL QEMU cxl2.0-v3 gitlab branch
https://lore.kernel.org/qemu-devel/20210202005948.241655-1-ben.widaw...@intel.com
which is located at
https://gitlab.com/bwidawsk/qemu
The changes from Ben’s latest cxl-2.0v3 are:
MAINTAINERS | 7 +
hw/cxl/cxl-component-utils.c | 151 +++
hw/mem/cxl_type3.c| 121 +
hw/pci/meson.build| 1 +
hw/pci/pcie.c | 4 +-
hw/pci/pcie_doe.c | 360 +
include/hw/cxl/cxl_component.h| 120 +
include/hw/cxl/cxl_pci.h | 428 ++
include/hw/pci/pcie.h | 5 +
include/hw/pci/pcie_doe.h | 130 +
include/hw/pci/pcie_regs.h| 4 +
include/standard-headers/linux/pci_regs.h | 3 +-
12 files changed, 1332 insertions(+), 2 deletions(-)
The DOE protocol defines a mailbox method that allows either UEFI or OS methods
to read the device and do further setup of ACPI tables, etc.
There are 2 PCIe DOE protocols (PCI-SIG ECN Data Object Exchange (DOE) March
2020)
- Discovery
- Component Measurement (CMA)
And 2 CXL specific ones:
- Compliance Mode (Compute Express Link Specification September 2, 2020
Revision: 2.0, Version 1.0)
- CDAT (Coherent Device Attribute Table (CDAT) Specification
October 2020 Revision 1.02)
For CXL, the CDAT table defines the memory device so that UEFI or OS can
read it out of device using DOE and then can configure the system’s ACPI SRAT/HMAT
tables for system memory, and DEVSEC, Component, and Device registers in CXL
device.
Current version provides fixed CDAT table defined in the CXL Type3 device
model. Updates are planned shortly to allow for user to provide CDAT tables
through -device option property to vary from run to run. The format will be
ASCII with structure/field-values pairs that are read by the device during
initialization as shown here:
-device
cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M,cdat_file=
For testing, a cxl_app.c user program is enhanced to test all supported
DOE protocols which are comprised of sequences of CFG RD/WR to various DOE cap
registers. The Linux kernel updates and CXL Type3 Device driver
provide sufficient ioctl() support to exercise the DOE protocol. See
https://lore.kernel.org/linux-cxl/20210130002438.1872527-1-ben.widaw...@intel.com
cxl_app.c
#include
#include
#include
#include
#include
#include "cxl_mem_wrapper.h"
const char* help= "\
-h help message\n\
-query IOCTL CXL_MEM_QUERY_COMMANDS\n\
-cfg_rd [0xoffset] IOCTL CXL_MEM_CONFIG_WR Read Hex\n\
-cfg_wr [0xoffset] [0xaddr] IOCTL CXL_MEM_CONFIG_WR Write Hex\n\
-doe_discovery [0xindex=0-3] IOCTL CXL_MEM_CONFIG_WR Write Hex\n\
-doe_cxl [0xprotocol=0 or 2] [0xreq_code=0,1 for protocol=0]\n\
-doe_cma [0xnum = 0] IOCTL CXL_MEM_CONFIG_WR Write Hex\n\
example:\n\
./cxl_app.exe -cfg_rd 0x00\n\
./cxl_app.exe -cfg_wr 0x10 0x00ff0004\n\
./cxl_app.exe -doe_discovery 0\n\
./cxl_app.exe -doe_cxl 2\n\
./cxl_app.exe -doe_cxl 0 1\n\
./cxl_app.exe -doe_cma 0\n\
";
#define READ 0
#define WRITE 1
int FD;
typedef struct cxl_pdev_config cxl_pdev_config;
int cxl_query() {
typedef struct cxl_mem_query_commands cxl_mem_query_commands;
typedef struct cxl_command_info cxl_command_info;
int n_cmds= 0;
// QUERY with n_commands == 0 to get command size
ioctl(FD, CXL_MEM_QUERY_COMMANDS, _cmds);
printf("Querying\n");
cxl_mem_query_commands* cmds= malloc(sizeof(cxl_mem_query_commands)
+ n_cmds * sizeof(cxl_command_info));
cmds->n_commands= n_cmds;
// QUERY with command size & pre-alloc memory
ioctl(FD, CXL_MEM_QUERY_COMMANDS, cmds);
for (int i= 0; i < (int)cmds->n_commands; i++) {
printf(" id %d", cmds->commands[i].id);
printf(" flags %d", cmds->commands[i].flags);
printf(" size_in %d", cmds->commands[i].size_in);
printf(" size_out %d\n", cmds->commands[i].size_out);
}
return 0;
};
int cxl_config(char* offset_s, char* data_s) {
int offset, data, is_write;
cxl_pdev_config* config_payload= malloc(sizeof(cxl_pdev_config));
if (data_s == NULL)
is_write= 0;
else {
is_write= 1;
data= strtol(data_s, NULL, 16);
}
offset= strtol(offset_s, NULL, 16);
config_payload->offset= offset;
config_payload->data= data;
config_payload->is_write= is_write;
ioctl(FD, CXL_MEM_CONFIG_WR, config_payload);
printf("CONFIG_WR %s [%0x] ", (is_write)? "write" : "read",
config_payload->offset);
for (int i= 0; i < 32; i += 8) printf("
Re: [RFC PATCH 00/25] Introduce CXL 2.0 Emulation
Hi Ben, Trying to bring up the environment using the latest developments as follows: 1. Linux kernel baseline version is cloned using git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git Using master branch. Merged the 9 CXL linux kernel patches manually and built kernel 2. QEMU baseline version is cloned using git clone https://gitlab.com/bwidawsk/qemu.git <https://gitlab.com/bwidawsk/qemu.git> 3. UEFI baseline is cloned using git clone https://github.com/tianocore/edk2.git <https://github.com/tianocore/edk2.git> Using master and built 4. Now can run qemu as follows: The qcow2 we use is based on Ubuntu 20.10 with updated with kernel from 1) above QEMU command: sudo qemu-system-x86_64 -nic \ user,hostfwd=tcp::-:22,hostfwd=tcp::1234-:1234 -machine \ type=pc-q35-4.0,hmat=on,accel=kvm -enable-kvm -cpu host -smp \ 6,cores=6,threads=1,sockets=1 -m 8G -boot order=d -k 'en-us' -vga virtio \ -drive file=/home/chris/Downloads/AQCXL/ubuntu_20.qcow,format=qcow2 -drive \ if=pflash,format=raw,readonly,file=/home/chris/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/home/chris/OVMF_VARS.fd \ -object memory-backend-file,id=cxl-mem1,share,mem-path=/tmp/cxl-test/cxl,size=512M \ -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=0,len-window-base=1,\ window-base[0]=0x4c000,memdev[0]=cxl-mem1 \ -device cxl-rp,id=rp0,bus=cxl.0,addr=0.0,chassis=0,slot=0 \ -device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M 2>&1 | tee -a \ /home/chris/Downloads/AQCXL/log/qemu.log The qemu options are derived from looking at the tests/qtests/cxl-test.c along with the -hmat=on which seemed to make sense. The system boots and lspci -vvv shows the CXL device is enumerated. But no DOE capability register for CDAT access though (see below). Otherwise the DVSEC registers are present. acpidump indicates the CXL0 and CXLM devices but no SRAT or HMAT tables are in the dump which is curious. 35:00.0 Memory controller [0502]: Intel Corporation Device 0d93 (rev 01) (prog-if 10) Subsystem: Red Hat, Inc. Device 1100 Physical Slot: 0 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Capabilities: [138 v1] Designated Vendor-Specific Kernel driver in use: cxl_mem Questions/Comments: --- 1. Linux a. Is there a gitlab for the linux kernel patches for CXL? This would facilitate review and code modifications. 2. UEFI (edk2 from tianocore) a. seems to only support CXL 1.1 which means only method #1 (Device option ROM) of Coherent Device Attribute Table_1.02 spec for CDAT handling is possible now. Does device option ROM need to be added to QEMU CXL setup? Can we add a CXL 1.1 emulated device? b. lspci doesn’t show the existence of the DOE extended capability register in the CXL CT3D (needed to support method #2). Are there more patches? 3. Do you have example user programs to share or better yet the CXL 2.0 Sec 14.3.6.1 Application Layer/ Transaction layer test for CXL.mem? 4. What are the userspace system APIs for targeting CXL HDM address domain? Usually you can mmap a SPA if you know how to look it up. Best Regards, Chris Browy
