Re: [PATCH v6 27/43] hw/cxl/host: Add support for CXL Fixed Memory Windows.
On Fri, 4 Mar 2022 15:56:38 +
Jonathan Cameron wrote:
> On Wed, 02 Mar 2022 07:55:45 +0100
> Markus Armbruster wrote:
>
> > Jonathan Cameron via writes:
> >
> > > From: Jonathan Cameron
> > >
> > > The concept of these is introduced in [1] in terms of the
> > > description the CEDT ACPI table. The principal is more general.
> > > Unlike once traffic hits the CXL root bridges, the host system
> > > memory address routing is implementation defined and effectively
> > > static once observable by standard / generic system software.
> > > Each CXL Fixed Memory Windows (CFMW) is a region of PA space
> > > which has fixed system dependent routing configured so that
> > > accesses can be routed to the CXL devices below a set of target
> > > root bridges. The accesses may be interleaved across multiple
> > > root bridges.
> > >
> > > For QEMU we could have fully specified these regions in terms
> > > of a base PA + size, but as the absolute address does not matter
> > > it is simpler to let individual platforms place the memory regions.
> > >
> > > ExampleS:
> > > -cxl-fixed-memory-window targets=cxl.0,size=128G
> > > -cxl-fixed-memory-window targets=cxl.1,size=128G
> > > -cxl-fixed-memory-window
> > > targets=cxl0,targets=cxl.1,size=256G,interleave-granularity=2k
> > >
> > > Specifies
> > > * 2x 128G regions not interleaved across root bridges, one for each of
> > > the root bridges with ids cxl.0 and cxl.1
> > > * 256G region interleaved across root bridges with ids cxl.0 and cxl.1
> > > with a 2k interleave granularity.
> > >
> > > When system software enumerates the devices below a given root bridge
> > > it can then decide which CFMW to use. If non interleave is desired
> > > (or possible) it can use the appropriate CFMW for the root bridge in
> > > question. If there are suitable devices to interleave across the
> > > two root bridges then it may use the 3rd CFMS.
> > >
> > > A number of other designs were considered but the following constraints
> > > made it hard to adapt existing QEMU approaches to this particular problem.
> > > 1) The size must be known before a specific architecture / board brings
> > >up it's PA memory map. We need to set up an appropriate region.
> > > 2) Using links to the host bridges provides a clean command line interface
> > >but these links cannot be established until command line devices have
> > >been added.
> > >
> > > Hence the two step process used here of first establishing the size,
> > > interleave-ways and granularity + caching the ids of the host bridges
> > > and then, once available finding the actual host bridges so they can
> > > be used later to support interleave decoding.
> > >
> > > [1] CXL 2.0 ECN: CEDT CFMWS & QTG DSM (computeexpresslink.org /
> > > specifications)
> > >
> > > Signed-off-by: Jonathan Cameron
> >
> > [...]
> >
> > > diff --git a/hw/cxl/cxl-host.c b/hw/cxl/cxl-host.c
> > > new file mode 100644
> > > index 00..9f303e6d8e
> > > --- /dev/null
> > > +++ b/hw/cxl/cxl-host.c
> >
> > [...]
> >
> > > +QemuOptsList qemu_cxl_fixed_window_opts = {
> > > +.name = "cxl-fixed-memory-window",
> > > +.implied_opt_name = "type",
> > > +.head = QTAILQ_HEAD_INITIALIZER(qemu_cxl_fixed_window_opts.head),
> > > +.desc = { { 0 } }
> > > +};
> > > +
> >
> > [...]
> >
> > > +static int parse_cxl_fixed_memory_window(void *opaque, QemuOpts *opts,
> > > + Error **errp)
> > > +{
> > > +CXLFixedMemoryWindowOptions *object = NULL;
> > > +MachineState *ms = MACHINE(opaque);
> > > +Error *err = NULL;
> > > +Visitor *v = opts_visitor_new(opts);
> > > +
> > > +visit_type_CXLFixedMemoryWindowOptions(v, NULL, , errp);
> > > +visit_free(v);
> > > +if (!object) {
> > > +return -1;
> > > +}
> > > +
> > > +set_cxl_fixed_memory_window_options(ms, object, );
> > > +
> > > +qapi_free_CXLFixedMemoryWindowOptions(object);
> > > +if (err) {
> > > +error_propagate(errp, err);
> > > +return -1;
> > > +}
> > > +
> > > +return 0;
> > > +}
> > > +
> > > +void parse_cxl_fixed_memory_window_opts(MachineState *ms)
> > > +{
> > > +qemu_opts_foreach(qemu_find_opts("cxl-fixed-memory-window"),
> > > + parse_cxl_fixed_memory_window, ms, _fatal);
> > > +}
> >
> > [...]
> >
> > > diff --git a/qapi/machine.json b/qapi/machine.json
> > > index 42fc68403d..0998a9128d 100644
> > > --- a/qapi/machine.json
> > > +++ b/qapi/machine.json
> > > @@ -504,6 +504,21 @@
> > > 'dst': 'uint16',
> > > 'val': 'uint8' }}
> > >
> > > +##
> > > +# @CXLFixedMemoryWindowOptions:
> > > +#
> > > +# Create a CXL Fixed Memory Window (for OptsVisitor)
> > > +#
> > > +# @targets: Target root bridge IDs
> >
> > Missing: @size, @targets.
> >
> > > +#
> > > +# Since X.X //fixme
> >
> > Well, "fix me, please".
> >
> > > +##
> > > +{ 'struct': 'CXLFixedMemoryWindowOptions',
> > > +
Re: [PATCH v6 27/43] hw/cxl/host: Add support for CXL Fixed Memory Windows.
On Wed, 02 Mar 2022 07:55:45 +0100
Markus Armbruster wrote:
> Jonathan Cameron via writes:
>
> > From: Jonathan Cameron
> >
> > The concept of these is introduced in [1] in terms of the
> > description the CEDT ACPI table. The principal is more general.
> > Unlike once traffic hits the CXL root bridges, the host system
> > memory address routing is implementation defined and effectively
> > static once observable by standard / generic system software.
> > Each CXL Fixed Memory Windows (CFMW) is a region of PA space
> > which has fixed system dependent routing configured so that
> > accesses can be routed to the CXL devices below a set of target
> > root bridges. The accesses may be interleaved across multiple
> > root bridges.
> >
> > For QEMU we could have fully specified these regions in terms
> > of a base PA + size, but as the absolute address does not matter
> > it is simpler to let individual platforms place the memory regions.
> >
> > ExampleS:
> > -cxl-fixed-memory-window targets=cxl.0,size=128G
> > -cxl-fixed-memory-window targets=cxl.1,size=128G
> > -cxl-fixed-memory-window
> > targets=cxl0,targets=cxl.1,size=256G,interleave-granularity=2k
> >
> > Specifies
> > * 2x 128G regions not interleaved across root bridges, one for each of
> > the root bridges with ids cxl.0 and cxl.1
> > * 256G region interleaved across root bridges with ids cxl.0 and cxl.1
> > with a 2k interleave granularity.
> >
> > When system software enumerates the devices below a given root bridge
> > it can then decide which CFMW to use. If non interleave is desired
> > (or possible) it can use the appropriate CFMW for the root bridge in
> > question. If there are suitable devices to interleave across the
> > two root bridges then it may use the 3rd CFMS.
> >
> > A number of other designs were considered but the following constraints
> > made it hard to adapt existing QEMU approaches to this particular problem.
> > 1) The size must be known before a specific architecture / board brings
> >up it's PA memory map. We need to set up an appropriate region.
> > 2) Using links to the host bridges provides a clean command line interface
> >but these links cannot be established until command line devices have
> >been added.
> >
> > Hence the two step process used here of first establishing the size,
> > interleave-ways and granularity + caching the ids of the host bridges
> > and then, once available finding the actual host bridges so they can
> > be used later to support interleave decoding.
> >
> > [1] CXL 2.0 ECN: CEDT CFMWS & QTG DSM (computeexpresslink.org /
> > specifications)
> >
> > Signed-off-by: Jonathan Cameron
>
> [...]
>
> > diff --git a/hw/cxl/cxl-host.c b/hw/cxl/cxl-host.c
> > new file mode 100644
> > index 00..9f303e6d8e
> > --- /dev/null
> > +++ b/hw/cxl/cxl-host.c
>
> [...]
>
> > +QemuOptsList qemu_cxl_fixed_window_opts = {
> > +.name = "cxl-fixed-memory-window",
> > +.implied_opt_name = "type",
> > +.head = QTAILQ_HEAD_INITIALIZER(qemu_cxl_fixed_window_opts.head),
> > +.desc = { { 0 } }
> > +};
> > +
>
> [...]
>
> > +static int parse_cxl_fixed_memory_window(void *opaque, QemuOpts *opts,
> > + Error **errp)
> > +{
> > +CXLFixedMemoryWindowOptions *object = NULL;
> > +MachineState *ms = MACHINE(opaque);
> > +Error *err = NULL;
> > +Visitor *v = opts_visitor_new(opts);
> > +
> > +visit_type_CXLFixedMemoryWindowOptions(v, NULL, , errp);
> > +visit_free(v);
> > +if (!object) {
> > +return -1;
> > +}
> > +
> > +set_cxl_fixed_memory_window_options(ms, object, );
> > +
> > +qapi_free_CXLFixedMemoryWindowOptions(object);
> > +if (err) {
> > +error_propagate(errp, err);
> > +return -1;
> > +}
> > +
> > +return 0;
> > +}
> > +
> > +void parse_cxl_fixed_memory_window_opts(MachineState *ms)
> > +{
> > +qemu_opts_foreach(qemu_find_opts("cxl-fixed-memory-window"),
> > + parse_cxl_fixed_memory_window, ms, _fatal);
> > +}
>
> [...]
>
> > diff --git a/qapi/machine.json b/qapi/machine.json
> > index 42fc68403d..0998a9128d 100644
> > --- a/qapi/machine.json
> > +++ b/qapi/machine.json
> > @@ -504,6 +504,21 @@
> > 'dst': 'uint16',
> > 'val': 'uint8' }}
> >
> > +##
> > +# @CXLFixedMemoryWindowOptions:
> > +#
> > +# Create a CXL Fixed Memory Window (for OptsVisitor)
> > +#
> > +# @targets: Target root bridge IDs
>
> Missing: @size, @targets.
>
> > +#
> > +# Since X.X //fixme
>
> Well, "fix me, please".
>
> > +##
> > +{ 'struct': 'CXLFixedMemoryWindowOptions',
> > + 'data': {
> > + 'size': 'size',
> > + '*interleave-granularity': 'size',
> > + 'targets': ['str'] }}
> > +
> > ##
> > # @X86CPURegister32:
> > #
>
> [...]
>
> > diff --git a/qemu-options.hx b/qemu-options.hx
> > index ba3ae6a42a..b4d2cc6f48 100644
> > --- a/qemu-options.hx
> > +++ b/qemu-options.hx
> > @@ -467,6 +467,43 @@ SRST
> >
Re: [PATCH v6 27/43] hw/cxl/host: Add support for CXL Fixed Memory Windows.
Jonathan Cameron via writes:
> From: Jonathan Cameron
>
> The concept of these is introduced in [1] in terms of the
> description the CEDT ACPI table. The principal is more general.
> Unlike once traffic hits the CXL root bridges, the host system
> memory address routing is implementation defined and effectively
> static once observable by standard / generic system software.
> Each CXL Fixed Memory Windows (CFMW) is a region of PA space
> which has fixed system dependent routing configured so that
> accesses can be routed to the CXL devices below a set of target
> root bridges. The accesses may be interleaved across multiple
> root bridges.
>
> For QEMU we could have fully specified these regions in terms
> of a base PA + size, but as the absolute address does not matter
> it is simpler to let individual platforms place the memory regions.
>
> ExampleS:
> -cxl-fixed-memory-window targets=cxl.0,size=128G
> -cxl-fixed-memory-window targets=cxl.1,size=128G
> -cxl-fixed-memory-window
> targets=cxl0,targets=cxl.1,size=256G,interleave-granularity=2k
>
> Specifies
> * 2x 128G regions not interleaved across root bridges, one for each of
> the root bridges with ids cxl.0 and cxl.1
> * 256G region interleaved across root bridges with ids cxl.0 and cxl.1
> with a 2k interleave granularity.
>
> When system software enumerates the devices below a given root bridge
> it can then decide which CFMW to use. If non interleave is desired
> (or possible) it can use the appropriate CFMW for the root bridge in
> question. If there are suitable devices to interleave across the
> two root bridges then it may use the 3rd CFMS.
>
> A number of other designs were considered but the following constraints
> made it hard to adapt existing QEMU approaches to this particular problem.
> 1) The size must be known before a specific architecture / board brings
>up it's PA memory map. We need to set up an appropriate region.
> 2) Using links to the host bridges provides a clean command line interface
>but these links cannot be established until command line devices have
>been added.
>
> Hence the two step process used here of first establishing the size,
> interleave-ways and granularity + caching the ids of the host bridges
> and then, once available finding the actual host bridges so they can
> be used later to support interleave decoding.
>
> [1] CXL 2.0 ECN: CEDT CFMWS & QTG DSM (computeexpresslink.org /
> specifications)
>
> Signed-off-by: Jonathan Cameron
[...]
> diff --git a/hw/cxl/cxl-host.c b/hw/cxl/cxl-host.c
> new file mode 100644
> index 00..9f303e6d8e
> --- /dev/null
> +++ b/hw/cxl/cxl-host.c
[...]
> +QemuOptsList qemu_cxl_fixed_window_opts = {
> +.name = "cxl-fixed-memory-window",
> +.implied_opt_name = "type",
> +.head = QTAILQ_HEAD_INITIALIZER(qemu_cxl_fixed_window_opts.head),
> +.desc = { { 0 } }
> +};
> +
[...]
> +static int parse_cxl_fixed_memory_window(void *opaque, QemuOpts *opts,
> + Error **errp)
> +{
> +CXLFixedMemoryWindowOptions *object = NULL;
> +MachineState *ms = MACHINE(opaque);
> +Error *err = NULL;
> +Visitor *v = opts_visitor_new(opts);
> +
> +visit_type_CXLFixedMemoryWindowOptions(v, NULL, , errp);
> +visit_free(v);
> +if (!object) {
> +return -1;
> +}
> +
> +set_cxl_fixed_memory_window_options(ms, object, );
> +
> +qapi_free_CXLFixedMemoryWindowOptions(object);
> +if (err) {
> +error_propagate(errp, err);
> +return -1;
> +}
> +
> +return 0;
> +}
> +
> +void parse_cxl_fixed_memory_window_opts(MachineState *ms)
> +{
> +qemu_opts_foreach(qemu_find_opts("cxl-fixed-memory-window"),
> + parse_cxl_fixed_memory_window, ms, _fatal);
> +}
[...]
> diff --git a/qapi/machine.json b/qapi/machine.json
> index 42fc68403d..0998a9128d 100644
> --- a/qapi/machine.json
> +++ b/qapi/machine.json
> @@ -504,6 +504,21 @@
> 'dst': 'uint16',
> 'val': 'uint8' }}
>
> +##
> +# @CXLFixedMemoryWindowOptions:
> +#
> +# Create a CXL Fixed Memory Window (for OptsVisitor)
> +#
> +# @targets: Target root bridge IDs
Missing: @size, @targets.
> +#
> +# Since X.X //fixme
Well, "fix me, please".
> +##
> +{ 'struct': 'CXLFixedMemoryWindowOptions',
> + 'data': {
> + 'size': 'size',
> + '*interleave-granularity': 'size',
> + 'targets': ['str'] }}
> +
> ##
> # @X86CPURegister32:
> #
[...]
> diff --git a/qemu-options.hx b/qemu-options.hx
> index ba3ae6a42a..b4d2cc6f48 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -467,6 +467,43 @@ SRST
> -numa
> hmat-cache,node-id=1,size=10K,level=1,associativity=direct,policy=write-back,line=8
> ERST
>
> +DEF("cxl-fixed-memory-window", HAS_ARG, QEMU_OPTION_cxl_fixed_memory_window,
> +"-cxl-fixed-memory-window
> targets=firsttarget,targets=secondtarget,size=size[,interleave-granularity=granularity]\n",
> +QEMU_ARCH_ALL)
> +SRST
> +``-cxl-fixed-memory-window
[PATCH v6 27/43] hw/cxl/host: Add support for CXL Fixed Memory Windows.
From: Jonathan Cameron
The concept of these is introduced in [1] in terms of the
description the CEDT ACPI table. The principal is more general.
Unlike once traffic hits the CXL root bridges, the host system
memory address routing is implementation defined and effectively
static once observable by standard / generic system software.
Each CXL Fixed Memory Windows (CFMW) is a region of PA space
which has fixed system dependent routing configured so that
accesses can be routed to the CXL devices below a set of target
root bridges. The accesses may be interleaved across multiple
root bridges.
For QEMU we could have fully specified these regions in terms
of a base PA + size, but as the absolute address does not matter
it is simpler to let individual platforms place the memory regions.
ExampleS:
-cxl-fixed-memory-window targets=cxl.0,size=128G
-cxl-fixed-memory-window targets=cxl.1,size=128G
-cxl-fixed-memory-window
targets=cxl0,targets=cxl.1,size=256G,interleave-granularity=2k
Specifies
* 2x 128G regions not interleaved across root bridges, one for each of
the root bridges with ids cxl.0 and cxl.1
* 256G region interleaved across root bridges with ids cxl.0 and cxl.1
with a 2k interleave granularity.
When system software enumerates the devices below a given root bridge
it can then decide which CFMW to use. If non interleave is desired
(or possible) it can use the appropriate CFMW for the root bridge in
question. If there are suitable devices to interleave across the
two root bridges then it may use the 3rd CFMS.
A number of other designs were considered but the following constraints
made it hard to adapt existing QEMU approaches to this particular problem.
1) The size must be known before a specific architecture / board brings
up it's PA memory map. We need to set up an appropriate region.
2) Using links to the host bridges provides a clean command line interface
but these links cannot be established until command line devices have
been added.
Hence the two step process used here of first establishing the size,
interleave-ways and granularity + caching the ids of the host bridges
and then, once available finding the actual host bridges so they can
be used later to support interleave decoding.
[1] CXL 2.0 ECN: CEDT CFMWS & QTG DSM (computeexpresslink.org / specifications)
Signed-off-by: Jonathan Cameron
---
hw/cxl/cxl-host-stubs.c | 22 +++
hw/cxl/cxl-host.c | 138
hw/cxl/meson.build | 6 ++
include/hw/cxl/cxl.h| 20 ++
qapi/machine.json | 15 +
qemu-options.hx | 37 +++
softmmu/vl.c| 11
7 files changed, 249 insertions(+)
diff --git a/hw/cxl/cxl-host-stubs.c b/hw/cxl/cxl-host-stubs.c
new file mode 100644
index 00..f942dda41b
--- /dev/null
+++ b/hw/cxl/cxl-host-stubs.c
@@ -0,0 +1,22 @@
+/*
+ * CXL host parameter parsing routine stubs
+ *
+ * Copyright (c) 2022 Huawei
+ */
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qemu/option.h"
+#include "hw/cxl/cxl.h"
+
+QemuOptsList qemu_cxl_fixed_window_opts = {
+.name = "cxl-fixed-memory-window",
+.implied_opt_name = "type",
+.head = QTAILQ_HEAD_INITIALIZER(qemu_cxl_fixed_window_opts.head),
+.desc = { { 0 } }
+};
+
+void parse_cxl_fixed_memory_window_opts(MachineState *ms) {};
+
+void cxl_fixed_memory_window_link_targets(Error **errp) {};
+
+const MemoryRegionOps cfmws_ops;
diff --git a/hw/cxl/cxl-host.c b/hw/cxl/cxl-host.c
new file mode 100644
index 00..9f303e6d8e
--- /dev/null
+++ b/hw/cxl/cxl-host.c
@@ -0,0 +1,138 @@
+/*
+ * CXL host parameter parsing routines
+ *
+ * Copyright (c) 2022 Huawei
+ * Modeled loosely on the NUMA options handling in hw/core/numa.c
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/units.h"
+#include "qemu/bitmap.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "sysemu/qtest.h"
+#include "hw/boards.h"
+
+#include "qapi/opts-visitor.h"
+#include "qapi/qapi-visit-machine.h"
+#include "qemu/option.h"
+#include "hw/cxl/cxl.h"
+#include "hw/pci/pci_bus.h"
+#include "hw/pci/pci_bridge.h"
+#include "hw/pci/pci_host.h"
+#include "hw/pci/pcie_port.h"
+
+QemuOptsList qemu_cxl_fixed_window_opts = {
+.name = "cxl-fixed-memory-window",
+.implied_opt_name = "type",
+.head = QTAILQ_HEAD_INITIALIZER(qemu_cxl_fixed_window_opts.head),
+.desc = { { 0 } }
+};
+
+static void set_cxl_fixed_memory_window_options(MachineState *ms,
+CXLFixedMemoryWindowOptions
*object,
+Error **errp)
+{
+CXLFixedWindow *fw = g_malloc0(sizeof(*fw));
+strList *target;
+int i;
+
+for (target = object->targets; target; target = target->next) {
+fw->num_targets++;
+}
+
+fw->enc_int_ways = cxl_interleave_ways_enc(fw->num_targets, errp);
+if (*errp) {
+return;
+}
+
+fw->targets = g_malloc0_n(fw->num_targets,
