On 11/13/20 1:49 AM, Ben Widawsky wrote:
> On 20-11-10 21:47:15, Ben Widawsky wrote:
>> In a bare metal CXL capable system, system firmware will program
>> physical address ranges on the host. This is done by programming
>> internal registers that aren't typically known to OS. These address
>> ranges might be contiguous or interleaved across host bridges.
>>
>> For a QEMU guest a new construct is introduced allowing passing a memory
>> backend to the host bridge for this same purpose. Each memory backend
>> needs to be passed to the host bridge as well as any device that will be
>> emulating that memory (not implemented here).
>>
>> I'm hopeful the interleaving work in the link can be re-purposed here
>> (see Link).
>>
>> An example to create a host bridges with a 512M window at 0x4c0000000
>> -object memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M
>> -device
>> pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=0,len-memory-base=1,memory-base\[0\]=0x4c0000000,memory\[0\]=cxl-mem1
>>
>> Link: https://lists.nongnu.org/archive/html/qemu-devel/2020-08/msg03680.html
>> Signed-off-by: Ben Widawsky <ben.widaw...@intel.com>
>
> Hi Phil, wanted to call you out specifically on this one.
>
> The newly released CXL 2.0 specification (which from a topology perspective
> can
> be thought of as very PCIe-like) allows for interleaving of memory access.
>
> Below is an example of two host bridges, each with two root ports, and 5
> devices
> (two of switch are behind a switch).
>
> RP: Root Port
> USP: Upstream Port
> DSP: Downstream Port
> Type 3 device: Memory Device with persistent or volatile memory.
>
> +-------------------------+ +-------------------------+
> | | | |
> | CXL 2.0 Host Bridge | | CXL 2.0 Host Bridge |
> | | | |
> | +------+ +------+ | | +------+ +------+ |
> | | RP | | RP | | | | RP | | RP | |
> +--+------+-----+------+--+ +--+------+-----+------+--+
> | | | \--
> | | | +-------+-\--+------+
> +------+ +-------+ +-------+ | |USP | |
> |Type 3| |Type 3 | |Type 3 | | +----+ |
> |Device| |Device | |Device | | |
> +------+ +-------+ +-------+ | +----+ +----+ |
> | |DSP | |DSP | |
> +-+----+-----+----+-+
> | |
> +------+ +-------+
> |Type 3| |Type 3 |
> |Device| |Device |
> +------+ +-------+
>
> Considering this picture... interleaving of memory access can happen in all 3
> layers in the topology.
>
> - Memory access can be interleaved across host bridges (this is accomplished
> based on the physical address chosen for the devices, those address ranges
> are
> platform specific and not part of the 2.0 spec, for now).
>
> - Memory access can be interleaved across root ports in a host bridge.
>
> - Finally, memory access can be interleaved across downstream ports.
>
> I'd like to start the discussion about how this might overlap with the patch
> series you've last been working on to interleave memory. Do you have any
> thoughts or ideas on how I should go about doing this?
Main case:
+-------------------------+
| |
| CXL 2.0 Host Bridge |
| |
| +------+ +------+ |
| | RP | | RP | |
+--+------+-----+------+--+
| |
| |
+------+ +-------+
|Type 3| |Type 3 |
|Device| |Device |
+------+ +-------+
// cxl device state
s = qdev_create(TYPE_CXL20_HB_DEV)
cxl_memsize = 2 * memsize(Type3Dev);
// container for cxl
memory_region_init(&s->container, OBJECT(s),
"container", cxl_memsize);
// create 2 slots, interleaved each 2k
s->interleaver = qdev_create(INTERLEAVER_DEV,
slotsize=2k,
max_slots=2)
qdev_prop_set_uint64(s->interleaver, "size",
cxl_memsize);
// connect each device to the interleaver
object_property_set_link(OBJECT(interleaver),
"mr0", OBJECT(RP0))
object_property_set_link(OBJECT(interleaver),
"mr1", OBJECT(RP1))
sysbus_realize_and_unref(SYS_BUS_DEVICE(interleaver))
// we can probably avoid this container
memory_region_add_subregion(&s->container, 0,
sysbus_mmio_get_region(interleaver, 0));
For the 2nd case, USP can be created the same way than case 1
(as a 2nd interleaver) then the main CXL is created with the
minor difference of mr1 now being the USP:
object_property_set_link(OBJECT(interleaver),
"mr1", OBJECT(USP))
sysbus_realize_and_unref(SYS_BUS_DEVICE(interleaver))
Regards,
Phil.
