> On Jan 30, 2025, at 2:16, Jonathan Cameron <jonathan.came...@huawei.com>
> wrote:
>
> On Wed, 29 Jan 2025 10:14:34 +0900
> Itaru Kitayama <itaru.kitay...@linux.dev> wrote:
>
>>> On Jan 22, 2025, at 23:07, Jonathan Cameron <jonathan.came...@huawei.com>
>>> wrote:
>>>
>>> On Fri, 17 Jan 2025 09:43:11 +0000
>>> Jonathan Cameron via <qemu-devel@nongnu.org> wrote:
>>>
>>>> On Fri, 17 Jan 2025 10:13:41 +0900
>>>> Itaru Kitayama <itaru.kitay...@linux.dev> wrote:
>>>>
>>>>>> On Jan 16, 2025, at 19:58, Jonathan Cameron
>>>>>> <jonathan.came...@huawei.com> wrote:
>>>>>>
>>>>>> On Thu, 16 Jan 2025 15:04:53 +0900
>>>>>> Itaru Kitayama <itaru.kitay...@linux.dev> wrote:
>>>>>>
>>>>>>> Hi Jonathan,
>>>>>>>
>>>>>>>> On Jan 14, 2025, at 19:26, Jonathan Cameron
>>>>>>>> <jonathan.came...@huawei.com> wrote:
>>>>>>>>
>>>>>>>> On Tue, 14 Jan 2025 12:03:03 +0900
>>>>>>>> Itaru Kitayama <itaru.kitay...@linux.dev> wrote:
>>>>>>>>
>>>>>>>>> Hi Jonathan,
>>>>>>>>>
>>>>>>>>>> On Jan 10, 2025, at 21:31, Jonathan Cameron
>>>>>>>>>> <jonathan.came...@huawei.com> wrote:
>>>>>>>>>>
>>>>>>>>>> On Fri, 10 Jan 2025 09:20:54 +0000
>>>>>>>>>> "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> On 10/01/2025 13:29, Itaru Kitayama wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> Is anybody working on the CXL emulation on aarch64?
>>>>>>>>>>>
>>>>>>>>>>> I'm not currently working on the CXL emulation on aarch64.
>>>>>>>>>>>
>>>>>>>>>>> However, IIRC the CXL maintainer's tree should work.
>>>>>>>>>>> https://gitlab.com/jic23/qemu/
>>>>>>>>>>
>>>>>>>>>> Pick up latest branch from there. I'm prepping a rebased version
>>>>>>>>>> with some new stuff but might take a few more days.
>>>>>>>>>
>>>>>>>>> Thanks for sharing your work with us. Your master and cxl-2024-11-27
>>>>>>>>> branches give:
>>>>>>>>>
>>>>>>>>> $ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl'
>>>>>>>>> not found
>>>>>>>>
>>>>>>>> cxl is a machine property not a accel one. So needs to be after virt
>>>>>>>> There are tests in the tree for bios tables. Copy the command line
>>>>>>>> from those.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> My commands are below:
>>>>>>>>> $HOME/projects/qemu/build/qemu-system-aarch64 \
>>>>>>>>> -M virt,virtualization=on,gic-version=3 \
>>>>>>>>> -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
>>>>>>>>> -accel tcg,cxl=on \
>>>>>>>>> -nographic \
>>>>>>>>> -bios $HOME/cca-v4/out/bin/flash.bin \
>>>>>>>>> -kernel Image-cca \
>>>>>>>>> -drive
>>>>>>>>> format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0
>>>>>>>>> \
>>>>>>>>> -device virtio-blk-pci,drive=hd0 \
>>>>>>>>> -append root=/dev/vda \
>>>>>>>>> -nodefaults \
>>>>>>>>> --serial tcp:localhost:54320 \
>>>>>>>>> -serial tcp:localhost:54321 \
>>>>>>>>> -append "root=/dev/vda earlycon console=hvc0" \
>>>>>>>>> -device virtio-net-pci,netdev=net0 \
>>>>>>>>> -netdev user,id=net0 \
>>>>>>>>> -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
>>>>>>>>> -fsdev local,security_model=none,path=../../,id=shr0
>>>>>>>>>
>>>>>>>>> Yes, I’m using Linaro’s CCA capable OP-TEE builds above.
>>>>>>>>
>>>>>>>> I'm a little curious why optee is relevant for this but shouldn't
>>>>>>>> matter as long
>>>>>>>> as an appropriate EDK2 is loaded.
>>>>>>>>
>>>>>>>
>>>>>>> I picked up your tree’s “master” and “cxl-next” as of today, and only
>>>>>>> the latter at least booted.
>>>>>>> The former gives:
>>>>>>>
>>>>>>> qemu-system-aarch64: Property 'virt-9.2-machine.cxl' not found
>>>>>>>
>>>>>>> Should I stick with the cxl-next? My concern is that the base QEMU
>>>>>>> version is a bit old
>>>>>>> 7.0.50.
>>>>>>
>>>>>> Always use the latest dated branch on that tree. I release whenever
>>>>>> there
>>>>>> is something new to carry or a major rebase needed.
>>>>>>
>>>>>> cxl-<date> is the right branch to use. Hope that helps.
>>>>>
>>>>> When do you think you want to get them (aarch64 specific?) merged
>>>>> mainline. Any reason you want to carry the patches by yourself?
>>>>
>>>> Nothing much has changed since I presented on this at Linaro connect in
>>>> 2023.
>>>> https://resources.linaro.org/en/resource/hM986DSHfoTrZ98UjpvLg1
>>>>
>>>> The issue is device tree bindings for PCI Expander bridgess and the fact
>>>> that
>>>> those need to be generated without the full enumeration that EDK2 is doing
>>>> prior to ACPI final table builds. In order to move forward with that it
>>>> needs a bunch of work to prove that we absolutely cannot get patches
>>>> upstream to support kernel base enumeration and breaking up of the
>>>> various resources (like EDK2 does).
>>>
>>> I was talking to Peter Maydell earlier and given developments in the last
>>> couple
>>> of years that have by necessity been ACPI only in arm virt he is less
>>> opposed to ACPI only features being added where device tree is challenging.
>>>
>>> So we may be able to move forwards without device tree support.
>>>
>>> The PXB enumeration question is also relevant for managing multiple
>>> vIOMMUs to represent multiple physical IOMMUs with the correct isolation
>>> and do it efficiently which is probably a more pressing usecase than CXL
>>> emulation.
>>> The discussion was mainly about that usecase, but maybe it also unblocks
>>> upstreaming this support.
>>>
>>> Thanks,
>>>
>>> Jonathan
>>
>> I finally made some CXL tests ran within the ndctl test framework along with
>> the kernel modules (QEMU is on your cxl-2024-11-27 branch) on aarch64.
>> However, the recent rebase cxl-2025-01-24 fails to start the system emulation
>>
>> qemu-system-aarch64: Property 'virt-10.0-machine.cxl' not found
>>
>> The build did not complain, what kind of tests you run against your
>> periodical QEMU rebase?
>
> On this run I was focused on chmu testing which will have included what you
> are running into here. It's made a little tricky by some local networking
> issues
> so getting a branch up on gitlab is far from straight forward.
>
> Ah. Looks like the push went wrong as that has 0 patches on top of main qemu
> branch
> so that branch is effectively empty. Typo in what I pushed.
> Sorry about that.
Not at all. I confirmed that the same branch now boots fine.
>
> Note this is far from a production tree as it carries many patches with
> outstanding
> review comments etc. Mainly exists for the purposes of testing specific
> kernel
> patches where the support is not ready for upstream QEMU yet. It is not in
> any
> sense a 'release' with the sort of testing that would imply, so mostly a case
> of local smoke tests.
I understand this.
Thanks,
Itaru.
>
> Any help with more comprehensive tests would be much appreciated. I know some
> other
> folk in the CXL community are talking about that, but I don't think anyone has
> it in place yet. It's hard as there are typically a lot of moving parts.
>
> Pushed now.
>
> Jonathan
>
>
>
>
>>
>> Itaru.
>>
>>>
>>>>
>>>> Given PXB enumeration in kernel has some issues on ARM anyway (that you
>>>> can paper
>>>> over with _DSM 5 - it self requiring an extra patch that isn't
>>>> upstreamable because
>>>> of IO port issues) there is quite a bit of work needed, mostly not in QEMU.
>>>> Or convince Peter and others that not all virt support needs DT bindings
>>>> (note that PXB for PCIE has been supported for years without an DT support,
>>>> just no one noticed!)
>>>>
>>>> After that we'd need to figure out CXL DT bindings in general and add
>>>> kernel
>>>> code support - despite there being no known DT based CXL systems out
>>>> there, so
>>>> that is going to be hard to do. Various CXL kernel maintainers have
>>>> expressed
>>>> they aren't against such support, but it's hardly going to be review
>>>> priority
>>>> (other than for me if someone else does the work!)
>>>>
>>>> For me this isn't particularly high priority. The ARM bit is fairly easy
>>>> to rebase.
>>>> I would like to see it solved, but it is behind various other items on my
>>>> backlog.
>>>>
>>>> There are SBSA machine patches on list, but it's not a useful platform for
>>>> CXL kernel code development because of the limited supported configurations
>>>> (in keeping with the more or less fixed model that SBSA-ref uses).
>>>>
>>>> Jonathan
>>>>
>>>>
>>>>
>>>>>
>>>>> Itaru.
>>>>>
>>>>>>
>>>>>> Jonathan
>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Itaru.
>>>>>>>
>>>>>>>> Jonathan
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Let me know which branch you were suggesting.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Itaru.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Note my main development work is on arm64 so that tends to work
>>>>>>>>>> more reliably than x86 which I only lightly test for stuff that
>>>>>>>>>> isn't ready for upstream yet.
>>>>>>>>>>
>>>>>>>>>> Give me a shout if you run into any problems.
>>>>>>>>>>
>>>>>>>>>> The main blocker on upstreaming this is resolving the missing device
>>>>>>>>>> tree
>>>>>>>>>> support for PCI expander bridges. I've not made any progress on
>>>>>>>>>> this since
>>>>>>>>>> talk at Linaro connect in 2023.
>>>>>>>>>>
>>>>>>>>>> Jonathan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Zhijian
>>>>>>>>>>>
>>>>>>>>>>>> If there’s a WIP branch, a pointer would be appreciated.
>>>>>>>>>>>>
>>>>>>>>>>>> Itaru
