Dan, thank you for the reply. I am curious how PCIe attached PMEM (Intel Optane memory attached to a PCIe 4.0 device) such as https://www.smartm.com/product/advanced-memory/AIC would be supported in the Linux kernel. Do you know if there is native support in the Linux kernel for this type of device?. If so, would you be able to point me to what modules (nd_pmem.ko, compat, device_dax.ko) would support this type of memory?.
Thanks, RK -----Original Message----- From: Dan Williams <[email protected]> Sent: Wednesday, April 27, 2022 7:27 PM To: An Sarpal <[email protected]> Cc: [email protected] Subject: Re: How to map my PCIe memory as a devdax device (/dev/daxX.Y) On Wed, Apr 27, 2022 at 2:34 PM An Sarpal <[email protected]> wrote: > > Dan, thank you very much for the reply. I appreciate your time on this. > > I talked to my application developer and he said that he can work around the > lack of coherency that you identified. Best of luck. > > I was under the impression that PCIe memory physical ranges were always > mapped as UC which obviously implies that they are not cacheable but I guess > I am wrong there. Not if you use memmap= to hack the memory map, that interface is "garbage in / garbage out" from a safety perspective. > > With that said, I still would like to create a /dev/daxX.Y character device > that would map to the PCIe BAR range. > > I am using driver/dax/device.c as my prototype. > a) I added memmap=X@Y to the kernel command line. When I rebooted the kernel, > I see /dev/pmem0. So far so good. > Ndctl list shows this memory with namespace0.0 > b) I then ran ndctl create namespace with --mode=devdax to convert from fsdax > to devdax and noticed that the probe() routine in device.c was invoked as I > expected. > > Now in my driver, I only have my PCIe function probe() routine. I was > wondering if there is an obvious way for me to make this work just like PMEM > (memmap). No, the obscurity of APIs for this is deliberate. It's broken outside of controlled scenarios, not something Linux will ever support in the general case. See the CXL specification for the hardware capability required to support memory over PCIe electricals. > > There is a lot of functionality provided in other source files of drivers/dax > and it all seems to depend on having that namespace created and lots of > initialization being done. > > I was wondering if there is an easier way for me to attach a struct > dev_dax to my own pcie functionality. It does not look like there is > an easy way but wanted to check with you Correct, I do not have any recommendations to offer here.
