I agree the feature discussed in TB. The original scheme is too vague and hard to understand, although after several round of clarification.
On 8/14/2025 12:46 AM, Vamsi Krishna Attunuru wrote: > Hi Thomas, Feng > > Can this feature be discussed in the next techboard meeting and decide on > supporting it > together with the ABI breaking changes in 25.11, rather than using the > versions scheme. > Since there are no further comments after we aligned with the Feng's > feedback, it would be > good to finalize the approach. > > Regards > Vamsi > >> -----Original Message----- >> From: Vamsi Krishna Attunuru <vattun...@marvell.com> >> Sent: Wednesday, July 30, 2025 10:07 AM >> To: fengchengwen <fengcheng...@huawei.com>; >> bruce.richard...@intel.com; Vladimir Medvedkin >> <vladimir.medved...@intel.com>; Anatoly Burakov >> <anatoly.bura...@intel.com> >> Cc: Jerin Jacob <jer...@marvell.com>; tho...@monjalon.net; >> dev@dpdk.org; Pavan Nikhilesh Bhagavatula <pbhagavat...@marvell.com>; >> kevin.la...@intel.com >> Subject: RE: [EXTERNAL] Re: [PATCH v0 1/1] doc: announce inter-device DMA >> capability support in dmadev >> >> Gentle reminder to share your feedback. >Hi Bruce, Vladimir, Anatoly > >>> Regarding inter-device or inter-domain DMA capability, could please clarify >>> if >>> Intel idxd driver will support this feature. >I believe the changes Feng has >> ZjQcmQRYFpfptBannerStart Prioritize security for external emails: >> Confirm sender and content safety before clicking links or opening >> attachments <https://us-phishalarm- >> ewt.proofpoint.com/EWT/v1/CRVmXkqW!uq3V- >> r_YrnsUWi9yeT3UAKXL4FFc4ONZAyvoCMlneUv-fN8An8AKohi4aES- >> ZG37anFsJG5Rq3NspRNGjPgt5I9U4c9cEVeu_P4sHAEFMopboItEI9hkXOMUrJ >> m_Pyoic-gg6kRF3p9wjJrTGA$> >> Report Suspicious >> >> ZjQcmQRYFpfptBannerEnd >> Gentle reminder to share your feedback. >> >>> Hi Bruce, Vladimir, Anatoly >>> >>> Regarding inter-device or inter-domain DMA capability, could please >>> clarify if Intel idxd driver will support this feature. >>> I believe the changes Feng has suggested here are in line with the >>> earlier "[PATCH v1 0/3] Add support for inter-domain DMA operations" >>> proposal. We are planning to implement this feature support in version >> 25.11. >>> >>> Your feedback would be appreciated, we are aiming for a more generic >>> solution. >>> >>> Regards >>> Vamsi >>> >>> >>>>> On 2025/7/16 18:59, Vamsi Krishna Attunuru wrote: >>>>>> >>>>>>> >>>>>>> Thanks for the explanation. >>>>>>> >>>>>>> Let me tell you what I understand: >>>>>>> 1\ Two dmadev (must belong to the same DMA controller?) each >>>>>>> passthrough to diffent domain (VM or container) 2\ The kernel DMA >>>>>>> controller driver could config access groups --- there is a secure >>>>>>> mechanism >>>>> (like Intel IDPTE) >>>>>>> and the two dmadev could communicate if the kernel DMA >>>>>>> controller driver has put them in the same access groups. >>>>>>> 3\ Application setup access group and get handle (maybe the new >>>>> 'dev_idx' >>>>>>> which you announce in this commit), >>>>>>> and then setup one vchan which config the handle. >>>>>>> and later launch copy request based on this vchan. >>>>>>> 4\ The driver will pass the request to dmadev-1 hardware, and >>>>>>> dmadev-1 hardware will do some verification, >>>>>>> and maybe use dmadev-2 stream ID for read/write operations? >>>>>>> >>>>>>> A few question about this: >>>>>>> 1\ What the prototype of 'dev_idx', is it uint16_t? >>>>>> Yes, it can be uint16_t and use two different dev_idx (src_dev_idx >>>>>> & >>>>>> dest_dev_idx) for read & write. >>>>>> >>>>>>> 2\ How to implement read/write between two dmadev ? use two >>>>>>> different dev_idx? the first for read and the second for write? >>>>>> Yes, two different dev_idx will be used. >>>>>> >>>>>>> >>>>>>> >>>>>>> I also re-read the patchset "[PATCH v1 0/3] Add support for >>>>>>> inter-domain DMA operations", it introduce: >>>>>>> 1\ One 'int controller-id' in the rte_dma_info. which maybe used >>>>>>> in >>>>>>> vendor- specific secure mechanism. >>>>>>> 2\ Two new OP_flag and two new datapath API. >>>>>>> The reason why this patch didn't continue (I guess) is whether >>>>>>> setup one new vchan. Yes, vchan was designed to represents >>>>>>> different transfer contexts. But each vchan has its own >>>>>>> enqueue/dequeue/ring, it more act like one logic dmadev, some of >>>>>>> the hardware can fit this model well, some may not (like Intel in >>>>>>> this >>> case). >>>>>>> >>>>>>> >>>>>>> So how about the following scheme: >>>>>>> 1\ Add inter-domain capability bits, for example: >>>>>>> RTE_DMA_CAPA_INTER_PROCESS_DOMAIN, >>>>>>> RTE_DMA_CAPA_INTER_OS_DOMAIN 2\ Add one >> domain_controller_id >>>> in >>>>> the >>>>>>> rte_dma_info which maybe used in vendor-specific secure >> mechanism. >>>>>>> 3\ Add four OP_FLAGs: >>>>>>> RTE_DMA_OP_FLAG_SRC_INTER_PROCESS_DOMAIN_HANDLE, >>>>>>> RTE_DMA_OP_FLAG_DST_INTER_PROCESS_DOMAIN_HANDLE >>>>>>> RTE_DMA_OP_FLAG_SRC_INTER_OS_DOMAIN_HANDLE, >>>>>>> RTE_DMA_OP_FLAG_DST_INTER_OS_DOMAIN_HANDLE >>>>>>> 4\ Reserved 32bit from flag parameter (which all enqueue API both >>>>>>> supports) as the src and dst handle. >>>>>>> or only reserved 16bit from flag parameter if we restrict don't >>>>>>> support 3rd transfer. >>>>>> >>>>>> Yes, the above approach seems acceptable to me. I believe src & dst >>>>>> handles require 16-bit values. Reserving 32-bits from flag >>>>>> parameter would leave 32 flags available, which should be fine. >>>>> >>>>> Great >>>>> tip: there are still 24bit flag reserved after apply this scheme. >>>>> >>>>> Would like more comments. >>>>> >>>> >>>> If there are no major comments at this time, can we proceed with >>>> accepting and merging this notice in this release. Further review can >>>> continue once the RFC is available next month. >>>> >>>> Thanks & Regards >>>> Vamsi >>>> >>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> On 2025/7/15 13:35, Vamsi Krishna Attunuru wrote: >>>>>>>> Hi Feng, >>>>>>>> >>>>>>>> Thanks for depicting the feature use case. >>>>>>>> >>>>>>>> From the application’s perspective, inter VM/process >>>>>>>> communication is >>>>>>> required to exchange the src & dst buffer details, however the >>>>>>> specifics of this communication mechanism are outside the scope in >>>>>>> this context. Regarding the address translations, these buffer >>>>>>> addresses can be either IOVA as PA or IOVA as VA. The DMA hardware >>>>>>> must use the appropriate IOMMU stream IDs when initiating the DMA >>>>>>> transfers. For example, in the use case shown in the diagram, >>>>>>> dmadev-1 and dmadev-2 would join an access group managed by the >>>>>>> kernel DMA controller driver. This controller driver will >>>>>>> configure the access group on the DMA hardware, enabling the >>>>>>> hardware to select the correct stream IDs for read/write >>>>>>> operations. New rte_dma APIs could be introduced to join or leave >>>>>>> the access group or to query the access group details. >>>>>>> Additionally, a secure token mechanism (similar to >>>>> vfio-pci token) can be implemented to validate any dmadev attempting >>>>> to join the access group. >>>>>>>> >>>>>>>> Regards. >>>>>>>> >>>>>>>> From: fengchengwen <fengcheng...@huawei.com> >>>>>>>> Sent: Tuesday, July 15, 2025 6:29 AM >>>>>>>> To: Vamsi Krishna Attunuru <vattun...@marvell.com>; >>> dev@dpdk.org; >>>>>>>> Pavan Nikhilesh Bhagavatula <pbhagavat...@marvell.com>; >>>>>>>> kevin.la...@intel.com; bruce.richard...@intel.com; >>>>>>>> m...@smartsharesystems.com >>>>>>>> Cc: Jerin Jacob <jer...@marvell.com>; tho...@monjalon.net >>>>>>>> Subject: [EXTERNAL] Re: [PATCH v0 1/1] doc: announce inter-device >>>>>>>> DMA capability support in dmadev >>>>>>>> >>>>>>>> Hi Vamsi, From the commit log, I guess this commit mainly want to >>>>>>>> meet following case: --------------- ---------------- | Container >>>>>>>> | >>>>>>>> | VirtMachine | | | | | | dmadev-1 | | dmadev2 | --------------- >>>>>>>> ---------------- | >>>>>>> | ------------------------------ ZjQcmQRYFpfptBannerStart >>>>>>> | Prioritize security for >>>>>>> external emails: >>>>>>>> Confirm sender and content safety before clicking links or >>>>>>>> opening >>>>>>> attachments >>>>>>>> Report Suspicious >>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__us- >>>> 2Dphishala >>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__us- >>>>>>>>>> 2Dphishala >>>>>>> r >>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__us- >>>> 2Dphishala >>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__us- >>>>>>>>>> 2Dphishala >>>>>>> r >>>>> m- >>>>> 2D&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=WllrYaumVkxaWjgKto6 >> E >>> _r >>>> t >>>>> DQsh >>>>>>>> >>>>> hIhik2jkvzFyRhW8&m=3uFGFVHxC4YLkjWHNg9s9rNDHd_ozbhLepOYCAki >> Z >>> K >>>> x >>>>> M0sQ0m >>>>>>>> >>>>> 43gqgTQl1cK9koZ&s=3_mvLYuMWu7RbHD3mj21CP65O5JY8L8AK8oVFutdT >>> W >>>> U >>>>> &e= >>>>>>> >>>>> ewt.proofpoint.com/EWT/v1/CRVmXkqW!tg3ZldV0Yr_wdSwWmT2aDdK >> Mi >>> - >>>>>>> >>>>> 4rn2z58vFaxwfOeocS1P19w1BeRdyGs5sjnhV2rU_6m8MOWj4KFbuXKkKJI >> v >>> c >>>> q >>>>>>> wWD2WEwJW_0$ > >>>>>>>> ZjQcmQRYFpfptBannerEnd >>>>>>>> >>>>>>>> Hi Vamsi, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> From the commit log, I guess this commit mainly want to meet >>>>>>>> following >>>>>>> case: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> --------------- ---------------- >>>>>>>> >>>>>>>> | Container | | VirtMachine | >>>>>>>> >>>>>>>> | | | | >>>>>>>> >>>>>>>> | dmadev-1 | | dmadev2 | >>>>>>>> >>>>>>>> --------------- ---------------- >>>>>>>> >>>>>>>> | | >>>>>>>> >>>>>>>> ------------------------------ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> App run in the container could launch DMA transfer from local >>>>>>>> buffer to the VirtMachine by config >>>>>>>> >>>>>>>> dmadev-1/2 (the dmadev-1/2 are passthrough to diffent OS domain). >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Could you explain how to use it from application perspective (for >>>>>>>> example address translation) and >>>>>>>> >>>>>>>> application & hardware restrictions? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> BTW: In this case, there are two OS domain communication, and I >>>>>>>> remember there are also inter-process >>>>>>>> >>>>>>>> DMA RFC, so maybe we could design more generic solution if you >>>>>>>> provide >>>>>>> more info. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 2025/7/10 16:51, Vamsi Krishna wrote: >>>>>>>> >>>>>>>>> From: Vamsi Attunuru >>>>>>>>> <vattun...@marvell.com<mailto:vattun...@marvell.com>> >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> Modern DMA hardware supports data transfer between multiple >>>>>>>> >>>>>>>>> DMA devices, enabling data communication across isolated domains >>>>>>>>> or >>>>>>>> >>>>>>>>> containers. To facilitate this, the ``dmadev`` library requires >>>>>>>>> changes >>>>>>>> >>>>>>>>> to allow devices to register with or unregisters from DMA groups >>>>>>>>> for >>>>>>>> >>>>>>>>> inter-device communication. This feature is planned for >>>>>>>>> inclusion >>>>>>>> >>>>>>>>> in DPDK 25.11. >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> Signed-off-by: Vamsi Attunuru >>>>>>>>> <vattun...@marvell.com<mailto:vattun...@marvell.com>> >>>>>>>> >>>>>>>>> --- >>>>>>>> >>>>>>>>> doc/guides/rel_notes/deprecation.rst | 7 +++++++ >>>>>>>> >>>>>>>>> 1 file changed, 7 insertions(+) >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> diff --git a/doc/guides/rel_notes/deprecation.rst >>>>>>>>> b/doc/guides/rel_notes/deprecation.rst >>>>>>>> >>>>>>>>> index e2d4125308..46836244dd 100644 >>>>>>>> >>>>>>>>> --- a/doc/guides/rel_notes/deprecation.rst >>>>>>>> >>>>>>>>> +++ b/doc/guides/rel_notes/deprecation.rst >>>>>>>> >>>>>>>>> @@ -152,3 +152,10 @@ Deprecation Notices >>>>>>>> >>>>>>>>> * bus/vmbus: Starting DPDK 25.11, all the vmbus API defined in >>>>>>>> >>>>>>>>> ``drivers/bus/vmbus/rte_bus_vmbus.h`` will become internal to >>>> DPDK. >>>>>>>> >>>>>>>>> Those API functions are used internally by DPDK core and >>>>>>>>> netvsc >>>> PMD. >>>>>>>> >>>>>>>>> + >>>>>>>> >>>>>>>>> +* dmadev: a new capability flag ``RTE_DMA_CAPA_INTER_DEV`` >> will >>>>>>>>> +be added >>>>>>>> >>>>>>>>> + to advertise DMA device's inter-device DMA copy capability. >>>>>>>>> + To enable >>>>>>>> >>>>>>>>> + this functionality, a few dmadev APIs will be added to >>>>>>>>> + configure the DMA >>>>>>>> >>>>>>>>> + access groups, facilitating coordinated data communication >>>>>>>>> + between >>>>>>> devices. >>>>>>>> >>>>>>>>> + A new ``dev_idx`` field will be added to the ``struct >>>>>>>>> + rte_dma_vchan_conf`` >>>>>>>> >>>>>>>>> + structure to configure a vchan for data transfers between any >>>>>>>>> + two DMA >>>>>>> devices. >>>>>>>> >>>>>>>> >>>>>> >
