Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive
On Fri, Sep 21, 2018 at 10:52:01AM -0400, Jerome Glisse wrote: > Received: from popscn.huawei.com [10.3.17.45] by Turing-Arch-b with POP3 > (fetchmail-6.3.26) for (single-drop); Fri, 21 Sep 2018 > 23:00:01 +0800 (CST) > Received: from DGGEMM406-HUB.china.huawei.com (10.3.20.214) by > DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server > (TLS) id 14.3.399.0; Fri, 21 Sep 2018 22:52:20 +0800 > Received: from dggwg01-in.huawei.com (172.30.65.38) by > DGGEMM406-HUB.china.huawei.com (10.3.20.214) with Microsoft SMTP Server id > 14.3.399.0; Fri, 21 Sep 2018 22:52:16 +0800 > Received: from mx1.redhat.com (unknown [209.132.183.28]) by Forcepoint > Email with ESMTPS id 912ECA2EC6662; Fri, 21 Sep 2018 22:52:12 +0800 (CST) > Received: from smtp.corp.redhat.com > (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using > TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client > certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id > 25BC0792BB; Fri, 21 Sep 2018 14:52:10 + (UTC) > Received: from redhat.com (ovpn-124-21.rdu2.redhat.com [10.10.124.21]) > by smtp.corp.redhat.com (Postfix) with ESMTPS id 67B25308BDA0; > Fri, 21 Sep 2018 14:52:03 + (UTC) > Date: Fri, 21 Sep 2018 10:52:01 -0400 > From: Jerome Glisse > To: Kenneth Lee > CC: Kenneth Lee , Herbert Xu > , k...@vger.kernel.org, Jonathan Corbet > , Greg Kroah-Hartman , Joerg > Roedel , linux-...@vger.kernel.org, Sanjay Kumar > , Hao Fang , > iommu@lists.linux-foundation.org, linux-ker...@vger.kernel.org, > linux...@huawei.com, Alex Williamson , Thomas > Gleixner , linux-cry...@vger.kernel.org, Zhou Wang > , Philippe Ombredanne , > Zaibo Xu , "David S . Miller" , > linux-accelerat...@lists.ozlabs.org, Lu Baolu > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > Message-ID: <20180921145201.ga3...@redhat.com> > References: <20180903005204.26041-1-nek.in...@gmail.com> > <20180917014244.ga27...@redhat.com> > <20180921100314.GH207969@Turing-Arch-b> > Content-Type: text/plain; charset="iso-8859-1" > Content-Disposition: inline > Content-Transfer-Encoding: 8bit > In-Reply-To: <20180921100314.GH207969@Turing-Arch-b> > User-Agent: Mutt/1.10.1 (2018-07-13) > X-Scanned-By: MIMEDefang 2.84 on 10.5.11.24 > X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 > (mx1.redhat.com [10.5.110.39]); Fri, 21 Sep 2018 14:52:10 + (UTC) > Return-Path: jgli...@redhat.com > X-MS-Exchange-Organization-AuthSource: DGGEMM406-HUB.china.huawei.com > X-MS-Exchange-Organization-AuthAs: Anonymous > MIME-Version: 1.0 > > On Fri, Sep 21, 2018 at 06:03:14PM +0800, Kenneth Lee wrote: > > On Sun, Sep 16, 2018 at 09:42:44PM -0400, Jerome Glisse wrote: > > > > > > So i want to summarize issues i have as this threads have dig deep into > > > details. For this i would like to differentiate two cases first the easy > > > one when relying on SVA/SVM. Then the second one when there is no SVA/SVM. > > > In both cases your objectives as i understand them: > > > > > > [R1]- expose a common user space API that make it easy to share boiler > > > plate code accross many devices (discovering devices, opening > > > device, creating context, creating command queue ...). > > > [R2]- try to share the device as much as possible up to device limits > > > (number of independant queues the device has) > > > [R3]- minimize syscall by allowing user space to directly schedule on the > > > device queue without a round trip to the kernel > > > > > > I don't think i missed any. > > > > > > > > > (1) Device with SVA/SVM > > > > > > For that case it is easy, you do not need to be in VFIO or part of any > > > thing specific in the kernel. There is no security risk (modulo bug in > > > the SVA/SVM silicon). Fork/exec is properly handle and binding a process > > > to a device is just couple dozen lines of code. > > > > > > > > > (2) Device does not have SVA/SVM (or it is disabled) > > > > > > You want to still allow device to be part of your framework. However > > > here i see fundamentals securities issues and you move the burden of > > > being careful to user space which i think is a bad idea. We should > > > never trus the userspace from kernel space. > > > > > > To keep the same API for the user space code you want a 1:1 mapping > > > between device physical address and process virtual address (ie if > > > device acc
Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive
On Tue, Sep 18, 2018 at 09:03:14AM -0400, Jerome Glisse wrote: > Date: Tue, 18 Sep 2018 09:03:14 -0400 > From: Jerome Glisse > To: Kenneth Lee > CC: Kenneth Lee , Alex Williamson > , Herbert Xu , > k...@vger.kernel.org, Jonathan Corbet , Greg Kroah-Hartman > , Joerg Roedel , > linux-...@vger.kernel.org, Sanjay Kumar , Hao > Fang , linux-ker...@vger.kernel.org, > linux...@huawei.com, iommu@lists.linux-foundation.org, "David S . Miller" > , linux-cry...@vger.kernel.org, Zhou Wang > , Philippe Ombredanne , > Thomas Gleixner , Zaibo Xu , > linux-accelerat...@lists.ozlabs.org, Lu Baolu > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > User-Agent: Mutt/1.10.1 (2018-07-13) > Message-ID: <20180918130314.ga3...@redhat.com> > > On Tue, Sep 18, 2018 at 02:00:14PM +0800, Kenneth Lee wrote: > > On Mon, Sep 17, 2018 at 08:37:45AM -0400, Jerome Glisse wrote: > > > On Mon, Sep 17, 2018 at 04:39:40PM +0800, Kenneth Lee wrote: > > > > On Sun, Sep 16, 2018 at 09:42:44PM -0400, Jerome Glisse wrote: > > > > > So i want to summarize issues i have as this threads have dig deep > > > > > into > > > > > details. For this i would like to differentiate two cases first the > > > > > easy > > > > > one when relying on SVA/SVM. Then the second one when there is no > > > > > SVA/SVM. > > > > > > > > Thank you very much for the summary. > > > > > > > > > In both cases your objectives as i understand them: > > > > > > > > > > [R1]- expose a common user space API that make it easy to share boiler > > > > > plate code accross many devices (discovering devices, opening > > > > > device, creating context, creating command queue ...). > > > > > [R2]- try to share the device as much as possible up to device limits > > > > > (number of independant queues the device has) > > > > > [R3]- minimize syscall by allowing user space to directly schedule on > > > > > the > > > > > device queue without a round trip to the kernel > > > > > > > > > > I don't think i missed any. > > > > > > > > > > > > > > > (1) Device with SVA/SVM > > > > > > > > > > For that case it is easy, you do not need to be in VFIO or part of any > > > > > thing specific in the kernel. There is no security risk (modulo bug in > > > > > the SVA/SVM silicon). Fork/exec is properly handle and binding a > > > > > process > > > > > to a device is just couple dozen lines of code. > > > > > > > > > > > > > This is right...logically. But the kernel has no clear definition about > > > > "Device > > > > with SVA/SVM" and no boiler plate for doing so. Then VFIO may become > > > > one of the > > > > boiler plate. > > > > > > > > VFIO is one of the wrappers for IOMMU for user space. And maybe it is > > > > the only > > > > one. If we add that support within VFIO, which solve most of the > > > > problem of > > > > SVA/SVM, it will save a lot of work in the future. > > > > > > You do not need to "wrap" IOMMU for SVA/SVM. Existing upstream SVA/SVM > > > user > > > all do the SVA/SVM setup in couple dozen lines and i failed to see how it > > > would require any more than that in your case. > > > > > > > > > > I think this is the key confliction between us. So could Alex please say > > > > something here? If the VFIO is going to take this into its scope, we > > > > can try > > > > together to solve all the problem on the way. If it it is not, it is > > > > also > > > > simple, we can just go to another way to fulfill this part of > > > > requirements even > > > > we have to duplicate most of the code. > > > > > > > > Another point I need to emphasis here: because we have to replace the > > > > hardware > > > > queue when fork, so it won't be very simple even in SVA/SVM case. > > > > > > I am assuming hardware queue can only be setup by the kernel and thus > > > you are totaly safe forkwise as the queue is setup against a PASID and > > > the child does not bind to any PASID and you use VM_DONTCOPY on the > > > mmap of the hardware
Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive
On Mon, Sep 17, 2018 at 08:37:45AM -0400, Jerome Glisse wrote: > Date: Mon, 17 Sep 2018 08:37:45 -0400 > From: Jerome Glisse > To: Kenneth Lee > CC: Kenneth Lee , Herbert Xu > , k...@vger.kernel.org, Jonathan Corbet > , Greg Kroah-Hartman , Joerg > Roedel , linux-...@vger.kernel.org, Sanjay Kumar > , Hao Fang , > iommu@lists.linux-foundation.org, linux-ker...@vger.kernel.org, > linux...@huawei.com, Alex Williamson , Thomas > Gleixner , linux-cry...@vger.kernel.org, Zhou Wang > , Philippe Ombredanne , > Zaibo Xu , "David S . Miller" , > linux-accelerat...@lists.ozlabs.org, Lu Baolu > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > User-Agent: Mutt/1.10.1 (2018-07-13) > Message-ID: <20180917123744.ga3...@redhat.com> > > On Mon, Sep 17, 2018 at 04:39:40PM +0800, Kenneth Lee wrote: > > On Sun, Sep 16, 2018 at 09:42:44PM -0400, Jerome Glisse wrote: > > > So i want to summarize issues i have as this threads have dig deep into > > > details. For this i would like to differentiate two cases first the easy > > > one when relying on SVA/SVM. Then the second one when there is no SVA/SVM. > > > > Thank you very much for the summary. > > > > > In both cases your objectives as i understand them: > > > > > > [R1]- expose a common user space API that make it easy to share boiler > > > plate code accross many devices (discovering devices, opening > > > device, creating context, creating command queue ...). > > > [R2]- try to share the device as much as possible up to device limits > > > (number of independant queues the device has) > > > [R3]- minimize syscall by allowing user space to directly schedule on the > > > device queue without a round trip to the kernel > > > > > > I don't think i missed any. > > > > > > > > > (1) Device with SVA/SVM > > > > > > For that case it is easy, you do not need to be in VFIO or part of any > > > thing specific in the kernel. There is no security risk (modulo bug in > > > the SVA/SVM silicon). Fork/exec is properly handle and binding a process > > > to a device is just couple dozen lines of code. > > > > > > > This is right...logically. But the kernel has no clear definition about > > "Device > > with SVA/SVM" and no boiler plate for doing so. Then VFIO may become one of > > the > > boiler plate. > > > > VFIO is one of the wrappers for IOMMU for user space. And maybe it is the > > only > > one. If we add that support within VFIO, which solve most of the problem of > > SVA/SVM, it will save a lot of work in the future. > > You do not need to "wrap" IOMMU for SVA/SVM. Existing upstream SVA/SVM user > all do the SVA/SVM setup in couple dozen lines and i failed to see how it > would require any more than that in your case. > > > > I think this is the key confliction between us. So could Alex please say > > something here? If the VFIO is going to take this into its scope, we can try > > together to solve all the problem on the way. If it it is not, it is also > > simple, we can just go to another way to fulfill this part of requirements > > even > > we have to duplicate most of the code. > > > > Another point I need to emphasis here: because we have to replace the > > hardware > > queue when fork, so it won't be very simple even in SVA/SVM case. > > I am assuming hardware queue can only be setup by the kernel and thus > you are totaly safe forkwise as the queue is setup against a PASID and > the child does not bind to any PASID and you use VM_DONTCOPY on the > mmap of the hardware MMIO queue because you should really use that flag > for that. > > > > > (2) Device does not have SVA/SVM (or it is disabled) > > > > > > You want to still allow device to be part of your framework. However > > > here i see fundamentals securities issues and you move the burden of > > > being careful to user space which i think is a bad idea. We should > > > never trus the userspace from kernel space. > > > > > > To keep the same API for the user space code you want a 1:1 mapping > > > between device physical address and process virtual address (ie if > > > device access device physical address A it is accessing the same > > > memory as what is backing the virtual address A in the process. > > > > > > Security issues are on two things: > > > [I1]- fork/exec, a process who opened any such device and created an > &
Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive
On Sun, Sep 16, 2018 at 09:42:44PM -0400, Jerome Glisse wrote: > Date: Sun, 16 Sep 2018 21:42:44 -0400 > From: Jerome Glisse > To: Kenneth Lee > CC: Jonathan Corbet , Herbert Xu > , "David S . Miller" , > Joerg Roedel , Alex Williamson > , Kenneth Lee , Hao > Fang , Zhou Wang , Zaibo Xu > , Philippe Ombredanne , Greg > Kroah-Hartman , Thomas Gleixner > , linux-...@vger.kernel.org, > linux-ker...@vger.kernel.org, linux-cry...@vger.kernel.org, > iommu@lists.linux-foundation.org, k...@vger.kernel.org, > linux-accelerat...@lists.ozlabs.org, Lu Baolu , > Sanjay Kumar , linux...@huawei.com > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > User-Agent: Mutt/1.10.1 (2018-07-13) > Message-ID: <20180917014244.ga27...@redhat.com> > > So i want to summarize issues i have as this threads have dig deep into > details. For this i would like to differentiate two cases first the easy > one when relying on SVA/SVM. Then the second one when there is no SVA/SVM. Thank you very much for the summary. > In both cases your objectives as i understand them: > > [R1]- expose a common user space API that make it easy to share boiler > plate code accross many devices (discovering devices, opening > device, creating context, creating command queue ...). > [R2]- try to share the device as much as possible up to device limits > (number of independant queues the device has) > [R3]- minimize syscall by allowing user space to directly schedule on the > device queue without a round trip to the kernel > > I don't think i missed any. > > > (1) Device with SVA/SVM > > For that case it is easy, you do not need to be in VFIO or part of any > thing specific in the kernel. There is no security risk (modulo bug in > the SVA/SVM silicon). Fork/exec is properly handle and binding a process > to a device is just couple dozen lines of code. > This is right...logically. But the kernel has no clear definition about "Device with SVA/SVM" and no boiler plate for doing so. Then VFIO may become one of the boiler plate. VFIO is one of the wrappers for IOMMU for user space. And maybe it is the only one. If we add that support within VFIO, which solve most of the problem of SVA/SVM, it will save a lot of work in the future. I think this is the key confliction between us. So could Alex please say something here? If the VFIO is going to take this into its scope, we can try together to solve all the problem on the way. If it it is not, it is also simple, we can just go to another way to fulfill this part of requirements even we have to duplicate most of the code. Another point I need to emphasis here: because we have to replace the hardware queue when fork, so it won't be very simple even in SVA/SVM case. > > (2) Device does not have SVA/SVM (or it is disabled) > > You want to still allow device to be part of your framework. However > here i see fundamentals securities issues and you move the burden of > being careful to user space which i think is a bad idea. We should > never trus the userspace from kernel space. > > To keep the same API for the user space code you want a 1:1 mapping > between device physical address and process virtual address (ie if > device access device physical address A it is accessing the same > memory as what is backing the virtual address A in the process. > > Security issues are on two things: > [I1]- fork/exec, a process who opened any such device and created an > active queue can transfer without its knowledge control of its > commands queue through COW. The parent map some anonymous region > to the device as a command queue buffer but because of COW the > parent can be the first to copy on write and thus the child can > inherit the original pages that are mapped to the hardware. > Here parent lose control and child gain it. This is indeed an issue. But it remains an issue only if you continue to use the queue and the memory after fork. We can use at_fork kinds of gadget to fix it in user space. >From some perspectives, I think the issue can be solved by iommu_notifier. For example, when the process is fork-ed, we can set the mapped device mmio space as COW for the child process, so a new queue can be created and set to the same state as the parent's if the space is accessed. Then we can have two separated queues for both the parent and the child. The memory part can be done in the same way. The thing is, the same strategy can be applied to VFIO without changing its original feature. > > [I2]- Because of [R3] you want to allow userspace to schedule commands > on the device without doing an ioctl and thus here user space > can schedule any commands to the device with a
Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive
On Fri, Sep 14, 2018 at 06:50:55AM +, Tian, Kevin wrote: > Date: Fri, 14 Sep 2018 06:50:55 + > From: "Tian, Kevin" > To: Jerome Glisse , Kenneth Lee > CC: Kenneth Lee , Alex Williamson > , Herbert Xu , > "k...@vger.kernel.org" , Jonathan Corbet > , Greg Kroah-Hartman , Zaibo > Xu , "linux-...@vger.kernel.org" > , "Kumar, Sanjay K" , > Hao Fang , "linux-ker...@vger.kernel.org" > , "linux...@huawei.com" > , "iommu@lists.linux-foundation.org" > , "David S . Miller" > , "linux-cry...@vger.kernel.org" > , Zhou Wang , > Philippe Ombredanne , Thomas Gleixner > , Joerg Roedel , > "linux-accelerat...@lists.ozlabs.org" > , Lu Baolu > Subject: RE: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > Message-ID: > > > > From: Jerome Glisse > > Sent: Thursday, September 13, 2018 10:52 PM > > > [...] > > AFAIK, on x86 and PPC at least, all PCIE devices are in the same group > > by default at boot or at least all devices behind the same bridge. > > the group thing reflects physical hierarchy limitation, not changed > cross boot. Please note iommu group defines the minimal isolation > boundary - all devices within same group must be attached to the > same iommu domain or address space, because physically IOMMU > cannot differentiate DMAs out of those devices. devices behind > legacy PCI-X bridge is one example. other examples include devices > behind a PCIe switch port which doesn't support ACS thus cannot > route p2p transaction to IOMMU. If talking about typical PCIe > endpoint (with upstreaming ports all supporting ACS), you'll get > one device per group. > > One iommu group today is attached to only one iommu domain. > In the future one group may attach to multiple domains, as the > aux domain concept being discussed in another thread. > > > > > Maybe they are kernel option to avoid that and userspace init program > > can definitly re-arrange that base on sysadmin policy). > > I don't think there is such option, as it may break isolation model > enabled by IOMMU. > > [...] > > > > That is why i am being pedantic :) on making sure there is good reasons > > > > to do what you do inside VFIO. I do believe that we want a common > > frame- > > > > work like the one you are proposing but i do not believe it should be > > > > part of VFIO given the baggages it comes with and that are not relevant > > > > to the use cases for this kind of devices. > > > > > The purpose of VFIO is clear - the kernel portal for granting generic > device resource (mmio, irq, etc.) to user space. VFIO doesn't care > what exactly a resource is used for (queue, cmd reg, etc.). If really > pursuing VFIO path is necessary, maybe such common framework > should lay down in user space, which gets all granted resource from > kernel driver thru VFIO and then provides accelerator services to > other processes? Yes. I think this is exactly what WarpDrive is now doing. This patch is just let the type1 driver use parent IOMMU for mdev. > > Thanks > Kevin -- -Kenneth(Hisilicon) 本邮件及其附件含有华为公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁 止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中 的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive
On Thu, Sep 13, 2018 at 10:51:50AM -0400, Jerome Glisse wrote: > Date: Thu, 13 Sep 2018 10:51:50 -0400 > From: Jerome Glisse > To: Kenneth Lee > CC: Kenneth Lee , Alex Williamson > , Herbert Xu , > k...@vger.kernel.org, Jonathan Corbet , Greg Kroah-Hartman > , Zaibo Xu , > linux-...@vger.kernel.org, Sanjay Kumar , Hao > Fang , linux-ker...@vger.kernel.org, > linux...@huawei.com, iommu@lists.linux-foundation.org, "David S . Miller" > , linux-cry...@vger.kernel.org, Zhou Wang > , Philippe Ombredanne , > Thomas Gleixner , Joerg Roedel , > linux-accelerat...@lists.ozlabs.org, Lu Baolu > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > User-Agent: Mutt/1.10.1 (2018-07-13) > Message-ID: <20180913145149.gb3...@redhat.com> > > On Thu, Sep 13, 2018 at 04:32:32PM +0800, Kenneth Lee wrote: > > On Tue, Sep 11, 2018 at 09:40:14AM -0400, Jerome Glisse wrote: > > > On Tue, Sep 11, 2018 at 02:40:43PM +0800, Kenneth Lee wrote: > > > > On Mon, Sep 10, 2018 at 11:33:59PM -0400, Jerome Glisse wrote: > > > > > On Tue, Sep 11, 2018 at 10:42:09AM +0800, Kenneth Lee wrote: > > > > > > On Mon, Sep 10, 2018 at 10:54:23AM -0400, Jerome Glisse wrote: > > > > > > > On Mon, Sep 10, 2018 at 11:28:09AM +0800, Kenneth Lee wrote: > > > > > > > > On Fri, Sep 07, 2018 at 12:53:06PM -0400, Jerome Glisse wrote: > > > > > > > > > On Fri, Sep 07, 2018 at 12:01:38PM +0800, Kenneth Lee wrote: > > > > > > > > > > On Thu, Sep 06, 2018 at 09:31:33AM -0400, Jerome Glisse > > > > > > > > > > wrote: > > > > > > > > > > > On Thu, Sep 06, 2018 at 05:45:32PM +0800, Kenneth Lee > > > > > > > > > > > wrote: > > > > > > > > > > > > On Tue, Sep 04, 2018 at 10:15:09AM -0600, Alex > > > > > > > > > > > > Williamson wrote: > > > > > > > > > > > > > On Tue, 4 Sep 2018 11:00:19 -0400 Jerome Glisse > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > On Mon, Sep 03, 2018 at 08:51:57AM +0800, Kenneth > > > > > > > > > > > > > > Lee wrote: > > > > > > > > > > > > > > [...] > > > > > > > > > > > > > > > > > I took a look at i915_gem_execbuffer_ioctl(). It seems it > > > > > > > > > > "copy_from_user" the > > > > > > > > > > user memory to the kernel. That is not what we need. What > > > > > > > > > > we try to get is: the > > > > > > > > > > user application do something on its data, and push it away > > > > > > > > > > to the accelerator, > > > > > > > > > > and says: "I'm tied, it is your turn to do the job...". > > > > > > > > > > Then the accelerator has > > > > > > > > > > the memory, referring any portion of it with the same VAs > > > > > > > > > > of the application, > > > > > > > > > > even the VAs are stored inside the memory itself. > > > > > > > > > > > > > > > > > > You were not looking at right place see > > > > > > > > > drivers/gpu/drm/i915/i915_gem_userptr.c > > > > > > > > > It does GUP and create GEM object AFAICR you can wrap that > > > > > > > > > GEM object into a > > > > > > > > > dma buffer object. > > > > > > > > > > > > > > > > > > > > > > > > > Thank you for directing me to this implementation. It is > > > > > > > > interesting:). > > > > > > > > > > > > > > > > But it is not yet solve my problem. If I understand it right, > > > > > > > > the userptr in > > > > > > > > i915 do the following: > > > > > > > > > > > > > > > > 1. The user process sets a user pointer with size to the kernel > > > > > > > > via ioctl. > > > > > > > > 2. The kernel wraps it as a dma-buf and keeps the process's mm > > > > > > > > for further > > > > > &g
Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive
On Tue, Sep 11, 2018 at 09:40:14AM -0400, Jerome Glisse wrote: > Date: Tue, 11 Sep 2018 09:40:14 -0400 > From: Jerome Glisse > To: Kenneth Lee > CC: Kenneth Lee , Alex Williamson > , Herbert Xu , > k...@vger.kernel.org, Jonathan Corbet , Greg Kroah-Hartman > , Zaibo Xu , > linux-...@vger.kernel.org, Sanjay Kumar , Hao > Fang , linux-ker...@vger.kernel.org, > linux...@huawei.com, iommu@lists.linux-foundation.org, "David S . Miller" > , linux-cry...@vger.kernel.org, Zhou Wang > , Philippe Ombredanne , > Thomas Gleixner , Joerg Roedel , > linux-accelerat...@lists.ozlabs.org, Lu Baolu > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > User-Agent: Mutt/1.10.1 (2018-07-13) > Message-ID: <20180911134013.ga3...@redhat.com> > > On Tue, Sep 11, 2018 at 02:40:43PM +0800, Kenneth Lee wrote: > > On Mon, Sep 10, 2018 at 11:33:59PM -0400, Jerome Glisse wrote: > > > On Tue, Sep 11, 2018 at 10:42:09AM +0800, Kenneth Lee wrote: > > > > On Mon, Sep 10, 2018 at 10:54:23AM -0400, Jerome Glisse wrote: > > > > > On Mon, Sep 10, 2018 at 11:28:09AM +0800, Kenneth Lee wrote: > > > > > > On Fri, Sep 07, 2018 at 12:53:06PM -0400, Jerome Glisse wrote: > > > > > > > On Fri, Sep 07, 2018 at 12:01:38PM +0800, Kenneth Lee wrote: > > > > > > > > On Thu, Sep 06, 2018 at 09:31:33AM -0400, Jerome Glisse wrote: > > > > > > > > > On Thu, Sep 06, 2018 at 05:45:32PM +0800, Kenneth Lee wrote: > > > > > > > > > > On Tue, Sep 04, 2018 at 10:15:09AM -0600, Alex Williamson > > > > > > > > > > wrote: > > > > > > > > > > > On Tue, 4 Sep 2018 11:00:19 -0400 Jerome Glisse > > > > > > > > > > > wrote: > > > > > > > > > > > > On Mon, Sep 03, 2018 at 08:51:57AM +0800, Kenneth Lee > > > > > > > > > > > > wrote: > > > > > > > > > > [...] > > > > > > > > > > > > > I took a look at i915_gem_execbuffer_ioctl(). It seems it > > > > > > > > "copy_from_user" the > > > > > > > > user memory to the kernel. That is not what we need. What we > > > > > > > > try to get is: the > > > > > > > > user application do something on its data, and push it away to > > > > > > > > the accelerator, > > > > > > > > and says: "I'm tied, it is your turn to do the job...". Then > > > > > > > > the accelerator has > > > > > > > > the memory, referring any portion of it with the same VAs of > > > > > > > > the application, > > > > > > > > even the VAs are stored inside the memory itself. > > > > > > > > > > > > > > You were not looking at right place see > > > > > > > drivers/gpu/drm/i915/i915_gem_userptr.c > > > > > > > It does GUP and create GEM object AFAICR you can wrap that GEM > > > > > > > object into a > > > > > > > dma buffer object. > > > > > > > > > > > > > > > > > > > Thank you for directing me to this implementation. It is > > > > > > interesting:). > > > > > > > > > > > > But it is not yet solve my problem. If I understand it right, the > > > > > > userptr in > > > > > > i915 do the following: > > > > > > > > > > > > 1. The user process sets a user pointer with size to the kernel via > > > > > > ioctl. > > > > > > 2. The kernel wraps it as a dma-buf and keeps the process's mm for > > > > > > further > > > > > >reference. > > > > > > 3. The user pages are allocated, GUPed or DMA mapped to the device. > > > > > > So the data > > > > > >can be shared between the user space and the hardware. > > > > > > > > > > > > But my scenario is: > > > > > > > > > > > > 1. The user process has some data in the user space, pointed by a > > > > > > pointer, say > > > > > >ptr1. And within the memory, there may be some other pointers, > > > > > > let's say one > > > > > >of them is ptr2. > > > &
Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive
On Mon, Sep 10, 2018 at 11:33:59PM -0400, Jerome Glisse wrote: > Date: Mon, 10 Sep 2018 23:33:59 -0400 > From: Jerome Glisse > To: Kenneth Lee > CC: Kenneth Lee , Zaibo Xu , > Herbert Xu , k...@vger.kernel.org, Jonathan > Corbet , Greg Kroah-Hartman , > Joerg Roedel , linux-...@vger.kernel.org, Sanjay Kumar > , Hao Fang , > iommu@lists.linux-foundation.org, linux-ker...@vger.kernel.org, > linux...@huawei.com, Alex Williamson , > linux-cry...@vger.kernel.org, Zhou Wang , > Philippe Ombredanne , Thomas Gleixner > , "David S . Miller" , > linux-accelerat...@lists.ozlabs.org, Lu Baolu > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > User-Agent: Mutt/1.10.1 (2018-07-13) > Message-ID: <20180911033358.ga4...@redhat.com> > > On Tue, Sep 11, 2018 at 10:42:09AM +0800, Kenneth Lee wrote: > > On Mon, Sep 10, 2018 at 10:54:23AM -0400, Jerome Glisse wrote: > > > On Mon, Sep 10, 2018 at 11:28:09AM +0800, Kenneth Lee wrote: > > > > On Fri, Sep 07, 2018 at 12:53:06PM -0400, Jerome Glisse wrote: > > > > > On Fri, Sep 07, 2018 at 12:01:38PM +0800, Kenneth Lee wrote: > > > > > > On Thu, Sep 06, 2018 at 09:31:33AM -0400, Jerome Glisse wrote: > > > > > > > On Thu, Sep 06, 2018 at 05:45:32PM +0800, Kenneth Lee wrote: > > > > > > > > On Tue, Sep 04, 2018 at 10:15:09AM -0600, Alex Williamson wrote: > > > > > > > > > On Tue, 4 Sep 2018 11:00:19 -0400 Jerome Glisse > > > > > > > > > wrote: > > > > > > > > > > On Mon, Sep 03, 2018 at 08:51:57AM +0800, Kenneth Lee wrote: > > > > > > [...] > > > > > > > > > I took a look at i915_gem_execbuffer_ioctl(). It seems it > > > > > > "copy_from_user" the > > > > > > user memory to the kernel. That is not what we need. What we try to > > > > > > get is: the > > > > > > user application do something on its data, and push it away to the > > > > > > accelerator, > > > > > > and says: "I'm tied, it is your turn to do the job...". Then the > > > > > > accelerator has > > > > > > the memory, referring any portion of it with the same VAs of the > > > > > > application, > > > > > > even the VAs are stored inside the memory itself. > > > > > > > > > > You were not looking at right place see > > > > > drivers/gpu/drm/i915/i915_gem_userptr.c > > > > > It does GUP and create GEM object AFAICR you can wrap that GEM object > > > > > into a > > > > > dma buffer object. > > > > > > > > > > > > > Thank you for directing me to this implementation. It is interesting:). > > > > > > > > But it is not yet solve my problem. If I understand it right, the > > > > userptr in > > > > i915 do the following: > > > > > > > > 1. The user process sets a user pointer with size to the kernel via > > > > ioctl. > > > > 2. The kernel wraps it as a dma-buf and keeps the process's mm for > > > > further > > > >reference. > > > > 3. The user pages are allocated, GUPed or DMA mapped to the device. So > > > > the data > > > >can be shared between the user space and the hardware. > > > > > > > > But my scenario is: > > > > > > > > 1. The user process has some data in the user space, pointed by a > > > > pointer, say > > > >ptr1. And within the memory, there may be some other pointers, let's > > > > say one > > > >of them is ptr2. > > > > 2. Now I need to assign ptr1 *directly* to the hardware MMIO space. And > > > > the > > > >hardware must refer ptr1 and ptr2 *directly* for data. > > > > > > > > Userptr lets the hardware and process share the same memory space. But > > > > I need > > > > them to share the same *address space*. So IOMMU is a MUST for > > > > WarpDrive, > > > > NOIOMMU mode, as Jean said, is just for verifying some of the procedure > > > > is OK. > > > > > > So to be 100% clear should we _ignore_ the non SVA/SVM case ? > > > If so then wait for necessary SVA/SVM to land and do warp drive > > > without non SVA/SVM path. > > > > > > > I think we should clear the concept of SVA/SVM
Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive
On Mon, Sep 10, 2018 at 10:54:23AM -0400, Jerome Glisse wrote: > Date: Mon, 10 Sep 2018 10:54:23 -0400 > From: Jerome Glisse > To: Kenneth Lee > CC: Kenneth Lee , Alex Williamson > , Herbert Xu , > k...@vger.kernel.org, Jonathan Corbet , Greg Kroah-Hartman > , Joerg Roedel , > linux-...@vger.kernel.org, Sanjay Kumar , Hao > Fang , linux-ker...@vger.kernel.org, > linux...@huawei.com, iommu@lists.linux-foundation.org, "David S . Miller" > , linux-cry...@vger.kernel.org, Zhou Wang > , Philippe Ombredanne , > Thomas Gleixner , Zaibo Xu , > linux-accelerat...@lists.ozlabs.org, Lu Baolu > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > User-Agent: Mutt/1.10.0 (2018-05-17) > Message-ID: <20180910145423.ga3...@redhat.com> > > On Mon, Sep 10, 2018 at 11:28:09AM +0800, Kenneth Lee wrote: > > On Fri, Sep 07, 2018 at 12:53:06PM -0400, Jerome Glisse wrote: > > > On Fri, Sep 07, 2018 at 12:01:38PM +0800, Kenneth Lee wrote: > > > > On Thu, Sep 06, 2018 at 09:31:33AM -0400, Jerome Glisse wrote: > > > > > On Thu, Sep 06, 2018 at 05:45:32PM +0800, Kenneth Lee wrote: > > > > > > On Tue, Sep 04, 2018 at 10:15:09AM -0600, Alex Williamson wrote: > > > > > > > On Tue, 4 Sep 2018 11:00:19 -0400 Jerome Glisse > > > > > > > wrote: > > > > > > > > On Mon, Sep 03, 2018 at 08:51:57AM +0800, Kenneth Lee wrote: > > [...] > > > > > I took a look at i915_gem_execbuffer_ioctl(). It seems it > > > > "copy_from_user" the > > > > user memory to the kernel. That is not what we need. What we try to get > > > > is: the > > > > user application do something on its data, and push it away to the > > > > accelerator, > > > > and says: "I'm tied, it is your turn to do the job...". Then the > > > > accelerator has > > > > the memory, referring any portion of it with the same VAs of the > > > > application, > > > > even the VAs are stored inside the memory itself. > > > > > > You were not looking at right place see > > > drivers/gpu/drm/i915/i915_gem_userptr.c > > > It does GUP and create GEM object AFAICR you can wrap that GEM object > > > into a > > > dma buffer object. > > > > > > > Thank you for directing me to this implementation. It is interesting:). > > > > But it is not yet solve my problem. If I understand it right, the userptr in > > i915 do the following: > > > > 1. The user process sets a user pointer with size to the kernel via ioctl. > > 2. The kernel wraps it as a dma-buf and keeps the process's mm for further > >reference. > > 3. The user pages are allocated, GUPed or DMA mapped to the device. So the > > data > >can be shared between the user space and the hardware. > > > > But my scenario is: > > > > 1. The user process has some data in the user space, pointed by a pointer, > > say > >ptr1. And within the memory, there may be some other pointers, let's say > > one > >of them is ptr2. > > 2. Now I need to assign ptr1 *directly* to the hardware MMIO space. And the > >hardware must refer ptr1 and ptr2 *directly* for data. > > > > Userptr lets the hardware and process share the same memory space. But I > > need > > them to share the same *address space*. So IOMMU is a MUST for WarpDrive, > > NOIOMMU mode, as Jean said, is just for verifying some of the procedure is > > OK. > > So to be 100% clear should we _ignore_ the non SVA/SVM case ? > If so then wait for necessary SVA/SVM to land and do warp drive > without non SVA/SVM path. > I think we should clear the concept of SVA/SVM here. As my understanding, Share Virtual Address/Memory means: any virtual address in a process can be used by device at the same time. This requires IOMMU device to support PASID. And optionally, it requires the feature of page-fault-from-device. But before the feature is settled down, IOMMU can be used immediately in the current kernel. That make it possible to assign ONE process's virtual addresses to the device's IOMMU page table with GUP. This make WarpDrive work well for one process. Now We are talking about SVA and PASID, just to make sure WarpDrive can benefit from the feature in the future. It dose not means WarpDrive is useless before that. And it works for our Zip and RSA accelerators in physical world. > If you still want non SVA/SVM path what you want to do only works > if both ptr1 and ptr2 are in a range that is DMA mapped to the > device (moreover you
Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive
On Fri, Sep 07, 2018 at 12:53:06PM -0400, Jerome Glisse wrote: > Date: Fri, 7 Sep 2018 12:53:06 -0400 > From: Jerome Glisse > To: Kenneth Lee > CC: Kenneth Lee , Herbert Xu > , k...@vger.kernel.org, Jonathan Corbet > , Greg Kroah-Hartman , Joerg > Roedel , linux-...@vger.kernel.org, Sanjay Kumar > , Hao Fang , > iommu@lists.linux-foundation.org, linux-ker...@vger.kernel.org, > linux...@huawei.com, Alex Williamson , Thomas > Gleixner , linux-cry...@vger.kernel.org, Zhou Wang > , Philippe Ombredanne , > Zaibo Xu , "David S . Miller" , > linux-accelerat...@lists.ozlabs.org, Lu Baolu > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > User-Agent: Mutt/1.10.0 (2018-05-17) > Message-ID: <20180907165303.ga3...@redhat.com> > > On Fri, Sep 07, 2018 at 12:01:38PM +0800, Kenneth Lee wrote: > > On Thu, Sep 06, 2018 at 09:31:33AM -0400, Jerome Glisse wrote: > > > Date: Thu, 6 Sep 2018 09:31:33 -0400 > > > From: Jerome Glisse > > > To: Kenneth Lee > > > CC: Alex Williamson , Kenneth Lee > > > , Jonathan Corbet , Herbert Xu > > > , "David S . Miller" , > > > Joerg Roedel , Hao Fang , Zhou > > > Wang > > > , Zaibo Xu , Philippe > > > Ombredanne , Greg Kroah-Hartman > > > , Thomas Gleixner , > > > linux-...@vger.kernel.org, linux-ker...@vger.kernel.org, > > > linux-cry...@vger.kernel.org, iommu@lists.linux-foundation.org, > > > k...@vger.kernel.org, linux-accelerat...@lists.ozlabs.org, Lu Baolu > > > , Sanjay Kumar , > > > linux...@huawei.com > > > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > > > User-Agent: Mutt/1.10.0 (2018-05-17) > > > Message-ID: <20180906133133.ga3...@redhat.com> > > > > > > On Thu, Sep 06, 2018 at 05:45:32PM +0800, Kenneth Lee wrote: > > > > On Tue, Sep 04, 2018 at 10:15:09AM -0600, Alex Williamson wrote: > > > > > Date: Tue, 4 Sep 2018 10:15:09 -0600 > > > > > From: Alex Williamson > > > > > To: Jerome Glisse > > > > > CC: Kenneth Lee , Jonathan Corbet > > > > > , > > > > > Herbert Xu , "David S . Miller" > > > > > , Joerg Roedel , Kenneth Lee > > > > > , Hao Fang , Zhou Wang > > > > > , Zaibo Xu , Philippe > > > > > Ombredanne , Greg Kroah-Hartman > > > > > , Thomas Gleixner , > > > > > linux-...@vger.kernel.org, linux-ker...@vger.kernel.org, > > > > > linux-cry...@vger.kernel.org, iommu@lists.linux-foundation.org, > > > > > k...@vger.kernel.org, linux-accelerat...@lists.ozlabs.org, Lu Baolu > > > > > , Sanjay Kumar , > > > > > linux...@huawei.com > > > > > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, > > > > > WarpDrive > > > > > Message-ID: <20180904101509.62314...@t450s.home> > > > > > > > > > > On Tue, 4 Sep 2018 11:00:19 -0400 > > > > > Jerome Glisse wrote: > > > > > > > > > > > On Mon, Sep 03, 2018 at 08:51:57AM +0800, Kenneth Lee wrote: > > > > > > > From: Kenneth Lee > > > > > > > > > > > > > > WarpDrive is an accelerator framework to expose the hardware > > > > > > > capabilities > > > > > > > directly to the user space. It makes use of the exist vfio and > > > > > > > vfio-mdev > > > > > > > facilities. So the user application can send request and DMA to > > > > > > > the > > > > > > > hardware without interaction with the kernel. This removes the > > > > > > > latency > > > > > > > of syscall. > > > > > > > > > > > > > > WarpDrive is the name for the whole framework. The component in > > > > > > > kernel > > > > > > > is called SDMDEV, Share Domain Mediated Device. Driver driver > > > > > > > exposes its > > > > > > > hardware resource by registering to SDMDEV as a VFIO-Mdev. So the > > > > > > > user > > > > > > > library of WarpDrive can access it via VFIO interface. > > > > > > > > > > > > > > The patchset contains document for the detail. Please refer to it > > > > > > > for more > > > > > > > inform
Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive
On Thu, Sep 06, 2018 at 09:31:33AM -0400, Jerome Glisse wrote: > Date: Thu, 6 Sep 2018 09:31:33 -0400 > From: Jerome Glisse > To: Kenneth Lee > CC: Alex Williamson , Kenneth Lee > , Jonathan Corbet , Herbert Xu > , "David S . Miller" , > Joerg Roedel , Hao Fang , Zhou Wang > , Zaibo Xu , Philippe > Ombredanne , Greg Kroah-Hartman > , Thomas Gleixner , > linux-...@vger.kernel.org, linux-ker...@vger.kernel.org, > linux-cry...@vger.kernel.org, iommu@lists.linux-foundation.org, > k...@vger.kernel.org, linux-accelerat...@lists.ozlabs.org, Lu Baolu > , Sanjay Kumar , > linux...@huawei.com > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > User-Agent: Mutt/1.10.0 (2018-05-17) > Message-ID: <20180906133133.ga3...@redhat.com> > > On Thu, Sep 06, 2018 at 05:45:32PM +0800, Kenneth Lee wrote: > > On Tue, Sep 04, 2018 at 10:15:09AM -0600, Alex Williamson wrote: > > > Date: Tue, 4 Sep 2018 10:15:09 -0600 > > > From: Alex Williamson > > > To: Jerome Glisse > > > CC: Kenneth Lee , Jonathan Corbet , > > > Herbert Xu , "David S . Miller" > > > , Joerg Roedel , Kenneth Lee > > > , Hao Fang , Zhou Wang > > > , Zaibo Xu , Philippe > > > Ombredanne , Greg Kroah-Hartman > > > , Thomas Gleixner , > > > linux-...@vger.kernel.org, linux-ker...@vger.kernel.org, > > > linux-cry...@vger.kernel.org, iommu@lists.linux-foundation.org, > > > k...@vger.kernel.org, linux-accelerat...@lists.ozlabs.org, Lu Baolu > > > , Sanjay Kumar , > > > linux...@huawei.com > > > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > > > Message-ID: <20180904101509.62314...@t450s.home> > > > > > > On Tue, 4 Sep 2018 11:00:19 -0400 > > > Jerome Glisse wrote: > > > > > > > On Mon, Sep 03, 2018 at 08:51:57AM +0800, Kenneth Lee wrote: > > > > > From: Kenneth Lee > > > > > > > > > > WarpDrive is an accelerator framework to expose the hardware > > > > > capabilities > > > > > directly to the user space. It makes use of the exist vfio and > > > > > vfio-mdev > > > > > facilities. So the user application can send request and DMA to the > > > > > hardware without interaction with the kernel. This removes the latency > > > > > of syscall. > > > > > > > > > > WarpDrive is the name for the whole framework. The component in kernel > > > > > is called SDMDEV, Share Domain Mediated Device. Driver driver exposes > > > > > its > > > > > hardware resource by registering to SDMDEV as a VFIO-Mdev. So the user > > > > > library of WarpDrive can access it via VFIO interface. > > > > > > > > > > The patchset contains document for the detail. Please refer to it for > > > > > more > > > > > information. > > > > > > > > > > This patchset is intended to be used with Jean Philippe Brucker's SVA > > > > > patch [1], which enables not only IO side page fault, but also PASID > > > > > support to IOMMU and VFIO. > > > > > > > > > > With these features, WarpDrive can support non-pinned memory and > > > > > multi-process in the same accelerator device. We tested it in our SoC > > > > > integrated Accelerator (board ID: D06, Chip ID: HIP08). A reference > > > > > work > > > > > tree can be found here: [2]. > > > > > > > > > > But it is not mandatory. This patchset is tested in the latest > > > > > mainline > > > > > kernel without the SVA patches. So it supports only one process for > > > > > each > > > > > accelerator. > > > > > > > > > > We have noticed the IOMMU aware mdev RFC announced recently [3]. > > > > > > > > > > The IOMMU aware mdev has similar idea but different intention > > > > > comparing to > > > > > WarpDrive. It intends to dedicate part of the hardware resource to a > > > > > VM. > > > > > And the design is supposed to be used with Scalable I/O > > > > > Virtualization. > > > > > While sdmdev is intended to share the hardware resource with a big > > > > > amount > > > > > of processes. It just requires the hardware supporting address > > > > > translation per process (PCI
Re: [PATCH 1/7] vfio/sdmdev: Add documents for WarpDrive framework
On Thu, Sep 06, 2018 at 11:36:36AM -0700, Randy Dunlap wrote: > Date: Thu, 6 Sep 2018 11:36:36 -0700 > From: Randy Dunlap > To: Kenneth Lee , Jonathan Corbet , > Herbert Xu , "David S . Miller" > , Joerg Roedel , Alex Williamson > , Kenneth Lee , Hao > Fang , Zhou Wang , Zaibo Xu > , Philippe Ombredanne , Greg > Kroah-Hartman , Thomas Gleixner > , linux-...@vger.kernel.org, > linux-ker...@vger.kernel.org, linux-cry...@vger.kernel.org, > iommu@lists.linux-foundation.org, k...@vger.kernel.org, > linux-accelerat...@lists.ozlabs.org, Lu Baolu , > Sanjay Kumar > CC: linux...@huawei.com > Subject: Re: [PATCH 1/7] vfio/sdmdev: Add documents for WarpDrive framework > User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 > Thunderbird/52.9.1 > Message-ID: <56f5f66d-f6d9-f4fa-40ca-e4a8bad17...@infradead.org> > > Hi, > > On 09/02/2018 05:51 PM, Kenneth Lee wrote: > > From: Kenneth Lee > > > > WarpDrive is a common user space accelerator framework. Its main component > > in Kernel is called sdmdev, Share Domain Mediated Device. It exposes > > the hardware capabilities to the user space via vfio-mdev. So processes in > > user land can obtain a "queue" by open the device and direct access the > > hardware MMIO space or do DMA operation via VFIO interface. > > > > WarpDrive is intended to be used with Jean Philippe Brucker's SVA > > patchset to support multi-process. But This is not a must. Without the > > SVA patches, WarpDrive can still work for one process for every hardware > > device. > > > > This patch add detail documents for the framework. > > > > Signed-off-by: Kenneth Lee > > --- > > Documentation/00-INDEX| 2 + > > Documentation/warpdrive/warpdrive.rst | 100 > > Documentation/warpdrive/wd-arch.svg | 728 ++ > > 3 files changed, 830 insertions(+) > > create mode 100644 Documentation/warpdrive/warpdrive.rst > > create mode 100644 Documentation/warpdrive/wd-arch.svg > > > diff --git a/Documentation/warpdrive/warpdrive.rst > > b/Documentation/warpdrive/warpdrive.rst > > new file mode 100644 > > index ..6d2a5d1e08c4 > > --- /dev/null > > +++ b/Documentation/warpdrive/warpdrive.rst > > @@ -0,0 +1,100 @@ > > +Introduction of WarpDrive > > += > > + > > +*WarpDrive* is a general accelerator framework for user space. It intends > > to > > +provide interface for the user process to send request to hardware > > +accelerator without heavy user-kernel interaction cost. > > + > > +The *WarpDrive* user library is supposed to provide a pipe-based API, such > > as: > > Do you say "is supposed to" because it doesn't do that (yet)? > Or you could just change that to say: > >The WarpDrive user library provides a pipe-based API, such as: > Actually, I tried to say it can be defined like this. But people can choose other implementation with the same kernel API. I will say it explicitly in the future version. Thank you. > > > +:: > > +int wd_request_queue(struct wd_queue *q); > > +void wd_release_queue(struct wd_queue *q); > > + > > +int wd_send(struct wd_queue *q, void *req); > > +int wd_recv(struct wd_queue *q, void **req); > > +int wd_recv_sync(struct wd_queue *q, void **req); > > +int wd_flush(struct wd_queue *q); > > + > > +*wd_request_queue* creates the pipe connection, *queue*, between the > > +application and the hardware. The application sends request and pulls the > > +answer back by asynchronized wd_send/wd_recv, which directly interact with > > the > > +hardware (by MMIO or share memory) without syscall. > > + > > +*WarpDrive* maintains a unified application address space among all > > involved > > +accelerators. With the following APIs: :: > > Seems like an extra '.' there. How about: > > accelerators with the following APIs: :: > Err, the "with..." clause belong to the following "The referred process space...". > > + > > +int wd_mem_share(struct wd_queue *q, const void *addr, > > + size_t size, int flags); > > +void wd_mem_unshare(struct wd_queue *q, const void *addr, size_t > > size); > > + > > +The referred process space shared by these APIs can be directly referred > > by the > > +hardware. The process can also dedicate its whole process space with flags, > > +*WD_SHARE_ALL* (not in this patch yet). >
Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive
On Tue, Sep 04, 2018 at 10:15:09AM -0600, Alex Williamson wrote: > Date: Tue, 4 Sep 2018 10:15:09 -0600 > From: Alex Williamson > To: Jerome Glisse > CC: Kenneth Lee , Jonathan Corbet , > Herbert Xu , "David S . Miller" > , Joerg Roedel , Kenneth Lee > , Hao Fang , Zhou Wang > , Zaibo Xu , Philippe > Ombredanne , Greg Kroah-Hartman > , Thomas Gleixner , > linux-...@vger.kernel.org, linux-ker...@vger.kernel.org, > linux-cry...@vger.kernel.org, iommu@lists.linux-foundation.org, > k...@vger.kernel.org, linux-accelerat...@lists.ozlabs.org, Lu Baolu > , Sanjay Kumar , > linux...@huawei.com > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > Message-ID: <20180904101509.62314...@t450s.home> > > On Tue, 4 Sep 2018 11:00:19 -0400 > Jerome Glisse wrote: > > > On Mon, Sep 03, 2018 at 08:51:57AM +0800, Kenneth Lee wrote: > > > From: Kenneth Lee > > > > > > WarpDrive is an accelerator framework to expose the hardware capabilities > > > directly to the user space. It makes use of the exist vfio and vfio-mdev > > > facilities. So the user application can send request and DMA to the > > > hardware without interaction with the kernel. This removes the latency > > > of syscall. > > > > > > WarpDrive is the name for the whole framework. The component in kernel > > > is called SDMDEV, Share Domain Mediated Device. Driver driver exposes its > > > hardware resource by registering to SDMDEV as a VFIO-Mdev. So the user > > > library of WarpDrive can access it via VFIO interface. > > > > > > The patchset contains document for the detail. Please refer to it for more > > > information. > > > > > > This patchset is intended to be used with Jean Philippe Brucker's SVA > > > patch [1], which enables not only IO side page fault, but also PASID > > > support to IOMMU and VFIO. > > > > > > With these features, WarpDrive can support non-pinned memory and > > > multi-process in the same accelerator device. We tested it in our SoC > > > integrated Accelerator (board ID: D06, Chip ID: HIP08). A reference work > > > tree can be found here: [2]. > > > > > > But it is not mandatory. This patchset is tested in the latest mainline > > > kernel without the SVA patches. So it supports only one process for each > > > accelerator. > > > > > > We have noticed the IOMMU aware mdev RFC announced recently [3]. > > > > > > The IOMMU aware mdev has similar idea but different intention comparing to > > > WarpDrive. It intends to dedicate part of the hardware resource to a VM. > > > And the design is supposed to be used with Scalable I/O Virtualization. > > > While sdmdev is intended to share the hardware resource with a big amount > > > of processes. It just requires the hardware supporting address > > > translation per process (PCIE's PASID or ARM SMMU's substream ID). > > > > > > But we don't see serious confliction on both design. We believe they can > > > be > > > normalized as one. > > > > > > > So once again i do not understand why you are trying to do things > > this way. Kernel already have tons of example of everything you > > want to do without a new framework. Moreover i believe you are > > confuse by VFIO. To me VFIO is for VM not to create general device > > driver frame work. > > VFIO is a userspace driver framework, the VM use case just happens to > be a rather prolific one. VFIO was never intended to be solely a VM > device interface and has several other userspace users, notably DPDK > and SPDK, an NVMe backend in QEMU, a userspace NVMe driver, a ruby > wrapper, and perhaps others that I'm not aware of. Whether vfio is > appropriate interface here might certainly still be a debatable topic, > but I would strongly disagree with your last sentence above. Thanks, > > Alex > Yes, that is also my standpoint here. > > So here is your use case as i understand it. You have a device > > with a limited number of command queues (can be just one) and in > > some case it can support SVA/SVM (when hardware support it and it > > is not disabled). Final requirement is being able to schedule cmds > > from userspace without ioctl. All of this exists already exists > > upstream in few device drivers. > > > > > > So here is how every body else is doing it. Please explain why > > this does not work. > > > > 1 Userspace open device file driver. Kernel device driver create > > a contex
Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive
On Mon, Sep 03, 2018 at 10:32:16AM +0800, Lu Baolu wrote: > Date: Mon, 3 Sep 2018 10:32:16 +0800 > From: Lu Baolu > To: Kenneth Lee , Jonathan Corbet , > Herbert Xu , "David S . Miller" > , Joerg Roedel , Alex Williamson > , Kenneth Lee , Hao > Fang , Zhou Wang , Zaibo Xu > , Philippe Ombredanne , Greg > Kroah-Hartman , Thomas Gleixner > , linux-...@vger.kernel.org, > linux-ker...@vger.kernel.org, linux-cry...@vger.kernel.org, > iommu@lists.linux-foundation.org, k...@vger.kernel.org, > linux-accelerat...@lists.ozlabs.org, Sanjay Kumar > > CC: baolu...@linux.intel.com, linux...@huawei.com > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 > Thunderbird/52.9.1 > Message-ID: <81edb8ff-d046-34e5-aee7-d8564e251...@linux.intel.com> > > Hi, > > On 09/03/2018 08:51 AM, Kenneth Lee wrote: > >From: Kenneth Lee > > > >WarpDrive is an accelerator framework to expose the hardware capabilities > >directly to the user space. It makes use of the exist vfio and vfio-mdev > >facilities. So the user application can send request and DMA to the > >hardware without interaction with the kernel. This removes the latency > >of syscall. > > > >WarpDrive is the name for the whole framework. The component in kernel > >is called SDMDEV, Share Domain Mediated Device. Driver driver exposes its > >hardware resource by registering to SDMDEV as a VFIO-Mdev. So the user > >library of WarpDrive can access it via VFIO interface. > > > >The patchset contains document for the detail. Please refer to it for more > >information. > > > >This patchset is intended to be used with Jean Philippe Brucker's SVA > >patch [1], which enables not only IO side page fault, but also PASID > >support to IOMMU and VFIO. > > > >With these features, WarpDrive can support non-pinned memory and > >multi-process in the same accelerator device. We tested it in our SoC > >integrated Accelerator (board ID: D06, Chip ID: HIP08). A reference work > >tree can be found here: [2]. > > > >But it is not mandatory. This patchset is tested in the latest mainline > >kernel without the SVA patches. So it supports only one process for each > >accelerator. > > > >We have noticed the IOMMU aware mdev RFC announced recently [3]. > > > >The IOMMU aware mdev has similar idea but different intention comparing to > >WarpDrive. It intends to dedicate part of the hardware resource to a VM. > >And the design is supposed to be used with Scalable I/O Virtualization. > >While sdmdev is intended to share the hardware resource with a big amount > >of processes. It just requires the hardware supporting address > >translation per process (PCIE's PASID or ARM SMMU's substream ID). > > > >But we don't see serious confliction on both design. We believe they can be > >normalized as one. > > > >The patch 1 is document of the framework. The patch 2 and 3 add sdmdev > >support. The patch 4, 5 and 6 is drivers for Hislicon's ZIP Accelerator > >which is registered to both crypto and warpdrive(sdmdev) and can be > >used from kernel or user space at the same time. The patch 7 is a user > >space sample demonstrating how WarpDrive works. > > > > > >Change History: > >V2 changed from V1: > > 1. Change kernel framework name from SPIMDEV (Share Parent IOMMU > >Mdev) to SDMDEV (Share Domain Mdev). > > 2. Allocate Hardware Resource when a new mdev is created (While > >it is allocated when the mdev is openned) > > 3. Unmap pages from the shared domain when the sdmdev iommu group is > >detached. (This procedure is necessary, but missed in V1) > > 4. Update document accordingly. > > 5. Rebase to the latest kernel (4.19.0-rc1) > > > > According the review comment on RFCv1, We did try to use dma-buf > > as back end of WarpDrive. It can work properly with the current > > solution [4], but it cannot make use of process's > > own memory address space directly. This is important to many > > acceleration scenario. So dma-buf will be taken as a backup > > alternative for noiommu scenario, it will be added in the future > > version. > > > > > >Refernces: > >[1] https://www.spinics.net/lists/kernel/msg2651481.html > >[2] > >https://github.com/Kenneth-Lee/linux-kernel-warpdrive/tree/warpdrive-sva-v0.5 > >[3] https://lkml.org/lkml/2018/7/22/34 > > Please refer to the latest version posted here for discussion. > >
Re: [PATCH 7/7] vfio/sdmdev: add user sample
On Sun, Sep 02, 2018 at 07:25:12PM -0700, Randy Dunlap wrote: > Date: Sun, 2 Sep 2018 19:25:12 -0700 > From: Randy Dunlap > To: Kenneth Lee , Jonathan Corbet , > Herbert Xu , "David S . Miller" > , Joerg Roedel , Alex Williamson > , Kenneth Lee , Hao > Fang , Zhou Wang , Zaibo Xu > , Philippe Ombredanne , Greg > Kroah-Hartman , Thomas Gleixner > , linux-...@vger.kernel.org, > linux-ker...@vger.kernel.org, linux-cry...@vger.kernel.org, > iommu@lists.linux-foundation.org, k...@vger.kernel.org, > linux-accelerat...@lists.ozlabs.org, Lu Baolu , > Sanjay Kumar > CC: linux...@huawei.com > Subject: Re: [PATCH 7/7] vfio/sdmdev: add user sample > User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 > Thunderbird/52.9.1 > Message-ID: > > On 09/02/2018 05:52 PM, Kenneth Lee wrote: > > From: Kenneth Lee > > > > This is the sample code to demostrate how WrapDrive user application > > should be. > > > > It contains: > > > > 1. wd.[ch], the common library to provide WrapDrive interface. > > WarpDrive > > > 2. drv/*, the user driver to access the hardware upon spimdev > > 3. test/*, the test application to use WrapDrive interface to access the > > WarpDrive > > >hardware queue(s) of the accelerator. > > > > The Hisilicon HIP08 ZIP accelerator is used in this sample. > > > > Signed-off-by: Zaibo Xu > > Signed-off-by: Kenneth Lee > > Signed-off-by: Hao Fang > > Signed-off-by: Zhou Wang > > --- > > samples/warpdrive/AUTHORS | 2 + > > samples/warpdrive/ChangeLog| 1 + > > samples/warpdrive/Makefile.am | 9 + > > samples/warpdrive/NEWS | 1 + > > samples/warpdrive/README | 32 +++ > > samples/warpdrive/autogen.sh | 3 + > > samples/warpdrive/cleanup.sh | 13 ++ > > samples/warpdrive/configure.ac | 52 + > > samples/warpdrive/drv/hisi_qm_udrv.c | 223 ++ > > samples/warpdrive/drv/hisi_qm_udrv.h | 53 + > > samples/warpdrive/test/Makefile.am | 7 + > > samples/warpdrive/test/comp_hw.h | 23 ++ > > samples/warpdrive/test/test_hisi_zip.c | 206 + > > samples/warpdrive/wd.c | 309 + > > samples/warpdrive/wd.h | 154 > > samples/warpdrive/wd_adapter.c | 74 ++ > > samples/warpdrive/wd_adapter.h | 43 > > 17 files changed, 1205 insertions(+) > > create mode 100644 samples/warpdrive/AUTHORS > > create mode 100644 samples/warpdrive/ChangeLog > > create mode 100644 samples/warpdrive/Makefile.am > > create mode 100644 samples/warpdrive/NEWS > > create mode 100644 samples/warpdrive/README > > create mode 100755 samples/warpdrive/autogen.sh > > create mode 100755 samples/warpdrive/cleanup.sh > > create mode 100644 samples/warpdrive/configure.ac > > create mode 100644 samples/warpdrive/drv/hisi_qm_udrv.c > > create mode 100644 samples/warpdrive/drv/hisi_qm_udrv.h > > create mode 100644 samples/warpdrive/test/Makefile.am > > create mode 100644 samples/warpdrive/test/comp_hw.h > > create mode 100644 samples/warpdrive/test/test_hisi_zip.c > > create mode 100644 samples/warpdrive/wd.c > > create mode 100644 samples/warpdrive/wd.h > > create mode 100644 samples/warpdrive/wd_adapter.c > > create mode 100644 samples/warpdrive/wd_adapter.h > > > diff --git a/samples/warpdrive/README b/samples/warpdrive/README > > new file mode 100644 > > index ..3adf66b112fc > > --- /dev/null > > +++ b/samples/warpdrive/README > > @@ -0,0 +1,32 @@ > > +WD User Land Demonstration > > +== > > + > > +This directory contains some applications and libraries to demonstrate how > > a > > + > > +WrapDrive application can be constructed. > >WarpDrive > > > + > > + > > +As a demo, we try to make it simple and clear for understanding. It is not > > + > > +supposed to be used in business scenario. > > + > > + > > +The directory contains the following elements: > > + > > +wd.[ch] > > + A demonstration WrapDrive fundamental library which wraps the basic > > WarpDrive > > > + operations to the WrapDrive-ed device. > > WarpDrive > > > + > > +wd_adapter.[ch] > > + User driver
Re: [PATCH 6/7] crypto: add sdmdev support to Hisilicon QM
On Sun, Sep 02, 2018 at 07:19:03PM -0700, Randy Dunlap wrote: > Date: Sun, 2 Sep 2018 19:19:03 -0700 > From: Randy Dunlap > To: Kenneth Lee , Jonathan Corbet , > Herbert Xu , "David S . Miller" > , Joerg Roedel , Alex Williamson > , Kenneth Lee , Hao > Fang , Zhou Wang , Zaibo Xu > , Philippe Ombredanne , Greg > Kroah-Hartman , Thomas Gleixner > , linux-...@vger.kernel.org, > linux-ker...@vger.kernel.org, linux-cry...@vger.kernel.org, > iommu@lists.linux-foundation.org, k...@vger.kernel.org, > linux-accelerat...@lists.ozlabs.org, Lu Baolu , > Sanjay Kumar > CC: linux...@huawei.com > Subject: Re: [PATCH 6/7] crypto: add sdmdev support to Hisilicon QM > User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 > Thunderbird/52.9.1 > Message-ID: <46e57b2b-756e-e256-5b3c-30749d865...@infradead.org> > > On 09/02/2018 05:52 PM, Kenneth Lee wrote: > > diff --git a/drivers/crypto/hisilicon/Kconfig > > b/drivers/crypto/hisilicon/Kconfig > > index 1d155708cd69..b85fab48fdab 100644 > > --- a/drivers/crypto/hisilicon/Kconfig > > +++ b/drivers/crypto/hisilicon/Kconfig > > @@ -17,6 +17,16 @@ config CRYPTO_DEV_HISI_SEC > > To compile this as a module, choose M here: the module > > will be called hisi_sec. > > > > +config CRYPTO_DEV_HISI_SDMDEV > > + bool "Enable SDMDEV interface" > > + depends on CRYPTO_DEV_HISILICON > > + select VFIO_SDMDEV > > + help > > + Enable this enable the SDMDEV, "shared IOMMU Domain Mediated Device" > > At a minimum: > Enable this to enable the SDMDEV, > > although that could be done better. Maybe just: > Enable the SDMDEV "shared IOMMU Domain Mediated Device" > > > > + interface for all Hisilicon accelerators if they can. The SDMDEV > > probably drop "if they can": accelerators. The SDMDEV interface > > > + enable the WarpDrive user space accelerator driver to access the > > enables Thank you, will change them all in the coming version. > > > + hardware function directly. > > + > > > -- > ~Randy -- -Kenneth(Hisilicon) 本邮件及其附件含有华为公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁 止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中 的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 4/7] crypto: add hisilicon Queue Manager driver
On Sun, Sep 02, 2018 at 07:15:07PM -0700, Randy Dunlap wrote: > Date: Sun, 2 Sep 2018 19:15:07 -0700 > From: Randy Dunlap > To: Kenneth Lee , Jonathan Corbet , > Herbert Xu , "David S . Miller" > , Joerg Roedel , Alex Williamson > , Kenneth Lee , Hao > Fang , Zhou Wang , Zaibo Xu > , Philippe Ombredanne , Greg > Kroah-Hartman , Thomas Gleixner > , linux-...@vger.kernel.org, > linux-ker...@vger.kernel.org, linux-cry...@vger.kernel.org, > iommu@lists.linux-foundation.org, k...@vger.kernel.org, > linux-accelerat...@lists.ozlabs.org, Lu Baolu , > Sanjay Kumar > CC: linux...@huawei.com > Subject: Re: [PATCH 4/7] crypto: add hisilicon Queue Manager driver > User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 > Thunderbird/52.9.1 > Message-ID: <4e46a451-d1cd-ac68-84b4-20792fdbc...@infradead.org> > > On 09/02/2018 05:52 PM, Kenneth Lee wrote: > > diff --git a/drivers/crypto/hisilicon/Kconfig > > b/drivers/crypto/hisilicon/Kconfig > > index 8ca9c503bcb0..02a6eef84101 100644 > > --- a/drivers/crypto/hisilicon/Kconfig > > +++ b/drivers/crypto/hisilicon/Kconfig > > @@ -1,4 +1,8 @@ > > # SPDX-License-Identifier: GPL-2.0 > > +config CRYPTO_DEV_HISILICON > > + tristate "Support for HISILICON CRYPTO ACCELERATOR" > > + help > > + Enable this to use Hisilicon Hardware Accelerators > > Accelerators. Thanks, will change it in next version. > > > -- > ~Randy -- -Kenneth(Hisilicon) 本邮件及其附件含有华为公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁 止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中 的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 3/7] vfio: add sdmdev support
On Mon, Sep 03, 2018 at 10:55:57AM +0800, Lu Baolu wrote:
> Date: Mon, 3 Sep 2018 10:55:57 +0800
> From: Lu Baolu
> To: Kenneth Lee , Jonathan Corbet ,
> Herbert Xu , "David S . Miller"
> , Joerg Roedel , Alex Williamson
> , Kenneth Lee , Hao
> Fang , Zhou Wang , Zaibo Xu
> , Philippe Ombredanne , Greg
> Kroah-Hartman , Thomas Gleixner
> , linux-...@vger.kernel.org,
> linux-ker...@vger.kernel.org, linux-cry...@vger.kernel.org,
> iommu@lists.linux-foundation.org, k...@vger.kernel.org,
> linux-accelerat...@lists.ozlabs.org, Sanjay Kumar
>
> CC: linux...@huawei.com, baolu...@linux.intel.com
> Subject: Re: [PATCH 3/7] vfio: add sdmdev support
> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
> Thunderbird/52.9.1
> Message-ID: <4ea51b20-dcc1-db32-18eb-24a004ab9...@linux.intel.com>
>
> Hi,
>
> On 09/03/2018 08:52 AM, Kenneth Lee wrote:
> >From: Kenneth Lee
> >
> >SDMDEV is "Share Domain Mdev". It is a vfio-mdev. But differ from
> >the general vfio-mdev, it shares its parent's IOMMU. If Multi-PASID
> >support is enabled in the IOMMU (not yet in the current kernel HEAD),
> >multiple process can share the IOMMU by different PASID. If it is not
> >support, only one process can share the IOMMU with the kernel driver.
> >
>
> If only for share domain purpose, I don't think it's necessary to create
> a new device type.
>
Yes, if ONLY for share domain purpose. But we need also to share the interrupt.
> >Currently only the vfio type-1 driver is updated to make it to be aware
> >of.
> >
> >Signed-off-by: Kenneth Lee
> >Signed-off-by: Zaibo Xu
> >Signed-off-by: Zhou Wang
> >---
> > drivers/vfio/Kconfig | 1 +
> > drivers/vfio/Makefile | 1 +
> > drivers/vfio/sdmdev/Kconfig | 10 +
> > drivers/vfio/sdmdev/Makefile | 3 +
> > drivers/vfio/sdmdev/vfio_sdmdev.c | 363 ++
> > drivers/vfio/vfio_iommu_type1.c | 151 -
> > include/linux/vfio_sdmdev.h | 96
> > include/uapi/linux/vfio_sdmdev.h | 29 +++
> > 8 files changed, 648 insertions(+), 6 deletions(-)
> > create mode 100644 drivers/vfio/sdmdev/Kconfig
> > create mode 100644 drivers/vfio/sdmdev/Makefile
> > create mode 100644 drivers/vfio/sdmdev/vfio_sdmdev.c
> > create mode 100644 include/linux/vfio_sdmdev.h
> > create mode 100644 include/uapi/linux/vfio_sdmdev.h
> >
>
> [--cut for short --]
>
> >diff --git a/drivers/vfio/vfio_iommu_type1.c
> >b/drivers/vfio/vfio_iommu_type1.c
> >index d9fd3188615d..ba73231d8692 100644
> >--- a/drivers/vfio/vfio_iommu_type1.c
> >+++ b/drivers/vfio/vfio_iommu_type1.c
> >@@ -41,6 +41,7 @@
> > #include
> > #include
> > #include
> >+#include
> > #define DRIVER_VERSION "0.2"
> > #define DRIVER_AUTHOR "Alex Williamson "
> >@@ -89,6 +90,8 @@ struct vfio_dma {
> > };
> > struct vfio_group {
> >+/* iommu_group of mdev's parent device */
> >+struct iommu_group *parent_group;
> > struct iommu_group *iommu_group;
> > struct list_headnext;
> > };
> >@@ -1327,6 +1330,109 @@ static bool vfio_iommu_has_sw_msi(struct iommu_group
> >*group, phys_addr_t *base)
> > return ret;
> > }
> >+/* return 0 if the device is not sdmdev.
> >+ * return 1 if the device is sdmdev, the data will be updated with parent
> >+ * device's group.
> >+ * return -errno if other error.
> >+ */
> >+static int vfio_sdmdev_type(struct device *dev, void *data)
> >+{
> >+struct iommu_group **group = data;
> >+struct iommu_group *pgroup;
> >+int (*_is_sdmdev)(struct device *dev);
> >+struct device *pdev;
> >+int ret = 1;
> >+
> >+/* vfio_sdmdev module is not configurated */
> >+_is_sdmdev = symbol_get(vfio_sdmdev_is_sdmdev);
> >+if (!_is_sdmdev)
> >+return 0;
> >+
> >+/* check if it belongs to vfio_sdmdev device */
> >+if (!_is_sdmdev(dev)) {
> >+ret = 0;
> >+goto out;
> >+}
> >+
> >+pdev = dev->parent;
> >+pgroup = iommu_group_get(pdev);
> >+if (!pgroup) {
> >+ret = -ENODEV;
> >+goto out;
> >+}
> >+
> >+if (group) {
> >+/* check if all parent devices is the same */
> >+if (*group && *group != pgroup)
> >+ret = -ENODEV;
Re: [PATCH 3/7] vfio: add sdmdev support
On Sun, Sep 02, 2018 at 07:11:12PM -0700, Randy Dunlap wrote: > Date: Sun, 2 Sep 2018 19:11:12 -0700 > From: Randy Dunlap > To: Kenneth Lee , Jonathan Corbet , > Herbert Xu , "David S . Miller" > , Joerg Roedel , Alex Williamson > , Kenneth Lee , Hao > Fang , Zhou Wang , Zaibo Xu > , Philippe Ombredanne , Greg > Kroah-Hartman , Thomas Gleixner > , linux-...@vger.kernel.org, > linux-ker...@vger.kernel.org, linux-cry...@vger.kernel.org, > iommu@lists.linux-foundation.org, k...@vger.kernel.org, > linux-accelerat...@lists.ozlabs.org, Lu Baolu , > Sanjay Kumar > CC: linux...@huawei.com > Subject: Re: [PATCH 3/7] vfio: add sdmdev support > User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 > Thunderbird/52.9.1 > Message-ID: <0574e7dc-bed2-d2d0-3aa6-93590d54c...@infradead.org> > > On 09/02/2018 05:52 PM, Kenneth Lee wrote: > > diff --git a/drivers/vfio/sdmdev/Kconfig b/drivers/vfio/sdmdev/Kconfig > > new file mode 100644 > > index ..51474272870d > > --- /dev/null > > +++ b/drivers/vfio/sdmdev/Kconfig > > @@ -0,0 +1,10 @@ > > +# SPDX-License-Identifier: GPL-2.0 > > +config VFIO_SDMDEV > > + tristate "Support for Share Domain MDEV" > > + depends on VFIO_MDEV_DEVICE > > + help > > + Support for VFIO Share Domain MDEV, which enables the kernel to > > + support light weight hardware accelerator framework, WarpDrive. > > lightweight > Thank you, will fix it. > > + > > + To compile this as a module, choose M here: the module will be called > > + sdmdev. > > > thanks, > -- > ~Randy -- -Kenneth(Hisilicon) ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
[PATCH 6/7] crypto: add sdmdev support to Hisilicon QM
From: Kenneth Lee
This commit add spimdev support to the Hislicon QM driver, any
accelerator that use QM can expose its queues to the user space.
Signed-off-by: Kenneth Lee
Signed-off-by: Zhou Wang
Signed-off-by: Hao Fang
Signed-off-by: Zaibo Xu
---
drivers/crypto/hisilicon/Kconfig | 10 ++
drivers/crypto/hisilicon/qm.c| 159 +++
drivers/crypto/hisilicon/qm.h| 12 +++
3 files changed, 181 insertions(+)
diff --git a/drivers/crypto/hisilicon/Kconfig b/drivers/crypto/hisilicon/Kconfig
index 1d155708cd69..b85fab48fdab 100644
--- a/drivers/crypto/hisilicon/Kconfig
+++ b/drivers/crypto/hisilicon/Kconfig
@@ -17,6 +17,16 @@ config CRYPTO_DEV_HISI_SEC
To compile this as a module, choose M here: the module
will be called hisi_sec.
+config CRYPTO_DEV_HISI_SDMDEV
+ bool "Enable SDMDEV interface"
+ depends on CRYPTO_DEV_HISILICON
+ select VFIO_SDMDEV
+ help
+ Enable this enable the SDMDEV, "shared IOMMU Domain Mediated Device"
+ interface for all Hisilicon accelerators if they can. The SDMDEV
+ enable the WarpDrive user space accelerator driver to access the
+ hardware function directly.
+
config CRYPTO_DEV_HISI_QM
tristate
depends on ARM64 && PCI
diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
index ea618b4d0929..c94e138e9b99 100644
--- a/drivers/crypto/hisilicon/qm.c
+++ b/drivers/crypto/hisilicon/qm.c
@@ -639,6 +639,155 @@ int hisi_qp_send(struct hisi_qp *qp, void *msg)
}
EXPORT_SYMBOL_GPL(hisi_qp_send);
+#ifdef CONFIG_CRYPTO_DEV_HISI_SDMDEV
+/* mdev->supported_type_groups */
+static struct attribute *hisi_qm_type_attrs[] = {
+ VFIO_SDMDEV_DEFAULT_MDEV_TYPE_ATTRS,
+ NULL,
+};
+static struct attribute_group hisi_qm_type_group = {
+ .attrs = hisi_qm_type_attrs,
+};
+static struct attribute_group *mdev_type_groups[] = {
+ _qm_type_group,
+ NULL,
+};
+
+static void qm_qp_event_notifier(struct hisi_qp *qp)
+{
+ vfio_sdmdev_wake_up(qp->sdmdev_q);
+}
+
+static int hisi_qm_get_queue(struct vfio_sdmdev *sdmdev,
+ struct vfio_sdmdev_queue **q)
+{
+ struct qm_info *qm = sdmdev->priv;
+ struct hisi_qp *qp = NULL;
+ struct vfio_sdmdev_queue *wd_q;
+ u8 alg_type = 0; /* fix me here */
+ int ret;
+
+ qp = hisi_qm_create_qp(qm, alg_type);
+ if (IS_ERR(qp))
+ return PTR_ERR(qp);
+
+ wd_q = kzalloc(sizeof(struct vfio_sdmdev_queue), GFP_KERNEL);
+ if (!wd_q) {
+ ret = -ENOMEM;
+ goto err_with_qp;
+ }
+
+ wd_q->priv = qp;
+ wd_q->sdmdev = sdmdev;
+ *q = wd_q;
+ qp->sdmdev_q = wd_q;
+ qp->event_cb = qm_qp_event_notifier;
+
+ return 0;
+
+err_with_qp:
+ hisi_qm_release_qp(qp);
+ return ret;
+}
+
+void hisi_qm_put_queue(struct vfio_sdmdev_queue *q)
+{
+ struct hisi_qp *qp = q->priv;
+
+ hisi_qm_release_qp(qp);
+ kfree(q);
+}
+
+/* map sq/cq/doorbell to user space */
+static int hisi_qm_mmap(struct vfio_sdmdev_queue *q,
+ struct vm_area_struct *vma)
+{
+ struct hisi_qp *qp = (struct hisi_qp *)q->priv;
+ struct qm_info *qm = qp->qm;
+ struct device *dev = >pdev->dev;
+ size_t sz = vma->vm_end - vma->vm_start;
+ u8 region;
+
+ vma->vm_flags |= (VM_IO | VM_LOCKED | VM_DONTEXPAND | VM_DONTDUMP);
+ region = _VFIO_SDMDEV_REGION(vma->vm_pgoff);
+
+ switch (region) {
+ case 0:
+ if (sz > PAGE_SIZE)
+ return -EINVAL;
+ /*
+* Warning: This is not safe as multiple queues use the same
+* doorbell, v1 hardware interface problem. v2 will fix it
+*/
+ return remap_pfn_range(vma, vma->vm_start,
+ qm->phys_base >> PAGE_SHIFT,
+ sz, pgprot_noncached(vma->vm_page_prot));
+ case 1:
+ vma->vm_pgoff = 0;
+ if (sz > qp->scqe.size)
+ return -EINVAL;
+
+ return dma_mmap_coherent(dev, vma, qp->scqe.addr, qp->scqe.dma,
+ sz);
+
+ default:
+ return -EINVAL;
+ }
+}
+
+static int hisi_qm_start_queue(struct vfio_sdmdev_queue *q)
+{
+ struct hisi_qp *qp = q->priv;
+
+#ifdef CONFIG_IOMMU_SVA
+ return hisi_qm_start_qp(qp, q->pasid);
+#else
+ return hisi_qm_start_qp(qp, 0);
+#endif
+}
+
+static void hisi_qm_stop_queue(struct vfio_sdmdev_queue *q)
+{
+ /* need to stop hardware, but can not support in v1 */
+}
+
+static const struct vfio_sdmdev_ops qm_ops = {
+ .get_queue = hisi_qm_get_queue,
+ .put_queue = hisi_qm_put_queue,
+ .start_que
[PATCH 7/7] vfio/sdmdev: add user sample
From: Kenneth Lee This is the sample code to demostrate how WrapDrive user application should be. It contains: 1. wd.[ch], the common library to provide WrapDrive interface. 2. drv/*, the user driver to access the hardware upon spimdev 3. test/*, the test application to use WrapDrive interface to access the hardware queue(s) of the accelerator. The Hisilicon HIP08 ZIP accelerator is used in this sample. Signed-off-by: Zaibo Xu Signed-off-by: Kenneth Lee Signed-off-by: Hao Fang Signed-off-by: Zhou Wang --- samples/warpdrive/AUTHORS | 2 + samples/warpdrive/ChangeLog| 1 + samples/warpdrive/Makefile.am | 9 + samples/warpdrive/NEWS | 1 + samples/warpdrive/README | 32 +++ samples/warpdrive/autogen.sh | 3 + samples/warpdrive/cleanup.sh | 13 ++ samples/warpdrive/configure.ac | 52 + samples/warpdrive/drv/hisi_qm_udrv.c | 223 ++ samples/warpdrive/drv/hisi_qm_udrv.h | 53 + samples/warpdrive/test/Makefile.am | 7 + samples/warpdrive/test/comp_hw.h | 23 ++ samples/warpdrive/test/test_hisi_zip.c | 206 + samples/warpdrive/wd.c | 309 + samples/warpdrive/wd.h | 154 samples/warpdrive/wd_adapter.c | 74 ++ samples/warpdrive/wd_adapter.h | 43 17 files changed, 1205 insertions(+) create mode 100644 samples/warpdrive/AUTHORS create mode 100644 samples/warpdrive/ChangeLog create mode 100644 samples/warpdrive/Makefile.am create mode 100644 samples/warpdrive/NEWS create mode 100644 samples/warpdrive/README create mode 100755 samples/warpdrive/autogen.sh create mode 100755 samples/warpdrive/cleanup.sh create mode 100644 samples/warpdrive/configure.ac create mode 100644 samples/warpdrive/drv/hisi_qm_udrv.c create mode 100644 samples/warpdrive/drv/hisi_qm_udrv.h create mode 100644 samples/warpdrive/test/Makefile.am create mode 100644 samples/warpdrive/test/comp_hw.h create mode 100644 samples/warpdrive/test/test_hisi_zip.c create mode 100644 samples/warpdrive/wd.c create mode 100644 samples/warpdrive/wd.h create mode 100644 samples/warpdrive/wd_adapter.c create mode 100644 samples/warpdrive/wd_adapter.h diff --git a/samples/warpdrive/AUTHORS b/samples/warpdrive/AUTHORS new file mode 100644 index ..fe7dc2413b0d --- /dev/null +++ b/samples/warpdrive/AUTHORS @@ -0,0 +1,2 @@ +Kenneth Lee +Zaibo Xu diff --git a/samples/warpdrive/ChangeLog b/samples/warpdrive/ChangeLog new file mode 100644 index ..b1b716105590 --- /dev/null +++ b/samples/warpdrive/ChangeLog @@ -0,0 +1 @@ +init diff --git a/samples/warpdrive/Makefile.am b/samples/warpdrive/Makefile.am new file mode 100644 index ..41154a880a97 --- /dev/null +++ b/samples/warpdrive/Makefile.am @@ -0,0 +1,9 @@ +ACLOCAL_AMFLAGS = -I m4 +AUTOMAKE_OPTIONS = foreign subdir-objects +AM_CFLAGS=-Wall -O0 -fno-strict-aliasing + +lib_LTLIBRARIES=libwd.la +libwd_la_SOURCES=wd.c wd_adapter.c wd.h wd_adapter.h \ +drv/hisi_qm_udrv.c drv/hisi_qm_udrv.h + +SUBDIRS=. test diff --git a/samples/warpdrive/NEWS b/samples/warpdrive/NEWS new file mode 100644 index ..b1b716105590 --- /dev/null +++ b/samples/warpdrive/NEWS @@ -0,0 +1 @@ +init diff --git a/samples/warpdrive/README b/samples/warpdrive/README new file mode 100644 index ..3adf66b112fc --- /dev/null +++ b/samples/warpdrive/README @@ -0,0 +1,32 @@ +WD User Land Demonstration +== + +This directory contains some applications and libraries to demonstrate how a + +WrapDrive application can be constructed. + + +As a demo, we try to make it simple and clear for understanding. It is not + +supposed to be used in business scenario. + + +The directory contains the following elements: + +wd.[ch] + A demonstration WrapDrive fundamental library which wraps the basic + operations to the WrapDrive-ed device. + +wd_adapter.[ch] + User driver adaptor for wd.[ch] + +wd_utils.[ch] + Some utitlities function used by WD and its drivers + +drv/* + User drivers. It helps to fulfill the semantic of wd.[ch] for + particular hardware + +test/* + Test applications to use the wrapdrive library + diff --git a/samples/warpdrive/autogen.sh b/samples/warpdrive/autogen.sh new file mode 100755 index ..58deaf49de2a --- /dev/null +++ b/samples/warpdrive/autogen.sh @@ -0,0 +1,3 @@ +#!/bin/sh -x + +autoreconf -i -f -v diff --git a/samples/warpdrive/cleanup.sh b/samples/warpdrive/cleanup.sh new file mode 100755 index ..c5f3d21e5dc1 --- /dev/null +++ b/samples/warpdrive/cleanup.sh @@ -0,0 +1,13 @@ +#!/bin/sh + +if [ -r Makefile ]; then + make distclean +fi + +FILES="aclocal.m4 autom4te.cache compile config.guess config.h.in config.log \ + config.status config.sub configure cscope.out depcomp insta
[PATCH 1/7] vfio/sdmdev: Add documents for WarpDrive framework
From: Kenneth Lee WarpDrive is a common user space accelerator framework. Its main component in Kernel is called sdmdev, Share Domain Mediated Device. It exposes the hardware capabilities to the user space via vfio-mdev. So processes in user land can obtain a "queue" by open the device and direct access the hardware MMIO space or do DMA operation via VFIO interface. WarpDrive is intended to be used with Jean Philippe Brucker's SVA patchset to support multi-process. But This is not a must. Without the SVA patches, WarpDrive can still work for one process for every hardware device. This patch add detail documents for the framework. Signed-off-by: Kenneth Lee --- Documentation/00-INDEX| 2 + Documentation/warpdrive/warpdrive.rst | 100 Documentation/warpdrive/wd-arch.svg | 728 ++ 3 files changed, 830 insertions(+) create mode 100644 Documentation/warpdrive/warpdrive.rst create mode 100644 Documentation/warpdrive/wd-arch.svg diff --git a/Documentation/00-INDEX b/Documentation/00-INDEX index 2754fe83f0d4..9959affab599 100644 --- a/Documentation/00-INDEX +++ b/Documentation/00-INDEX @@ -410,6 +410,8 @@ vm/ - directory with info on the Linux vm code. w1/ - directory with documents regarding the 1-wire (w1) subsystem. +warpdrive/ + - directory with documents about WarpDrive accelerator framework. watchdog/ - how to auto-reboot Linux if it has "fallen and can't get up". ;-) wimax/ diff --git a/Documentation/warpdrive/warpdrive.rst b/Documentation/warpdrive/warpdrive.rst new file mode 100644 index ..6d2a5d1e08c4 --- /dev/null +++ b/Documentation/warpdrive/warpdrive.rst @@ -0,0 +1,100 @@ +Introduction of WarpDrive += + +*WarpDrive* is a general accelerator framework for user space. It intends to +provide interface for the user process to send request to hardware +accelerator without heavy user-kernel interaction cost. + +The *WarpDrive* user library is supposed to provide a pipe-based API, such as: +:: +int wd_request_queue(struct wd_queue *q); +void wd_release_queue(struct wd_queue *q); + +int wd_send(struct wd_queue *q, void *req); +int wd_recv(struct wd_queue *q, void **req); +int wd_recv_sync(struct wd_queue *q, void **req); +int wd_flush(struct wd_queue *q); + +*wd_request_queue* creates the pipe connection, *queue*, between the +application and the hardware. The application sends request and pulls the +answer back by asynchronized wd_send/wd_recv, which directly interact with the +hardware (by MMIO or share memory) without syscall. + +*WarpDrive* maintains a unified application address space among all involved +accelerators. With the following APIs: :: + +int wd_mem_share(struct wd_queue *q, const void *addr, + size_t size, int flags); +void wd_mem_unshare(struct wd_queue *q, const void *addr, size_t size); + +The referred process space shared by these APIs can be directly referred by the +hardware. The process can also dedicate its whole process space with flags, +*WD_SHARE_ALL* (not in this patch yet). + +The name *WarpDrive* is simply a cool and general name meaning the framework +makes the application faster. As it will be explained in this text later, the +facility in kernel is called *SDMDEV*, namely "Share Domain Mediated Device". + + +How does it work + + +*WarpDrive* is built upon *VFIO-MDEV*. The queue is wrapped as *mdev* in VFIO. +So memory sharing can be done via standard VFIO standard DMA interface. + +The architecture is illustrated as follow figure: + +.. image:: wd-arch.svg +:alt: WarpDrive Architecture + +Accelerator driver shares its capability via *SDMDEV* API: :: + +vfio_sdmdev_register(struct vfio_sdmdev *sdmdev); +vfio_sdmdev_unregister(struct vfio_sdmdev *sdmdev); +vfio_sdmdev_wake_up(struct spimdev_queue *q); + +*vfio_sdmdev_register* is a helper function to register the hardware to the +*VFIO_MDEV* framework. The queue creation is done by *mdev* creation interface. + +*WarpDrive* User library mmap the mdev to access its mmio space and shared +memory. Request can be sent to, or receive from, hardware in this mmap-ed +space until the queue is full or empty. + +The user library can wait on the queue by ioctl(VFIO_SDMDEV_CMD_WAIT) the mdev +if the queue is full or empty. If the queue status is changed, the hardware +driver use *vfio_sdmdev_wake_up* to wake up the waiting process. + + +Multiple processes support +== + +In the latest mainline kernel (4.18) when this document is written, +multi-process is not supported in VFIO yet. + +Jean Philippe Brucker has a patchset to enable it[1]_. We have tested it +with our hardware (which is known as *D06*). It works well. *WarpDrive* rely +on them to support multiple processes. If it is not enabled, *WarpDrive* can +still
[RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive
From: Kenneth Lee WarpDrive is an accelerator framework to expose the hardware capabilities directly to the user space. It makes use of the exist vfio and vfio-mdev facilities. So the user application can send request and DMA to the hardware without interaction with the kernel. This removes the latency of syscall. WarpDrive is the name for the whole framework. The component in kernel is called SDMDEV, Share Domain Mediated Device. Driver driver exposes its hardware resource by registering to SDMDEV as a VFIO-Mdev. So the user library of WarpDrive can access it via VFIO interface. The patchset contains document for the detail. Please refer to it for more information. This patchset is intended to be used with Jean Philippe Brucker's SVA patch [1], which enables not only IO side page fault, but also PASID support to IOMMU and VFIO. With these features, WarpDrive can support non-pinned memory and multi-process in the same accelerator device. We tested it in our SoC integrated Accelerator (board ID: D06, Chip ID: HIP08). A reference work tree can be found here: [2]. But it is not mandatory. This patchset is tested in the latest mainline kernel without the SVA patches. So it supports only one process for each accelerator. We have noticed the IOMMU aware mdev RFC announced recently [3]. The IOMMU aware mdev has similar idea but different intention comparing to WarpDrive. It intends to dedicate part of the hardware resource to a VM. And the design is supposed to be used with Scalable I/O Virtualization. While sdmdev is intended to share the hardware resource with a big amount of processes. It just requires the hardware supporting address translation per process (PCIE's PASID or ARM SMMU's substream ID). But we don't see serious confliction on both design. We believe they can be normalized as one. The patch 1 is document of the framework. The patch 2 and 3 add sdmdev support. The patch 4, 5 and 6 is drivers for Hislicon's ZIP Accelerator which is registered to both crypto and warpdrive(sdmdev) and can be used from kernel or user space at the same time. The patch 7 is a user space sample demonstrating how WarpDrive works. Change History: V2 changed from V1: 1. Change kernel framework name from SPIMDEV (Share Parent IOMMU Mdev) to SDMDEV (Share Domain Mdev). 2. Allocate Hardware Resource when a new mdev is created (While it is allocated when the mdev is openned) 3. Unmap pages from the shared domain when the sdmdev iommu group is detached. (This procedure is necessary, but missed in V1) 4. Update document accordingly. 5. Rebase to the latest kernel (4.19.0-rc1) According the review comment on RFCv1, We did try to use dma-buf as back end of WarpDrive. It can work properly with the current solution [4], but it cannot make use of process's own memory address space directly. This is important to many acceleration scenario. So dma-buf will be taken as a backup alternative for noiommu scenario, it will be added in the future version. Refernces: [1] https://www.spinics.net/lists/kernel/msg2651481.html [2] https://github.com/Kenneth-Lee/linux-kernel-warpdrive/tree/warpdrive-sva-v0.5 [3] https://lkml.org/lkml/2018/7/22/34 [4] https://github.com/Kenneth-Lee/linux-kernel-warpdrive/tree/warpdrive-v0.7-dmabuf Best Regards Kenneth Lee Kenneth Lee (7): vfio/sdmdev: Add documents for WarpDrive framework iommu: Add share domain interface in iommu for sdmdev vfio: add sdmdev support crypto: add hisilicon Queue Manager driver crypto: Add Hisilicon Zip driver crypto: add sdmdev support to Hisilicon QM vfio/sdmdev: add user sample Documentation/00-INDEX| 2 + Documentation/warpdrive/warpdrive.rst | 100 +++ Documentation/warpdrive/wd-arch.svg | 728 drivers/crypto/Makefile | 2 +- drivers/crypto/hisilicon/Kconfig | 25 + drivers/crypto/hisilicon/Makefile | 2 + drivers/crypto/hisilicon/qm.c | 979 ++ drivers/crypto/hisilicon/qm.h | 122 +++ drivers/crypto/hisilicon/zip/Makefile | 2 + drivers/crypto/hisilicon/zip/zip.h| 57 ++ drivers/crypto/hisilicon/zip/zip_crypto.c | 353 drivers/crypto/hisilicon/zip/zip_crypto.h | 8 + drivers/crypto/hisilicon/zip/zip_main.c | 195 + drivers/iommu/iommu.c | 29 +- drivers/vfio/Kconfig | 1 + drivers/vfio/Makefile | 1 + drivers/vfio/sdmdev/Kconfig | 10 + drivers/vfio/sdmdev/Makefile | 3 + drivers/vfio/sdmdev/vfio_sdmdev.c | 363 drivers/vfio/vfio_iommu_type1.c | 151 +++- include/linux/iommu.h | 15 + include/linux/vfio_sdmdev.h | 96 +++ include/uapi/linux/vfio_sdmdev.h | 29 + samples/warpdrive
[PATCH 4/7] crypto: add hisilicon Queue Manager driver
From: Kenneth Lee Hisilicon QM is a general IP used by some Hisilicon accelerators. It provides a general PCIE interface for the CPU and the accelerator to share a group of queues. This commit includes a library used by the accelerator driver to access the QM hardware. Signed-off-by: Kenneth Lee Signed-off-by: Zhou Wang Signed-off-by: Hao Fang --- drivers/crypto/Makefile | 2 +- drivers/crypto/hisilicon/Kconfig | 8 + drivers/crypto/hisilicon/Makefile | 1 + drivers/crypto/hisilicon/qm.c | 820 ++ drivers/crypto/hisilicon/qm.h | 110 5 files changed, 940 insertions(+), 1 deletion(-) create mode 100644 drivers/crypto/hisilicon/qm.c create mode 100644 drivers/crypto/hisilicon/qm.h diff --git a/drivers/crypto/Makefile b/drivers/crypto/Makefile index c23396f32c8a..f3a7abe42424 100644 --- a/drivers/crypto/Makefile +++ b/drivers/crypto/Makefile @@ -46,4 +46,4 @@ obj-$(CONFIG_CRYPTO_DEV_VMX) += vmx/ obj-$(CONFIG_CRYPTO_DEV_BCM_SPU) += bcm/ obj-$(CONFIG_CRYPTO_DEV_SAFEXCEL) += inside-secure/ obj-$(CONFIG_CRYPTO_DEV_ARTPEC6) += axis/ -obj-y += hisilicon/ +obj-$(CONFIG_CRYPTO_DEV_HISILICON) += hisilicon/ diff --git a/drivers/crypto/hisilicon/Kconfig b/drivers/crypto/hisilicon/Kconfig index 8ca9c503bcb0..02a6eef84101 100644 --- a/drivers/crypto/hisilicon/Kconfig +++ b/drivers/crypto/hisilicon/Kconfig @@ -1,4 +1,8 @@ # SPDX-License-Identifier: GPL-2.0 +config CRYPTO_DEV_HISILICON + tristate "Support for HISILICON CRYPTO ACCELERATOR" + help + Enable this to use Hisilicon Hardware Accelerators config CRYPTO_DEV_HISI_SEC tristate "Support for Hisilicon SEC crypto block cipher accelerator" @@ -12,3 +16,7 @@ config CRYPTO_DEV_HISI_SEC To compile this as a module, choose M here: the module will be called hisi_sec. + +config CRYPTO_DEV_HISI_QM + tristate + depends on ARM64 && PCI diff --git a/drivers/crypto/hisilicon/Makefile b/drivers/crypto/hisilicon/Makefile index 463f46ace182..05e9052e0f52 100644 --- a/drivers/crypto/hisilicon/Makefile +++ b/drivers/crypto/hisilicon/Makefile @@ -1,2 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_CRYPTO_DEV_HISI_SEC) += sec/ +obj-$(CONFIG_CRYPTO_DEV_HISI_QM) += qm.o diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c new file mode 100644 index ..ea618b4d0929 --- /dev/null +++ b/drivers/crypto/hisilicon/qm.c @@ -0,0 +1,820 @@ +// SPDX-License-Identifier: GPL-2.0+ +#include +#include +#include +#include +#include +#include +#include "qm.h" + +#define QM_DEF_Q_NUM 128 + +/* eq/aeq irq enable */ +#define QM_VF_AEQ_INT_SOURCE 0x0 +#define QM_VF_AEQ_INT_MASK 0x4 +#define QM_VF_EQ_INT_SOURCE0x8 +#define QM_VF_EQ_INT_MASK 0xc + +/* mailbox */ +#define MAILBOX_CMD_SQC0x0 +#define MAILBOX_CMD_CQC0x1 +#define MAILBOX_CMD_EQC0x2 +#define MAILBOX_CMD_SQC_BT 0x4 +#define MAILBOX_CMD_CQC_BT 0x5 + +#define MAILBOX_CMD_SEND_BASE 0x300 +#define MAILBOX_EVENT_SHIFT8 +#define MAILBOX_STATUS_SHIFT 9 +#define MAILBOX_BUSY_SHIFT 13 +#define MAILBOX_OP_SHIFT 14 +#define MAILBOX_QUEUE_SHIFT16 + +/* sqc shift */ +#define SQ_HEAD_SHIFT 0 +#define SQ_TAIL_SHIFI 16 +#define SQ_HOP_NUM_SHIFT 0 +#define SQ_PAGE_SIZE_SHIFT 4 +#define SQ_BUF_SIZE_SHIFT 8 +#define SQ_SQE_SIZE_SHIFT 12 +#define SQ_HEAD_IDX_SIG_SHIFT 0 +#define SQ_TAIL_IDX_SIG_SHIFT 0 +#define SQ_CQN_SHIFT 0 +#define SQ_PRIORITY_SHIFT 0 +#define SQ_ORDERS_SHIFT4 +#define SQ_TYPE_SHIFT 8 + +#define SQ_TYPE_MASK 0xf + +/* cqc shift */ +#define CQ_HEAD_SHIFT 0 +#define CQ_TAIL_SHIFI 16 +#define CQ_HOP_NUM_SHIFT 0 +#define CQ_PAGE_SIZE_SHIFT 4 +#define CQ_BUF_SIZE_SHIFT 8 +#define CQ_SQE_SIZE_SHIFT 12 +#define CQ_PASID 0 +#define CQ_HEAD_IDX_SIG_SHIFT 0 +#define CQ_TAIL_IDX_SIG_SHIFT 0 +#define CQ_CQN_SHIFT 0 +#define CQ_PRIORITY_SHIFT 16 +#define CQ_ORDERS_SHIFT0 +#define CQ_TYPE_SHIFT 0 +#define CQ_PHASE_SHIFT 0 +#define CQ_FLAG_SHIFT 1 + +#define CQC_HEAD_INDEX(cqc)((cqc)->cq_head) +#define CQC_PHASE(cqc) (((cqc)->dw6) & 0x1) +#define CQC_CQ_ADDRESS(cqc)(((u64)((cqc)->cq_base_h) << 32) | \ +((cqc)->cq_base_l)) +#define CQC_PHASE_BIT 0x1 + +/* eqc shift */ +#define M
[PATCH 3/7] vfio: add sdmdev support
From: Kenneth Lee
SDMDEV is "Share Domain Mdev". It is a vfio-mdev. But differ from
the general vfio-mdev, it shares its parent's IOMMU. If Multi-PASID
support is enabled in the IOMMU (not yet in the current kernel HEAD),
multiple process can share the IOMMU by different PASID. If it is not
support, only one process can share the IOMMU with the kernel driver.
Currently only the vfio type-1 driver is updated to make it to be aware
of.
Signed-off-by: Kenneth Lee
Signed-off-by: Zaibo Xu
Signed-off-by: Zhou Wang
---
drivers/vfio/Kconfig | 1 +
drivers/vfio/Makefile | 1 +
drivers/vfio/sdmdev/Kconfig | 10 +
drivers/vfio/sdmdev/Makefile | 3 +
drivers/vfio/sdmdev/vfio_sdmdev.c | 363 ++
drivers/vfio/vfio_iommu_type1.c | 151 -
include/linux/vfio_sdmdev.h | 96
include/uapi/linux/vfio_sdmdev.h | 29 +++
8 files changed, 648 insertions(+), 6 deletions(-)
create mode 100644 drivers/vfio/sdmdev/Kconfig
create mode 100644 drivers/vfio/sdmdev/Makefile
create mode 100644 drivers/vfio/sdmdev/vfio_sdmdev.c
create mode 100644 include/linux/vfio_sdmdev.h
create mode 100644 include/uapi/linux/vfio_sdmdev.h
diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
index c84333eb5eb5..5af7d1db505e 100644
--- a/drivers/vfio/Kconfig
+++ b/drivers/vfio/Kconfig
@@ -47,4 +47,5 @@ menuconfig VFIO_NOIOMMU
source "drivers/vfio/pci/Kconfig"
source "drivers/vfio/platform/Kconfig"
source "drivers/vfio/mdev/Kconfig"
+source "drivers/vfio/sdmdev/Kconfig"
source "virt/lib/Kconfig"
diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile
index de67c4725cce..678592360a7a 100644
--- a/drivers/vfio/Makefile
+++ b/drivers/vfio/Makefile
@@ -9,3 +9,4 @@ obj-$(CONFIG_VFIO_SPAPR_EEH) += vfio_spapr_eeh.o
obj-$(CONFIG_VFIO_PCI) += pci/
obj-$(CONFIG_VFIO_PLATFORM) += platform/
obj-$(CONFIG_VFIO_MDEV) += mdev/
+obj-$(CONFIG_VFIO_SDMDEV) += sdmdev/
diff --git a/drivers/vfio/sdmdev/Kconfig b/drivers/vfio/sdmdev/Kconfig
new file mode 100644
index ..51474272870d
--- /dev/null
+++ b/drivers/vfio/sdmdev/Kconfig
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0
+config VFIO_SDMDEV
+ tristate "Support for Share Domain MDEV"
+ depends on VFIO_MDEV_DEVICE
+ help
+ Support for VFIO Share Domain MDEV, which enables the kernel to
+ support light weight hardware accelerator framework, WarpDrive.
+
+ To compile this as a module, choose M here: the module will be called
+ sdmdev.
diff --git a/drivers/vfio/sdmdev/Makefile b/drivers/vfio/sdmdev/Makefile
new file mode 100644
index ..ccaaa03f3184
--- /dev/null
+++ b/drivers/vfio/sdmdev/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+sdmdev-y := sdmdev.o
+obj-$(CONFIG_VFIO_SDMDEV) += vfio_sdmdev.o
diff --git a/drivers/vfio/sdmdev/vfio_sdmdev.c
b/drivers/vfio/sdmdev/vfio_sdmdev.c
new file mode 100644
index ..c6eb5d4bdab0
--- /dev/null
+++ b/drivers/vfio/sdmdev/vfio_sdmdev.c
@@ -0,0 +1,363 @@
+// SPDX-License-Identifier: GPL-2.0+
+#include
+#include
+
+static struct class *sdmdev_class;
+
+static int vfio_sdmdev_dev_exist(struct device *dev, void *data)
+{
+ return !strcmp(dev_name(dev), dev_name((struct device *)data));
+}
+
+#ifdef CONFIG_IOMMU_SVA
+static bool vfio_sdmdev_is_valid_pasid(int pasid)
+{
+ struct mm_struct *mm;
+
+ mm = iommu_sva_find(pasid);
+ if (mm) {
+ mmput(mm);
+ return mm == current->mm;
+ }
+
+ return false;
+}
+#endif
+
+/* Check if the device is a mediated device belongs to vfio_sdmdev */
+int vfio_sdmdev_is_sdmdev(struct device *dev)
+{
+ struct mdev_device *mdev;
+ struct device *pdev;
+
+ mdev = mdev_from_dev(dev);
+ if (!mdev)
+ return 0;
+
+ pdev = mdev_parent_dev(mdev);
+ if (!pdev)
+ return 0;
+
+ return class_for_each_device(sdmdev_class, NULL, pdev,
+ vfio_sdmdev_dev_exist);
+}
+EXPORT_SYMBOL_GPL(vfio_sdmdev_is_sdmdev);
+
+struct vfio_sdmdev *vfio_sdmdev_pdev_sdmdev(struct device *dev)
+{
+ struct device *class_dev;
+
+ if (!dev)
+ return ERR_PTR(-EINVAL);
+
+ class_dev = class_find_device(sdmdev_class, NULL, dev,
+ (int(*)(struct device *, const void *))vfio_sdmdev_dev_exist);
+ if (!class_dev)
+ return ERR_PTR(-ENODEV);
+
+ return container_of(class_dev, struct vfio_sdmdev, cls_dev);
+}
+EXPORT_SYMBOL_GPL(vfio_sdmdev_pdev_sdmdev);
+
+struct vfio_sdmdev *mdev_sdmdev(struct mdev_device *mdev)
+{
+ struct device *pdev = mdev_parent_dev(mdev);
+
+ return vfio_sdmdev_pdev_sdmdev(pdev);
+}
+EXPORT_SYMBOL_GPL(mdev_sdmdev);
+
+static ssize_t iommu_type_show(struct device *dev,
+ struct device_attribute *attr, char
[PATCH 2/7] iommu: Add share domain interface in iommu for sdmdev
From: Kenneth Lee
This patch add sharing interface for a iommu_group. The new interface:
iommu_group_share_domain()
iommu_group_unshare_domain()
can be used by some virtual iommu_group (such as iommu_group of sdmdev)
to share their parent's iommu_group.
When the domain of a group is shared, it cannot be changed before
being unshared. By this way, all domain users can assume the shared
IOMMU have the same configuration. In the future, notification can be
added if update is required.
Signed-off-by: Kenneth Lee
---
drivers/iommu/iommu.c | 29 -
include/linux/iommu.h | 15 +++
2 files changed, 43 insertions(+), 1 deletion(-)
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index 8c15c5980299..8e567e1037dd 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -58,6 +58,9 @@ struct iommu_group {
int id;
struct iommu_domain *default_domain;
struct iommu_domain *domain;
+ atomic_t domain_shared_ref; /* Number of user of current domain.
+* The domain cannot be modified if ref > 0
+*/
};
struct group_device {
@@ -385,6 +388,7 @@ struct iommu_group *iommu_group_alloc(void)
return ERR_PTR(ret);
}
group->id = ret;
+ atomic_set(>domain_shared_ref, 0);
ret = kobject_init_and_add(>kobj, _group_ktype,
NULL, "%d", group->id);
@@ -518,6 +522,26 @@ int iommu_group_set_name(struct iommu_group *group, const
char *name)
}
EXPORT_SYMBOL_GPL(iommu_group_set_name);
+struct iommu_domain *iommu_group_share_domain(struct iommu_group *group)
+{
+ /* the domain can be shared only when the default domain is used */
+ /* todo: more shareable check */
+ if (group->domain != group->default_domain)
+ return ERR_PTR(-EINVAL);
+
+ atomic_inc(>domain_shared_ref);
+ return group->domain;
+}
+EXPORT_SYMBOL_GPL(iommu_group_share_domain);
+
+struct iommu_domain *iommu_group_unshare_domain(struct iommu_group *group)
+{
+ atomic_dec(>domain_shared_ref);
+ WARN_ON(atomic_read(>domain_shared_ref) < 0);
+ return group->domain;
+}
+EXPORT_SYMBOL_GPL(iommu_group_unshare_domain);
+
static int iommu_group_create_direct_mappings(struct iommu_group *group,
struct device *dev)
{
@@ -1437,7 +1461,8 @@ static int __iommu_attach_group(struct iommu_domain
*domain,
{
int ret;
- if (group->default_domain && group->domain != group->default_domain)
+ if ((group->default_domain && group->domain != group->default_domain) ||
+atomic_read(>domain_shared_ref) > 0)
return -EBUSY;
ret = __iommu_group_for_each_dev(group, domain,
@@ -1474,6 +1499,8 @@ static void __iommu_detach_group(struct iommu_domain
*domain,
{
int ret;
+ WARN_ON(atomic_read(>domain_shared_ref) > 0);
+
if (!group->default_domain) {
__iommu_group_for_each_dev(group, domain,
iommu_group_do_detach_device);
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 87994c265bf5..013ac400b643 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -344,6 +344,9 @@ extern int iommu_domain_get_attr(struct iommu_domain
*domain, enum iommu_attr,
void *data);
extern int iommu_domain_set_attr(struct iommu_domain *domain, enum iommu_attr,
void *data);
+extern struct iommu_domain *iommu_group_share_domain(struct iommu_group
*group);
+extern struct iommu_domain *iommu_group_unshare_domain(
+ struct iommu_group *group);
/* Window handling function prototypes */
extern int iommu_domain_window_enable(struct iommu_domain *domain, u32 wnd_nr,
@@ -616,6 +619,18 @@ static inline int iommu_domain_set_attr(struct
iommu_domain *domain,
return -EINVAL;
}
+static inline struct iommu_domain *iommu_group_share_domain(
+ struct iommu_group *group)
+{
+ return NULL;
+}
+
+static inline struct iommu_domain *iommu_group_unshare_domain(
+ struct iommu_group *group)
+{
+ return NULL;
+}
+
static inline int iommu_device_register(struct iommu_device *iommu)
{
return -ENODEV;
--
2.17.1
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive
On Mon, Aug 13, 2018 at 03:23:01PM -0400, Jerome Glisse wrote: > Received: from popscn.huawei.com [10.3.17.45] by Turing-Arch-b with POP3 > (fetchmail-6.3.26) for (single-drop); Tue, 14 Aug 2018 > 03:30:02 +0800 (CST) > Received: from DGGEMM401-HUB.china.huawei.com (10.3.20.209) by > DGGEML402-HUB.china.huawei.com (10.3.17.38) with Microsoft SMTP Server > (TLS) id 14.3.399.0; Tue, 14 Aug 2018 03:23:25 +0800 > Received: from dggwg01-in.huawei.com (172.30.65.34) by > DGGEMM401-HUB.china.huawei.com (10.3.20.209) with Microsoft SMTP Server id > 14.3.399.0; Tue, 14 Aug 2018 03:23:21 +0800 > Received: from mx1.redhat.com (unknown [66.187.233.73]) by Forcepoint > Email with ESMTPS id 301B5D4F60895 for ; Tue, 14 > Aug 2018 03:23:16 +0800 (CST) > Received: from smtp.corp.redhat.com > (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using > TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client > certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id > 4D0A14023461;Mon, 13 Aug 2018 19:23:05 + (UTC) > Received: from redhat.com (unknown [10.20.6.215]) by > smtp.corp.redhat.com (Postfix) with ESMTPS id 20D072156712; Mon, 13 Aug > 2018 19:23:03 + (UTC) > Date: Mon, 13 Aug 2018 15:23:01 -0400 > From: Jerome Glisse > To: Kenneth Lee > CC: Kenneth Lee , Jean-Philippe Brucker > , Herbert Xu , > "k...@vger.kernel.org" , Jonathan Corbet > , Greg Kroah-Hartman , Zaibo > Xu , "linux-...@vger.kernel.org" > , "Kumar, Sanjay K" , > "Tian, Kevin" , "iommu@lists.linux-foundation.org" > , "linux-ker...@vger.kernel.org" > , "linux...@huawei.com" > , Alex Williamson , > "linux-cry...@vger.kernel.org" , Philippe > Ombredanne , Thomas Gleixner , > Hao Fang , "David S . Miller" , > "linux-accelerat...@lists.ozlabs.org" > > Subject: Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive > Message-ID: <20180813192301.gc3...@redhat.com> > References: <20180806031252.GG91035@Turing-Arch-b> > <20180806153257.gb6...@redhat.com> > <11bace0e-dc14-5d2c-f65c-25b852f4e...@gmail.com> > <20180808151835.ga3...@redhat.com> <20180809080352.GI91035@Turing-Arch-b> > <20180809144613.gb3...@redhat.com> <20180810033913.GK91035@Turing-Arch-b> > <0f6bac9b-8381-1874-9367-46b5f4cef...@arm.com> > <6ea4dcfd-d539-93e4-acf1-d09ea35f0...@gmail.com> > <20180813092931.GL91035@Turing-Arch-b> > Content-Type: text/plain; charset="iso-8859-1" > Content-Disposition: inline > Content-Transfer-Encoding: 8bit > In-Reply-To: <20180813092931.GL91035@Turing-Arch-b> > User-Agent: Mutt/1.10.0 (2018-05-17) > X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 > X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 > (mx1.redhat.com [10.11.55.6]); Mon, 13 Aug 2018 19:23:05 + (UTC) > X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com > [10.11.55.6]); Mon, 13 Aug 2018 19:23:05 + (UTC) for IP:'10.11.54.6' > DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' > HELO:'smtp.corp.redhat.com' FROM:'jgli...@redhat.com' RCPT:'' > Return-Path: jgli...@redhat.com > X-MS-Exchange-Organization-AuthSource: DGGEMM401-HUB.china.huawei.com > X-MS-Exchange-Organization-AuthAs: Anonymous > MIME-Version: 1.0 > > On Mon, Aug 13, 2018 at 05:29:31PM +0800, Kenneth Lee wrote: > > > > I made a quick change basing on the RFCv1 here: > > > > https://github.com/Kenneth-Lee/linux-kernel-warpdrive/commits/warpdrive-v0.6 > > > > I just made it compilable and not test it yet. But it shows how the idea is > > going to be. > > > > The Pros is: most of the virtual device stuff can be removed. Resource > > management is on the openned files only. > > > > The Cons is: as Jean said, we have to redo something that has been done by > > VFIO. > > These mainly are: > > > > 1. Track the dma operation and remove them on resource releasing > > 2. Pin the memory with gup and do accounting > > > > It not going to be easy to make a decision... > > > > Maybe it would be good to list things you want do. Looking at your tree > it seems you are re-inventing what dma-buf is already doing. My English is quite limited;). I think I did not explain it well in the WrapDrive document. Please let me try again here: The requirement of WrapDrive is simple. Many acceleration requirements are from user space, such as OpenSSL, AI, and so on. We want to provide a framework for the user land application to summon the accelerator easily. So the scenario is simple: The application i
Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive
On Sat, Aug 11, 2018 at 11:26:48PM +0800, Kenneth Lee wrote: > Date: Sat, 11 Aug 2018 23:26:48 +0800 > From: Kenneth Lee > To: Jean-Philippe Brucker , Kenneth Lee > , Jerome Glisse > CC: Herbert Xu , "k...@vger.kernel.org" > , Jonathan Corbet , Greg > Kroah-Hartman , Zaibo Xu , > "linux-...@vger.kernel.org" , "Kumar, Sanjay K" > , "Tian, Kevin" , > "iommu@lists.linux-foundation.org" , > "linux-ker...@vger.kernel.org" , > "linux...@huawei.com" , Alex Williamson > , "linux-cry...@vger.kernel.org" > , Philippe Ombredanne > , Thomas Gleixner , Hao Fang > , "David S . Miller" , > "linux-accelerat...@lists.ozlabs.org" > > Subject: Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive > User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 > Thunderbird/52.9.1 > Message-ID: <6ea4dcfd-d539-93e4-acf1-d09ea35f0...@gmail.com> > > > > 在 2018年08月10日 星期五 09:12 下午, Jean-Philippe Brucker 写道: > >Hi Kenneth, > > > >On 10/08/18 04:39, Kenneth Lee wrote: > >>>You can achieve everything you want to achieve with existing upstream > >>>solution. Re-inventing a whole new driver infrastructure should really > >>>be motivated with strong and obvious reasons. > >>I want to understand better of your idea. If I create some unified helper > >>APIs in drivers/iommu/, say: > >> > >>wd_create_dev(parent_dev, wd_dev) > >>wd_release_dev(wd_dev) > >> > >>The API create chrdev to take request from user space for open(resource > >>allocation), iomap, epoll (irq), and dma_map(with pasid automatically). > >> > >>Do you think it is acceptable? > >Maybe not drivers/iommu/ :) That subsystem only contains tools for > >dealing with DMA, I don't think epoll, resource enumeration or iomap fit > >in there. > Yes. I should consider where to put it carefully. > > > >Creating new helpers seems to be precisely what we're trying to avoid in > >this thread, and vfio-mdev does provide the components that you > >describe, so I wouldn't discard it right away. When the GPU, net, block > >or another subsystem doesn't fit your needs, either because your > >accelerator provides some specialized function, or because for > >performance reasons your client wants direct MMIO access, you can at > >least build your driver and library on top of those existing VFIO > >components: > > > >* open allocates a partition of an accelerator. > >* vfio_device_info, vfio_region_info and vfio_irq_info enumerates > >available resources. > >* vfio_irq_set deals with epoll. > >* mmap gives you a private MMIO doorbell. > >* vfio_iommu_type1 provides the DMA operations. > > > >Currently missing: > > > >* Sharing the parent IOMMU between mdev, which is also what the "IOMMU > >aware mediated device" series tackles, and seems like a logical addition > >to VFIO. I'd argue that the existing IOMMU ops (or ones implemented by > >the SVA series) can be used to deal with this > > > >* The interface to discover an accelerator near your memory node, or one > >that you can chain with other devices. If I understood correctly the > >conclusion was that the API (a topology description in sysfs?) should be > >common to various subsystems, in which case vfio-mdev (or the mediating > >driver) could also use it. > > > >* The queue abstraction discussed on patch 3/7. Perhaps the current vfio > >resource description of MMIO and IRQ is sufficient here as well, since > >vendors tend to each implement their own queue schemes. If you need > >additional features, read/write fops give the mediating driver a lot of > >freedom. To support features that are too specific for drivers/vfio/ you > >can implement a config space with capabilities and registers of your > >choice. If you're versioning the capabilities, the code to handle them > >could even be shared between different accelerator drivers and libraries. > Thank you, Jean, > > The major reason that I want to remove dependency to VFIO is: I > accepted that the whole logic of VFIO was built on the idea of > creating virtual device. > > Let's consider it in this way: We have hardware with IOMMU support. > So we create a default_domain to the particular IOMMU (unit) in the > group for the kernel driver to use it. Now the device is going to be > used by a VM or a Container. So we unbind it from the original > driver, and put the default_domain away, create a new domain for > this particular use case.
Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive
在 2018年08月10日 星期五 09:12 下午, Jean-Philippe Brucker 写道: Hi Kenneth, On 10/08/18 04:39, Kenneth Lee wrote: You can achieve everything you want to achieve with existing upstream solution. Re-inventing a whole new driver infrastructure should really be motivated with strong and obvious reasons. I want to understand better of your idea. If I create some unified helper APIs in drivers/iommu/, say: wd_create_dev(parent_dev, wd_dev) wd_release_dev(wd_dev) The API create chrdev to take request from user space for open(resource allocation), iomap, epoll (irq), and dma_map(with pasid automatically). Do you think it is acceptable? Maybe not drivers/iommu/ :) That subsystem only contains tools for dealing with DMA, I don't think epoll, resource enumeration or iomap fit in there. Yes. I should consider where to put it carefully. Creating new helpers seems to be precisely what we're trying to avoid in this thread, and vfio-mdev does provide the components that you describe, so I wouldn't discard it right away. When the GPU, net, block or another subsystem doesn't fit your needs, either because your accelerator provides some specialized function, or because for performance reasons your client wants direct MMIO access, you can at least build your driver and library on top of those existing VFIO components: * open allocates a partition of an accelerator. * vfio_device_info, vfio_region_info and vfio_irq_info enumerates available resources. * vfio_irq_set deals with epoll. * mmap gives you a private MMIO doorbell. * vfio_iommu_type1 provides the DMA operations. Currently missing: * Sharing the parent IOMMU between mdev, which is also what the "IOMMU aware mediated device" series tackles, and seems like a logical addition to VFIO. I'd argue that the existing IOMMU ops (or ones implemented by the SVA series) can be used to deal with this * The interface to discover an accelerator near your memory node, or one that you can chain with other devices. If I understood correctly the conclusion was that the API (a topology description in sysfs?) should be common to various subsystems, in which case vfio-mdev (or the mediating driver) could also use it. * The queue abstraction discussed on patch 3/7. Perhaps the current vfio resource description of MMIO and IRQ is sufficient here as well, since vendors tend to each implement their own queue schemes. If you need additional features, read/write fops give the mediating driver a lot of freedom. To support features that are too specific for drivers/vfio/ you can implement a config space with capabilities and registers of your choice. If you're versioning the capabilities, the code to handle them could even be shared between different accelerator drivers and libraries. Thank you, Jean, The major reason that I want to remove dependency to VFIO is: I accepted that the whole logic of VFIO was built on the idea of creating virtual device. Let's consider it in this way: We have hardware with IOMMU support. So we create a default_domain to the particular IOMMU (unit) in the group for the kernel driver to use it. Now the device is going to be used by a VM or a Container. So we unbind it from the original driver, and put the default_domain away, create a new domain for this particular use case. So now the device shows up as a platform or pci device to the user space. This is what VFIO try to provide. Mdev extends the scenario but dose not change the intention. And I think that is why Alex emphasis pre-allocating resource to the mdev. But what WarpDrive need is to get service from the hardware itself and set mapping to its current domain, aka defaut_domain. If we do it in VFIO-mdev, it looks like the VFIO framework takes all the effort to put the default_domain away and create a new one and be ready for user space to use. But I tell him stop using the new domain and try the original one... It is not reasonable, isn't it:) So why don't I just take the request and set it into the default_domain directly? The true requirement of WarpDrive is to let process set the page table for particular pasid or substream id, so it can accept command with address in the process space. It needs no device. From this perspective, it seems there is no reason to keep it in VFIO. Thanks Kenneth Thanks, Jean ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive
在 2018年08月10日 星期五 10:32 下午, Jerome Glisse 写道: On Fri, Aug 10, 2018 at 11:39:13AM +0800, Kenneth Lee wrote: On Thu, Aug 09, 2018 at 10:46:13AM -0400, Jerome Glisse wrote: Date: Thu, 9 Aug 2018 10:46:13 -0400 From: Jerome Glisse To: Kenneth Lee CC: Kenneth Lee , "Tian, Kevin" , Alex Williamson , Herbert Xu , "k...@vger.kernel.org" , Jonathan Corbet , Greg Kroah-Hartman , Zaibo Xu , "linux-...@vger.kernel.org" , "Kumar, Sanjay K" , Hao Fang , "linux-ker...@vger.kernel.org" , "linux...@huawei.com" , "iommu@lists.linux-foundation.org" , "linux-cry...@vger.kernel.org" , Philippe Ombredanne , Thomas Gleixner , "David S . Miller" , "linux-accelerat...@lists.ozlabs.org" Subject: Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive User-Agent: Mutt/1.10.0 (2018-05-17) Message-ID: <20180809144613.gb3...@redhat.com> On Thu, Aug 09, 2018 at 04:03:52PM +0800, Kenneth Lee wrote: On Wed, Aug 08, 2018 at 11:18:35AM -0400, Jerome Glisse wrote: On Wed, Aug 08, 2018 at 09:08:42AM +0800, Kenneth Lee wrote: 在 2018年08月06日 星期一 11:32 下午, Jerome Glisse 写道: On Mon, Aug 06, 2018 at 11:12:52AM +0800, Kenneth Lee wrote: On Fri, Aug 03, 2018 at 10:39:44AM -0400, Jerome Glisse wrote: On Fri, Aug 03, 2018 at 11:47:21AM +0800, Kenneth Lee wrote: On Thu, Aug 02, 2018 at 10:22:43AM -0400, Jerome Glisse wrote: On Thu, Aug 02, 2018 at 12:05:57PM +0800, Kenneth Lee wrote: [...] your mechanisms the userspace must have a specific userspace drivers for each hardware and thus there are virtually no differences between having this userspace driver open a device file in vfio or somewhere else in the device filesystem. This is just a different path. The basic problem WarpDrive want to solve it to avoid syscall. This is important to accelerators. We have some data here: https://www.slideshare.net/linaroorg/progress-and-demonstration-of-wrapdrive-a-accelerator-framework-sfo17317 (see page 3) The performance is different on using kernel and user drivers. Yes and example i point to is exactly that. You have a one time setup cost (creating command buffer binding PASID with command buffer and couple other setup steps). Then userspace no longer have to do any ioctl to schedule work on the GPU. It is all down from userspace and it use a doorbell to notify hardware when it should go look at command buffer for new thing to execute. My point stands on that. You have existing driver already doing so with no new framework and in your scheme you need a userspace driver. So i do not see the value add, using one path or the other in the userspace driver is litteraly one line to change. Sorry, I'd got confuse here. I partially agree that the user driver is redundance of kernel driver. (But for WarpDrive, the kernel driver is a full driver include all preparation and setup stuff for the hardware, the user driver is simply to send request and receive answer). Yes, it is just a choice of path. But the user path is faster if the request come from use space. And to do that, we need user land DMA support. Then why is it invaluable to let VFIO involved? Some drivers in the kernel already do exactly what you said. The user space emit commands without ever going into kernel by directly scheduling commands and ringing a doorbell. They do not need VFIO either and they can map userspace address into the DMA address space of the device and again they do not need VFIO for that. Could you please directly point out which driver you refer to here? Thank you. drivers/gpu/drm/amd/ Sub-directory of interest is amdkfd Because it is a big driver here is a highlevel overview of how it works (this is a simplification): - Process can allocate GPUs buffer (through ioclt) and map them into its address space (through mmap of device file at buffer object specific offset). - Process can map any valid range of virtual address space into device address space (IOMMU mapping). This must be regular memory ie not an mmap of a device file or any special file (this is the non PASID path) - Process can create a command queue and bind its process to it aka PASID, this is done through an ioctl. - Process can schedule commands onto queues it created from userspace without ioctl. For that it just write command into a ring buffer that it mapped during the command queue creation process and it rings a doorbell when commands are ready to be consume by the hardware. - Commands can reference (access) all 3 types of object above ie either full GPUs buffer, process regular memory maped as object (non PASID) and PASID memory all at the same time ie you can mix all of the above in same commands queue. - Kernel can evict, unbind any process command queues, unbind commands queue are still valid from process point of view but commands process schedules
Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive
On Thu, Aug 09, 2018 at 10:46:13AM -0400, Jerome Glisse wrote: > Date: Thu, 9 Aug 2018 10:46:13 -0400 > From: Jerome Glisse > To: Kenneth Lee > CC: Kenneth Lee , "Tian, Kevin" > , Alex Williamson , > Herbert Xu , "k...@vger.kernel.org" > , Jonathan Corbet , Greg > Kroah-Hartman , Zaibo Xu , > "linux-...@vger.kernel.org" , "Kumar, Sanjay K" > , Hao Fang , > "linux-ker...@vger.kernel.org" , > "linux...@huawei.com" , > "iommu@lists.linux-foundation.org" , > "linux-cry...@vger.kernel.org" , Philippe > Ombredanne , Thomas Gleixner , > "David S . Miller" , > "linux-accelerat...@lists.ozlabs.org" > > Subject: Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive > User-Agent: Mutt/1.10.0 (2018-05-17) > Message-ID: <20180809144613.gb3...@redhat.com> > > On Thu, Aug 09, 2018 at 04:03:52PM +0800, Kenneth Lee wrote: > > On Wed, Aug 08, 2018 at 11:18:35AM -0400, Jerome Glisse wrote: > > > On Wed, Aug 08, 2018 at 09:08:42AM +0800, Kenneth Lee wrote: > > > > 在 2018年08月06日 星期一 11:32 下午, Jerome Glisse 写道: > > > > > On Mon, Aug 06, 2018 at 11:12:52AM +0800, Kenneth Lee wrote: > > > > > > On Fri, Aug 03, 2018 at 10:39:44AM -0400, Jerome Glisse wrote: > > > > > > > On Fri, Aug 03, 2018 at 11:47:21AM +0800, Kenneth Lee wrote: > > > > > > > > On Thu, Aug 02, 2018 at 10:22:43AM -0400, Jerome Glisse wrote: > > > > > > > > > On Thu, Aug 02, 2018 at 12:05:57PM +0800, Kenneth Lee wrote: > > [...] > > > > > > > > > > your mechanisms the userspace must have a specific userspace > > > > > > > > > drivers for each hardware and thus there are virtually no > > > > > > > > > differences between having this userspace driver open a device > > > > > > > > > file in vfio or somewhere else in the device filesystem. This > > > > > > > > > is > > > > > > > > > just a different path. > > > > > > > > > > > > > > > > > The basic problem WarpDrive want to solve it to avoid syscall. > > > > > > > > This is important > > > > > > > > to accelerators. We have some data here: > > > > > > > > https://www.slideshare.net/linaroorg/progress-and-demonstration-of-wrapdrive-a-accelerator-framework-sfo17317 > > > > > > > > > > > > > > > > (see page 3) > > > > > > > > > > > > > > > > The performance is different on using kernel and user drivers. > > > > > > > Yes and example i point to is exactly that. You have a one time > > > > > > > setup > > > > > > > cost (creating command buffer binding PASID with command buffer > > > > > > > and > > > > > > > couple other setup steps). Then userspace no longer have to do any > > > > > > > ioctl to schedule work on the GPU. It is all down from userspace > > > > > > > and > > > > > > > it use a doorbell to notify hardware when it should go look at > > > > > > > command > > > > > > > buffer for new thing to execute. > > > > > > > > > > > > > > My point stands on that. You have existing driver already doing so > > > > > > > with no new framework and in your scheme you need a userspace > > > > > > > driver. > > > > > > > So i do not see the value add, using one path or the other in the > > > > > > > userspace driver is litteraly one line to change. > > > > > > > > > > > > > Sorry, I'd got confuse here. I partially agree that the user driver > > > > > > is > > > > > > redundance of kernel driver. (But for WarpDrive, the kernel driver > > > > > > is a full > > > > > > driver include all preparation and setup stuff for the hardware, > > > > > > the user driver > > > > > > is simply to send request and receive answer). Yes, it is just a > > > > > > choice of path. > > > > > > But the user path is faster if the request come from use space. And > > > > > > to do that, > > > > > > we need user land DMA support. Then why is it invaluable to let > > > &
Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive
On Thu, Aug 09, 2018 at 08:31:31AM +, Tian, Kevin wrote: > Date: Thu, 9 Aug 2018 08:31:31 + > From: "Tian, Kevin" > To: Kenneth Lee , Jerome Glisse > CC: Kenneth Lee , Alex Williamson > , Herbert Xu , > "k...@vger.kernel.org" , Jonathan Corbet > , Greg Kroah-Hartman , Zaibo > Xu , "linux-...@vger.kernel.org" > , "Kumar, Sanjay K" , > Hao Fang , "linux-ker...@vger.kernel.org" > , "linux...@huawei.com" > , "iommu@lists.linux-foundation.org" > , "linux-cry...@vger.kernel.org" > , Philippe Ombredanne > , Thomas Gleixner , "David S . > Miller" , "linux-accelerat...@lists.ozlabs.org" > > Subject: RE: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive > Message-ID: > > > > From: Kenneth Lee [mailto:liguo...@hisilicon.com] > > Sent: Thursday, August 9, 2018 4:04 PM > > > > But we have another requirement which is to combine some device > > together to > > share the same address space. This is a little like these kinds of solution: > > > > http://tce.technion.ac.il/wp-content/uploads/sites/8/2015/06/SC-7.2-M.- > > Silberstein.pdf > > > > With that, the application can directly pass the NiC packet pointer to the > > decryption accelerator, and get the bare data in place. This is the feature > > that > > the VFIO container can provide. > > above is not a good argument, at least in the context of your discussion. > If each device has their own interface (similar to GPU) for process to bind > with, then having the process binding to multiple devices one-by-one then > you still get same address space shared cross them... If we consider this from the VFIO container perspective, with a container, you can do DMA to the container applying it to all devices, even the device is added after the DMA operation. So your argument remains true only when SVM is enabled and the whole process space is devoted to the devices. Yes, the process can do the same all by itself. But if we agree with that, it makes no sense to keep the container concept in VFIO;) > > Thanks > Kevin -- -Kenneth(Hisilicon) 本邮件及其附件含有华为公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁 止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中 的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive
On Wed, Aug 08, 2018 at 11:18:35AM -0400, Jerome Glisse wrote: > Date: Wed, 8 Aug 2018 11:18:35 -0400 > From: Jerome Glisse > To: Kenneth Lee > CC: Kenneth Lee , "Tian, Kevin" > , Alex Williamson , > Herbert Xu , "k...@vger.kernel.org" > , Jonathan Corbet , Greg > Kroah-Hartman , Zaibo Xu , > "linux-...@vger.kernel.org" , "Kumar, Sanjay K" > , Hao Fang , > "linux-ker...@vger.kernel.org" , > "linux...@huawei.com" , > "iommu@lists.linux-foundation.org" , > "linux-cry...@vger.kernel.org" , Philippe > Ombredanne , Thomas Gleixner , > "David S . Miller" , > "linux-accelerat...@lists.ozlabs.org" > > Subject: Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive > User-Agent: Mutt/1.10.0 (2018-05-17) > Message-ID: <20180808151835.ga3...@redhat.com> > > On Wed, Aug 08, 2018 at 09:08:42AM +0800, Kenneth Lee wrote: > > > > > > 在 2018年08月06日 星期一 11:32 下午, Jerome Glisse 写道: > > > On Mon, Aug 06, 2018 at 11:12:52AM +0800, Kenneth Lee wrote: > > > > On Fri, Aug 03, 2018 at 10:39:44AM -0400, Jerome Glisse wrote: > > > > > On Fri, Aug 03, 2018 at 11:47:21AM +0800, Kenneth Lee wrote: > > > > > > On Thu, Aug 02, 2018 at 10:22:43AM -0400, Jerome Glisse wrote: > > > > > > > On Thu, Aug 02, 2018 at 12:05:57PM +0800, Kenneth Lee wrote: > > > > > > > > On Thu, Aug 02, 2018 at 02:33:12AM +, Tian, Kevin wrote: > > > > > > > > > > On Wed, Aug 01, 2018 at 06:22:14PM +0800, Kenneth Lee wrote: > > [...] > > > > > > > > > > > My more general question is do we want to grow VFIO to > > > > > > > > > > become > > > > > > > > > > a more generic device driver API. This patchset adds a > > > > > > > > > > command > > > > > > > > > > queue concept to it (i don't think it exist today but i have > > > > > > > > > > not follow VFIO closely). > > > > > > > > > > > > > > > > > > The thing is, VFIO is the only place to support DMA from user > > > > > > > > land. If we don't > > > > > > > > put it here, we have to create another similar facility to > > > > > > > > support the same. > > > > > > > No it is not, network device, GPU, block device, ... they all do > > > > > > > support DMA. The point i am trying to make here is that even in > > > > > > Sorry, wait a minute, are we talking the same thing? I meant "DMA > > > > > > from user > > > > > > land", not "DMA from kernel driver". To do that we have to > > > > > > manipulate the > > > > > > IOMMU(Unit). I think it can only be done by default_domain or vfio > > > > > > domain. Or > > > > > > the user space have to directly access the IOMMU. > > > > > GPU do DMA in the sense that you pass to the kernel a valid > > > > > virtual address (kernel driver do all the proper check) and > > > > > then you can use the GPU to copy from or to that range of > > > > > virtual address. Exactly how you want to use this compression > > > > > engine. It does not rely on SVM but SVM going forward would > > > > > still be the prefered option. > > > > > > > > > No, SVM is not the reason why we rely on Jean's SVM(SVA) series. We > > > > rely on > > > > Jean's series because of multi-process (PASID or substream ID) support. > > > > > > > > But of couse, WarpDrive can still benefit from the SVM feature. > > > We are getting side tracked here. PASID/ID do not require VFIO. > > > > > Yes, PASID itself do not require VFIO. But what if: > > > > 1. Support DMA from user space. > > 2. The hardware makes use of standard IOMMU/SMMU for IO address translation. > > 3. The IOMMU facility is shared by both kernel and user drivers. > > 4. Support PASID with the current IOMMU facility > > I do not see how any of this means it has to be in VFIO. > Other devices do just that. GPUs driver for instance share > DMA engine (that copy data around) between kernel and user > space. Sometime kernel use it to move things around. Evict > some memory to make room for a new process is the common > example. Same DMA engines is often use by userspace its
Re: [RFC PATCH 2/7] iommu: Add share domain interface in iommu for spimdev
On Wed, Aug 08, 2018 at 11:13:54AM +0200, Joerg Roedel wrote: > Date: Wed, 8 Aug 2018 11:13:54 +0200 > From: Joerg Roedel > To: Kenneth Lee > CC: Jonathan Corbet , Herbert Xu > , "David S . Miller" , > Alex Williamson , Kenneth Lee > , Hao Fang , Zhou Wang > , Zaibo Xu , Philippe > Ombredanne , Greg Kroah-Hartman > , Thomas Gleixner , > linux-...@vger.kernel.org, linux-ker...@vger.kernel.org, > linux-cry...@vger.kernel.org, iommu@lists.linux-foundation.org, > k...@vger.kernel.org, linux-accelerat...@lists.ozlabs.org, Lu Baolu > , Sanjay Kumar , > linux...@huawei.com > Subject: Re: [RFC PATCH 2/7] iommu: Add share domain interface in iommu for > spimdev > User-Agent: NeoMutt/20170421 (1.8.2) > Message-ID: <20180808091354.ppqgineql3puf...@8bytes.org> > > On Wed, Aug 01, 2018 at 06:22:16PM +0800, Kenneth Lee wrote: > > From: Kenneth Lee > > > > This patch add sharing interface for a iommu_group. The new interface: > > > > iommu_group_share_domain() > > iommu_group_unshare_domain() > > > > can be used by some virtual iommu_group (such as iommu_group for spimdev) > > to share their parent's iommu_group. > > > > When the domain of the group is shared, it cannot be changed before > > unshared. In the future, notification can be added if update is required. > > >From the description or the patch I don't really get why this is needed. > Please update the description and make a case for this API addition. > Yes, I will add more description in next version. The idea here is: 1. iommu_domain is the setting of the IOMMU(the unit) of an iommu_group. 2. The iommu_group originally uses a default domain or a created one (e.g. by vfio) as its effective domain. There is only one user for a domain. If a device is bound to vfio driver, it uses the new created domain. And if it is unbound from vfio driver, it is switched back to the default domain. 3. But for spimdev, we create an mdev which share it parent device's IOMMU. That means, the parent device (along with its iommu group) will continue use its iommu_domain, and the mdev will use the same at the same time. 4. This API, is to tell the iommu_group that the domain is shared by the spimdev. And it should be changed before it is unshared. Cheers > > Joerg -- -Kenneth(Hisilicon) ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 1/7] vfio/spimdev: Add documents for WarpDrive framework
在 2018年08月06日 星期一 08:27 下午, Pavel Machek 写道: Hi! WarpDrive is a common user space accelerator framework. Its main component in Kernel is called spimdev, Share Parent IOMMU Mediated Device. It exposes spimdev is really unfortunate name. It looks like it has something to do with SPI, but it does not. Yes. Let me change it to Share (IOMMU) Domain MDev, SDMdev:) +++ b/Documentation/warpdrive/warpdrive.rst @@ -0,0 +1,153 @@ +Introduction of WarpDrive += + +*WarpDrive* is a general accelerator framework built on top of vfio. +It can be taken as a light weight virtual function, which you can use without +*SR-IOV* like facility and can be shared among multiple processes. + +It can be used as the quick channel for accelerators, network adaptors or +other hardware in user space. It can make some implementation simpler. E.g. +you can reuse most of the *netdev* driver and just share some ring buffer to +the user space driver for *DPDK* or *ODP*. Or you can combine the RSA +accelerator with the *netdev* in the user space as a Web reversed proxy, etc. What is DPDK? ODP? DPDK：https://www.dpdk.org/about/ ODP: https://www.opendataplane.org/ will add the reference in the next RFC +How does it work + + +*WarpDrive* takes the Hardware Accelerator as a heterogeneous processor which +can share some load for the CPU: + +.. image:: wd.svg +:alt: This is a .svg image, if your browser cannot show it, +try to download and view it locally + +So it provides the capability to the user application to: + +1. Send request to the hardware +2. Share memory with the application and other accelerators + +These requirements can be fulfilled by VFIO if the accelerator can serve each +application with a separated Virtual Function. But a *SR-IOV* like VF (we will +call it *HVF* hereinafter) design is too heavy for the accelerator which +service thousands of processes. VFIO? VF? HVF? Also "gup" might be worth spelling out. But I think the reference [1] has explained this. +References +== +.. [1] Accroding to the comment in in mm/gup.c, The *gup* is only safe within + a syscall. Because it can only keep the physical memory in place + without making sure the VMA will always point to it. Maybe we should + raise the VM_PINNED patchset (see + https://lists.gt.net/linux/kernel/1931993) again to solve this probl I went through the docs, but I still don't know what it does. Will refine the doc in next RFC, hope it will help. Pavel ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 3/7] vfio: add spimdev support
在 2018年08月07日 星期二 01:05 上午, Alex Williamson 写道: On Mon, 6 Aug 2018 09:34:28 -0700 "Raj, Ashok" wrote: On Mon, Aug 06, 2018 at 09:49:40AM -0600, Alex Williamson wrote: On Mon, 6 Aug 2018 09:40:04 +0800 Kenneth Lee wrote: 1. It supports thousands of processes. Take zip accelerator as an example, any application need data compression/decompression will need to interact with the accelerator. To support that, you have to create tens of thousands of mdev for their usage. I don't think it is a good idea to have so many devices in the system. Each mdev is a device, regardless of whether there are hardware resources committed to the device, so I don't understand this argument. 2. The application does not want to own the mdev for long. It just need an access point for the hardware service. If it has to interact with an management agent for allocation and release, this makes the problem complex. I don't see how the length of the usage plays a role here either. Are you concerned that the time it takes to create and remove an mdev is significant compared to the usage time? Userspace is certainly welcome to create a pool of devices, but why should it be the kernel's responsibility to dynamically assign resources to an mdev? What's the usage model when resources are unavailable? It seems there's complexity in either case, but it's generally userspace's responsibility to impose a policy. Can vfio dev's created representing an mdev be shared between several processes? It doesn't need to be exclusive. The path to hardware is established by the processes binding to SVM and IOMMU ensuring that the PASID is plummed properly. One can think the same hardware is shared between several processes, hardware knows the isolation is via the PASID. For these cases it isn't required to create a dev per process. The iommu group is the unit of ownership, a vfio group mirrors an iommu group, therefore a vfio group only allows a single open(2). A group also represents the minimum isolation set of devices, therefore devices within a group are not considered isolated and must share the same address space represented by the vfio container. Beyond that, it is possible to share devices among processes, but (I think) it generally implies a hierarchical rather than peer relationship between processes. Thanks, Actually, this is the key problem we concerned. Our logic was: The PASID refer to the connection between the device and the process. So the resource should be allocated only when the process "make use of" the device. This strategy also bring another advantage that the kernel driver can also make use of the resource if no user application open it. We do have another branch that allocate resource to mdev directly. It looks not so nice (many mdevs and user agent is required for resource management). If the conclusion here is to keep the mdev's original semantics, we will send that branch for discussion in next RFC. Cheers Kenneth Alex ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive
在 2018年08月06日 星期一 11:32 下午, Jerome Glisse 写道: On Mon, Aug 06, 2018 at 11:12:52AM +0800, Kenneth Lee wrote: On Fri, Aug 03, 2018 at 10:39:44AM -0400, Jerome Glisse wrote: On Fri, Aug 03, 2018 at 11:47:21AM +0800, Kenneth Lee wrote: On Thu, Aug 02, 2018 at 10:22:43AM -0400, Jerome Glisse wrote: On Thu, Aug 02, 2018 at 12:05:57PM +0800, Kenneth Lee wrote: On Thu, Aug 02, 2018 at 02:33:12AM +, Tian, Kevin wrote: On Wed, Aug 01, 2018 at 06:22:14PM +0800, Kenneth Lee wrote: [...] But doorbell is just a notification. Except for DOS (to make hardware busy) it cannot actually take or change anything from the kernel space. And the DOS problem can be always taken as the problem that a group of processes share the same kernel entity. In the coming HIP09 hardware, the doorbell will come with a random number so only the process who allocated the queue can knock it correctly. When doorbell is ring the hardware start fetching commands from the queue and execute them ? If so than a rogue process B might ring the doorbell of process A which would starts execution of random commands (ie whatever random memory value there is left inside the command buffer memory, could be old commands i guess). If this is not how this doorbell works then, yes it can only do a denial of service i guess. Issue i have with doorbell is that i have seen 10 differents implementations in 10 differents hw and each are different as to what ringing or value written to the doorbell does. It is painfull to track what is what for each hw. In our implementation, doorbell is simply a notification, just like an interrupt to the accelerator. The command is all about what's in the queue. I agree that there is no simple and standard way to track the shared IO space. But I think we have to trust the driver in some way. If the driver is malicious, even a simple ioctl can become an attack. Trusting kernel space driver is fine, trusting user space driver is not in my view. AFAICT every driver developer so far always made sure that someone could not abuse its device to do harmfull thing to other process. Fully agree. That is why this driver shares only the doorbell space. There is only the doorbell is shared in the whole page, nothing else. Maybe you are concerning the user driver will give malicious command to the hardware? But these commands cannot influence the other process. If we can trust the hardware design, the process cannot do any harm. My questions was what happens if a process B ring the doorbell of process A. On some hardware the value written in the doorbell is use as an index in command buffer. On other it just wakeup the hardware to go look at a structure private to the process. They are other variations of those themes. If it is the former ie the value is use to advance in the command buffer then a rogue process can force another process to advance its command buffer and what is in the command buffer can be some random old memory values which can be more harmfull than just Denial Of Service. Yes. We have considered that. There is no other information in the doorbell. The indexes, such as head and tail pointers, are all in the shared memory between the hardware and the user process. The other process cannot touch it. My more general question is do we want to grow VFIO to become a more generic device driver API. This patchset adds a command queue concept to it (i don't think it exist today but i have not follow VFIO closely). The thing is, VFIO is the only place to support DMA from user land. If we don't put it here, we have to create another similar facility to support the same. No it is not, network device, GPU, block device, ... they all do support DMA. The point i am trying to make here is that even in Sorry, wait a minute, are we talking the same thing? I meant "DMA from user land", not "DMA from kernel driver". To do that we have to manipulate the IOMMU(Unit). I think it can only be done by default_domain or vfio domain. Or the user space have to directly access the IOMMU. GPU do DMA in the sense that you pass to the kernel a valid virtual address (kernel driver do all the proper check) and then you can use the GPU to copy from or to that range of virtual address. Exactly how you want to use this compression engine. It does not rely on SVM but SVM going forward would still be the prefered option. No, SVM is not the reason why we rely on Jean's SVM(SVA) series. We rely on Jean's series because of multi-process (PASID or substream ID) support. But of couse, WarpDrive can still benefit from the SVM feature. We are getting side tracked here. PASID/ID do not require VFIO. Yes, PASID itself do not require VFIO. But what if: 1. Support DMA from user space. 2. The hardware makes use of standard IOMMU/SMMU for IO address translation. 3. The IOMMU facility is shared by both kernel and user drivers. 4. Support PASID with the current IOMMU facility your mechanisms the userspace
Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive
On Fri, Aug 03, 2018 at 10:39:44AM -0400, Jerome Glisse wrote: > Date: Fri, 3 Aug 2018 10:39:44 -0400 > From: Jerome Glisse > To: Kenneth Lee > CC: "Tian, Kevin" , Herbert Xu > , "k...@vger.kernel.org" > , Jonathan Corbet , Greg > Kroah-Hartman , Zaibo Xu , > "linux-...@vger.kernel.org" , "Kumar, Sanjay K" > , Hao Fang , > "iommu@lists.linux-foundation.org" , > "linux-ker...@vger.kernel.org" , > "linux...@huawei.com" , Alex Williamson > , "linux-cry...@vger.kernel.org" > , Philippe Ombredanne > , Thomas Gleixner , Kenneth Lee > , "David S . Miller" , > "linux-accelerat...@lists.ozlabs.org" > > Subject: Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive > User-Agent: Mutt/1.10.0 (2018-05-17) > Message-ID: <20180803143944.ga4...@redhat.com> > > On Fri, Aug 03, 2018 at 11:47:21AM +0800, Kenneth Lee wrote: > > On Thu, Aug 02, 2018 at 10:22:43AM -0400, Jerome Glisse wrote: > > > Date: Thu, 2 Aug 2018 10:22:43 -0400 > > > From: Jerome Glisse > > > To: Kenneth Lee > > > CC: "Tian, Kevin" , Hao Fang , > > > Alex Williamson , Herbert Xu > > > , "k...@vger.kernel.org" > > > , Jonathan Corbet , Greg > > > Kroah-Hartman , Zaibo Xu > > > , > > > "linux-...@vger.kernel.org" , "Kumar, Sanjay > > > K" > > > , Kenneth Lee , > > > "iommu@lists.linux-foundation.org" , > > > "linux-ker...@vger.kernel.org" , > > > "linux...@huawei.com" , > > > "linux-cry...@vger.kernel.org" , Philippe > > > Ombredanne , Thomas Gleixner , > > > "David S . Miller" , > > > "linux-accelerat...@lists.ozlabs.org" > > > > > > Subject: Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive > > > User-Agent: Mutt/1.10.0 (2018-05-17) > > > Message-ID: <20180802142243.ga3...@redhat.com> > > > > > > On Thu, Aug 02, 2018 at 12:05:57PM +0800, Kenneth Lee wrote: > > > > On Thu, Aug 02, 2018 at 02:33:12AM +, Tian, Kevin wrote: > > > > > Date: Thu, 2 Aug 2018 02:33:12 + > > > > > > From: Jerome Glisse > > > > > > On Wed, Aug 01, 2018 at 06:22:14PM +0800, Kenneth Lee wrote: > > > > > > > From: Kenneth Lee > > > > > > > > > > > > > > WarpDrive is an accelerator framework to expose the hardware > > > > > > capabilities > > > > > > > directly to the user space. It makes use of the exist vfio and > > > > > > > vfio-mdev > > > > > > > facilities. So the user application can send request and DMA to > > > > > > > the > > > > > > > hardware without interaction with the kernel. This remove the > > > > > > > latency > > > > > > > of syscall and context switch. > > > > > > > > > > > > > > The patchset contains documents for the detail. Please refer to > > > > > > > it for > > > > > > more > > > > > > > information. > > > > > > > > > > > > > > This patchset is intended to be used with Jean Philippe Brucker's > > > > > > > SVA > > > > > > > patch [1] (Which is also in RFC stage). But it is not mandatory. > > > > > > > This > > > > > > > patchset is tested in the latest mainline kernel without the SVA > > > > > > > patches. > > > > > > > So it support only one process for each accelerator. > > > > > > > > > > > > > > With SVA support, WarpDrive can support multi-process in the same > > > > > > > accelerator device. We tested it in our SoC integrated > > > > > > > Accelerator (board > > > > > > > ID: D06, Chip ID: HIP08). A reference work tree can be found > > > > > > > here: [2]. > > > > > > > > > > > > I have not fully inspected things nor do i know enough about > > > > > > this Hisilicon ZIP accelerator to ascertain, but from glimpsing > > > > > > at the code it seems that it is unsafe to use even with SVA due > > > > > > to the doorbell. There is a comment talking about safetyness > > > > > > in patch 7. >
Re: [RFC PATCH 3/7] vfio: add spimdev support
On Thu, Aug 02, 2018 at 12:43:27PM -0600, Alex Williamson wrote: > Date: Thu, 2 Aug 2018 12:43:27 -0600 > From: Alex Williamson > To: Cornelia Huck > CC: Kenneth Lee , "Tian, Kevin" > , Kenneth Lee , Jonathan Corbet > , Herbert Xu , "David S . > Miller" , Joerg Roedel , Hao Fang > , Zhou Wang , Zaibo Xu > , Philippe Ombredanne , "Greg > Kroah-Hartman" , Thomas Gleixner > , "linux-...@vger.kernel.org" > , "linux-ker...@vger.kernel.org" > , "linux-cry...@vger.kernel.org" > , "iommu@lists.linux-foundation.org" > , "k...@vger.kernel.org" > , "linux-accelerat...@lists.ozlabs.org\" > , Lu Baolu > , Kumar", , " linux...@huawei.com " > "> > Subject: Re: [RFC PATCH 3/7] vfio: add spimdev support > Message-ID: <20180802124327.403b1...@t450s.home> > > On Thu, 2 Aug 2018 10:35:28 +0200 > Cornelia Huck wrote: > > > On Thu, 2 Aug 2018 15:34:40 +0800 > > Kenneth Lee wrote: > > > > > On Thu, Aug 02, 2018 at 04:24:22AM +, Tian, Kevin wrote: > > > > > > > From: Kenneth Lee [mailto:liguo...@hisilicon.com] > > > > > Sent: Thursday, August 2, 2018 11:47 AM > > > > > > > > > > > > > > > > > > From: Kenneth Lee > > > > > > > Sent: Wednesday, August 1, 2018 6:22 PM > > > > > > > > > > > > > > From: Kenneth Lee > > > > > > > > > > > > > > SPIMDEV is "Share Parent IOMMU Mdev". It is a vfio-mdev. But > > > > > > > differ > > > > > from > > > > > > > the general vfio-mdev: > > > > > > > > > > > > > > 1. It shares its parent's IOMMU. > > > > > > > 2. There is no hardware resource attached to the mdev is created. > > > > > > > The > > > > > > > hardware resource (A `queue') is allocated only when the mdev is > > > > > > > opened. > > > > > > > > > > > > Alex has concern on doing so, as pointed out in: > > > > > > > > > > > > https://www.spinics.net/lists/kvm/msg172652.html > > > > > > > > > > > > resource allocation should be reserved at creation time. > > > > > > > > > > Yes. That is why I keep telling that SPIMDEV is not for "VM", it is > > > > > for "many > > > > > processes", it is just an access point to the process. Not a device > > > > > to VM. I > > > > > hope > > > > > Alex can accept it:) > > > > > > > > > > > > > VFIO is just about assigning device resource to user space. It doesn't > > > > care > > > > whether it's native processes or VM using the device so far. Along the > > > > direction > > > > which you described, looks VFIO needs to support the configuration that > > > > some mdevs are used for native process only, while others can be used > > > > for both native and VM. I'm not sure whether there is a clean way to > > > > enforce it... > > > > > > I had the same idea at the beginning. But finally I found that the life > > > cycle > > > of the virtual device for VM and process were different. Consider you > > > create > > > some mdevs for VM use, you will give all those mdevs to lib-virt, which > > > distribute those mdev to VMs or containers. If the VM or container exits, > > > the > > > mdev is returned to the lib-virt and used for next allocation. It is the > > > administrator who controlled every mdev's allocation. > > Libvirt currently does no management of mdev devices, so I believe > this example is fictitious. The extent of libvirt's interaction with > mdev is that XML may specify an mdev UUID as the source for a hostdev > and set the permissions on the device files appropriately. Whether > mdevs are created in advance and re-used or created and destroyed > around a VM instance (for example via qemu hooks scripts) is not a > policy that libvirt imposes. > > > > But for process, it is different. There is no lib-virt in control. The > > > administrator's intension is to grant some type of application to access > > > the > > > hardware. The application can get a handle of the hardware, send request > > > and ge
Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive
On Fri, Aug 03, 2018 at 03:20:43PM +0100, Alan Cox wrote: > Date: Fri, 3 Aug 2018 15:20:43 +0100 > From: Alan Cox > To: Jerome Glisse > CC: "Tian, Kevin" , Kenneth Lee > , Hao Fang , Herbert Xu > , "k...@vger.kernel.org" > , Jonathan Corbet , Greg > Kroah-Hartman , "linux-...@vger.kernel.org" > , "Kumar, Sanjay K" , > "iommu@lists.linux-foundation.org" , > "linux-ker...@vger.kernel.org" , > "linux...@huawei.com" , Alex Williamson > , Thomas Gleixner , > "linux-cry...@vger.kernel.org" , Philippe > Ombredanne , Zaibo Xu , Kenneth > Lee , "David S . Miller" , > Ross Zwisler > Subject: Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive > Organization: Intel Corporation > X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; x86_64-redhat-linux-gnu) > Message-ID: <20180803152043.40f88947@alans-desktop> > > > If we are going to have any kind of general purpose accelerator API then > > > it has to be able to implement things like > > > > Why is the existing driver model not good enough ? So you want > > a device with function X you look into /dev/X (for instance > > for GPU you look in /dev/dri) > > Except when my GPU is in an FPGA in which case it might be somewhere else > or it's a general purpose accelerator that happens to be usable as a GPU. > Unusual today in big computer space but you'll find it in > microcontrollers. > > > Each of those device need a userspace driver and thus this > > user space driver can easily knows where to look. I do not > > expect that every application will reimplement those drivers > > but instead use some kind of library that provide a high > > level API for each of those devices. > > Think about it from the user level. You have a pipeline of things you > wish to execute, you need to get the right accelerator combinations and > they need to fit together to meet system constraints like number of > IOMMU ids the accelerator supports, where they are connected. > > > Now you have a hierarchy of memory for the CPU (HBM, local > > node main memory aka you DDR dimm, persistent memory) each > > It's not a heirarchy, it's a graph. There's no fundamental reason two > accelerators can't be close to two different CPU cores but have shared > HBM that is far from each processor. There are physical reasons it tends > to look more like a heirarchy today. > > > Anyway i think finding devices and finding relation between > > devices and memory is 2 separate problems and as such should > > be handled separatly. > > At a certain level they are deeply intertwined because you need a common > API. It's not good if I want a particular accelerator and need to then > see which API its under on this machine and which interface I have to > use, and maybe have a mix of FPGA, WarpDrive and Google ASIC interfaces > all different. > > The job of the kernel is to impose some kind of sanity and unity on this > lot. > > All of it in the end comes down to > > 'Somehow glue some chunk of memory into my address space and find any > supporting driver I need' > Agree. This is also our intension on WarpDrive. And it looks VFIO is the best place to fulfill this requirement. > plus virtualization of the above. > > That bit's easy - but making it usable is a different story. > > Alan -- -Kenneth(Hisilicon) 本邮件及其附件含有华为公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁 止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中 的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 3/7] vfio: add spimdev support
On Thu, Aug 02, 2018 at 04:24:22AM +, Tian, Kevin wrote: > Date: Thu, 2 Aug 2018 04:24:22 + > From: "Tian, Kevin" > To: Kenneth Lee > CC: Kenneth Lee , Jonathan Corbet , > Herbert Xu , "David S . Miller" > , Joerg Roedel , Alex Williamson > , Hao Fang , Zhou Wang > , Zaibo Xu , Philippe > Ombredanne , Greg Kroah-Hartman > , Thomas Gleixner , > "linux-...@vger.kernel.org" , > "linux-ker...@vger.kernel.org" , > "linux-cry...@vger.kernel.org" , > "iommu@lists.linux-foundation.org" , > "k...@vger.kernel.org" , > "linux-accelerat...@lists.ozlabs.org" > , Lu Baolu > , "Kumar, Sanjay K" , > "linux...@huawei.com" > Subject: RE: [RFC PATCH 3/7] vfio: add spimdev support > Message-ID: > > > > From: Kenneth Lee [mailto:liguo...@hisilicon.com] > > Sent: Thursday, August 2, 2018 11:47 AM > > > > > > > > > From: Kenneth Lee > > > > Sent: Wednesday, August 1, 2018 6:22 PM > > > > > > > > From: Kenneth Lee > > > > > > > > SPIMDEV is "Share Parent IOMMU Mdev". It is a vfio-mdev. But differ > > from > > > > the general vfio-mdev: > > > > > > > > 1. It shares its parent's IOMMU. > > > > 2. There is no hardware resource attached to the mdev is created. The > > > > hardware resource (A `queue') is allocated only when the mdev is > > > > opened. > > > > > > Alex has concern on doing so, as pointed out in: > > > > > > https://www.spinics.net/lists/kvm/msg172652.html > > > > > > resource allocation should be reserved at creation time. > > > > Yes. That is why I keep telling that SPIMDEV is not for "VM", it is for > > "many > > processes", it is just an access point to the process. Not a device to VM. I > > hope > > Alex can accept it:) > > > > VFIO is just about assigning device resource to user space. It doesn't care > whether it's native processes or VM using the device so far. Along the > direction > which you described, looks VFIO needs to support the configuration that > some mdevs are used for native process only, while others can be used > for both native and VM. I'm not sure whether there is a clean way to > enforce it... I had the same idea at the beginning. But finally I found that the life cycle of the virtual device for VM and process were different. Consider you create some mdevs for VM use, you will give all those mdevs to lib-virt, which distribute those mdev to VMs or containers. If the VM or container exits, the mdev is returned to the lib-virt and used for next allocation. It is the administrator who controlled every mdev's allocation. But for process, it is different. There is no lib-virt in control. The administrator's intension is to grant some type of application to access the hardware. The application can get a handle of the hardware, send request and get the result. That's all. He/She dose not care which mdev is allocated to that application. If it crashes, it should be the kernel's responsibility to withdraw the resource, the system administrator does not want to do it by hand. > > Let's hear from Alex's thought. Sure:) > > Thanks > Kevin -- -Kenneth(Hisilicon) 本邮件及其附件含有华为公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁 止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中 的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive
On Thu, Aug 02, 2018 at 04:36:52AM +, Tian, Kevin wrote: > Date: Thu, 2 Aug 2018 04:36:52 + > From: "Tian, Kevin" > To: Kenneth Lee > CC: Kenneth Lee , Herbert Xu > , "k...@vger.kernel.org" > , Jonathan Corbet , Greg > Kroah-Hartman , "linux-...@vger.kernel.org" > , "Kumar, Sanjay K" , > Hao Fang , "iommu@lists.linux-foundation.org" > , "linux-ker...@vger.kernel.org" > , "linux...@huawei.com" > , Alex Williamson , > Thomas Gleixner , "linux-cry...@vger.kernel.org" > , Philippe Ombredanne > , Zaibo Xu , "David S . Miller" > , "linux-accelerat...@lists.ozlabs.org" > > Subject: RE: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive > Message-ID: > > > > From: Kenneth Lee > > Sent: Thursday, August 2, 2018 11:40 AM > > > > On Thu, Aug 02, 2018 at 02:59:33AM +, Tian, Kevin wrote: > > > > From: Kenneth Lee > > > > Sent: Wednesday, August 1, 2018 6:22 PM > > > > > > > > From: Kenneth Lee > > > > > > > > WarpDrive is an accelerator framework to expose the hardware > > capabilities > > > > directly to the user space. It makes use of the exist vfio and vfio-mdev > > > > facilities. So the user application can send request and DMA to the > > > > hardware without interaction with the kernel. This remove the latency > > > > of syscall and context switch. > > > > > > > > The patchset contains documents for the detail. Please refer to it for > > more > > > > information. > > > > > > > > This patchset is intended to be used with Jean Philippe Brucker's SVA > > > > patch [1] (Which is also in RFC stage). But it is not mandatory. This > > > > patchset is tested in the latest mainline kernel without the SVA > > > > patches. > > > > So it support only one process for each accelerator. > > > > > > If no sharing, then why not just assigning the whole parent device to > > > the process? IMO if SVA usage is the clear goal of your series, it > > > might be made clearly so then Jean's series is mandatory dependency... > > > > > > > We don't know how SVA will be finally. But the feature, "make use of > > per-PASID/substream ID IOMMU page table", should be able to be enabled > > in the > > kernel. So we don't want to enforce it here. After we have this serial > > ready, > > it > > can be hooked to any implementation. > > "any" or "only queue-based" implementation? some devices may not > have queue concept, e.g. GPU. Here, "any implementation" refer to the "per PASID page table" implementatin:) > > > > > Further more, even without "per-PASID IOMMU page table", this series has > > its > > value. It is not simply dedicate the whole device to the process. It > > "shares" > > the device with the kernel driver. So you can support crypto and a user > > application at the same time. > > OK. > > Thanks > Kevin -- -Kenneth(Hisilicon) 本邮件及其附件含有华为公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁 止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中 的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 1/7] vfio/spimdev: Add documents for WarpDrive framework
On Thu, Aug 02, 2018 at 03:14:38AM +, Tian, Kevin wrote: > Date: Thu, 2 Aug 2018 03:14:38 + > From: "Tian, Kevin" > To: Kenneth Lee , Jonathan Corbet , > Herbert Xu , "David S . Miller" > , Joerg Roedel , Alex Williamson > , Kenneth Lee , Hao > Fang , Zhou Wang , Zaibo Xu > , Philippe Ombredanne , Greg > Kroah-Hartman , Thomas Gleixner > , "linux-...@vger.kernel.org" > , "linux-ker...@vger.kernel.org" > , "linux-cry...@vger.kernel.org" > , "iommu@lists.linux-foundation.org" > , "k...@vger.kernel.org" > , "linux-accelerat...@lists.ozlabs.org" > , Lu Baolu > , "Kumar, Sanjay K" > CC: "linux...@huawei.com" > Subject: RE: [RFC PATCH 1/7] vfio/spimdev: Add documents for WarpDrive > framework > Message-ID: > > > > From: Kenneth Lee > > Sent: Wednesday, August 1, 2018 6:22 PM > > > > From: Kenneth Lee > > > > WarpDrive is a common user space accelerator framework. Its main > > component > > in Kernel is called spimdev, Share Parent IOMMU Mediated Device. It > > Not sure whether "share parent IOMMU" is a good term here. better > stick to what capabity you bring to user space, instead of describing > internal trick... Agree. The "SPI" is also strongly hint to "Share Peripheral Interrupt" or "Serial Peripheral Interface". Personally I hate this name. But I cannot find a better one. In the user space, we can use a impassible name (WarpDrive). But kernel require a name point to its feature... maybe I should use "Process-Shared Mdev" in next version? > > > exposes > > the hardware capabilities to the user space via vfio-mdev. So processes in > > user land can obtain a "queue" by open the device and direct access the > > hardware MMIO space or do DMA operation via VFIO interface. > > > > WarpDrive is intended to be used with Jean Philippe Brucker's SVA patchset > > (it is still in RFC stage) to support multi-process. But This is not a must. > > Without the SVA patches, WarpDrive can still work for one process for > > every > > hardware device. > > > > This patch add detail documents for the framework. > > > > Signed-off-by: Kenneth Lee > > --- > > Documentation/00-INDEX| 2 + > > Documentation/warpdrive/warpdrive.rst | 153 ++ > > Documentation/warpdrive/wd-arch.svg | 732 > > ++ > > Documentation/warpdrive/wd.svg| 526 ++ > > 4 files changed, 1413 insertions(+) > > create mode 100644 Documentation/warpdrive/warpdrive.rst > > create mode 100644 Documentation/warpdrive/wd-arch.svg > > create mode 100644 Documentation/warpdrive/wd.svg > > > > diff --git a/Documentation/00-INDEX b/Documentation/00-INDEX > > index 2754fe83f0d4..9959affab599 100644 > > --- a/Documentation/00-INDEX > > +++ b/Documentation/00-INDEX > > @@ -410,6 +410,8 @@ vm/ > > - directory with info on the Linux vm code. > > w1/ > > - directory with documents regarding the 1-wire (w1) subsystem. > > +warpdrive/ > > + - directory with documents about WarpDrive accelerator > > framework. > > watchdog/ > > - how to auto-reboot Linux if it has "fallen and can't get up". ;-) > > wimax/ > > diff --git a/Documentation/warpdrive/warpdrive.rst > > b/Documentation/warpdrive/warpdrive.rst > > new file mode 100644 > > index ..3792b2780ea6 > > --- /dev/null > > +++ b/Documentation/warpdrive/warpdrive.rst > > @@ -0,0 +1,153 @@ > > +Introduction of WarpDrive > > += > > + > > +*WarpDrive* is a general accelerator framework built on top of vfio. > > +It can be taken as a light weight virtual function, which you can use > > without > > +*SR-IOV* like facility and can be shared among multiple processes. > > + > > +It can be used as the quick channel for accelerators, network adaptors or > > +other hardware in user space. It can make some implementation simpler. > > E.g. > > +you can reuse most of the *netdev* driver and just share some ring buffer > > to > > +the user space driver for *DPDK* or *ODP*. Or you can combine the RSA > > +accelerator with the *netdev* in the user space as a Web reversed proxy, > > etc. > > + > > +The name *WarpDrive* is simply a cool and general name meaning the > > framework > > +makes the application faster. In kernel, the framework is called SPIMDEV, > > +namely "Share Parent IOMMU
Re: [RFC PATCH 2/7] iommu: Add share domain interface in iommu for spimdev
On Thu, Aug 02, 2018 at 03:17:03AM +, Tian, Kevin wrote:
> Date: Thu, 2 Aug 2018 03:17:03 +
> From: "Tian, Kevin"
> To: Kenneth Lee , Jonathan Corbet ,
> Herbert Xu , "David S . Miller"
> , Joerg Roedel , Alex Williamson
> , Kenneth Lee , Hao
> Fang , Zhou Wang , Zaibo Xu
> , Philippe Ombredanne , Greg
> Kroah-Hartman , Thomas Gleixner
> , "linux-...@vger.kernel.org"
> , "linux-ker...@vger.kernel.org"
> , "linux-cry...@vger.kernel.org"
> , "iommu@lists.linux-foundation.org"
> , "k...@vger.kernel.org"
> , "linux-accelerat...@lists.ozlabs.org"
> , Lu Baolu
> , "Kumar, Sanjay K"
> CC: "linux...@huawei.com"
> Subject: RE: [RFC PATCH 2/7] iommu: Add share domain interface in iommu for
> spimdev
> Message-ID:
>
>
> > From: Kenneth Lee
> > Sent: Wednesday, August 1, 2018 6:22 PM
> >
> > From: Kenneth Lee
> >
> > This patch add sharing interface for a iommu_group. The new interface:
> >
> > iommu_group_share_domain()
> > iommu_group_unshare_domain()
> >
> > can be used by some virtual iommu_group (such as iommu_group for
> > spimdev)
> > to share their parent's iommu_group.
> >
> > When the domain of the group is shared, it cannot be changed before
> > unshared. In the future, notification can be added if update is required.
>
> Is it necessary or just asking VFIO to use parent domain directly?
>
Even we add to VFIO, the iommu still need to be changed. We can move the type1
part to VFIO if we have agreement in RFC stage.
> >
> > Signed-off-by: Kenneth Lee
> > ---
> > drivers/iommu/iommu.c | 28 +++-
> > include/linux/iommu.h | 2 ++
> > 2 files changed, 29 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> > index 63b37563db7e..a832aafe660d 100644
> > --- a/drivers/iommu/iommu.c
> > +++ b/drivers/iommu/iommu.c
> > @@ -54,6 +54,9 @@ struct iommu_group {
> > int id;
> > struct iommu_domain *default_domain;
> > struct iommu_domain *domain;
> > + atomic_t domain_shared_ref; /* Number of user of current domain.
> > +* The domain cannot be modified if ref >
> > 0
> > +*/
> > };
> >
> > struct group_device {
> > @@ -353,6 +356,7 @@ struct iommu_group *iommu_group_alloc(void)
> > return ERR_PTR(ret);
> > }
> > group->id = ret;
> > + atomic_set(>domain_shared_ref, 0);
> >
> > ret = kobject_init_and_add(>kobj, _group_ktype,
> >NULL, "%d", group->id);
> > @@ -482,6 +486,25 @@ int iommu_group_set_name(struct iommu_group
> > *group, const char *name)
> > }
> > EXPORT_SYMBOL_GPL(iommu_group_set_name);
> >
> > +struct iommu_domain *iommu_group_share_domain(struct
> > iommu_group *group)
> > +{
> > + /* the domain can be shared only when the default domain is used
> > */
> > + /* todo: more shareable check */
> > + if (group->domain != group->default_domain)
> > + return ERR_PTR(-EINVAL);
> > +
> > + atomic_inc(>domain_shared_ref);
> > + return group->domain;
> > +}
> > +EXPORT_SYMBOL_GPL(iommu_group_share_domain);
> > +
> > +void iommu_group_unshare_domain(struct iommu_group *group)
> > +{
> > + atomic_dec(>domain_shared_ref);
> > + WARN_ON(atomic_read(>domain_shared_ref) < 0);
> > +}
> > +EXPORT_SYMBOL_GPL(iommu_group_unshare_domain);
> > +
> > static int iommu_group_create_direct_mappings(struct iommu_group
> > *group,
> > struct device *dev)
> > {
> > @@ -1401,7 +1424,8 @@ static int __iommu_attach_group(struct
> > iommu_domain *domain,
> > {
> > int ret;
> >
> > - if (group->default_domain && group->domain != group-
> > >default_domain)
> > + if ((group->default_domain && group->domain != group-
> > >default_domain) ||
> > +atomic_read(>domain_shared_ref) > 0)
> > return -EBUSY;
> >
> > ret = __iommu_group_for_each_dev(group, domain,
> > @@ -1438,6 +1462,8 @@ static void __iommu_detach_group(struct
> > iommu_domain *domain,
> > {
> > int ret;
> >
> > + WARN_ON(atomic_read(>domain_shared_ref) > 0);
> > +
> >
Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive
On Thu, Aug 02, 2018 at 02:33:12AM +, Tian, Kevin wrote: > Date: Thu, 2 Aug 2018 02:33:12 + > From: "Tian, Kevin" > To: Jerome Glisse , Kenneth Lee > CC: Hao Fang , Herbert Xu > , "k...@vger.kernel.org" > , Jonathan Corbet , Greg > Kroah-Hartman , "linux-...@vger.kernel.org" > , "Kumar, Sanjay K" , > "iommu@lists.linux-foundation.org" , > "linux-ker...@vger.kernel.org" , > "linux...@huawei.com" , Alex Williamson > , Thomas Gleixner , > "linux-cry...@vger.kernel.org" , Philippe > Ombredanne , Zaibo Xu , Kenneth > Lee , "David S . Miller" , > "linux-accelerat...@lists.ozlabs.org" > > Subject: RE: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive > Message-ID: > > > > From: Jerome Glisse > > Sent: Thursday, August 2, 2018 12:57 AM > > > > On Wed, Aug 01, 2018 at 06:22:14PM +0800, Kenneth Lee wrote: > > > From: Kenneth Lee > > > > > > WarpDrive is an accelerator framework to expose the hardware > > capabilities > > > directly to the user space. It makes use of the exist vfio and vfio-mdev > > > facilities. So the user application can send request and DMA to the > > > hardware without interaction with the kernel. This remove the latency > > > of syscall and context switch. > > > > > > The patchset contains documents for the detail. Please refer to it for > > more > > > information. > > > > > > This patchset is intended to be used with Jean Philippe Brucker's SVA > > > patch [1] (Which is also in RFC stage). But it is not mandatory. This > > > patchset is tested in the latest mainline kernel without the SVA patches. > > > So it support only one process for each accelerator. > > > > > > With SVA support, WarpDrive can support multi-process in the same > > > accelerator device. We tested it in our SoC integrated Accelerator (board > > > ID: D06, Chip ID: HIP08). A reference work tree can be found here: [2]. > > > > I have not fully inspected things nor do i know enough about > > this Hisilicon ZIP accelerator to ascertain, but from glimpsing > > at the code it seems that it is unsafe to use even with SVA due > > to the doorbell. There is a comment talking about safetyness > > in patch 7. > > > > Exposing thing to userspace is always enticing, but if it is > > a security risk then it should clearly say so and maybe a > > kernel boot flag should be necessary to allow such device to > > be use. > > But doorbell is just a notification. Except for DOS (to make hardware busy) it cannot actually take or change anything from the kernel space. And the DOS problem can be always taken as the problem that a group of processes share the same kernel entity. In the coming HIP09 hardware, the doorbell will come with a random number so only the process who allocated the queue can knock it correctly. > > > > My more general question is do we want to grow VFIO to become > > a more generic device driver API. This patchset adds a command > > queue concept to it (i don't think it exist today but i have > > not follow VFIO closely). > > The thing is, VFIO is the only place to support DMA from user land. If we don't put it here, we have to create another similar facility to support the same. > > Why is that any better that existing driver model ? Where a > > device create a device file (can be character device, block > > device, ...). Such models also allow for direct hardware > > access from userspace. For instance see the AMD KFD driver > > inside drivers/gpu/drm/amd > > One motivation I guess, is that most accelerators lack of a > well-abstracted high level APIs similar to GPU side (e.g. OpenCL > clearly defines Shared Virtual Memory models). VFIO mdev > might be an alternative common interface to enable SVA usages > on various accelerators... > Yes. > > > > So you can already do what you are doing with the Hisilicon > > driver today without this new infrastructure. This only need > > hardware that have command queue and doorbell like mechanisms. > > > > > > Unlike mdev which unify a very high level concept, it seems > > to me spimdev just introduce low level concept (namely command > > queue) and i don't see the intrinsic value here. > > As I have said, VFIO is the only place support DMA from user space now. > > > > Cheers, > > Jérôme > > ___ > > iommu mailing list > > iommu@lists.linux-foundation.org > > ht
Re: [RFC PATCH 3/7] vfio: add spimdev support
On Thu, Aug 02, 2018 at 03:21:25AM +, Tian, Kevin wrote: > Date: Thu, 2 Aug 2018 03:21:25 + > From: "Tian, Kevin" > To: Kenneth Lee , Jonathan Corbet , > Herbert Xu , "David S . Miller" > , Joerg Roedel , Alex Williamson > , Kenneth Lee , Hao > Fang , Zhou Wang , Zaibo Xu > , Philippe Ombredanne , Greg > Kroah-Hartman , Thomas Gleixner > , "linux-...@vger.kernel.org" > , "linux-ker...@vger.kernel.org" > , "linux-cry...@vger.kernel.org" > , "iommu@lists.linux-foundation.org" > , "k...@vger.kernel.org" > , "linux-accelerat...@lists.ozlabs.org" > , Lu Baolu > , "Kumar, Sanjay K" > CC: "linux...@huawei.com" > Subject: RE: [RFC PATCH 3/7] vfio: add spimdev support > Message-ID: > > > > From: Kenneth Lee > > Sent: Wednesday, August 1, 2018 6:22 PM > > > > From: Kenneth Lee > > > > SPIMDEV is "Share Parent IOMMU Mdev". It is a vfio-mdev. But differ from > > the general vfio-mdev: > > > > 1. It shares its parent's IOMMU. > > 2. There is no hardware resource attached to the mdev is created. The > > hardware resource (A `queue') is allocated only when the mdev is > > opened. > > Alex has concern on doing so, as pointed out in: > > https://www.spinics.net/lists/kvm/msg172652.html > > resource allocation should be reserved at creation time. Yes. That is why I keep telling that SPIMDEV is not for "VM", it is for "many processes", it is just an access point to the process. Not a device to VM. I hope Alex can accept it:) > > > > > Currently only the vfio type-1 driver is updated to make it to be aware > > of. > > > > Signed-off-by: Kenneth Lee > > Signed-off-by: Zaibo Xu > > Signed-off-by: Zhou Wang > > --- > > drivers/vfio/Kconfig| 1 + > > drivers/vfio/Makefile | 1 + > > drivers/vfio/spimdev/Kconfig| 10 + > > drivers/vfio/spimdev/Makefile | 3 + > > drivers/vfio/spimdev/vfio_spimdev.c | 421 > > > > drivers/vfio/vfio_iommu_type1.c | 136 - > > include/linux/vfio_spimdev.h| 95 +++ > > include/uapi/linux/vfio_spimdev.h | 28 ++ > > 8 files changed, 689 insertions(+), 6 deletions(-) > > create mode 100644 drivers/vfio/spimdev/Kconfig > > create mode 100644 drivers/vfio/spimdev/Makefile > > create mode 100644 drivers/vfio/spimdev/vfio_spimdev.c > > create mode 100644 include/linux/vfio_spimdev.h > > create mode 100644 include/uapi/linux/vfio_spimdev.h > > > > diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig > > index c84333eb5eb5..3719eba72ef1 100644 > > --- a/drivers/vfio/Kconfig > > +++ b/drivers/vfio/Kconfig > > @@ -47,4 +47,5 @@ menuconfig VFIO_NOIOMMU > > source "drivers/vfio/pci/Kconfig" > > source "drivers/vfio/platform/Kconfig" > > source "drivers/vfio/mdev/Kconfig" > > +source "drivers/vfio/spimdev/Kconfig" > > source "virt/lib/Kconfig" > > diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile > > index de67c4725cce..28f3ef0cdce1 100644 > > --- a/drivers/vfio/Makefile > > +++ b/drivers/vfio/Makefile > > @@ -9,3 +9,4 @@ obj-$(CONFIG_VFIO_SPAPR_EEH) += vfio_spapr_eeh.o > > obj-$(CONFIG_VFIO_PCI) += pci/ > > obj-$(CONFIG_VFIO_PLATFORM) += platform/ > > obj-$(CONFIG_VFIO_MDEV) += mdev/ > > +obj-$(CONFIG_VFIO_SPIMDEV) += spimdev/ > > diff --git a/drivers/vfio/spimdev/Kconfig b/drivers/vfio/spimdev/Kconfig > > new file mode 100644 > > index ..1226301f9d0e > > --- /dev/null > > +++ b/drivers/vfio/spimdev/Kconfig > > @@ -0,0 +1,10 @@ > > +# SPDX-License-Identifier: GPL-2.0 > > +config VFIO_SPIMDEV > > + tristate "Support for Share Parent IOMMU MDEV" > > + depends on VFIO_MDEV_DEVICE > > + help > > + Support for VFIO Share Parent IOMMU MDEV, which enable the > > kernel to > > + support for the light weight hardware accelerator framework, > > WrapDrive. > > + > > + To compile this as a module, choose M here: the module will be > > called > > + spimdev. > > diff --git a/drivers/vfio/spimdev/Makefile b/drivers/vfio/spimdev/Makefile > > new file mode 100644 > > index ..d02fb69c37e4 > > --- /dev/null > > +++ b/drivers/vfio/spimdev/Makefile > > @@ -0,0 +1,3 @@ > > +# SPDX-License-Identifier: GPL-2.0 > > +spimdev-y := spimdev.o > &g
Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive
On Thu, Aug 02, 2018 at 02:59:33AM +, Tian, Kevin wrote: > Date: Thu, 2 Aug 2018 02:59:33 + > From: "Tian, Kevin" > To: Kenneth Lee , Jonathan Corbet , > Herbert Xu , "David S . Miller" > , Joerg Roedel , Alex Williamson > , Kenneth Lee , Hao > Fang , Zhou Wang , Zaibo Xu > , Philippe Ombredanne , Greg > Kroah-Hartman , Thomas Gleixner > , "linux-...@vger.kernel.org" > , "linux-ker...@vger.kernel.org" > , "linux-cry...@vger.kernel.org" > , "iommu@lists.linux-foundation.org" > , "k...@vger.kernel.org" > , "linux-accelerat...@lists.ozlabs.org" > , Lu Baolu > , "Kumar, Sanjay K" > CC: "linux...@huawei.com" > Subject: RE: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive > Message-ID: > > > > From: Kenneth Lee > > Sent: Wednesday, August 1, 2018 6:22 PM > > > > From: Kenneth Lee > > > > WarpDrive is an accelerator framework to expose the hardware capabilities > > directly to the user space. It makes use of the exist vfio and vfio-mdev > > facilities. So the user application can send request and DMA to the > > hardware without interaction with the kernel. This remove the latency > > of syscall and context switch. > > > > The patchset contains documents for the detail. Please refer to it for more > > information. > > > > This patchset is intended to be used with Jean Philippe Brucker's SVA > > patch [1] (Which is also in RFC stage). But it is not mandatory. This > > patchset is tested in the latest mainline kernel without the SVA patches. > > So it support only one process for each accelerator. > > If no sharing, then why not just assigning the whole parent device to > the process? IMO if SVA usage is the clear goal of your series, it > might be made clearly so then Jean's series is mandatory dependency... > We don't know how SVA will be finally. But the feature, "make use of per-PASID/substream ID IOMMU page table", should be able to be enabled in the kernel. So we don't want to enforce it here. After we have this serial ready, it can be hooked to any implementation. Further more, even without "per-PASID IOMMU page table", this series has its value. It is not simply dedicate the whole device to the process. It "shares" the device with the kernel driver. So you can support crypto and a user application at the same time. > > > > With SVA support, WarpDrive can support multi-process in the same > > accelerator device. We tested it in our SoC integrated Accelerator (board > > ID: D06, Chip ID: HIP08). A reference work tree can be found here: [2]. > > > > We have noticed the IOMMU aware mdev RFC announced recently [3]. > > > > The IOMMU aware mdev has similar idea but different intention comparing > > to > > WarpDrive. It intends to dedicate part of the hardware resource to a VM. > > Not just to VM, though I/O Virtualization is in the name. You can assign > such mdev to either VMs, containers, or bare metal processes. It's just > a fully-isolated device from user space p.o.v. Oh, yes. Thank you for clarification. > > > And the design is supposed to be used with Scalable I/O Virtualization. > > While spimdev is intended to share the hardware resource with a big > > amount > > of processes. It just requires the hardware supporting address > > translation per process (PCIE's PASID or ARM SMMU's substream ID). > > > > But we don't see serious confliction on both design. We believe they can be > > normalized as one. > > yes there are something which can be shared, e.g. regarding to > the interface to IOMMU. > > Conceptually I see them different mindset on device resource sharing: > > WarpDrive more aims to provide a generic framework to enable SVA > usages on various accelerators, which lack of a well-abstracted user > API like OpenCL. SVA is a hardware capability - sort of exposing resources > composing ONE capability to user space through mdev framework. It is > not like a VF which naturally carries most capabilities as PF. > Yes. But we believe the user abstraction layer will be enabled soon when the channel is opened. WarpDrive gives the hardware the chance to serve the application directly. For example, an AI engine can be called by many processes for inference. The resource need not to be dedicated to one particular process. > Intel Scalable I/O virtualization is a thorough design to partition the > device into minimal sharable copies (queue, queue pair, context), > while each copy carries most PF capabilities (including SVA) similar to > VF. Also with IOMMU scalable mode support, the copy can be > independently assigned to any client (process, container, VM, etc.) > Yes, we can see this intension. > Thanks > Kevin Thank you. -- -Kenneth(Hisilicon) ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 3/7] vfio: add spimdev support
On Wed, Aug 01, 2018 at 09:23:52AM -0700, Randy Dunlap wrote: > Date: Wed, 1 Aug 2018 09:23:52 -0700 > From: Randy Dunlap > To: Kenneth Lee , Jonathan Corbet , > Herbert Xu , "David S . Miller" > , Joerg Roedel , Alex Williamson > , Kenneth Lee , Hao > Fang , Zhou Wang , Zaibo Xu > , Philippe Ombredanne , Greg > Kroah-Hartman , Thomas Gleixner > , linux-...@vger.kernel.org, > linux-ker...@vger.kernel.org, linux-cry...@vger.kernel.org, > iommu@lists.linux-foundation.org, k...@vger.kernel.org, > linux-accelerat...@lists.ozlabs.org, Lu Baolu , > Sanjay Kumar > CC: linux...@huawei.com > Subject: Re: [RFC PATCH 3/7] vfio: add spimdev support > User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 > Thunderbird/52.9.1 > Message-ID: > > On 08/01/2018 03:22 AM, Kenneth Lee wrote: > > From: Kenneth Lee > > > > SPIMDEV is "Share Parent IOMMU Mdev". It is a vfio-mdev. But differ from > > the general vfio-mdev: > > > > 1. It shares its parent's IOMMU. > > 2. There is no hardware resource attached to the mdev is created. The > > hardware resource (A `queue') is allocated only when the mdev is > > opened. > > > > Currently only the vfio type-1 driver is updated to make it to be aware > > of. > > > > Signed-off-by: Kenneth Lee > > Signed-off-by: Zaibo Xu > > Signed-off-by: Zhou Wang > > --- > > drivers/vfio/Kconfig| 1 + > > drivers/vfio/Makefile | 1 + > > drivers/vfio/spimdev/Kconfig| 10 + > > drivers/vfio/spimdev/Makefile | 3 + > > drivers/vfio/spimdev/vfio_spimdev.c | 421 > > drivers/vfio/vfio_iommu_type1.c | 136 - > > include/linux/vfio_spimdev.h| 95 +++ > > include/uapi/linux/vfio_spimdev.h | 28 ++ > > 8 files changed, 689 insertions(+), 6 deletions(-) > > create mode 100644 drivers/vfio/spimdev/Kconfig > > create mode 100644 drivers/vfio/spimdev/Makefile > > create mode 100644 drivers/vfio/spimdev/vfio_spimdev.c > > create mode 100644 include/linux/vfio_spimdev.h > > create mode 100644 include/uapi/linux/vfio_spimdev.h > > > > diff --git a/drivers/vfio/spimdev/Kconfig b/drivers/vfio/spimdev/Kconfig > > new file mode 100644 > > index ..1226301f9d0e > > --- /dev/null > > +++ b/drivers/vfio/spimdev/Kconfig > > @@ -0,0 +1,10 @@ > > +# SPDX-License-Identifier: GPL-2.0 > > +config VFIO_SPIMDEV > > + tristate "Support for Share Parent IOMMU MDEV" > > + depends on VFIO_MDEV_DEVICE > > + help > > + Support for VFIO Share Parent IOMMU MDEV, which enable the kernel to > > enables > > > + support for the light weight hardware accelerator framework, > > WrapDrive. > > support the lightweight hardware accelerator framework, WrapDrive. > > > + > > + To compile this as a module, choose M here: the module will be called > > + spimdev. > > > -- > ~Randy Thanks, I will update it in next version. -- -Kenneth(Hisilicon) 本邮件及其附件含有华为公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁 止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中 的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
[RFC PATCH 7/7] vfio/spimdev: add user sample for spimdev
From: Kenneth Lee This is the sample code to demostrate how WrapDrive user application should be. It contains: 1. wd.[ch], the common library to provide WrapDrive interface. 2. drv/*, the user driver to access the hardware upon spimdev 3. test/*, the test application to use WrapDrive interface to access the hardware queue(s) of the accelerator. The Hisilicon HIP08 ZIP accelerator is used in this sample. Signed-off-by: Zaibo Xu Signed-off-by: Kenneth Lee Signed-off-by: Hao Fang Signed-off-by: Zhou Wang --- samples/warpdrive/AUTHORS | 2 + samples/warpdrive/ChangeLog| 1 + samples/warpdrive/Makefile.am | 9 + samples/warpdrive/NEWS | 1 + samples/warpdrive/README | 32 +++ samples/warpdrive/autogen.sh | 3 + samples/warpdrive/cleanup.sh | 13 + samples/warpdrive/configure.ac | 52 samples/warpdrive/drv/hisi_qm_udrv.c | 223 + samples/warpdrive/drv/hisi_qm_udrv.h | 53 samples/warpdrive/test/Makefile.am | 7 + samples/warpdrive/test/comp_hw.h | 23 ++ samples/warpdrive/test/test_hisi_zip.c | 204 samples/warpdrive/wd.c | 325 + samples/warpdrive/wd.h | 153 samples/warpdrive/wd_adapter.c | 74 ++ samples/warpdrive/wd_adapter.h | 43 17 files changed, 1218 insertions(+) create mode 100644 samples/warpdrive/AUTHORS create mode 100644 samples/warpdrive/ChangeLog create mode 100644 samples/warpdrive/Makefile.am create mode 100644 samples/warpdrive/NEWS create mode 100644 samples/warpdrive/README create mode 100755 samples/warpdrive/autogen.sh create mode 100755 samples/warpdrive/cleanup.sh create mode 100644 samples/warpdrive/configure.ac create mode 100644 samples/warpdrive/drv/hisi_qm_udrv.c create mode 100644 samples/warpdrive/drv/hisi_qm_udrv.h create mode 100644 samples/warpdrive/test/Makefile.am create mode 100644 samples/warpdrive/test/comp_hw.h create mode 100644 samples/warpdrive/test/test_hisi_zip.c create mode 100644 samples/warpdrive/wd.c create mode 100644 samples/warpdrive/wd.h create mode 100644 samples/warpdrive/wd_adapter.c create mode 100644 samples/warpdrive/wd_adapter.h diff --git a/samples/warpdrive/AUTHORS b/samples/warpdrive/AUTHORS new file mode 100644 index ..fe7dc2413b0d --- /dev/null +++ b/samples/warpdrive/AUTHORS @@ -0,0 +1,2 @@ +Kenneth Lee +Zaibo Xu diff --git a/samples/warpdrive/ChangeLog b/samples/warpdrive/ChangeLog new file mode 100644 index ..b1b716105590 --- /dev/null +++ b/samples/warpdrive/ChangeLog @@ -0,0 +1 @@ +init diff --git a/samples/warpdrive/Makefile.am b/samples/warpdrive/Makefile.am new file mode 100644 index ..41154a880a97 --- /dev/null +++ b/samples/warpdrive/Makefile.am @@ -0,0 +1,9 @@ +ACLOCAL_AMFLAGS = -I m4 +AUTOMAKE_OPTIONS = foreign subdir-objects +AM_CFLAGS=-Wall -O0 -fno-strict-aliasing + +lib_LTLIBRARIES=libwd.la +libwd_la_SOURCES=wd.c wd_adapter.c wd.h wd_adapter.h \ +drv/hisi_qm_udrv.c drv/hisi_qm_udrv.h + +SUBDIRS=. test diff --git a/samples/warpdrive/NEWS b/samples/warpdrive/NEWS new file mode 100644 index ..b1b716105590 --- /dev/null +++ b/samples/warpdrive/NEWS @@ -0,0 +1 @@ +init diff --git a/samples/warpdrive/README b/samples/warpdrive/README new file mode 100644 index ..3adf66b112fc --- /dev/null +++ b/samples/warpdrive/README @@ -0,0 +1,32 @@ +WD User Land Demonstration +== + +This directory contains some applications and libraries to demonstrate how a + +WrapDrive application can be constructed. + + +As a demo, we try to make it simple and clear for understanding. It is not + +supposed to be used in business scenario. + + +The directory contains the following elements: + +wd.[ch] + A demonstration WrapDrive fundamental library which wraps the basic + operations to the WrapDrive-ed device. + +wd_adapter.[ch] + User driver adaptor for wd.[ch] + +wd_utils.[ch] + Some utitlities function used by WD and its drivers + +drv/* + User drivers. It helps to fulfill the semantic of wd.[ch] for + particular hardware + +test/* + Test applications to use the wrapdrive library + diff --git a/samples/warpdrive/autogen.sh b/samples/warpdrive/autogen.sh new file mode 100755 index ..58deaf49de2a --- /dev/null +++ b/samples/warpdrive/autogen.sh @@ -0,0 +1,3 @@ +#!/bin/sh -x + +autoreconf -i -f -v diff --git a/samples/warpdrive/cleanup.sh b/samples/warpdrive/cleanup.sh new file mode 100755 index ..c5f3d21e5dc1 --- /dev/null +++ b/samples/warpdrive/cleanup.sh @@ -0,0 +1,13 @@ +#!/bin/sh + +if [ -r Makefile ]; then + make distclean +fi + +FILES="aclocal.m4 autom4te.cache compile config.guess config.h.in config.log \ + config.status config.sub configure cscope.out depcomp install-sh \ + l
[RFC PATCH 6/7] crypto: add spimdev support to Hisilicon QM
From: Kenneth Lee
The commit add spimdev support to the Hislicon QM driver, any
accelerator that use QM can share its queues to the user space.
Signed-off-by: Kenneth Lee
Signed-off-by: Zhou Wang
Signed-off-by: Hao Fang
Signed-off-by: Zaibo Xu
---
drivers/crypto/hisilicon/qm.c | 150 ++
drivers/crypto/hisilicon/qm.h | 12 +++
2 files changed, 162 insertions(+)
diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
index e779bc661500..06da8387dc58 100644
--- a/drivers/crypto/hisilicon/qm.c
+++ b/drivers/crypto/hisilicon/qm.c
@@ -667,6 +667,146 @@ int hisi_qp_send(struct hisi_qp *qp, void *msg)
}
EXPORT_SYMBOL_GPL(hisi_qp_send);
+#ifdef CONFIG_CRYPTO_DEV_HISI_SPIMDEV
+/* mdev->supported_type_groups */
+static struct attribute *hisi_qm_type_attrs[] = {
+ VFIO_SPIMDEV_DEFAULT_MDEV_TYPE_ATTRS,
+ NULL,
+};
+static struct attribute_group hisi_qm_type_group = {
+ .attrs = hisi_qm_type_attrs,
+};
+static struct attribute_group *mdev_type_groups[] = {
+ _qm_type_group,
+ NULL,
+};
+
+static void qm_qp_event_notifier(struct hisi_qp *qp)
+{
+ vfio_spimdev_wake_up(qp->spimdev_q);
+}
+
+static int hisi_qm_get_queue(struct vfio_spimdev *spimdev, unsigned long arg,
+ struct vfio_spimdev_queue **q)
+{
+ struct qm_info *qm = spimdev->priv;
+ struct hisi_qp *qp = NULL;
+ struct vfio_spimdev_queue *wd_q;
+ u8 alg_type = 0; /* fix me here */
+ int ret;
+ int pasid = arg;
+
+ qp = hisi_qm_create_qp(qm, alg_type);
+ if (IS_ERR(qp))
+ return PTR_ERR(qp);
+
+ wd_q = kzalloc(sizeof(struct vfio_spimdev_queue), GFP_KERNEL);
+ if (!wd_q) {
+ ret = -ENOMEM;
+ goto err_with_qp;
+ }
+
+ wd_q->priv = qp;
+ wd_q->spimdev = spimdev;
+ wd_q->qid = (u16)ret;
+ *q = wd_q;
+ qp->spimdev_q = wd_q;
+ qp->event_cb = qm_qp_event_notifier;
+
+ ret = hisi_qm_start_qp(qp, arg);
+ if (ret < 0)
+ goto err_with_wd_q;
+
+ return ret;
+
+err_with_wd_q:
+ kfree(wd_q);
+err_with_qp:
+ hisi_qm_release_qp(qp);
+ return ret;
+}
+
+static int hisi_qm_put_queue(struct vfio_spimdev_queue *q)
+{
+ struct hisi_qp *qp = q->priv;
+
+ /* need to stop hardware, but can not support in v1 */
+ hisi_qm_release_qp(qp);
+ kfree(q);
+ return 0;
+}
+
+/* map sq/cq/doorbell to user space */
+static int hisi_qm_mmap(struct vfio_spimdev_queue *q,
+ struct vm_area_struct *vma)
+{
+ struct hisi_qp *qp = (struct hisi_qp *)q->priv;
+ struct qm_info *qm = qp->qm;
+ struct device *dev = >pdev->dev;
+ size_t sz = vma->vm_end - vma->vm_start;
+ u8 region;
+
+ vma->vm_flags |= (VM_IO | VM_LOCKED | VM_DONTEXPAND | VM_DONTDUMP);
+ region = _VFIO_SPIMDEV_REGION(vma->vm_pgoff);
+
+ switch (region) {
+ case 0:
+ if (sz > PAGE_SIZE)
+ return -EINVAL;
+ /*
+* Warning: This is not safe as multiple queues use the same
+* doorbell, v1 hardware interface problem. v2 will fix it
+*/
+ return remap_pfn_range(vma, vma->vm_start,
+ qm->phys_base >> PAGE_SHIFT,
+ sz, pgprot_noncached(vma->vm_page_prot));
+ case 1:
+ vma->vm_pgoff = 0;
+ if (sz > qp->scqe.size)
+ return -EINVAL;
+
+ return dma_mmap_coherent(dev, vma, qp->scqe.addr, qp->scqe.dma,
+ sz);
+
+ default:
+ return -EINVAL;
+ }
+}
+
+static const struct vfio_spimdev_ops qm_ops = {
+ .get_queue = hisi_qm_get_queue,
+ .put_queue = hisi_qm_put_queue,
+ .mmap = hisi_qm_mmap,
+};
+
+static int qm_register_spimdev(struct qm_info *qm)
+{
+ struct pci_dev *pdev = qm->pdev;
+ struct vfio_spimdev *spimdev = >spimdev;
+
+ spimdev->iommu_type = VFIO_TYPE1_IOMMU;
+#ifdef CONFIG_IOMMU_SVA
+ spimdev->dma_flag = VFIO_SPIMDEV_DMA_MULTI_PROC_MAP;
+#else
+ spimdev->dma_flag = VFIO_SPIMDEV_DMA_SINGLE_PROC_MAP;
+#endif
+ spimdev->owner = THIS_MODULE;
+ spimdev->name = qm->dev_name;
+ spimdev->dev = >dev;
+ spimdev->is_vf = pdev->is_virtfn;
+ spimdev->priv = qm;
+ spimdev->api_ver = "hisi_qm_v1";
+ spimdev->flags = 0;
+
+ spimdev->mdev_fops.mdev_attr_groups = qm->mdev_dev_groups;
+ hisi_qm_type_group.name = qm->dev_name;
+ spimdev->mdev_fops.supported_type_groups = mdev_type_groups;
+ spimdev->ops = _ops;
+
+ return vfio_spimdev_register(spimdev);
+}
+#endif
[RFC PATCH 5/7] crypto: Add Hisilicon Zip driver
From: Kenneth Lee
The Hisilicon ZIP accelerator implements zlib and gzip algorithm support
for the software. It uses Hisilicon QM as the interface to the CPU, so it
is shown up as a PCIE device to the CPU with a group of queues.
This commit provides PCIE driver to the accelerator and register it to
the crypto subsystem.
Signed-off-by: Kenneth Lee
Signed-off-by: Zhou Wang
Signed-off-by: Hao Fang
---
drivers/crypto/hisilicon/Kconfig | 7 +
drivers/crypto/hisilicon/Makefile | 1 +
drivers/crypto/hisilicon/zip/Makefile | 2 +
drivers/crypto/hisilicon/zip/zip.h| 55
drivers/crypto/hisilicon/zip/zip_crypto.c | 358 ++
drivers/crypto/hisilicon/zip/zip_crypto.h | 18 ++
drivers/crypto/hisilicon/zip/zip_main.c | 182 +++
7 files changed, 623 insertions(+)
create mode 100644 drivers/crypto/hisilicon/zip/Makefile
create mode 100644 drivers/crypto/hisilicon/zip/zip.h
create mode 100644 drivers/crypto/hisilicon/zip/zip_crypto.c
create mode 100644 drivers/crypto/hisilicon/zip/zip_crypto.h
create mode 100644 drivers/crypto/hisilicon/zip/zip_main.c
diff --git a/drivers/crypto/hisilicon/Kconfig b/drivers/crypto/hisilicon/Kconfig
index 0dd30f84b90e..48ad682e0a52 100644
--- a/drivers/crypto/hisilicon/Kconfig
+++ b/drivers/crypto/hisilicon/Kconfig
@@ -6,3 +6,10 @@ config CRYPTO_DEV_HISILICON
config CRYPTO_DEV_HISI_QM
tristate
depends on ARM64 && PCI
+
+config CRYPTO_DEV_HISI_ZIP
+ tristate "Support for HISI ZIP Driver"
+ depends on ARM64 && CRYPTO_DEV_HISILICON
+ select CRYPTO_DEV_HISI_QM
+ help
+ Support for HiSilicon HIP08 ZIP Driver
diff --git a/drivers/crypto/hisilicon/Makefile
b/drivers/crypto/hisilicon/Makefile
index 3378afc11703..62e40b093c49 100644
--- a/drivers/crypto/hisilicon/Makefile
+++ b/drivers/crypto/hisilicon/Makefile
@@ -1 +1,2 @@
obj-$(CONFIG_CRYPTO_DEV_HISI_QM) += qm.o
+obj-$(CONFIG_CRYPTO_DEV_HISI_ZIP) += zip/
diff --git a/drivers/crypto/hisilicon/zip/Makefile
b/drivers/crypto/hisilicon/zip/Makefile
new file mode 100644
index ..a936f099ee22
--- /dev/null
+++ b/drivers/crypto/hisilicon/zip/Makefile
@@ -0,0 +1,2 @@
+obj-$(CONFIG_CRYPTO_DEV_HISI_ZIP) += hisi_zip.o
+hisi_zip-objs = zip_main.o zip_crypto.o
diff --git a/drivers/crypto/hisilicon/zip/zip.h
b/drivers/crypto/hisilicon/zip/zip.h
new file mode 100644
index ..a0c56e4aeb51
--- /dev/null
+++ b/drivers/crypto/hisilicon/zip/zip.h
@@ -0,0 +1,55 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+#ifndef HISI_ZIP_H
+#define HISI_ZIP_H
+
+#include
+#include "../qm.h"
+
+#define HZIP_SQE_SIZE 128
+#define HZIP_SQ_SIZE (HZIP_SQE_SIZE * QM_Q_DEPTH)
+#define QM_CQ_SIZE (QM_CQE_SIZE * QM_Q_DEPTH)
+#define HZIP_PF_DEF_Q_NUM 64
+#define HZIP_PF_DEF_Q_BASE 0
+
+struct hisi_zip {
+ struct qm_info qm;
+ struct list_head list;
+
+#ifdef CONFIG_CRYPTO_DEV_HISI_SPIMDEV
+ struct vfio_spimdev *spimdev;
+#endif
+};
+
+struct hisi_zip_sqe {
+ __u32 consumed;
+ __u32 produced;
+ __u32 comp_data_length;
+ __u32 dw3;
+ __u32 input_data_length;
+ __u32 lba_l;
+ __u32 lba_h;
+ __u32 dw7;
+ __u32 dw8;
+ __u32 dw9;
+ __u32 dw10;
+ __u32 priv_info;
+ __u32 dw12;
+ __u32 tag;
+ __u32 dest_avail_out;
+ __u32 rsvd0;
+ __u32 comp_head_addr_l;
+ __u32 comp_head_addr_h;
+ __u32 source_addr_l;
+ __u32 source_addr_h;
+ __u32 dest_addr_l;
+ __u32 dest_addr_h;
+ __u32 stream_ctx_addr_l;
+ __u32 stream_ctx_addr_h;
+ __u32 cipher_key1_addr_l;
+ __u32 cipher_key1_addr_h;
+ __u32 cipher_key2_addr_l;
+ __u32 cipher_key2_addr_h;
+ __u32 rsvd1[4];
+};
+
+#endif
diff --git a/drivers/crypto/hisilicon/zip/zip_crypto.c
b/drivers/crypto/hisilicon/zip/zip_crypto.c
new file mode 100644
index ..a3a5a6a6554d
--- /dev/null
+++ b/drivers/crypto/hisilicon/zip/zip_crypto.c
@@ -0,0 +1,358 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright 2018 (c) HiSilicon Limited.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include
+#include
+#include
+#include
+#include "../qm.h"
+#include "zip.h"
+
+#define INPUT_BUFFER_SIZE (64 * 1024)
+#define OUTPUT_BUFFER_SIZE (64 * 1024)
+
+extern struct list_head hisi_zip_list;
+
+#define COMP_NAME_TO_TYPE(alg_name)\
+ (!strcmp((alg_name), "zlib-deflate") ? 0x02 : \
+!strcmp((alg_name), "gzip") ? 0x03 : 0)\
+
+struct hisi_zip_buffer {
+ u8 *input;
+ dma_addr_t input_dma;
+
[RFC PATCH 4/7] crypto: add hisilicon Queue Manager driver
From: Kenneth Lee Hisilicon QM is a general IP used by some Hisilicon accelerators. It provides a general PCIE interface for the CPU and the accelerator to share a group of queues. This commit includes a library used by the accelerator driver to access the QM hardware. Signed-off-by: Kenneth Lee Signed-off-by: Zhou Wang Signed-off-by: Hao Fang --- drivers/crypto/Kconfig| 2 + drivers/crypto/Makefile | 1 + drivers/crypto/hisilicon/Kconfig | 8 + drivers/crypto/hisilicon/Makefile | 1 + drivers/crypto/hisilicon/qm.c | 855 ++ drivers/crypto/hisilicon/qm.h | 111 6 files changed, 978 insertions(+) create mode 100644 drivers/crypto/hisilicon/Kconfig create mode 100644 drivers/crypto/hisilicon/Makefile create mode 100644 drivers/crypto/hisilicon/qm.c create mode 100644 drivers/crypto/hisilicon/qm.h diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig index 43cccf6aff61..8da1e3170eb4 100644 --- a/drivers/crypto/Kconfig +++ b/drivers/crypto/Kconfig @@ -746,4 +746,6 @@ config CRYPTO_DEV_CCREE cryptographic operations on the system REE. If unsure say Y. +source "drivers/crypto/hisilicon/Kconfig" + endif # CRYPTO_HW diff --git a/drivers/crypto/Makefile b/drivers/crypto/Makefile index 7ae87b4f6c8d..32e9bf64a42f 100644 --- a/drivers/crypto/Makefile +++ b/drivers/crypto/Makefile @@ -45,3 +45,4 @@ obj-$(CONFIG_CRYPTO_DEV_VMX) += vmx/ obj-$(CONFIG_CRYPTO_DEV_BCM_SPU) += bcm/ obj-$(CONFIG_CRYPTO_DEV_SAFEXCEL) += inside-secure/ obj-$(CONFIG_CRYPTO_DEV_ARTPEC6) += axis/ +obj-$(CONFIG_CRYPTO_DEV_HISILICON) += hisilicon/ diff --git a/drivers/crypto/hisilicon/Kconfig b/drivers/crypto/hisilicon/Kconfig new file mode 100644 index ..0dd30f84b90e --- /dev/null +++ b/drivers/crypto/hisilicon/Kconfig @@ -0,0 +1,8 @@ +config CRYPTO_DEV_HISILICON + tristate "Support for HISILICON CRYPTO ACCELERATOR" + help + Enable this to use Hisilicon Hardware Accelerators + +config CRYPTO_DEV_HISI_QM + tristate + depends on ARM64 && PCI diff --git a/drivers/crypto/hisilicon/Makefile b/drivers/crypto/hisilicon/Makefile new file mode 100644 index ..3378afc11703 --- /dev/null +++ b/drivers/crypto/hisilicon/Makefile @@ -0,0 +1 @@ +obj-$(CONFIG_CRYPTO_DEV_HISI_QM) += qm.o diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c new file mode 100644 index ..e779bc661500 --- /dev/null +++ b/drivers/crypto/hisilicon/qm.c @@ -0,0 +1,855 @@ +// SPDX-License-Identifier: GPL-2.0+ +#include +#include +#include +#include +#include +#include +#include "qm.h" + +#define QM_DEF_Q_NUM 128 + +/* eq/aeq irq enable */ +#define QM_VF_AEQ_INT_SOURCE 0x0 +#define QM_VF_AEQ_INT_MASK 0x4 +#define QM_VF_EQ_INT_SOURCE0x8 +#define QM_VF_EQ_INT_MASK 0xc + +/* mailbox */ +#define MAILBOX_CMD_SQC0x0 +#define MAILBOX_CMD_CQC0x1 +#define MAILBOX_CMD_EQC0x2 +#define MAILBOX_CMD_AEQC 0x3 +#define MAILBOX_CMD_SQC_BT 0x4 +#define MAILBOX_CMD_CQC_BT 0x5 + +#define MAILBOX_CMD_SEND_BASE 0x300 +#define MAILBOX_EVENT_SHIFT8 +#define MAILBOX_STATUS_SHIFT 9 +#define MAILBOX_BUSY_SHIFT 13 +#define MAILBOX_OP_SHIFT 14 +#define MAILBOX_QUEUE_SHIFT16 + +/* sqc shift */ +#define SQ_HEAD_SHIFT 0 +#define SQ_TAIL_SHIFI 16 +#define SQ_HOP_NUM_SHIFT 0 +#define SQ_PAGE_SIZE_SHIFT 4 +#define SQ_BUF_SIZE_SHIFT 8 +#define SQ_SQE_SIZE_SHIFT 12 +#define SQ_HEAD_IDX_SIG_SHIFT 0 +#define SQ_TAIL_IDX_SIG_SHIFT 0 +#define SQ_CQN_SHIFT 0 +#define SQ_PRIORITY_SHIFT 0 +#define SQ_ORDERS_SHIFT4 +#define SQ_TYPE_SHIFT 8 + +#define SQ_TYPE_MASK 0xf + +/* cqc shift */ +#define CQ_HEAD_SHIFT 0 +#define CQ_TAIL_SHIFI 16 +#define CQ_HOP_NUM_SHIFT 0 +#define CQ_PAGE_SIZE_SHIFT 4 +#define CQ_BUF_SIZE_SHIFT 8 +#define CQ_SQE_SIZE_SHIFT 12 +#define CQ_PASID 0 +#define CQ_HEAD_IDX_SIG_SHIFT 0 +#define CQ_TAIL_IDX_SIG_SHIFT 0 +#define CQ_CQN_SHIFT 0 +#define CQ_PRIORITY_SHIFT 16 +#define CQ_ORDERS_SHIFT0 +#define CQ_TYPE_SHIFT 0 +#define CQ_PHASE_SHIFT 0 +#define CQ_FLAG_SHIFT 1 + +#define CQC_HEAD_INDEX(cqc)((cqc)->cq_head) +#define CQC_PHASE(cqc) (((cqc)->dw6) & 0x1) +#define CQC_CQ_ADDRESS(cqc)(((u64)((cqc)->cq_base_h) << 32) | \ +
[RFC PATCH 3/7] vfio: add spimdev support
From: Kenneth Lee
SPIMDEV is "Share Parent IOMMU Mdev". It is a vfio-mdev. But differ from
the general vfio-mdev:
1. It shares its parent's IOMMU.
2. There is no hardware resource attached to the mdev is created. The
hardware resource (A `queue') is allocated only when the mdev is
opened.
Currently only the vfio type-1 driver is updated to make it to be aware
of.
Signed-off-by: Kenneth Lee
Signed-off-by: Zaibo Xu
Signed-off-by: Zhou Wang
---
drivers/vfio/Kconfig| 1 +
drivers/vfio/Makefile | 1 +
drivers/vfio/spimdev/Kconfig| 10 +
drivers/vfio/spimdev/Makefile | 3 +
drivers/vfio/spimdev/vfio_spimdev.c | 421
drivers/vfio/vfio_iommu_type1.c | 136 -
include/linux/vfio_spimdev.h| 95 +++
include/uapi/linux/vfio_spimdev.h | 28 ++
8 files changed, 689 insertions(+), 6 deletions(-)
create mode 100644 drivers/vfio/spimdev/Kconfig
create mode 100644 drivers/vfio/spimdev/Makefile
create mode 100644 drivers/vfio/spimdev/vfio_spimdev.c
create mode 100644 include/linux/vfio_spimdev.h
create mode 100644 include/uapi/linux/vfio_spimdev.h
diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
index c84333eb5eb5..3719eba72ef1 100644
--- a/drivers/vfio/Kconfig
+++ b/drivers/vfio/Kconfig
@@ -47,4 +47,5 @@ menuconfig VFIO_NOIOMMU
source "drivers/vfio/pci/Kconfig"
source "drivers/vfio/platform/Kconfig"
source "drivers/vfio/mdev/Kconfig"
+source "drivers/vfio/spimdev/Kconfig"
source "virt/lib/Kconfig"
diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile
index de67c4725cce..28f3ef0cdce1 100644
--- a/drivers/vfio/Makefile
+++ b/drivers/vfio/Makefile
@@ -9,3 +9,4 @@ obj-$(CONFIG_VFIO_SPAPR_EEH) += vfio_spapr_eeh.o
obj-$(CONFIG_VFIO_PCI) += pci/
obj-$(CONFIG_VFIO_PLATFORM) += platform/
obj-$(CONFIG_VFIO_MDEV) += mdev/
+obj-$(CONFIG_VFIO_SPIMDEV) += spimdev/
diff --git a/drivers/vfio/spimdev/Kconfig b/drivers/vfio/spimdev/Kconfig
new file mode 100644
index ..1226301f9d0e
--- /dev/null
+++ b/drivers/vfio/spimdev/Kconfig
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0
+config VFIO_SPIMDEV
+ tristate "Support for Share Parent IOMMU MDEV"
+ depends on VFIO_MDEV_DEVICE
+ help
+ Support for VFIO Share Parent IOMMU MDEV, which enable the kernel to
+ support for the light weight hardware accelerator framework,
WrapDrive.
+
+ To compile this as a module, choose M here: the module will be called
+ spimdev.
diff --git a/drivers/vfio/spimdev/Makefile b/drivers/vfio/spimdev/Makefile
new file mode 100644
index ..d02fb69c37e4
--- /dev/null
+++ b/drivers/vfio/spimdev/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+spimdev-y := spimdev.o
+obj-$(CONFIG_VFIO_SPIMDEV) += vfio_spimdev.o
diff --git a/drivers/vfio/spimdev/vfio_spimdev.c
b/drivers/vfio/spimdev/vfio_spimdev.c
new file mode 100644
index ..1b6910c9d27d
--- /dev/null
+++ b/drivers/vfio/spimdev/vfio_spimdev.c
@@ -0,0 +1,421 @@
+// SPDX-License-Identifier: GPL-2.0+
+#include
+#include
+#include
+#include
+#include
+
+struct spimdev_mdev_state {
+ struct vfio_spimdev *spimdev;
+};
+
+static struct class *spimdev_class;
+static DEFINE_IDR(spimdev_idr);
+
+static int vfio_spimdev_dev_exist(struct device *dev, void *data)
+{
+ return !strcmp(dev_name(dev), dev_name((struct device *)data));
+}
+
+#ifdef CONFIG_IOMMU_SVA
+static bool vfio_spimdev_is_valid_pasid(int pasid)
+{
+ struct mm_struct *mm;
+
+ mm = iommu_sva_find(pasid);
+ if (mm) {
+ mmput(mm);
+ return mm == current->mm;
+ }
+
+ return false;
+}
+#endif
+
+/* Check if the device is a mediated device belongs to vfio_spimdev */
+int vfio_spimdev_is_spimdev(struct device *dev)
+{
+ struct mdev_device *mdev;
+ struct device *pdev;
+
+ mdev = mdev_from_dev(dev);
+ if (!mdev)
+ return 0;
+
+ pdev = mdev_parent_dev(mdev);
+ if (!pdev)
+ return 0;
+
+ return class_for_each_device(spimdev_class, NULL, pdev,
+ vfio_spimdev_dev_exist);
+}
+EXPORT_SYMBOL_GPL(vfio_spimdev_is_spimdev);
+
+struct vfio_spimdev *vfio_spimdev_pdev_spimdev(struct device *dev)
+{
+ struct device *class_dev;
+
+ if (!dev)
+ return ERR_PTR(-EINVAL);
+
+ class_dev = class_find_device(spimdev_class, NULL, dev,
+ (int(*)(struct device *, const void *))vfio_spimdev_dev_exist);
+ if (!class_dev)
+ return ERR_PTR(-ENODEV);
+
+ return container_of(class_dev, struct vfio_spimdev, cls_dev);
+}
+EXPORT_SYMBOL_GPL(vfio_spimdev_pdev_spimdev);
+
+struct vfio_spimdev *mdev_spimdev(struct mdev_device *mdev)
+{
+ struct device *pdev = mdev_parent_dev(mdev);
+
+ return vfio_spimdev_pdev_spimdev(pdev);
+}
+EXPORT_SYMBO
[RFC PATCH 1/7] vfio/spimdev: Add documents for WarpDrive framework
From: Kenneth Lee WarpDrive is a common user space accelerator framework. Its main component in Kernel is called spimdev, Share Parent IOMMU Mediated Device. It exposes the hardware capabilities to the user space via vfio-mdev. So processes in user land can obtain a "queue" by open the device and direct access the hardware MMIO space or do DMA operation via VFIO interface. WarpDrive is intended to be used with Jean Philippe Brucker's SVA patchset (it is still in RFC stage) to support multi-process. But This is not a must. Without the SVA patches, WarpDrive can still work for one process for every hardware device. This patch add detail documents for the framework. Signed-off-by: Kenneth Lee --- Documentation/00-INDEX| 2 + Documentation/warpdrive/warpdrive.rst | 153 ++ Documentation/warpdrive/wd-arch.svg | 732 ++ Documentation/warpdrive/wd.svg| 526 ++ 4 files changed, 1413 insertions(+) create mode 100644 Documentation/warpdrive/warpdrive.rst create mode 100644 Documentation/warpdrive/wd-arch.svg create mode 100644 Documentation/warpdrive/wd.svg diff --git a/Documentation/00-INDEX b/Documentation/00-INDEX index 2754fe83f0d4..9959affab599 100644 --- a/Documentation/00-INDEX +++ b/Documentation/00-INDEX @@ -410,6 +410,8 @@ vm/ - directory with info on the Linux vm code. w1/ - directory with documents regarding the 1-wire (w1) subsystem. +warpdrive/ + - directory with documents about WarpDrive accelerator framework. watchdog/ - how to auto-reboot Linux if it has "fallen and can't get up". ;-) wimax/ diff --git a/Documentation/warpdrive/warpdrive.rst b/Documentation/warpdrive/warpdrive.rst new file mode 100644 index ..3792b2780ea6 --- /dev/null +++ b/Documentation/warpdrive/warpdrive.rst @@ -0,0 +1,153 @@ +Introduction of WarpDrive += + +*WarpDrive* is a general accelerator framework built on top of vfio. +It can be taken as a light weight virtual function, which you can use without +*SR-IOV* like facility and can be shared among multiple processes. + +It can be used as the quick channel for accelerators, network adaptors or +other hardware in user space. It can make some implementation simpler. E.g. +you can reuse most of the *netdev* driver and just share some ring buffer to +the user space driver for *DPDK* or *ODP*. Or you can combine the RSA +accelerator with the *netdev* in the user space as a Web reversed proxy, etc. + +The name *WarpDrive* is simply a cool and general name meaning the framework +makes the application faster. In kernel, the framework is called SPIMDEV, +namely "Share Parent IOMMU Mediated Device". + + +How does it work + + +*WarpDrive* takes the Hardware Accelerator as a heterogeneous processor which +can share some load for the CPU: + +.. image:: wd.svg +:alt: This is a .svg image, if your browser cannot show it, +try to download and view it locally + +So it provides the capability to the user application to: + +1. Send request to the hardware +2. Share memory with the application and other accelerators + +These requirements can be fulfilled by VFIO if the accelerator can serve each +application with a separated Virtual Function. But a *SR-IOV* like VF (we will +call it *HVF* hereinafter) design is too heavy for the accelerator which +service thousands of processes. + +And the *HVF* is not good for the scenario that a device keep most of its +resource but share partial of the function to the user space. E.g. a *NIC* +works as a *netdev* but share some hardware queues to the user application to +send packets direct to the hardware. + +*VFIO-mdev* can solve some of the problem here. But *VFIO-mdev* has two problem: + +1. it cannot make use of its parent device's IOMMU. +2. it is assumed to be openned only once. + +So it will need some add-on for better resource control and let the VFIO +driver be aware of this. + + +Architecture + + +The full *WarpDrive* architecture is represented in the following class +diagram: + +.. image:: wd-arch.svg +:alt: This is a .svg image, if your browser cannot show it, +try to download and view it locally + +The idea is: when a device is probed, it can be registered to the general +framework, e.g. *netdev* or *crypto*, and the *SPIMDEV* at the same time. + +If *SPIMDEV* is registered. A *mdev* creation interface is created. Then the +system administrator can create a *mdev* in the user space and set its +parameters via its sysfs interfacev. But not like the other mdev +implementation, hardware resource will not be allocated until it is opened by +an application. + +With this strategy, the hardware resource can be easily scheduled among +multiple processes. + + +The user API + + +We adopt a polling style interface in the user space: :: + +int wd_request_queue(int
[RFC PATCH 0/7] A General Accelerator Framework, WarpDrive
From: Kenneth Lee WarpDrive is an accelerator framework to expose the hardware capabilities directly to the user space. It makes use of the exist vfio and vfio-mdev facilities. So the user application can send request and DMA to the hardware without interaction with the kernel. This remove the latency of syscall and context switch. The patchset contains documents for the detail. Please refer to it for more information. This patchset is intended to be used with Jean Philippe Brucker's SVA patch [1] (Which is also in RFC stage). But it is not mandatory. This patchset is tested in the latest mainline kernel without the SVA patches. So it support only one process for each accelerator. With SVA support, WarpDrive can support multi-process in the same accelerator device. We tested it in our SoC integrated Accelerator (board ID: D06, Chip ID: HIP08). A reference work tree can be found here: [2]. We have noticed the IOMMU aware mdev RFC announced recently [3]. The IOMMU aware mdev has similar idea but different intention comparing to WarpDrive. It intends to dedicate part of the hardware resource to a VM. And the design is supposed to be used with Scalable I/O Virtualization. While spimdev is intended to share the hardware resource with a big amount of processes. It just requires the hardware supporting address translation per process (PCIE's PASID or ARM SMMU's substream ID). But we don't see serious confliction on both design. We believe they can be normalized as one. The patch 1 is document. The patch 2 and 3 add spimdev support. The patch 4, 5 and 6 is drivers for Hislicon's ZIP Accelerator which is registered to both crypto and warpdrive(spimdev) and can be used from kernel or user space at the same time. The patch 7 is a user space sample demonstrating how WarpDrive works. Refernces: [1] https://www.spinics.net/lists/kernel/msg2651481.html [2] https://github.com/Kenneth-Lee/linux-kernel-warpdrive/tree/warpdrive-sva-v0.5 [3] https://lkml.org/lkml/2018/7/22/34 Best Regards Kenneth Lee Kenneth Lee (7): vfio/spimdev: Add documents for WarpDrive framework iommu: Add share domain interface in iommu for spimdev vfio: add spimdev support crypto: add hisilicon Queue Manager driver crypto: Add Hisilicon Zip driver crypto: add spimdev support to Hisilicon QM vfio/spimdev: add user sample for spimdev Documentation/00-INDEX|2 + Documentation/warpdrive/warpdrive.rst | 153 Documentation/warpdrive/wd-arch.svg | 732 +++ Documentation/warpdrive/wd.svg| 526 +++ drivers/crypto/Kconfig|2 + drivers/crypto/Makefile |1 + drivers/crypto/hisilicon/Kconfig | 15 + drivers/crypto/hisilicon/Makefile |2 + drivers/crypto/hisilicon/qm.c | 1005 + drivers/crypto/hisilicon/qm.h | 123 +++ drivers/crypto/hisilicon/zip/Makefile |2 + drivers/crypto/hisilicon/zip/zip.h| 55 ++ drivers/crypto/hisilicon/zip/zip_crypto.c | 358 drivers/crypto/hisilicon/zip/zip_crypto.h | 18 + drivers/crypto/hisilicon/zip/zip_main.c | 182 drivers/iommu/iommu.c | 28 +- drivers/vfio/Kconfig |1 + drivers/vfio/Makefile |1 + drivers/vfio/spimdev/Kconfig | 10 + drivers/vfio/spimdev/Makefile |3 + drivers/vfio/spimdev/vfio_spimdev.c | 421 + drivers/vfio/vfio_iommu_type1.c | 136 ++- include/linux/iommu.h |2 + include/linux/vfio_spimdev.h | 95 ++ include/uapi/linux/vfio_spimdev.h | 28 + samples/warpdrive/AUTHORS |2 + samples/warpdrive/ChangeLog |1 + samples/warpdrive/Makefile.am |9 + samples/warpdrive/NEWS|1 + samples/warpdrive/README | 32 + samples/warpdrive/autogen.sh |3 + samples/warpdrive/cleanup.sh | 13 + samples/warpdrive/configure.ac| 52 ++ samples/warpdrive/drv/hisi_qm_udrv.c | 223 + samples/warpdrive/drv/hisi_qm_udrv.h | 53 ++ samples/warpdrive/test/Makefile.am|7 + samples/warpdrive/test/comp_hw.h | 23 + samples/warpdrive/test/test_hisi_zip.c| 204 + samples/warpdrive/wd.c| 325 +++ samples/warpdrive/wd.h| 153 samples/warpdrive/wd_adapter.c| 74 ++ samples/warpdrive/wd_adapter.h| 43 + 42 files changed, 5112 insertions(+), 7 deletions(-) create mode 100644 Documentation/warpdrive/warpdrive.rst create mode 100644 Documentation/warpdrive/wd-arch.svg create mode 100644 Documentation/warpdrive/wd.svg create mode 100644 drivers/crypto/hisilicon/Kconfig create mode 100644 drivers/crypto/hisilicon/Makefile create mode 100644 drivers/crypto
Re: [PATCH v2 03/40] iommu/sva: Manage process address spaces
On Fri, May 25, 2018 at 09:39:59AM +0100, Jonathan Cameron wrote: > Date: Fri, 25 May 2018 09:39:59 +0100 > From: Jonathan Cameron > To: Ilias Apalodimas > CC: Jean-Philippe Brucker , > "xieyishe...@huawei.com" , "k...@vger.kernel.org" > , "linux-...@vger.kernel.org" > , "xuza...@huawei.com" , > Will Deacon , "ok...@codeaurora.org" > , "linux...@kvack.org" , > "yi.l@intel.com" , "ashok@intel.com" > , "t...@semihalf.com" , > "j...@8bytes.org" , "robdcl...@gmail.com" > , "bhara...@xilinx.com" , > "linux-a...@vger.kernel.org" , > "liudongdo...@huawei.com" , "rfr...@cavium.com" > , "devicet...@vger.kernel.org" > , "kevin.t...@intel.com" > , Jacob Pan , > "alex.william...@redhat.com" , > "rgum...@xilinx.com" , "thunder.leiz...@huawei.com" > , "linux-arm-ker...@lists.infradead.org" > , "shunyong.y...@hxt-semitech.com" > , "dw...@infradead.org" > , "liub...@huawei.com" , > "jcro...@codeaurora.org" , > "iommu@lists.linux-foundation.org" , > Robin Murphy , "christian.koe...@amd.com" > , "nwatt...@codeaurora.org" > , "baolu...@linux.intel.com" > , liguo...@hisilicon.com, > kenneth-lee-2...@foxmail.com > Subject: Re: [PATCH v2 03/40] iommu/sva: Manage process address spaces > Message-ID: <20180525093959.4...@huawei.com> > Organization: Huawei > X-Mailer: Claws Mail 3.15.0 (GTK+ 2.24.31; x86_64-w64-mingw32) > > +CC Kenneth Lee > > On Fri, 25 May 2018 09:33:11 +0300 > Ilias Apalodimas wrote: > > > On Thu, May 24, 2018 at 04:04:39PM +0100, Jean-Philippe Brucker wrote: > > > On 24/05/18 12:50, Ilias Apalodimas wrote: > > > >> Interesting, I hadn't thought about this use-case before. At first I > > > >> thought you were talking about mdev devices assigned to VMs, but I > > > >> think > > > >> you're referring to mdevs assigned to userspace drivers instead? Out of > > > >> curiosity, is it only theoretical or does someone actually need this? > > > > > > > > There has been some non upstreamed efforts to have mdev and produce > > > > userspace > > > > drivers. Huawei is using it on what they call "wrapdrive" for crypto > > > > devices and > > > > we did a proof of concept for ethernet interfaces. At the time we > > > > choose not to > > > > involve the IOMMU for the reason you mentioned, but having it there > > > > would be > > > > good. > > > > > > I'm guessing there were good reasons to do it that way but I wonder, is > > > it not simpler to just have the kernel driver create a /dev/foo, with a > > > standard ioctl/mmap/poll interface? Here VFIO adds a layer of > > > indirection, and since the mediating driver has to implement these > > > operations already, what is gained? > > The best reason i can come up with is "common code". You already have one > > API > > doing that for you so we replicate it in a /dev file? > > The mdev approach still needs extentions to support what we tried to do (i.e > > mdev bus might need yo have access on iommu_ops), but as far as i undestand > > it's > > a possible case. Hi, Jean, Please allow me to share my understanding here: https://zhuanlan.zhihu.com/p/35489035 The reason we do not use the /dev/foo scheme is that the devices to be shared are programmable accelerators. We cannot fix up the kernel driver for them. > > > > > > Thanks, > > > Jean > > (p.s. I sent this mail on May 26 from my public email count. But it seems the email server is blocked. I resent it from my company count until my colleague told me just now. Sorry for inconvenience) -- -Kenneth(Hisilicon) ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v2 03/40] iommu/sva: Manage process address spaces
On Sat, May 26, 2018 at 10:24:45AM +0800, Kenneth Lee wrote: > Date: Sat, 26 May 2018 10:24:45 +0800 > From: Kenneth Lee > To: Jonathan Cameron > Cc: Ilias Apalodimas , Jean-Philippe Brucker > , "xieyishe...@huawei.com" > , "k...@vger.kernel.org" , > "linux-...@vger.kernel.org" , > "xuza...@huawei.com" , Will Deacon > , "ok...@codeaurora.org" , > "linux...@kvack.org" , "yi.l@intel.com" > , "ashok@intel.com" , > "t...@semihalf.com" , "j...@8bytes.org" , > "robdcl...@gmail.com" , "bhara...@xilinx.com" > , "linux-a...@vger.kernel.org" > , "liudongdo...@huawei.com" > , "rfr...@cavium.com" , > "devicet...@vger.kernel.org" , > "kevin.t...@intel.com" , Jacob Pan > , "alex.william...@redhat.com" > , "rgum...@xilinx.com" , > "thunder.leiz...@huawei.com" , > "linux-arm-ker...@lists.infradead.org" > , "shunyong.y...@hxt-semitech.com" > , "dw...@infradead.org" > , "liub...@huawei.com" , > "jcro...@codeaurora.org" , > "iommu@lists.linux-foundation.org" , > Robin Murphy , "christian.koe...@amd.com" > , "nwatt...@codeaurora.org" > , "baolu...@linux.intel.com" > , liguo...@hisilicon.com > Subject: Re: [PATCH v2 03/40] iommu/sva: Manage process address spaces > Message-ID: <20180526022445.GA6069@kllp05> > > On Fri, May 25, 2018 at 09:39:59AM +0100, Jonathan Cameron wrote: > > Date: Fri, 25 May 2018 09:39:59 +0100 > > From: Jonathan Cameron > > To: Ilias Apalodimas > > CC: Jean-Philippe Brucker , > > "xieyishe...@huawei.com" , "k...@vger.kernel.org" > > , "linux-...@vger.kernel.org" > > , "xuza...@huawei.com" , > > Will Deacon , "ok...@codeaurora.org" > > , "linux...@kvack.org" , > > "yi.l@intel.com" , "ashok@intel.com" > > , "t...@semihalf.com" , > > "j...@8bytes.org" , "robdcl...@gmail.com" > > , "bhara...@xilinx.com" , > > "linux-a...@vger.kernel.org" , > > "liudongdo...@huawei.com" , "rfr...@cavium.com" > > , "devicet...@vger.kernel.org" > > , "kevin.t...@intel.com" > > , Jacob Pan , > > "alex.william...@redhat.com" , > > "rgum...@xilinx.com" , "thunder.leiz...@huawei.com" > > , "linux-arm-ker...@lists.infradead.org" > > , "shunyong.y...@hxt-semitech.com" > > , "dw...@infradead.org" > > , "liub...@huawei.com" , > > "jcro...@codeaurora.org" , > > "iommu@lists.linux-foundation.org" , > > Robin Murphy , "christian.koe...@amd.com" > > , "nwatt...@codeaurora.org" > > , "baolu...@linux.intel.com" > > , liguo...@hisilicon.com, > > kenneth-lee-2...@foxmail.com > > Subject: Re: [PATCH v2 03/40] iommu/sva: Manage process address spaces > > Message-ID: <20180525093959.4...@huawei.com> > > > > +CC Kenneth Lee > > > > On Fri, 25 May 2018 09:33:11 +0300 > > Ilias Apalodimas wrote: > > > > > On Thu, May 24, 2018 at 04:04:39PM +0100, Jean-Philippe Brucker wrote: > > > > On 24/05/18 12:50, Ilias Apalodimas wrote: > > > > >> Interesting, I hadn't thought about this use-case before. At first I > > > > >> thought you were talking about mdev devices assigned to VMs, but I > > > > >> think > > > > >> you're referring to mdevs assigned to userspace drivers instead? Out > > > > >> of > > > > >> curiosity, is it only theoretical or does someone actually need > > > > >> this? > > > > > > > > > > There has been some non upstreamed efforts to have mdev and produce > > > > > userspace > > > > > drivers. Huawei is using it on what they call "wrapdrive" for crypto > > > > > devices and > > > > > we did a proof of concept for ethernet interfaces. At the time we > > > > > choose not to > > > > > involve the IOMMU for the reason you mentioned, but having it there > > > > > would be > > > > > good. > > > > > > > > I'm guessing there were good reasons to do it that way but I wonder, is > > > > it not simpler to just have the kernel driver create a /dev/foo, with a > > > > standard ioctl/mmap/poll interface? Here VFIO adds a layer of > > > > indirection, and since the mediating driver has to implement these > > > > operations already, what is gained? > > > The best reason i can come up with is "common code". You already have one > > > API > > > doing that for you so we replicate it in a /dev file? > > > The mdev approach still needs extentions to support what we tried to do > > > (i.e > > > mdev bus might need yo have access on iommu_ops), but as far as i > > > undestand it's > > > a possible case. > > Hi, Jean, Please allow me to share my understanding here: > https://zhuanlan.zhihu.com/p/35489035 > > The reason we do not use the /dev/foo scheme is that the devices to be > shared are programmable accelerators. We cannot fix up the kernel driver for > them. > > > > > > > > Thanks, > > > > Jean > > > > > > -- > -Kenneth Lee (Hisilicon) I just found this mail was missed in the mailing list. I tried it once again. -- -Kenneth Lee (Hisilicon) ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v2 03/40] iommu/sva: Manage process address spaces
On Fri, May 25, 2018 at 09:39:59AM +0100, Jonathan Cameron wrote: > Date: Fri, 25 May 2018 09:39:59 +0100 > From: Jonathan Cameron <jonathan.came...@huawei.com> > To: Ilias Apalodimas <ilias.apalodi...@linaro.org> > CC: Jean-Philippe Brucker <jean-philippe.bruc...@arm.com>, > "xieyishe...@huawei.com" <xieyishe...@huawei.com>, "k...@vger.kernel.org" > <k...@vger.kernel.org>, "linux-...@vger.kernel.org" > <linux-...@vger.kernel.org>, "xuza...@huawei.com" <xuza...@huawei.com>, > Will Deacon <will.dea...@arm.com>, "ok...@codeaurora.org" > <ok...@codeaurora.org>, "linux...@kvack.org" <linux...@kvack.org>, > "yi.l@intel.com" <yi.l@intel.com>, "ashok@intel.com" > <ashok@intel.com>, "t...@semihalf.com" <t...@semihalf.com>, > "j...@8bytes.org" <j...@8bytes.org>, "robdcl...@gmail.com" > <robdcl...@gmail.com>, "bhara...@xilinx.com" <bhara...@xilinx.com>, > "linux-a...@vger.kernel.org" <linux-a...@vger.kernel.org>, > "liudongdo...@huawei.com" <liudongdo...@huawei.com>, "rfr...@cavium.com" > <rfr...@cavium.com>, "devicet...@vger.kernel.org" > <devicet...@vger.kernel.org>, "kevin.t...@intel.com" > <kevin.t...@intel.com>, Jacob Pan <jacob.jun@linux.intel.com>, > "alex.william...@redhat.com" <alex.william...@redhat.com>, > "rgum...@xilinx.com" <rgum...@xilinx.com>, "thunder.leiz...@huawei.com" > <thunder.leiz...@huawei.com>, "linux-arm-ker...@lists.infradead.org" > <linux-arm-ker...@lists.infradead.org>, "shunyong.y...@hxt-semitech.com" > <shunyong.y...@hxt-semitech.com>, "dw...@infradead.org" > <dw...@infradead.org>, "liub...@huawei.com" <liub...@huawei.com>, > "jcro...@codeaurora.org" <jcro...@codeaurora.org>, > "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>, > Robin Murphy <robin.mur...@arm.com>, "christian.koe...@amd.com" > <christian.koe...@amd.com>, "nwatt...@codeaurora.org" > <nwatt...@codeaurora.org>, "baolu...@linux.intel.com" > <baolu...@linux.intel.com>, liguo...@hisilicon.com, > kenneth-lee-2...@foxmail.com > Subject: Re: [PATCH v2 03/40] iommu/sva: Manage process address spaces > Message-ID: <20180525093959.4...@huawei.com> > > +CC Kenneth Lee > > On Fri, 25 May 2018 09:33:11 +0300 > Ilias Apalodimas <ilias.apalodi...@linaro.org> wrote: > > > On Thu, May 24, 2018 at 04:04:39PM +0100, Jean-Philippe Brucker wrote: > > > On 24/05/18 12:50, Ilias Apalodimas wrote: > > > >> Interesting, I hadn't thought about this use-case before. At first I > > > >> thought you were talking about mdev devices assigned to VMs, but I > > > >> think > > > >> you're referring to mdevs assigned to userspace drivers instead? Out of > > > >> curiosity, is it only theoretical or does someone actually need this? > > > > > > > > There has been some non upstreamed efforts to have mdev and produce > > > > userspace > > > > drivers. Huawei is using it on what they call "wrapdrive" for crypto > > > > devices and > > > > we did a proof of concept for ethernet interfaces. At the time we > > > > choose not to > > > > involve the IOMMU for the reason you mentioned, but having it there > > > > would be > > > > good. > > > > > > I'm guessing there were good reasons to do it that way but I wonder, is > > > it not simpler to just have the kernel driver create a /dev/foo, with a > > > standard ioctl/mmap/poll interface? Here VFIO adds a layer of > > > indirection, and since the mediating driver has to implement these > > > operations already, what is gained? > > The best reason i can come up with is "common code". You already have one > > API > > doing that for you so we replicate it in a /dev file? > > The mdev approach still needs extentions to support what we tried to do (i.e > > mdev bus might need yo have access on iommu_ops), but as far as i undestand > > it's > > a possible case. Hi, Jean, Please allow me to share my understanding here: https://zhuanlan.zhihu.com/p/35489035 The reason we do not use the /dev/foo scheme is that the devices to be shared are programmable accelerators. We cannot fix up the kernel driver for them. > > > > > > Thanks, > > > Jean > > -- -Kenneth Lee (Hisilicon) ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
