Re: Ping? Re: [PATCH rc] kvm: Prevent compiling virt/kvm/vfio.c unless VFIO is selected

On Wed, 29 Nov 2023 10:31:03 -0800 Sean Christopherson wrote: > On Wed, Nov 29, 2023, Jason Gunthorpe wrote: > > On Tue, Nov 28, 2023 at 06:21:42PM -0800, Sean Christopherson wrote: > > > diff --git a/include/linux/vfio.h b/include/linux/vfio.h > > > index 454e9295970c..a65b2513f8cd 100644 > >

Re: [PATCH 02/26] vfio: Move KVM get/put helpers to colocate it with other KVM related code

> Signed-off-by: Sean Christopherson > --- > drivers/vfio/vfio_main.c | 104 +++ > 1 file changed, 52 insertions(+), 52 deletions(-) Reviewed-by: Alex Williamson > diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c > in

Re: [PATCH 05/26] vfio: KVM: Pass get/put helpers from KVM to VFIO, don't do circular lookup

t; virt/kvm/vfio.c | 9 +++-- > 4 files changed, 47 insertions(+), 42 deletions(-) Reviewed-by: Alex Williamson > diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h > index a1f741365075..eec51c7ee822 100644 > --- a/drivers/vfio/vfio.h > +++ b/drivers/vfio/vfio.h > @@

Re: [PATCH 03/26] virt: Declare and define vfio_file_set_kvm() iff CONFIG_KVM is enabled

-- > drivers/vfio/vfio_main.c | 2 +- > include/linux/vfio.h | 2 ++ > 2 files changed, 3 insertions(+), 1 deletion(-) As Jason noted, s/virt/vfio/ in title. Reviewed-by: Alex Williamson > diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c > index 6368eed7b7b2..12

Re: [PATCH 01/26] vfio: Wrap KVM helpers with CONFIG_KVM instead of CONFIG_HAVE_KVM

- > drivers/vfio/vfio.h | 2 +- > drivers/vfio/vfio_main.c | 4 +--- > 2 files changed, 2 insertions(+), 4 deletions(-) Reviewed-by: Alex Williamson > diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h > index 307e3f29b527..c26d1ad68105 100644 > --- a/drivers/vfio/vf

Re: [PATCH 04/26] vfio: Add struct to hold KVM assets and dedup group vs. iommufd code

+++-- > drivers/vfio/vfio_main.c | 43 +++--- > 4 files changed, 45 insertions(+), 47 deletions(-) Reviewed-by: Alex Williamson > diff --git a/drivers/vfio/device_cdev.c b/drivers/vfio/device_cdev.c > index e75da0a70d1f..e484d6d64

Re: [PATCH 06/26] KVM: Drop CONFIG_KVM_VFIO and just look at KVM+VFIO

fig | 3 --- > virt/kvm/Makefile.kvm| 4 +++- > virt/kvm/vfio.h | 2 +- > 7 files changed, 4 insertions(+), 9 deletions(-) Reviewed-by: Alex Williamson > diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig > index 83c1e09be42e..2b5c332f157d 10

Re: [PATCH 0/2] eventfd: simplify signal helpers

On Mon, 17 Jul 2023 19:12:16 -0300 Jason Gunthorpe wrote: > On Mon, Jul 17, 2023 at 01:08:31PM -0600, Alex Williamson wrote: > > > What would that mechanism be? We've been iterating on getting the > > serialization and buffering correct, but I don't know of another means

Re: [PATCH 0/2] eventfd: simplify signal helpers

On Mon, 17 Jul 2023 10:29:34 +0200 Grzegorz Jaszczyk wrote: > pt., 14 lip 2023 o 09:05 Christian Brauner napisał(a): > > > > On Thu, Jul 13, 2023 at 11:10:54AM -0600, Alex Williamson wrote: > > > On Thu, 13 Jul 2023 12:05:36 +0200 > > > Christian Brauner wr

Re: [PATCH 0/2] eventfd: simplify signal helpers

On Thu, 13 Jul 2023 12:05:36 +0200 Christian Brauner wrote: > Hey everyone, > > This simplifies the eventfd_signal() and eventfd_signal_mask() helpers > by removing the count argument which is effectively unused. We have a patch under review which does in fact make use of the signaling value:

Re: [PATCH v4 0/7] introduce vm_flags modifier functions

On Fri, 17 Mar 2023 12:08:32 -0700 Suren Baghdasaryan wrote: > On Tue, Mar 14, 2023 at 1:11 PM Alex Williamson > wrote: > > > > On Thu, 26 Jan 2023 11:37:45 -0800 > > Suren Baghdasaryan wrote: > > > > > This patchset was originally pu

Re: [PATCH v4 0/7] introduce vm_flags modifier functions

On Thu, 26 Jan 2023 11:37:45 -0800 Suren Baghdasaryan wrote: > This patchset was originally published as a part of per-VMA locking [1] and > was split after suggestion that it's viable on its own and to facilitate > the review process. It is now a preprequisite for the next version of per-VMA >

Re: [PATCH v2 0/4] Reenable VFIO support on POWER systems

On Mon, 6 Mar 2023 18:35:22 -0600 (CST) Timothy Pearson wrote: > - Original Message - > > From: "Alex Williamson" > > To: "Timothy Pearson" > > Cc: "kvm" , "linuxppc-dev" > > > > Sent: Monday, March 6, 2023

Re: [PATCH v2 0/4] Reenable VFIO support on POWER systems

drivers/vfio/vfio_iommu_spapr_tce.c | 96 ++--- > 10 files changed, 338 insertions(+), 94 deletions(-) > For vfio and MAINTAINERS portions, Acked-by: Alex Williamson I'll note though that spapr_tce_take_ownership() looks like it copied a bug from the old tce_iommu_t

[PATCH] vfio/pci: Revert nvlink removal uAPI breakage

Revert the uAPI changes from the below commit with notice that these regions and capabilities are no longer provided. Fixes: b392a1989170 ("vfio/pci: remove vfio_pci_nvlink2") Reported-by: Greg Kurz Signed-off-by: Alex Williamson --- Greg (Kurz), please double check this resolves

Re: remove the nvlink2 pci_vfio subdriver v2

On Tue, 4 May 2021 16:11:31 +0200 Greg Kurz wrote: > On Tue, 4 May 2021 15:30:15 +0200 > Greg Kroah-Hartman wrote: > > > On Tue, May 04, 2021 at 03:20:34PM +0200, Greg Kurz wrote: > > > On Tue, 4 May 2021 14:59:07 +0200 > > > Greg Kroah-Hartman wrote: > > > > > > > On Tue, May 04, 2021

Re: [PATCH 1/2] vfio/pci: remove vfio_pci_nvlink2

On Mon, 12 Apr 2021 19:41:41 +1000 Michael Ellerman wrote: > Alex Williamson writes: > > On Fri, 26 Mar 2021 07:13:10 +0100 > > Christoph Hellwig wrote: > > > >> This driver never had any open userspace (which for VFIO would include > >> VM kernel driv

Re: [PATCH 1/2] vfio/pci: remove vfio_pci_nvlink2

On Fri, 26 Mar 2021 07:13:10 +0100 Christoph Hellwig wrote: > This driver never had any open userspace (which for VFIO would include > VM kernel drivers) that use it, and thus should never have been added > by our normal userspace ABI rules. > > Signed-off-by: Christoph Hellwig > Acked-by:

Re: [PATCH 1/2] vfio/pci: remove vfio_pci_nvlink2

On Mon, 22 Mar 2021 16:01:54 +0100 Christoph Hellwig wrote: > diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h > index 8ce36c1d53ca11..db7e782419d5d9 100644 > --- a/include/uapi/linux/vfio.h > +++ b/include/uapi/linux/vfio.h > @@ -332,19 +332,6 @@ struct

Re: [PATCH kernel v2] vfio/pci/nvlink2: Do not attempt NPU2 setup on POWER8NVL NPU

On Sun, 22 Nov 2020 18:39:50 +1100 Alexey Kardashevskiy wrote: > We execute certain NPU2 setup code (such as mapping an LPID to a device > in NPU2) unconditionally if an Nvlink bridge is detected. However this > cannot succeed on POWER8NVL machines as the init helpers return an error > other

Re: [PATCH v2 1/1] vfio-pci/nvlink2: Allow fallback to ibm,mmio-atsd[0]

On Tue, 31 Mar 2020 15:12:46 +1100 Sam Bobroff wrote: > Older versions of skiboot only provide a single value in the device > tree property "ibm,mmio-atsd", even when multiple Address Translation > Shoot Down (ATSD) registers are present. This prevents NVLink2 devices > (other than the first)

Re: [PATCH kernel 5/5] vfio/spapr_tce: Advertise and allow a huge DMA windows at 4GB

On Tue, 18 Feb 2020 18:36:50 +1100 Alexey Kardashevskiy wrote: > So far the only option for a big 64big DMA window was a window located > at 0x800... (1<<59) which creates problems for devices > supporting smaller DMA masks. > > This exploits a POWER9 PHB option to allow the second

Re: [PATCH kernel] vfio/spapr/nvlink2: Skip unpinning pages on error exit

On Mon, 23 Dec 2019 12:09:27 +1100 Alexey Kardashevskiy wrote: > The nvlink2 subdriver for IBM Witherspoon machines preregisters > GPU memory in the IOMMI API so KVM TCE code can map this memory > for DMA as well. This is done by mm_iommu_newdev() called from > vfio_pci_nvgpu_regops::mmap. > >

Re: [PATCH v7 09/24] vfio, mm: fix get_user_pages_remote() and FOLL_LONGTERM

sertions(+), 30 deletions(-) Tested with device assignment and Intel mdev vGPU assignment with QEMU userspace: Tested-by: Alex Williamson Acked-by: Alex Williamson Feel free to include for 19/24 as well. Thanks, Alex > diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iomm

Re: [PATCH kernel] vfio/spapr_tce: Fix incorrect tce_iommu_group memory free

On Mon, 19 Aug 2019 11:51:17 +1000 Alexey Kardashevskiy wrote: > The @tcegrp variable is used in 1) a loop over attached groups > 2) it stores a pointer to a newly allocated tce_iommu_group if 1) found > nothing. However the error handler does not distinguish how we got there > and incorrectly

Re: [PATCH kernel] vfio/spapr_tce: Fix incorrect tce_iommu_group memory free

On Fri, 23 Aug 2019 15:32:41 +1000 Paul Mackerras wrote: > On Mon, Aug 19, 2019 at 11:51:17AM +1000, Alexey Kardashevskiy wrote: > > The @tcegrp variable is used in 1) a loop over attached groups > > 2) it stores a pointer to a newly allocated tce_iommu_group if 1) found > > nothing. However the

Re: [PATCH v3] mm: add account_locked_vm utility function

same, so user-visible behavior does too. The one > exception is that the -EPERM case in tce_account_locked_vm is removed > because Alexey has never seen it triggered. > > Signed-off-by: Daniel Jordan > Tested-by: Alexey Kardashevskiy > Cc: Alan Tull > Cc: Alex Williamson > Cc

Re: [PATCH v2] mm: add account_locked_vm utility function

On Tue, 28 May 2019 11:04:24 -0400 Daniel Jordan wrote: > On Sat, May 25, 2019 at 02:51:18PM -0700, Andrew Morton wrote: > > On Fri, 24 May 2019 13:50:45 -0400 Daniel Jordan > > wrote: > > > > > locked_vm accounting is done roughly the same way in five places, so > > > unify them in a

Re: [PATCH] vfio-pci/nvlink2: Fix potential VMA leak

On Tue, 7 May 2019 09:01:45 +0200 Greg Kurz wrote: > On Tue, 7 May 2019 11:52:44 +1000 > Sam Bobroff wrote: > > > On Mon, May 06, 2019 at 03:58:45PM -0600, Alex Williamson wrote: > > > On Fri, 19 Apr 2019 17:37:17 +020

Re: [PATCH] vfio-pci/nvlink2: Fix potential VMA leak

On Fri, 19 Apr 2019 17:37:17 +0200 Greg Kurz wrote: > If vfio_pci_register_dev_region() fails then we should rollback > previous changes, ie. unmap the ATSD registers. > > Signed-off-by: Greg Kurz > --- Applied to vfio next branch for v5.2 with Alexey's R-b. Thanks! Alex >

Re: [PATCH kernel v3] powerpc/powernv: Isolate NVLinks between GV100GL on Witherspoon

On Tue, 30 Apr 2019 16:14:35 +1000 Alexey Kardashevskiy wrote: > On 30/04/2019 15:45, Alistair Popple wrote: > > Alexey, > > > > +void pnv_try_isolate_nvidia_v100(struct pci_dev *bridge) > > +{ > > + u32 mask, val; > > + void __iomem *bar0_0, *bar0_12,

Re: [PATCH kernel v3] powerpc/powernv: Isolate NVLinks between GV100GL on Witherspoon

On Thu, 11 Apr 2019 16:48:44 +1000 Alexey Kardashevskiy wrote: > The NVIDIA V100 SXM2 GPUs are connected to the CPU via PCIe links and > (on POWER9) NVLinks. In addition to that, GPUs themselves have direct > peer-to-peer NVLinks in groups of 2 to 4 GPUs with no buffers/latches > between GPUs. >

Re: [RFC PATCH kernel v2] powerpc/powernv: Isolate NVLinks between GV100GL on Witherspoon

On Thu, 4 Apr 2019 16:23:24 +1100 Alexey Kardashevskiy wrote: > The NVIDIA V100 SXM2 GPUs are connected to the CPU via PCIe links and > (on POWER9) NVLinks. In addition to that, GPUs themselves have direct > peer to peer NVLinks in groups of 2 to 4 GPUs. At the moment the POWERNV > platform

Re: [PATCH kernel RFC 2/2] vfio-pci-nvlink2: Implement interconnect isolation

On Fri, 22 Mar 2019 14:08:38 +1100 David Gibson wrote: > On Thu, Mar 21, 2019 at 12:19:34PM -0600, Alex Williamson wrote: > > On Thu, 21 Mar 2019 10:56:00 +1100 > > David Gibson wrote: > > > > > On Wed, Mar 20, 2019 at 01:09:08PM -0600, Alex Williamson wrote:

Re: [PATCH kernel RFC 2/2] vfio-pci-nvlink2: Implement interconnect isolation

On Thu, 21 Mar 2019 10:56:00 +1100 David Gibson wrote: > On Wed, Mar 20, 2019 at 01:09:08PM -0600, Alex Williamson wrote: > > On Wed, 20 Mar 2019 15:38:24 +1100 > > David Gibson wrote: > > > > > On Tue, Mar 19, 2019 at 10:36:19AM -0600, Alex Williamson wrote:

Re: [PATCH kernel RFC 2/2] vfio-pci-nvlink2: Implement interconnect isolation

On Wed, 20 Mar 2019 15:38:24 +1100 David Gibson wrote: > On Tue, Mar 19, 2019 at 10:36:19AM -0600, Alex Williamson wrote: > > On Fri, 15 Mar 2019 19:18:35 +1100 > > Alexey Kardashevskiy wrote: > > > > > The NVIDIA V100 SXM2 GPUs are connected to the CPU via P

Re: [PATCH kernel RFC 2/2] vfio-pci-nvlink2: Implement interconnect isolation

On Fri, 15 Mar 2019 19:18:35 +1100 Alexey Kardashevskiy wrote: > The NVIDIA V100 SXM2 GPUs are connected to the CPU via PCIe links and > (on POWER9) NVLinks. In addition to that, GPUs themselves have direct > peer to peer NVLinks in groups of 2 to 4 GPUs. At the moment the POWERNV > platform

Re: [PATCH kernel] vfio/spapr_tce: Skip unsetting already unset table

On Wed, 13 Feb 2019 11:18:21 +1100 Alexey Kardashevskiy wrote: > On 13/02/2019 07:52, Alex Williamson wrote: > > On Mon, 11 Feb 2019 18:49:17 +1100 > > Alexey Kardashevskiy wrote: > > > >> VFIO TCE IOMMU v2 owns IOMMU tables so when detach a IOMMU group

Re: [PATCH 1/5] vfio/type1: use pinned_vm instead of locked_vm to account pinned pages

On Tue, 12 Feb 2019 19:26:50 -0500 Daniel Jordan wrote: > On Tue, Feb 12, 2019 at 11:41:10AM -0700, Alex Williamson wrote: > > Daniel Jordan wrote: > > > On Mon, Feb 11, 2019 at 03:56:20PM -0700, Jason Gunthorpe wrote: > > > > I haven't looked at this super cl

Re: [PATCH kernel] vfio/spapr_tce: Skip unsetting already unset table

On Mon, 11 Feb 2019 18:49:17 +1100 Alexey Kardashevskiy wrote: > VFIO TCE IOMMU v2 owns IOMMU tables so when detach a IOMMU group from > a container, we need to unset those from a group so we call unset_window() > so do we unconditionally. We also unset tables when removing a DMA window Patch

Re: [PATCH 2/5] vfio/spapr_tce: use pinned_vm instead of locked_vm to account pinned pages

On Tue, 12 Feb 2019 17:56:18 +1100 Alexey Kardashevskiy wrote: > On 12/02/2019 09:44, Daniel Jordan wrote: > > Beginning with bc3e53f682d9 ("mm: distinguish between mlocked and pinned > > pages"), locked and pinned pages are accounted separately. The SPAPR > > TCE VFIO IOMMU driver accounts

Re: [PATCH 1/5] vfio/type1: use pinned_vm instead of locked_vm to account pinned pages

On Mon, 11 Feb 2019 18:11:53 -0500 Daniel Jordan wrote: > On Mon, Feb 11, 2019 at 03:56:20PM -0700, Jason Gunthorpe wrote: > > On Mon, Feb 11, 2019 at 05:44:33PM -0500, Daniel Jordan wrote: > > > @@ -266,24 +267,15 @@ static int vfio_lock_acct(struct vfio_dma *dma, > > > long npage, bool

Re: [PATCH kernel] vfio-pci/nvlink2: Fix ancient gcc warnings

On Wed, 23 Jan 2019 15:07:11 +1100 Alexey Kardashevskiy wrote: > Using the {0} construct as a generic initializer is perfectly fine in C, > however due to a bug in old gcc there is a warning: > > + /kisskb/src/drivers/vfio/pci/vfio_pci_nvlink2.c: warning: (near > initialization for

Re: [PATCH kernel] vfio-pci/nvlink2: Fix ancient gcc warnings

Hi Geert, The below patch comes about from the build regressions and improvements list you've sent out, but something doesn't add up that we'd be testing with an old compiler where initialization with { 0 } generates a "missing braces around initialization" warning. Is this really the case or

Re: [PATCH kernel v7 20/20] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] subdriver

On Fri, 21 Dec 2018 12:50:00 +1100 Alexey Kardashevskiy wrote: > On 21/12/2018 12:37, Alex Williamson wrote: > > On Fri, 21 Dec 2018 12:23:16 +1100 > > Alexey Kardashevskiy wrote: > > > >> On 21/12/2018 03:46, Alex Williamson wrote: > >>> On Th

Re: [PATCH kernel v7 20/20] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] subdriver

On Fri, 21 Dec 2018 12:23:16 +1100 Alexey Kardashevskiy wrote: > On 21/12/2018 03:46, Alex Williamson wrote: > > On Thu, 20 Dec 2018 19:23:50 +1100 > > Alexey Kardashevskiy wrote: > > > >> POWER9 Witherspoon machines come with 4 or 6 V100 GPUs which are n

Re: [PATCH kernel v7 20/20] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] subdriver

On Thu, 20 Dec 2018 19:23:50 +1100 Alexey Kardashevskiy wrote: > POWER9 Witherspoon machines come with 4 or 6 V100 GPUs which are not > pluggable PCIe devices but still have PCIe links which are used > for config space and MMIO. In addition to that the GPUs have 6 NVLinks > which are connected

Re: [PATCH kernel v6 20/20] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] subdriver

_ID_IBM && > + IS_ENABLED(CONFIG_VFIO_PCI_NVLINK2)) { > + ret = vfio_pci_ibm_npu2_init(vdev); > + if (ret && ret != -ENODEV) { > + dev_warn(>pdev->dev, > + "Failed t

Re: [PATCH kernel v6 19/20] vfio_pci: Allow regions to add own capabilities

> This adds an add_capability() hook to vfio_pci_regops. > > Signed-off-by: Alexey Kardashevskiy > Acked-by: Alex Williamson > --- > Changes: > v3: > * removed confusing rationale for the patch, the next patch makes > use of it anyway > --- > drivers/vfio/pci/v

Re: [PATCH kernel v6 18/20] vfio_pci: Allow mapping extra regions

ing > them to the userspace. > > Signed-off-by: Alexey Kardashevskiy > Reviewed-by: David Gibson > Acked-by: Alex Williamson > --- > Changes: > v2: > * reverted one of mistakenly removed error checks > --- > drivers/vfio/pci/vfio_pci_private.h | 3 +++ > drive

Re: [PATCH kernel v5 20/20] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] subdriver

On Thu, 13 Dec 2018 17:17:34 +1100 Alexey Kardashevskiy wrote: > POWER9 Witherspoon machines come with 4 or 6 V100 GPUs which are not > pluggable PCIe devices but still have PCIe links which are used > for config space and MMIO. In addition to that the GPUs have 6 NVLinks > which are connected

Re: [PATCH kernel v4 19/19] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] subdriver

On Tue, 11 Dec 2018 11:57:20 +1100 Alexey Kardashevskiy wrote: > On 11/12/2018 11:08, Alex Williamson wrote: > > On Fri, 23 Nov 2018 16:53:04 +1100 > > Alexey Kardashevskiy wrote: > > > >> POWER9 Witherspoon machines come with 4 or 6 V100 GPUs which are n

Re: [PATCH kernel v4 18/19] vfio_pci: Allow regions to add own capabilities

ps. > > Signed-off-by: Alexey Kardashevskiy > --- > Changes: > v3: > * removed confusing rationale for the patch, the next patch makes > use of it anyway > --- > drivers/vfio/pci/vfio_pci_private.h | 3 +++ > drivers/vfio/pci/vfio_pci.c | 6 ++ >

Re: [PATCH kernel v4 17/19] vfio_pci: Allow mapping extra regions

On Fri, 23 Nov 2018 16:53:02 +1100 Alexey Kardashevskiy wrote: > So far we only allowed mapping of MMIO BARs to the userspace. However > there there are GPUs with on-board coherent RAM accessible via side s/there there/there/ Otherwise: Acked-by: Alex Williamson > channels whic

Re: [PATCH kernel v4 19/19] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] subdriver

On Fri, 23 Nov 2018 16:53:04 +1100 Alexey Kardashevskiy wrote: > POWER9 Witherspoon machines come with 4 or 6 V100 GPUs which are not > pluggable PCIe devices but still have PCIe links which are used > for config space and MMIO. In addition to that the GPUs have 6 NVLinks > which are connected

Re: [PATCH kernel 3/3] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] [10de:1db1] subdriver

On Thu, 18 Oct 2018 10:37:46 -0700 Piotr Jaroszynski wrote: > On 10/18/18 9:55 AM, Alex Williamson wrote: > > On Thu, 18 Oct 2018 11:31:33 +1100 > > Alexey Kardashevskiy wrote: > > > >> On 18/10/2018 08:52, Alex Williamson wrote: > >>> On We

Re: [PATCH kernel 3/3] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] [10de:1db1] subdriver

On Thu, 18 Oct 2018 11:31:33 +1100 Alexey Kardashevskiy wrote: > On 18/10/2018 08:52, Alex Williamson wrote: > > On Wed, 17 Oct 2018 12:19:20 +1100 > > Alexey Kardashevskiy wrote: > > > >> On 17/10/2018 06:08, Alex Williamson wrote: > >>> On Mo

Re: [PATCH kernel 3/3] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] [10de:1db1] subdriver

On Wed, 17 Oct 2018 12:19:20 +1100 Alexey Kardashevskiy wrote: > On 17/10/2018 06:08, Alex Williamson wrote: > > On Mon, 15 Oct 2018 20:42:33 +1100 > > Alexey Kardashevskiy wrote: > > > >> POWER9 Witherspoon machines come with 4 or 6 V100 GPUs which are n

Re: [PATCH kernel 3/3] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] [10de:1db1] subdriver

gt; b/drivers/vfio/pci/vfio_pci_nvlink2.c > new file mode 100644 > index 000..c9d2b55 > --- /dev/null > +++ b/drivers/vfio/pci/vfio_pci_nvlink2.c > @@ -0,0 +1,409 @@ > +// SPDX-License-Identifier: GPL-2.0+ > +/* > + * VFIO PCI NVIDIA Whitherspoon GPU support a.k.a. NVL

Re: [RFC PATCH kernel] vfio/spapr_tce: Get rid of possible infinite loop

es tag? Looks like it's relevant to: 4b6fad7097f8 powerpc/mm/iommu, vfio/spapr: Put pages on VFIO container shutdown Also, not sure who you're wanting to take this since it was sent to ppc lists. If it's me, let me know, otherwise Acked-by: Alex Williamson Thanks, Alex > diff --git a/driv

Re: [RFC PATCH kernel 0/5] powerpc/P9/vfio: Pass through NVIDIA Tesla V100

On Thu, 9 Aug 2018 14:21:29 +1000 Alexey Kardashevskiy wrote: > On 08/08/2018 18:39, Alexey Kardashevskiy wrote: > > > > > > On 02/08/2018 02:16, Alex Williamson wrote: > >> On Wed, 1 Aug 2018 18:37:35 +1000 > >> Alexey Kardashevskiy wrote: > >&

Re: [RFC PATCH kernel 0/5] powerpc/P9/vfio: Pass through NVIDIA Tesla V100

On Wed, 1 Aug 2018 18:37:35 +1000 Alexey Kardashevskiy wrote: > On 01/08/2018 00:29, Alex Williamson wrote: > > On Tue, 31 Jul 2018 14:03:35 +1000 > > Alexey Kardashevskiy wrote: > > > >> On 31/07/2018 02:29, Alex Williamson wrote: > >>> On Mo

Re: [RFC PATCH kernel 0/5] powerpc/P9/vfio: Pass through NVIDIA Tesla V100

On Tue, 31 Jul 2018 14:03:35 +1000 Alexey Kardashevskiy wrote: > On 31/07/2018 02:29, Alex Williamson wrote: > > On Mon, 30 Jul 2018 18:58:49 +1000 > > Alexey Kardashevskiy wrote: > >> After some local discussions, it was pointed out that force disabling > >

Re: [RFC PATCH kernel 0/5] powerpc/P9/vfio: Pass through NVIDIA Tesla V100

On Mon, 30 Jul 2018 18:58:49 +1000 Alexey Kardashevskiy wrote: > On 11/07/2018 19:26, Alexey Kardashevskiy wrote: > > On Tue, 10 Jul 2018 16:37:15 -0600 > > Alex Williamson wrote: > > > >> On Tue, 10 Jul 2018 14:10:20 +1000 > >> Alexey Kardashevskiy wr

Re: [RFC PATCH kernel 0/5] powerpc/P9/vfio: Pass through NVIDIA Tesla V100

On Tue, 10 Jul 2018 14:10:20 +1000 Alexey Kardashevskiy wrote: > On Thu, 7 Jun 2018 23:03:23 -0600 > Alex Williamson wrote: > > > On Fri, 8 Jun 2018 14:14:23 +1000 > > Alexey Kardashevskiy wrote: > > > > > On 8/6/18 1:44 pm, Alex Williamson wrote:

Re: [PATCH kernel v2 1/2] vfio/spapr: Use IOMMU pageshift rather than pagesize

t; Reviewed-by: David Gibson > Signed-off-by: Alexey Kardashevskiy > --- > drivers/vfio/vfio_iommu_spapr_tce.c | 8 > 1 file changed, 4 insertions(+), 4 deletions(-) I assume a v3+ will go in through the ppc tree since the bulk of the series is there. For this, Acked-b

Re: [RFC PATCH kernel 0/5] powerpc/P9/vfio: Pass through NVIDIA Tesla V100

On Fri, 8 Jun 2018 14:14:23 +1000 Alexey Kardashevskiy wrote: > On 8/6/18 1:44 pm, Alex Williamson wrote: > > On Fri, 8 Jun 2018 13:08:54 +1000 > > Alexey Kardashevskiy wrote: > > > >> On 8/6/18 8:15 am, Alex Williamson wrote: > >>> On Fri,

Re: [RFC PATCH kernel 5/5] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] [10de:1db1] subdriver

On Fri, 8 Jun 2018 13:52:05 +1000 Alexey Kardashevskiy wrote: > On 8/6/18 1:35 pm, Alex Williamson wrote: > > On Fri, 8 Jun 2018 13:09:13 +1000 > > Alexey Kardashevskiy wrote: > >> On 8/6/18 3:04 am, Alex Williamson wrote: > >>> On Thu, 7 Jun 2018 18:44

Re: [RFC PATCH kernel 0/5] powerpc/P9/vfio: Pass through NVIDIA Tesla V100

On Fri, 8 Jun 2018 13:08:54 +1000 Alexey Kardashevskiy wrote: > On 8/6/18 8:15 am, Alex Williamson wrote: > > On Fri, 08 Jun 2018 07:54:02 +1000 > > Benjamin Herrenschmidt wrote: > > > >> On Thu, 2018-06-07 at 11:04 -0600, Alex Williamson wrote: > >

Re: [RFC PATCH kernel 5/5] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] [10de:1db1] subdriver

On Fri, 8 Jun 2018 13:09:13 +1000 Alexey Kardashevskiy wrote: > On 8/6/18 3:04 am, Alex Williamson wrote: > > On Thu, 7 Jun 2018 18:44:20 +1000 > > Alexey Kardashevskiy wrote: > >> diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c > >&

Re: [RFC PATCH kernel 0/5] powerpc/P9/vfio: Pass through NVIDIA Tesla V100

On Fri, 08 Jun 2018 10:58:54 +1000 Benjamin Herrenschmidt wrote: > On Thu, 2018-06-07 at 18:34 -0600, Alex Williamson wrote: > > > We *can* allow individual GPUs to be passed through, either if somebody > > > designs a system without cross links, or if the user is ok with th

Re: [RFC PATCH kernel 0/5] powerpc/P9/vfio: Pass through NVIDIA Tesla V100

On Fri, 08 Jun 2018 09:20:30 +1000 Benjamin Herrenschmidt wrote: > On Thu, 2018-06-07 at 16:15 -0600, Alex Williamson wrote: > > On Fri, 08 Jun 2018 07:54:02 +1000 > > Benjamin Herrenschmidt wrote: > > > > > On Thu, 2018-06-07 at 11:04 -0600, Alex Williamson wr

Re: [RFC PATCH kernel 0/5] powerpc/P9/vfio: Pass through NVIDIA Tesla V100

On Fri, 08 Jun 2018 07:54:02 +1000 Benjamin Herrenschmidt wrote: > On Thu, 2018-06-07 at 11:04 -0600, Alex Williamson wrote: > > > > Can we back up and discuss whether the IOMMU grouping of NVLink > > connected devices makes sense? AIUI we have a PCI view of these

Re: [RFC PATCH kernel 4/5] vfio_pci: Allow mapping extra regions

On Thu, 7 Jun 2018 18:44:19 +1000 Alexey Kardashevskiy wrote: What's an "extra region", -ENOCOMMITLOG > Signed-off-by: Alexey Kardashevskiy > --- > drivers/vfio/pci/vfio_pci_private.h | 3 +++ > drivers/vfio/pci/vfio_pci.c | 10 -- > 2 files changed, 11 insertions(+), 2

Re: [RFC PATCH kernel 5/5] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] [10de:1db1] subdriver

spoon GPU support a.k.a. NVLink2. > + * > + * Copyright (C) 2018 IBM Corp. All rights reserved. > + * Author: Alexey Kardashevskiy > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License

Re: [RFC PATCH kernel 0/5] powerpc/P9/vfio: Pass through NVIDIA Tesla V100

On Thu, 7 Jun 2018 18:44:15 +1000 Alexey Kardashevskiy wrote: > Here is an rfc of some patches adding psaa-through support > for NVIDIA V100 GPU found in some POWER9 boxes. > > The example P9 system has 6 GPUs, each accompanied with 2 bridges > representing the hardware links (aka NVLink2): >

Re: [PATCH kernel] vfio/spapr: Add trace points for map/unmap

On Thu, 23 Nov 2017 15:13:37 +1100 Alexey Kardashevskiy <a...@ozlabs.ru> wrote: > On 17/11/17 17:58, Alexey Kardashevskiy wrote: > > On 17/11/17 11:13, Alex Williamson wrote: > >> On Tue, 14 Nov 2017 10:47:12 +1100 > >> Alexey Kardashevskiy <a...@ozlabs.ru>

Re: [PATCH kernel] vfio/spapr: Add trace points for map/unmap

On Tue, 14 Nov 2017 10:47:12 +1100 Alexey Kardashevskiy wrote: > On 27/10/17 14:00, Alexey Kardashevskiy wrote: > > This adds trace_map/trace_unmap tracepoints to spapr driver. Type1 already > > uses these via the IOMMU API (iommu_map/__iommu_unmap). > > > > Signed-off-by:

Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device

On Fri, 13 Oct 2017 07:01:48 -0500 Steven Royer wrote: > On 2017-10-13 06:53, Steven Royer wrote: > > On 2017-10-12 22:34, Bjorn Helgaas wrote: > >> [+cc Alex, Bodong, Eli, Saeed] > >> > >> On Thu, Oct 12, 2017 at 02:59:23PM -0500, Bryant G. Ly wrote: > >>> On

Re: [PATCH kernel v3] vfio/spapr: Add cond_resched() for huge updates

On Thu, 28 Sep 2017 19:16:12 +1000 Alexey Kardashevskiy wrote: > Clearing very big IOMMU tables can trigger soft lockups. This adds > cond_resched() to allow the scheduler to do context switching when > it decides to. > > Signed-off-by: Alexey Kardashevskiy >

Re: [RFC PATCH v5 0/5] vfio-pci: Add support for mmapping MSI-X table

On Thu, 17 Aug 2017 10:56:35 + David Laight <david.lai...@aculab.com> wrote: > From: Alex Williamson > > Sent: 16 August 2017 17:56 > ... > > Firmware pissing match... Processors running with 8k or less page size > > fall within the recommendations of the P

Re: [RFC PATCH v5 0/5] vfio-pci: Add support for mmapping MSI-X table

On Wed, 16 Aug 2017 10:35:49 +1000 Benjamin Herrenschmidt <b...@kernel.crashing.org> wrote: > On Tue, 2017-08-15 at 10:37 -0600, Alex Williamson wrote: > > Of course I don't think either of those are worth imposing a > > performance penalty where we don't otherwise need

Re: [RFC PATCH v5 0/5] vfio-pci: Add support for mmapping MSI-X table

On Mon, 14 Aug 2017 14:12:33 +0100 Robin Murphy wrote: > On 14/08/17 10:45, Alexey Kardashevskiy wrote: > > Folks, > > > > Is there anything to change besides those compiler errors and David's > > comment in 5/5? Or the while patchset is too bad? Thanks. > > While I now

Re: [PATCH v2] include/linux/vfio.h: Guard powerpc-specific functions with CONFIG_VFIO_SPAPR_EEH

On Tue, 18 Jul 2017 14:22:20 -0300 Murilo Opsfelder Araujo wrote: > When CONFIG_EEH=y and CONFIG_VFIO_SPAPR_EEH=n, build fails with the > following: > > drivers/vfio/pci/vfio_pci.o: In function `.vfio_pci_release': > vfio_pci.c:(.text+0xa98): undefined reference to

Re: [PATCH v2] include/linux/vfio.h: Guard powerpc-specific functions with CONFIG_VFIO_SPAPR_EEH

[cc +Alexey, David] Any comments from the usual suspects for vfio/spapr? Thanks, Alex On Tue, 25 Jul 2017 10:56:38 -0300 Murilo Opsfelder Araújo wrote: > On 07/18/2017 02:22 PM, Murilo Opsfelder Araujo wrote: > > When CONFIG_EEH=y and CONFIG_VFIO_SPAPR_EEH=n, build

Re: [PATCH kernel 0/3 REPOST] vfio-pci: Add support for mmapping MSI-X table

On Wed, 28 Jun 2017 17:27:32 +1000 Alexey Kardashevskiy <a...@ozlabs.ru> wrote: > On 24/06/17 01:17, Alex Williamson wrote: > > On Fri, 23 Jun 2017 15:06:37 +1000 > > Alexey Kardashevskiy <a...@ozlabs.ru> wrote: > > > >> On 23/06/17 07:11, Alex Willia

Re: [PATCH kernel 0/3 REPOST] vfio-pci: Add support for mmapping MSI-X table

On Fri, 23 Jun 2017 15:06:37 +1000 Alexey Kardashevskiy <a...@ozlabs.ru> wrote: > On 23/06/17 07:11, Alex Williamson wrote: > > On Thu, 15 Jun 2017 15:48:42 +1000 > > Alexey Kardashevskiy <a...@ozlabs.ru> wrote: > > > >> Here is a patchset which Yon

Re: [PATCH kernel 0/3 REPOST] vfio-pci: Add support for mmapping MSI-X table

On Thu, 15 Jun 2017 15:48:42 +1000 Alexey Kardashevskiy wrote: > Here is a patchset which Yongji was working on before > leaving IBM LTC. Since we still want to have this functionality > in the kernel (DPDK is the first user), here is a rebase > on the current upstream. > > >

Re: [PATCH guest kernel] vfio/powerpc/spapr_tce: Enforce IOMMU type compatibility check

On Tue, 4 Apr 2017 20:12:45 +1000 Alexey Kardashevskiy <a...@ozlabs.ru> wrote: > On 25/03/17 23:25, Alexey Kardashevskiy wrote: > > On 25/03/17 07:29, Alex Williamson wrote: > >> On Fri, 24 Mar 2017 17:44:06 +1100 > >> Alexey Kardashevskiy <a...@ozlabs.ru>

Re: [PATCH guest kernel] vfio/powerpc/spapr_tce: Enforce IOMMU type compatibility check

On Fri, 24 Mar 2017 17:44:06 +1100 Alexey Kardashevskiy wrote: > The existing SPAPR TCE driver advertises both VFIO_SPAPR_TCE_IOMMU and > VFIO_SPAPR_TCE_v2_IOMMU types to the userspace and the userspace usually > picks the v2. > > Normally the userspace would create a container,

Re: [PATCH kernel v11 10/10] KVM: PPC: VFIO: Add in-kernel acceleration for VFIO

return; > + > + kvm_spapr_tce_release_iommu_group(kvm, grp); > + iommu_group_put(grp); > +} > +#endif > + > /* > * Groups can use the same or different IOMMU domains. If the same then > * adding a new group may change the coherency of groups we've previously > @@ -211,6 +256,9 @@ static int kvm_vfio_set_group(struct kvm_device *dev, > long attr, u64 arg) > > mutex_unlock(>lock); > > +#ifdef CONFIG_SPAPR_TCE_IOMMU > + kvm_spapr_tce_release_vfio_group(dev->kvm, vfio_group); > +#endif > kvm_vfio_group_set_kvm(vfio_group, NULL); > > kvm_vfio_group_put_external_user(vfio_group); > @@ -218,6 +266,57 @@ static int kvm_vfio_set_group(struct kvm_device *dev, > long attr, u64 arg) > kvm_vfio_update_coherency(dev); > > return ret; > + > +#ifdef CONFIG_SPAPR_TCE_IOMMU > + case KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE: { > + struct kvm_vfio_spapr_tce param; > + struct kvm_vfio *kv = dev->private; > + struct vfio_group *vfio_group; > + struct kvm_vfio_group *kvg; > + struct fd f; > + struct iommu_group *grp; > + > + if (copy_from_user(, (void __user *)arg, > + sizeof(struct kvm_vfio_spapr_tce))) > + return -EFAULT; > + > + f = fdget(param.groupfd); > + if (!f.file) > + return -EBADF; > + > + vfio_group = kvm_vfio_group_get_external_user(f.file); > + fdput(f); > + > + if (IS_ERR(vfio_group)) > + return PTR_ERR(vfio_group); > + > + grp = kvm_vfio_group_get_iommu_group(vfio_group); > + if (WARN_ON_ONCE(!grp)) { > + kvm_vfio_group_put_external_user(vfio_group); > + return -EIO; > + } > + > + ret = -ENOENT; > + > + mutex_lock(>lock); > + > + list_for_each_entry(kvg, >group_list, node) { > + if (kvg->vfio_group != vfio_group) > + continue; > + > + ret = kvm_spapr_tce_attach_iommu_group(dev->kvm, > + param.tablefd, grp); > + break; > + } > + > + mutex_unlock(>lock); > + > + iommu_group_put(grp); > + kvm_vfio_group_put_external_user(vfio_group); > + > + return ret; > + } > +#endif /* CONFIG_SPAPR_TCE_IOMMU */ > } > > return -ENXIO; > @@ -242,6 +341,9 @@ static int kvm_vfio_has_attr(struct kvm_device *dev, > switch (attr->attr) { > case KVM_DEV_VFIO_GROUP_ADD: > case KVM_DEV_VFIO_GROUP_DEL: > +#ifdef CONFIG_SPAPR_TCE_IOMMU > + case KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE: > +#endif > return 0; > } > > @@ -257,6 +359,9 @@ static void kvm_vfio_destroy(struct kvm_device *dev) > struct kvm_vfio_group *kvg, *tmp; > > list_for_each_entry_safe(kvg, tmp, >group_list, node) { > +#ifdef CONFIG_SPAPR_TCE_IOMMU > + kvm_spapr_tce_release_vfio_group(dev->kvm, kvg->vfio_group); > +#endif > kvm_vfio_group_set_kvm(kvg->vfio_group, NULL); > kvm_vfio_group_put_external_user(kvg->vfio_group); > list_del(>node); Acked-by: Alex Williamson <alex.william...@redhat.com>

Re: [PATCH kernel v11 04/10] powerpc/vfio_spapr_tce: Add reference counting to iommu_table

s/pseries/vio.c > b/arch/powerpc/platforms/pseries/vio.c > index 720493932486..28b09fd797ec 100644 > --- a/arch/powerpc/platforms/pseries/vio.c > +++ b/arch/powerpc/platforms/pseries/vio.c > @@ -1318,7 +1318,7 @@ static void vio_dev_release(struct device *dev) > struct iommu_table *tbl = get_iommu_table_base(dev); > > if (tbl) > - iommu_free_table(tbl, of_node_full_name(dev->of_node)); > + iommu_tce_table_put(tbl); > of_node_put(dev->of_node); > kfree(to_vio_dev(dev)); > } > diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c > b/drivers/vfio/vfio_iommu_spapr_tce.c > index fbec7348a7e5..8031d3a55a17 100644 > --- a/drivers/vfio/vfio_iommu_spapr_tce.c > +++ b/drivers/vfio/vfio_iommu_spapr_tce.c > @@ -680,7 +680,7 @@ static void tce_iommu_free_table(struct tce_container > *container, > unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT; > > tce_iommu_userspace_view_free(tbl, container->mm); > - iommu_free_table(tbl, ""); > + iommu_tce_table_put(tbl); > decrement_locked_vm(container->mm, pages); > } > Acked-by: Alex Williamson <alex.william...@redhat.com>

Re: [PATCH kernel v10 10/10] KVM: PPC: VFIO: Add in-kernel acceleration for VFIO

On Fri, 17 Mar 2017 16:09:59 +1100 Alexey Kardashevskiy wrote: > This allows the host kernel to handle H_PUT_TCE, H_PUT_TCE_INDIRECT > and H_STUFF_TCE requests targeted an IOMMU TCE table used for VFIO > without passing them to user space which saves time on switching > to user

Re: [PATCH kernel v9 10/10] KVM: PPC: VFIO: Add in-kernel acceleration for VFIO

On Thu, 16 Mar 2017 18:09:32 +1100 Alexey Kardashevskiy wrote: > This allows the host kernel to handle H_PUT_TCE, H_PUT_TCE_INDIRECT > and H_STUFF_TCE requests targeted an IOMMU TCE table used for VFIO > without passing them to user space which saves time on switching > to user

Re: [PATCH kernel v8 10/10] KVM: PPC: VFIO: Add in-kernel acceleration for VFIO

On Thu, 16 Mar 2017 00:21:07 +1100 Alexey Kardashevskiy <a...@ozlabs.ru> wrote: > On 15/03/17 08:05, Alex Williamson wrote: > > On Fri, 10 Mar 2017 14:53:37 +1100 > > Alexey Kardashevskiy <a...@ozlabs.ru> wrote: > > > >> This allows the host kern

Re: [PATCH kernel v8 10/10] KVM: PPC: VFIO: Add in-kernel acceleration for VFIO

On Wed, 15 Mar 2017 15:40:14 +1100 David Gibson wrote: > > > diff --git a/arch/powerpc/kvm/book3s_64_vio.c > > > b/arch/powerpc/kvm/book3s_64_vio.c > > > index e96a4590464c..be18cda01e1b 100644 > > > --- a/arch/powerpc/kvm/book3s_64_vio.c > > > +++

Re: [PATCH kernel v8 10/10] KVM: PPC: VFIO: Add in-kernel acceleration for VFIO

On Fri, 10 Mar 2017 14:53:37 +1100 Alexey Kardashevskiy wrote: > This allows the host kernel to handle H_PUT_TCE, H_PUT_TCE_INDIRECT > and H_STUFF_TCE requests targeted an IOMMU TCE table used for VFIO > without passing them to user space which saves time on switching > to user

Re: [PATCH kernel v8 04/10] powerpc/vfio_spapr_tce: Add reference counting to iommu_table

group); > } > diff --git a/arch/powerpc/platforms/pseries/vio.c > b/arch/powerpc/platforms/pseries/vio.c > index 720493932486..744d639da92c 100644 > --- a/arch/powerpc/platforms/pseries/vio.c > +++ b/arch/powerpc/platforms/pseries/vio.c > @@ -1318,7 +1318,7 @@ static void vio_d

Re: [PATCH kernel v8 03/10] powerpc/iommu/vfio_spapr_tce: Cleanup iommu_table disposal

> static void pnv_ioda2_release_ownership(struct iommu_table_group > *table_group) > diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c > b/drivers/vfio/vfio_iommu_spapr_tce.c > index cf3de91fbfe7..fbec7348a7e5 100644 > --- a/drivers/vfio/vfio_iommu_spapr_tce.c > +++ b/drivers/vfio/vfio_iommu_spapr_tce.c > @@ -680,7 +680,7 @@ static void tce_iommu_free_table(struct tce_container > *container, > unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT; > > tce_iommu_userspace_view_free(tbl, container->mm); > - tbl->it_ops->free(tbl); > + iommu_free_table(tbl, ""); > decrement_locked_vm(container->mm, pages); > } > Acked-by: Alex Williamson <alex.william...@redhat.com>

Re: [PATCH kernel v8 00/10] powerpc/kvm/vfio: Enable in-kernel acceleration

On Tue, 14 Mar 2017 11:55:33 +1100 David Gibson wrote: > On Tue, Mar 14, 2017 at 11:54:03AM +1100, Alexey Kardashevskiy wrote: > > On 10/03/17 15:48, David Gibson wrote: > > > On Fri, Mar 10, 2017 at 02:53:27PM +1100, Alexey Kardashevskiy wrote: > > >> This is my

  1   2   3   4   >