Re: [Intel-gfx] [PATCH v2 01/21] scatterlist: Introduce sg_map helper functions

2017-05-03 Thread Jason Gunthorpe
On Thu, Apr 27, 2017 at 08:53:38AM +0200, Christoph Hellwig wrote: > > The main difficulty we > > have now is that neither of those functions are expected to fail and we > > need them to be able to in cases where the page doesn't map to system > > RAM. This patch series is trying to address it

Re: [Intel-gfx] [PATCH v2 15/21] xen-blkfront: Make use of the new sg_map helper function

2017-05-03 Thread Jason Gunthorpe
On Thu, Apr 27, 2017 at 05:03:45PM -0600, Logan Gunthorpe wrote: > > > On 27/04/17 04:11 PM, Jason Gunthorpe wrote: > > On Thu, Apr 27, 2017 at 03:53:37PM -0600, Logan Gunthorpe wrote: > > Well, that is in the current form, with more users it would make sense > > to o

Re: [Intel-gfx] [PATCH v2 15/21] xen-blkfront: Make use of the new sg_map helper function

2017-05-03 Thread Jason Gunthorpe
On Thu, Apr 27, 2017 at 02:19:24PM -0600, Logan Gunthorpe wrote: > > > On 26/04/17 01:37 AM, Roger Pau Monné wrote: > > On Tue, Apr 25, 2017 at 12:21:02PM -0600, Logan Gunthorpe wrote: > >> Straightforward conversion to the new helper, except due to the lack > >> of error path, we have to use

Re: [Intel-gfx] [PATCH v2 15/21] xen-blkfront: Make use of the new sg_map helper function

2017-05-03 Thread Jason Gunthorpe
On Thu, Apr 27, 2017 at 03:53:37PM -0600, Logan Gunthorpe wrote: > On 27/04/17 02:53 PM, Jason Gunthorpe wrote: > > blkfront is one of the drivers I looked at, and it appears to only be > > memcpying with the bvec_data pointer, so I wonder why it does not use > > sg_

Re: [Intel-gfx] [PATCH rdma-next 01/21] drm/i915: Move u64-to-ptr helpers to general header

2018-05-15 Thread Jason Gunthorpe
On Thu, May 03, 2018 at 04:36:55PM +0300, Leon Romanovsky wrote: > From: Leon Romanovsky > > The macro u64_to_ptr() and function ptr_to_u64() are useful enough > to be part of general header, so move them there and allow RDMA > subsystem reuse them. > > Signed-off-by: Leon

Re: [Intel-gfx] [PATCH v3 hmm 04/11] misc/sgi-gru: use mmu_notifier_get/put for struct gru_mm_struct

2019-08-14 Thread Jason Gunthorpe
On Thu, Aug 08, 2019 at 12:25:56PM +0200, Christoph Hellwig wrote: > Looks good, > > Reviewed-by: Christoph Hellwig Dimitri, are you OK with this patch? Thanks, Jason ___ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org

Re: [Intel-gfx] [PATCH v3 hmm 08/11] drm/radeon: use mmu_notifier_get/put for struct radeon_mn

2019-08-15 Thread Jason Gunthorpe
On Thu, Aug 15, 2019 at 10:28:21AM +0200, Christian König wrote: > Am 07.08.19 um 01:15 schrieb Jason Gunthorpe: > > From: Jason Gunthorpe > > > > radeon is using a device global hash table to track what mmu_notifiers > > have been registered on struct mm. This is

Re: [Intel-gfx] [PATCH v3 hmm 10/11] drm/amdkfd: use mmu_notifier_put

2019-08-09 Thread Jason Gunthorpe
On Tue, Aug 06, 2019 at 11:47:44PM +, Kuehling, Felix wrote: > On 2019-08-06 19:15, Jason Gunthorpe wrote: > > From: Jason Gunthorpe > > > > The sequence of mmu_notifier_unregister_no_release(), > > mmu_notifier_call_srcu() is identical to mmu_notifier_put()

Re: [Intel-gfx] [PATCH V13 4/6] mdev: introduce mediated virtio bus

2019-11-18 Thread Jason Gunthorpe
On Mon, Nov 18, 2019 at 06:59:21PM +0800, Jason Wang wrote: > +struct bus_type mdev_virtio_bus_type; > + > +struct mdev_virtio_device { > + struct mdev_device mdev; > + const struct mdev_virtio_ops *ops; > + u16 class_id; > +}; This seems to share nothing with mdev (ie mdev-vfio), why

Re: [Intel-gfx] [PATCH V13 4/6] mdev: introduce mediated virtio bus

2019-11-18 Thread Jason Gunthorpe
On Mon, Nov 18, 2019 at 03:27:13PM -0500, Michael S. Tsirkin wrote: > On Mon, Nov 18, 2019 at 01:41:00PM +0000, Jason Gunthorpe wrote: > > On Mon, Nov 18, 2019 at 06:59:21PM +0800, Jason Wang wrote: > > > +struct bus_type mdev_virtio_bus_type; > > > + >

Re: [Intel-gfx] [RFC 06/13] drm/i915/svm: Page table mirroring support

2019-11-22 Thread Jason Gunthorpe
On Fri, Nov 22, 2019 at 12:01:17PM -0800, Niranjan Vishwanathapura wrote: > On Fri, Nov 22, 2019 at 11:54:45AM -0800, Niranjana Vishwanathapura wrote: > > Use HMM page table mirroring support to build device page table. > > Implement the bind ioctl and bind the process address range in the > >

Re: [Intel-gfx] [RFC 06/13] drm/i915/svm: Page table mirroring support

2019-11-22 Thread Jason Gunthorpe
On Fri, Nov 22, 2019 at 12:57:27PM -0800, Niranjana Vishwanathapura wrote: > +static inline bool > +i915_range_done(struct hmm_range *range) > +{ > + bool ret = hmm_range_valid(range); > + > + hmm_range_unregister(range); > + return ret; > +} This needs to be updated to follow the

Re: [Intel-gfx] [RFC v2 02/12] drm/i915/svm: Runtime (RT) allocator support

2019-12-17 Thread Jason Gunthorpe
On Fri, Dec 13, 2019 at 01:56:04PM -0800, Niranjana Vishwanathapura wrote: > + ctx = i915_gem_context_lookup(file->driver_priv, args->rsvd1); > + if (!ctx || !rcu_access_pointer(ctx->vm)) > + return -ENOENT; > + > + rcu_read_lock(); > + vm = i915_vm_get(ctx->vm); > +

Re: [Intel-gfx] [RFC v2 05/12] drm/i915/svm: Page table mirroring support

2019-12-17 Thread Jason Gunthorpe
On Fri, Dec 13, 2019 at 01:56:07PM -0800, Niranjana Vishwanathapura wrote: > +static struct i915_svm *vm_get_svm(struct i915_address_space *vm) > +{ > + struct i915_svm *svm = vm->svm; > + > + mutex_lock(>svm_mutex); > + if (svm && !kref_get_unless_zero(>ref)) > + svm =

Re: [Intel-gfx] [RFC v2 06/12] drm/i915/svm: Device memory support

2019-12-17 Thread Jason Gunthorpe
On Fri, Dec 13, 2019 at 01:56:08PM -0800, Niranjana Vishwanathapura wrote: > @@ -169,6 +170,11 @@ static int i915_range_fault(struct svm_notifier *sn, > return ret; > } > > + /* For dgfx, ensure the range is in device local memory only */ > +

Re: [Intel-gfx] [RFC 06/13] drm/i915/svm: Page table mirroring support

2019-12-08 Thread Jason Gunthorpe
On Wed, Dec 04, 2019 at 04:51:36PM -0500, Jerome Glisse wrote: > On Tue, Dec 03, 2019 at 11:19:43AM -0800, Niranjan Vishwanathapura wrote: > > On Tue, Nov 26, 2019 at 06:32:52PM +0000, Jason Gunthorpe wrote: > > > On Mon, Nov 25, 2019 at 11:33:27AM -0500, Jerome Glisse wrote: &

Re: [Intel-gfx] [PATCH V13 4/6] mdev: introduce mediated virtio bus

2019-11-20 Thread Jason Gunthorpe
On Wed, Nov 20, 2019 at 10:14:26AM +0800, Jason Wang wrote: > > > I don't quite get the question here. > > In the driver model the bus_type and foo_device are closely > > linked. > > I don't get the definition of "closely linked" here. Do you think the bus > and device implement virtual bus

Re: [Intel-gfx] [PATCH V13 4/6] mdev: introduce mediated virtio bus

2019-11-19 Thread Jason Gunthorpe
On Tue, Nov 19, 2019 at 10:41:31AM +0800, Jason Wang wrote: > > On 2019/11/19 上午4:28, Jason Gunthorpe wrote: > > On Mon, Nov 18, 2019 at 03:27:13PM -0500, Michael S. Tsirkin wrote: > > > On Mon, Nov 18, 2019 at 01:41:00PM +, Jason Gunthorpe wrote: > > > > O

Re: [Intel-gfx] [PATCH V13 6/6] docs: sample driver to demonstrate how to implement virtio-mdev framework

2019-11-19 Thread Jason Gunthorpe
On Tue, Nov 19, 2019 at 11:03:39AM +0800, Jason Wang wrote: > > Also, see the other conversations we are having about a "virtual" bus > > and devices. I do not want to have two different ways of doing the same > > thing in the kernel at the same time please. Please work together with > > the

Re: [Intel-gfx] [PATCH V13 4/6] mdev: introduce mediated virtio bus

2019-11-19 Thread Jason Gunthorpe
On Tue, Nov 19, 2019 at 10:02:08PM +0800, Jason Wang wrote: > > On 2019/11/19 下午8:38, Jason Gunthorpe wrote: > > On Tue, Nov 19, 2019 at 10:41:31AM +0800, Jason Wang wrote: > > > On 2019/11/19 上午4:28, Jason Gunthorpe wrote: > > > > On Mon, Nov 18, 2019 at 03:

Re: [Intel-gfx] [PATCH V13 6/6] docs: sample driver to demonstrate how to implement virtio-mdev framework

2019-11-19 Thread Jason Gunthorpe
On Tue, Nov 19, 2019 at 10:07:18PM +0800, Jason Wang wrote: > > On 2019/11/19 下午8:40, Jason Gunthorpe wrote: > > On Tue, Nov 19, 2019 at 11:03:39AM +0800, Jason Wang wrote: > > > > Also, see the other conversations we are having about a "virtual" bus > >

Re: [Intel-gfx] [RFC 06/13] drm/i915/svm: Page table mirroring support

2019-11-26 Thread Jason Gunthorpe
On Mon, Nov 25, 2019 at 08:32:58AM -0800, Niranjan Vishwanathapura wrote: > > And putting the cpu PFN of a ZONE_DEVICE device page into > > sg_dma_address still looks very wrong to me > > The below call in patch 7 does convert any cpu PFN to device address. > So, it won't be CPU PFN. >

Re: [Intel-gfx] [RFC 06/13] drm/i915/svm: Page table mirroring support

2019-11-26 Thread Jason Gunthorpe
On Mon, Nov 25, 2019 at 11:33:27AM -0500, Jerome Glisse wrote: > On Fri, Nov 22, 2019 at 11:33:12PM +0000, Jason Gunthorpe wrote: > > On Fri, Nov 22, 2019 at 12:57:27PM -0800, Niranjana Vishwanathapura wrote: > > [...] > > > > +static int > > > +i915_rang

Re: [Intel-gfx] [RFC 06/13] drm/i915/svm: Page table mirroring support

2019-11-25 Thread Jason Gunthorpe
On Sun, Nov 24, 2019 at 01:12:47PM -0800, Niranjan Vishwanathapura wrote: > > > > Using a temporary range is the pattern from nouveau, is it really > > > > necessary in this driver? > > > > > > Yah, not required. In my local build I tried with proper default_flags > > > and set pfn_flags_mask to

Re: [Intel-gfx] [RFC 06/13] drm/i915/svm: Page table mirroring support

2019-11-23 Thread Jason Gunthorpe
On Fri, Nov 22, 2019 at 08:44:18PM -0800, Niranjan Vishwanathapura wrote: > On Fri, Nov 22, 2019 at 11:33:12PM +0000, Jason Gunthorpe wrote: > > On Fri, Nov 22, 2019 at 12:57:27PM -0800, Niranjana Vishwanathapura wrote: > > > > > +static inline bool > > > +i915_r

Re: [Intel-gfx] [RFC v2 05/12] drm/i915/svm: Page table mirroring support

2019-12-20 Thread Jason Gunthorpe
On Wed, Dec 18, 2019 at 02:41:47PM -0800, Niranjana Vishwanathapura wrote: > > > +static u32 i915_svm_build_sg(struct i915_address_space *vm, > > > + struct hmm_range *range, > > > + struct sg_table *st) > > > +{ > > > + struct scatterlist *sg; > > > + u32

Re: [Intel-gfx] [PATCH rdma-next v4 1/4] lib/scatterlist: Add support in dynamic allocation of SG table from pages

2020-10-02 Thread Jason Gunthorpe
On Fri, Oct 02, 2020 at 07:11:33PM +0300, Maor Gottlieb wrote: > > On 10/2/2020 6:02 PM, Jason Gunthorpe wrote: > > On Sun, Sep 27, 2020 at 09:46:44AM +0300, Leon Romanovsky wrote: > > > +struct scatterlist *__sg_alloc_table_from_pages(struct sg_table *sgt, > > >

Re: [Intel-gfx] [PATCH rdma-next v4 1/4] lib/scatterlist: Add support in dynamic allocation of SG table from pages

2020-10-02 Thread Jason Gunthorpe
On Sun, Sep 27, 2020 at 09:46:44AM +0300, Leon Romanovsky wrote: > +struct scatterlist *__sg_alloc_table_from_pages(struct sg_table *sgt, > + struct page **pages, unsigned int n_pages, unsigned int offset, > + unsigned long size, unsigned int max_segment, > +

Re: [Intel-gfx] [PATCH rdma-next v5 0/4] Dynamicaly allocate SG table from the pages

2020-10-06 Thread Jason Gunthorpe
On Sun, Oct 04, 2020 at 06:43:36PM +0300, Leon Romanovsky wrote: > This series extends __sg_alloc_table_from_pages to allow chaining of > new pages to already initialized SG table. > > This allows for the drivers to utilize the optimization of merging contiguous > pages without a need to pre

Re: [Intel-gfx] [PATCH rdma-next v4 4/4] RDMA/umem: Move to allocate SG table from pages

2020-09-29 Thread Jason Gunthorpe
On Sun, Sep 27, 2020 at 09:46:47AM +0300, Leon Romanovsky wrote: > @@ -296,11 +223,17 @@ static struct ib_umem *__ib_umem_get(struct ib_device > *device, > goto umem_release; > > cur_base += ret * PAGE_SIZE; > - npages -= ret; > - > -

Re: [Intel-gfx] [PATCH rdma-next v4 4/4] RDMA/umem: Move to allocate SG table from pages

2020-09-30 Thread Jason Gunthorpe
On Wed, Sep 30, 2020 at 12:53:21PM +0300, Leon Romanovsky wrote: > On Tue, Sep 29, 2020 at 04:59:29PM -0300, Jason Gunthorpe wrote: > > On Sun, Sep 27, 2020 at 09:46:47AM +0300, Leon Romanovsky wrote: > > > @@ -296,11 +223,17 @@ static struct ib_umem *__ib_umem_get(struct >

Re: [Intel-gfx] [PATCH rdma-next v4 4/4] RDMA/umem: Move to allocate SG table from pages

2020-09-30 Thread Jason Gunthorpe
On Wed, Sep 30, 2020 at 02:53:58PM +0300, Maor Gottlieb wrote: > > On 9/30/2020 2:45 PM, Jason Gunthorpe wrote: > > On Wed, Sep 30, 2020 at 12:53:21PM +0300, Leon Romanovsky wrote: > > > On Tue, Sep 29, 2020 at 04:59:29PM -0300, Jason Gunthorpe wrote: > > > > O

Re: [Intel-gfx] [PATCH rdma-next v3 1/2] lib/scatterlist: Add support in dynamic allocation of SG table from pages

2020-09-25 Thread Jason Gunthorpe
On Fri, Sep 25, 2020 at 12:41:29PM +0100, Tvrtko Ursulin wrote: > > On 25/09/2020 08:13, Leon Romanovsky wrote: > > On Thu, Sep 24, 2020 at 09:21:20AM +0100, Tvrtko Ursulin wrote: > > > > > > On 22/09/2020 09:39, Leon Romanovsky wrote: > > > > From: Maor Gottlieb > > > > > > > > Extend

Re: [Intel-gfx] [PATCH rdma-next v3 1/2] lib/scatterlist: Add support in dynamic allocation of SG table from pages

2020-09-25 Thread Jason Gunthorpe
On Fri, Sep 25, 2020 at 01:29:49PM +0100, Tvrtko Ursulin wrote: > > On 25/09/2020 12:58, Jason Gunthorpe wrote: > > On Fri, Sep 25, 2020 at 12:41:29PM +0100, Tvrtko Ursulin wrote: > > > > > > On 25/09/2020 08:13, Leon Romanovsky wrote: > > > > On Thu

Re: [Intel-gfx] [PATCH rdma-next v3 1/2] lib/scatterlist: Add support in dynamic allocation of SG table from pages

2020-09-25 Thread Jason Gunthorpe
On Fri, Sep 25, 2020 at 10:13:30AM +0300, Leon Romanovsky wrote: > > > diff --git a/tools/testing/scatterlist/main.c > > > b/tools/testing/scatterlist/main.c > > > index 0a1464181226..4899359a31ac 100644 > > > +++ b/tools/testing/scatterlist/main.c > > > @@ -55,14 +55,13 @@ int main(void) > > >

Re: [Intel-gfx] [PATCH rdma-next v4 4/4] RDMA/umem: Move to allocate SG table from pages

2020-09-30 Thread Jason Gunthorpe
On Wed, Sep 30, 2020 at 06:05:15PM +0300, Maor Gottlieb wrote: > This is right only for the last iteration. E.g. in the first iteration in > case that there are more pages (left_pages), then we allocate > SG_MAX_SINGLE_ALLOC.  We don't know how many pages from the second iteration > will be

Re: [Intel-gfx] [PATCH 02/25] dma-fence: prime lockdep annotations

2020-07-10 Thread Jason Gunthorpe
On Fri, Jul 10, 2020 at 03:01:10PM +0200, Christian König wrote: > Am 10.07.20 um 14:54 schrieb Jason Gunthorpe: > > On Fri, Jul 10, 2020 at 02:48:16PM +0200, Christian König wrote: > > > Am 10.07.20 um 14:43 schrieb Jason Gunthorpe: > > > > On Thu, Jul 09, 2020 at 1

Re: [Intel-gfx] [PATCH 02/25] dma-fence: prime lockdep annotations

2020-07-10 Thread Jason Gunthorpe
On Fri, Jul 10, 2020 at 02:48:16PM +0200, Christian König wrote: > Am 10.07.20 um 14:43 schrieb Jason Gunthorpe: > > On Thu, Jul 09, 2020 at 10:09:11AM +0200, Daniel Vetter wrote: > > > Hi Jason, > > > > > > Below the paragraph I've added after our discuss

Re: [Intel-gfx] [PATCH 02/25] dma-fence: prime lockdep annotations

2020-07-10 Thread Jason Gunthorpe
On Thu, Jul 09, 2020 at 10:09:11AM +0200, Daniel Vetter wrote: > Hi Jason, > > Below the paragraph I've added after our discussions around dma-fences > outside of drivers/gpu. Good enough for an ack on this, or want something > changed? > > Thanks, Daniel > > > + * Note that only GPU drivers

Re: [Intel-gfx] [PATCH 02/25] dma-fence: prime lockdep annotations

2020-07-10 Thread Jason Gunthorpe
On Fri, Jul 10, 2020 at 04:02:35PM +0200, Daniel Vetter wrote: > > dma_fence only possibly makes some sense if you intend to expose the > > completion outside a single driver. > > > > The prefered kernel design pattern for this is to connect things with > > a function callback. > > > > So the

Re: [Intel-gfx] [PATCH 000/141] Fix fall-through warnings for Clang

2020-11-24 Thread Jason Gunthorpe
On Fri, Nov 20, 2020 at 12:21:39PM -0600, Gustavo A. R. Silva wrote: > IB/hfi1: Fix fall-through warnings for Clang > IB/mlx4: Fix fall-through warnings for Clang > IB/qedr: Fix fall-through warnings for Clang > RDMA/mlx5: Fix fall-through warnings for Clang I picked these four to the

Re: [Intel-gfx] [PATCH] drm: Remove SCATTERLIST_MAX_SEGMENT

2020-10-29 Thread Jason Gunthorpe
On Wed, Oct 28, 2020 at 08:49:11PM +0100, Daniel Vetter wrote: > On Wed, Oct 28, 2020 at 04:15:26PM -0300, Jason Gunthorpe wrote: > > Since commit 9a40401cfa13 ("lib/scatterlist: Do not limit max_segment to > > PAGE_ALIGNED values") the max_segment input to sg_alloc_ta

[Intel-gfx] [PATCH] drm: Remove SCATTERLIST_MAX_SEGMENT

2020-10-29 Thread Jason Gunthorpe
, UINT_MAX) == SCATTERLIST_MAX_SEGMENT - drm_prime_pages_to_sg uses it as a default if max_segment is zero, UINT_MAX is fine to use directly. Cc: Gerd Hoffmann Cc: Daniel Vetter Cc: Thomas Hellstrom Cc: Qian Cai Cc: "Ursulin, Tvrtko" Suggested-by: Christoph Hellwig Signed-off

Re: [Intel-gfx] [PATCH 06/10] vfio/mdev: Remove CONFIG_VFIO_MDEV_DEVICE

2021-06-14 Thread Jason Gunthorpe
On Fri, Jun 11, 2021 at 02:40:41PM +0200, Cornelia Huck wrote: > On Mon, Jun 07 2021, Jason Gunthorpe wrote: > > > For some reason the vfio_mdev shim mdev_driver has its own module and > > kconfig. As the next patch requires access to it from mdev.ko merge the > > two mod

Re: [Intel-gfx] Allow mdev drivers to directly create the vfio_device (v3)

2021-06-16 Thread Jason Gunthorpe
On Tue, Jun 15, 2021 at 01:35:49PM -0600, Alex Williamson wrote: > On Tue, 15 Jun 2021 15:35:09 +0200 > Christoph Hellwig wrote: > > > This is my alternative take on this series from Jason: > > > > https://lore.kernel.org/dri-devel/87czsszi9i@redhat.com/T/ > > > > The mdev/vfio parts are

Re: [Intel-gfx] [PATCH 07/10] vfio/mdev: Allow the mdev_parent_ops to specify the device driver to bind

2021-06-16 Thread Jason Gunthorpe
On Tue, Jun 15, 2021 at 04:11:29PM +0200, Greg Kroah-Hartman wrote: > On Tue, Jun 15, 2021 at 03:35:16PM +0200, Christoph Hellwig wrote: > > From: Jason Gunthorpe > > > > This allows a mdev driver to opt out of using vfio_mdev.c, instead the > > driver will pr

Re: [Intel-gfx] Allow mdev drivers to directly create the vfio_device (v4)

2021-06-23 Thread Jason Gunthorpe
On Thu, Jun 17, 2021 at 04:22:08PM +0200, Christoph Hellwig wrote: > This is my alternative take on this series from Jason: > > https://lore.kernel.org/dri-devel/87czsszi9i@redhat.com/T/ > > The mdev/vfio parts are exactly the same, but this solves the driver core > changes for the direct

Re: [Intel-gfx] [PATCH 00/10] Allow mdev drivers to directly create the vfio_device

2021-06-15 Thread Jason Gunthorpe
On Mon, Jun 14, 2021 at 08:04:03PM +0530, Kirti Wankhede wrote: > Jason, > > I couldn't find patch 1,2,4 and 5 of these series. Can you please keep > k...@vger.kernel.org cc for all patches? It is an error, sorry > Also it will be helpful if you can add version prefix, eg. 'v3' for this >

Re: [Intel-gfx] [PATCH 04/10] driver core: Don't return EPROBE_DEFER to userspace during sysfs bind

2021-06-15 Thread Jason Gunthorpe
On Mon, Jun 14, 2021 at 05:08:40PM +0200, Christoph Hellwig wrote: > @@ -679,8 +666,6 @@ static int really_probe(struct device *dev, struct > device_driver *drv) > dev->pm_domain->dismiss(dev); > pm_runtime_reinit(dev); > dev_pm_set_driver_flags(dev, 0); > - if

Re: [Intel-gfx] Allow mdev drivers to directly create the vfio_device (v2 / alternative)

2021-06-15 Thread Jason Gunthorpe
On Tue, Jun 15, 2021 at 07:50:21AM +0200, Christoph Hellwig wrote: > On Tue, Jun 15, 2021 at 07:21:57AM +0200, Greg Kroah-Hartman wrote: > > This looks much better as far as the driver core changes go, thank you > > for doing this. > > > > I'm guessing there will be at least one more revision of

[Intel-gfx] [PATCH 06/10] vfio/mdev: Remove CONFIG_VFIO_MDEV_DEVICE

2021-06-08 Thread Jason Gunthorpe
For some reason the vfio_mdev shim mdev_driver has its own module and kconfig. As the next patch requires access to it from mdev.ko merge the two modules together and remove VFIO_MDEV_DEVICE. A later patch deletes this driver entirely. Signed-off-by: Jason Gunthorpe --- Documentation/s390/vfio

[Intel-gfx] [PATCH 00/10] Allow mdev drivers to directly create the vfio_device

2021-06-08 Thread Jason Gunthorpe
egister()/bus_unregister() - Reflow long doc lines v1: https://lore.kernel.org/r/0-v1-d88406ed308e+418-vfio3_...@nvidia.com Jason Gunthorpe (10): driver core: Do not continue searching for drivers if deferred probe is used driver core: Pull required checks into driver_probe_device() driver co

[Intel-gfx] [PATCH 08/12] vfio/gvt: Convert to use vfio_register_group_dev()

2021-04-26 Thread Jason Gunthorpe
While there is a confusing mess of pointers and structs in this driver, the struct kvmgt_vdev (which in turn is 1:1 with a struct intel_vgpu) is what holds the vfio_device. Replace all the drvdata's and weird derivations of vgpu and vdev with container_of() or vdev->vgpu. Signed-off-by: Ja

[Intel-gfx] [PATCH 10/12] vfio/mdev: Remove mdev_parent_ops

2021-04-26 Thread Jason Gunthorpe
The last useful member in this struct is the supported_type_groups, move it to the mdev_driver and delete mdev_parent_ops. Replace it with mdev_driver as an argument to mdev_register_device() Signed-off-by: Jason Gunthorpe --- .../driver-api/vfio-mediated-device.rst | 36

[Intel-gfx] [PATCH 00/12] Remove vfio_mdev.c, mdev_parent_ops and more

2021-04-26 Thread Jason Gunthorpe
n part, the only VFIO special part of mdev that remains is the mdev specific iommu intervention. Thanks, Jason Jason Gunthorpe (12): vfio/mdev: Remove CONFIG_VFIO_MDEV_DEVICE vfio/mdev: Allow the mdev_parent_ops to specify the device driver to bind vfio/mtty: Convert to use vfio_register_gro

[Intel-gfx] [PATCH 01/12] vfio/mdev: Remove CONFIG_VFIO_MDEV_DEVICE

2021-04-26 Thread Jason Gunthorpe
prevented the samples from being built in. Signed-off-by: Jason Gunthorpe --- Documentation/s390/vfio-ap.rst | 1 - arch/s390/Kconfig| 2 +- drivers/gpu/drm/i915/Kconfig | 2 +- drivers/vfio/mdev/Kconfig| 7 --- drivers/vfio/mdev/Makefile | 3 +-- drivers

[Intel-gfx] [PATCH v2 00/13] Remove vfio_mdev.c, mdev_parent_ops and more

2021-04-27 Thread Jason Gunthorpe
l bus_register()/bus_unregister() - Reflow long doc lines v1: https://lore.kernel.org/r/0-v1-d88406ed308e+418-vfio3_...@nvidia.com Jason Cc: Leon Romanovsky Cc: "Raj, Ashok" Cc: Dan Williams Cc: Max Gurtovoy Cc: Christoph Hellwig Cc: Tarun Gupta Cc: Daniel Vetter Jason Gunthorpe (

[Intel-gfx] [PATCH v2 11/13] vfio/mdev: Remove mdev_parent_ops

2021-04-27 Thread Jason Gunthorpe
The last useful member in this struct is the supported_type_groups, move it to the mdev_driver and delete mdev_parent_ops. Replace it with mdev_driver as an argument to mdev_register_device() Reviewed-by: Christoph Hellwig Signed-off-by: Jason Gunthorpe --- .../driver-api/vfio-mediated

Re: [Intel-gfx] [PATCH 10/12] vfio/mdev: Remove mdev_parent_ops

2021-04-27 Thread Jason Gunthorpe
On Mon, Apr 26, 2021 at 04:19:11PM +0200, Christoph Hellwig wrote: > > +The mediated bus driver's probe function should create a vfio_device on > > top of > > +the mdev_device and connect it to an appropriate implementation of > > vfio_device_ops. > > Overly long line. > > > +This will provide

Re: [Intel-gfx] [PATCH 00/12] Remove vfio_mdev.c, mdev_parent_ops and more

2021-04-27 Thread Jason Gunthorpe
On Mon, Apr 26, 2021 at 06:43:14PM +0200, Christian Borntraeger wrote: > On 24.04.21 01:02, Jason Gunthorpe wrote: > > Prologue > > > > > > This is series #3 in part of a larger work that arose from the minor > > remark that the mdev_paren

Re: [Intel-gfx] [PATCH 01/12] vfio/mdev: Remove CONFIG_VFIO_MDEV_DEVICE

2021-04-27 Thread Jason Gunthorpe
On Fri, Apr 23, 2021 at 05:08:10PM -0700, Randy Dunlap wrote: > On 4/23/21 4:02 PM, Jason Gunthorpe wrote: > > @@ -171,7 +171,7 @@ config SAMPLE_VFIO_MDEV_MDPY_FB > > > > config SAMPLE_VFIO_MDEV_MBOCHS > > tristate "Build VFIO mdpy example mediated dev

Re: [Intel-gfx] [PATCH 08/12] vfio/gvt: Convert to use vfio_register_group_dev()

2021-04-27 Thread Jason Gunthorpe
that are connected to it. Preserve VFIO's design of allowing mdev drivers to be !GPL by allowing the three functions that replace this module for !GPL usage. This goes along with the other 19 symbols that are already marked !GPL in VFIO. Signed-off-by: Jaso

[Intel-gfx] [PATCH v2 01/13] vfio/mdev: Remove CONFIG_VFIO_MDEV_DEVICE

2021-04-27 Thread Jason Gunthorpe
For some reason the vfio_mdev shim mdev_driver has its own module and kconfig. As the next patch requires access to it from mdev.ko merge the two modules together and remove VFIO_MDEV_DEVICE. A later patch deletes this driver entirely. Signed-off-by: Jason Gunthorpe --- Documentation/s390/vfio

[Intel-gfx] [PATCH v2 08/13] vfio/gvt: Convert to use vfio_register_group_dev()

2021-04-27 Thread Jason Gunthorpe
While there is a confusing mess of pointers and structs in this driver, the struct kvmgt_vdev (which in turn is 1:1 with a struct intel_vgpu) is what holds the vfio_device. Replace all the drvdata's and weird derivations of vgpu and vdev with container_of() or vdev->vgpu. Signed-off-by: Ja

Re: [Intel-gfx] [PATCH v2 00/13] Remove vfio_mdev.c, mdev_parent_ops and more

2021-04-28 Thread Jason Gunthorpe
On Tue, Apr 27, 2021 at 03:30:42PM -0600, Alex Williamson wrote: > It'd be really helpful if you could consistently copy at least one > list, preferably one monitored by patchwork, for an entire series. The > kvm list is missing patches 06 and 08. I can find the latter hopping > over to the

Re: [Intel-gfx] [PATCH 00/12] Remove vfio_mdev.c, mdev_parent_ops and more

2021-04-28 Thread Jason Gunthorpe
On Tue, Apr 27, 2021 at 09:33:56AM +0200, Christian Borntraeger wrote: > On 26.04.21 19:42, Jason Gunthorpe wrote: > > On Mon, Apr 26, 2021 at 06:43:14PM +0200, Christian Borntraeger wrote: > > > On 24.04.21 01:02, Jason Gunthorpe wrote

Re: [Intel-gfx] [PATCH 15/18] vfio/gvt: Make DRM_I915_GVT depend on VFIO_MDEV

2021-03-24 Thread Jason Gunthorpe
On Tue, Mar 23, 2021 at 08:26:30PM +0100, Christoph Hellwig wrote: > On Tue, Mar 23, 2021 at 02:55:32PM -0300, Jason Gunthorpe wrote: > > Ideally all of this would be moved to kvmgt.c, but it is entangled with > > the rest of the "generic" code in an odd way. Thus put in

Re: [Intel-gfx] [PATCH 18/18] vfio/mdev: Correct the function signatures for the mdev_type_attributes

2021-04-06 Thread Jason Gunthorpe
On Tue, Mar 23, 2021 at 08:31:03PM +0100, Christoph Hellwig wrote: > > - type = intel_gvt_find_vgpu_type(gvt, mtype_get_type_group_id(kobj)); > > + type = intel_gvt_find_vgpu_type(gvt, mtype_get_type_group_id(mtype)); > > Somewhere in this series you should probably switch >

[Intel-gfx] [PATCH v2 00/18] Make vfio_mdev type safe

2021-04-06 Thread Jason Gunthorpe
r/0-v1-7dedf20b2b75+4f785-vfio2_...@nvidia.com Jason Gunthorpe (18): vfio/mdev: Fix missing static's on MDEV_TYPE_ATTR's vfio/mdev: Do not allow a mdev_type to have a NULL parent pointer vfio/mdev: Add missing typesafety around mdev_device vfio/mdev: Simplify driver registration vfio/m

[Intel-gfx] [PATCH v2 16/18] vfio/gvt: Use mdev_get_type_group_id()

2021-04-06 Thread Jason Gunthorpe
intel_gvt_init_vgpu_type_groups() makes gvt->types 1:1 with the supported_type_groups array, so the type_group_id is also the index into gvt->types. Use it directly and remove the string matching. Reviewed-by: Kevin Tian Reviewed-by: Christoph Hellwig Signed-off-by: Jason Gun

[Intel-gfx] [PATCH v2 18/18] vfio/mdev: Correct the function signatures for the mdev_type_attributes

2021-04-06 Thread Jason Gunthorpe
would be hard to add in retroactively, so do it now. Reviewed-by: Kevin Tian Reviewed-by: Cornelia Huck Reviewed-by: Christoph Hellwig Signed-off-by: Jason Gunthorpe --- drivers/gpu/drm/i915/gvt/gvt.c| 21 +++-- drivers/s390/cio/vfio_ccw_ops.c | 15 +-- driver

[Intel-gfx] [PATCH v2 15/18] vfio/gvt: Make DRM_I915_GVT depend on VFIO_MDEV

2021-04-06 Thread Jason Gunthorpe
randconfig failures when the next patch creates a link time dependency related to the use of MDEV_TYPE. Reviewed-by: Kevin Tian Reviewed-by: Christoph Hellwig Signed-off-by: Jason Gunthorpe --- drivers/gpu/drm/i915/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/gpu/drm/i915/K

[Intel-gfx] [PATCH 15/18] vfio/gvt: Make DRM_I915_GVT depend on VFIO_MDEV

2021-03-23 Thread Jason Gunthorpe
randconfig failures when the next patch creates a link time dependency related to the use of MDEV_TYPE. Signed-off-by: Jason Gunthorpe --- drivers/gpu/drm/i915/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/gpu/drm/i915/Kconfig b/drivers/gpu/drm/i915/Kconfig index 1e1cb245fc

[Intel-gfx] [PATCH 18/18] vfio/mdev: Correct the function signatures for the mdev_type_attributes

2021-03-23 Thread Jason Gunthorpe
would be hard to add in retroactively, so do it now. Signed-off-by: Jason Gunthorpe --- drivers/gpu/drm/i915/gvt/gvt.c| 21 +++-- drivers/s390/cio/vfio_ccw_ops.c | 15 +-- drivers/s390/crypto/vfio_ap_ops.c | 12 +++- drivers/vfio/mdev/mdev_core.c

[Intel-gfx] [PATCH 16/18] vfio/gvt: Use mdev_get_type_group_id()

2021-03-23 Thread Jason Gunthorpe
intel_gvt_init_vgpu_type_groups() makes gvt->types 1:1 with the supported_type_groups array, so the type_group_id is also the index into gvt->types. Use it directly and remove the string matching. Signed-off-by: Jason Gunthorpe --- drivers/gpu/drm/i915/gvt/gvt.c

[Intel-gfx] [PATCH 17/18] vfio/mdev: Remove kobj from mdev_parent_ops->create()

2021-03-23 Thread Jason Gunthorpe
The kobj here is a type-erased version of mdev_type, which is already stored in the struct mdev_device being passed in. It was only ever used to compute the type_group_id, which is now extracted directly from the mdev. Signed-off-by: Jason Gunthorpe --- drivers/gpu/drm/i915/gvt/kvmgt.c | 2

[Intel-gfx] [PATCH 00/18] Make vfio_mdev type safe

2021-03-23 Thread Jason Gunthorpe
kobj->name' matching by realizing the the kobj is a mdev_type, which is linked to the supported_types_list provided by the driver, and thus the core code can directly return the array indexes all the drivers actually want. Jason Jason Gunthorpe (18): vfio/mdev: Fix missing static's on MDEV_TYPE_

[Intel-gfx] [PATCH v2 17/18] vfio/mdev: Remove kobj from mdev_parent_ops->create()

2021-04-07 Thread Jason Gunthorpe
Hellwig Signed-off-by: Jason Gunthorpe --- drivers/gpu/drm/i915/gvt/kvmgt.c | 2 +- drivers/s390/cio/vfio_ccw_ops.c | 2 +- drivers/s390/crypto/vfio_ap_ops.c | 2 +- drivers/vfio/mdev/mdev_core.c | 2 +- include/linux/mdev.h | 3 +-- samples/vfio-mdev/mbochs.c| 2

Re: [Intel-gfx] [vfio:next 33/38] drivers/gpu/drm/i915/i915_pci.c:975:2: warning: missing field 'override_only' initializer

2021-08-27 Thread Jason Gunthorpe
On Fri, Aug 27, 2021 at 03:12:36PM +, kernel test robot wrote: > tree: https://github.com/awilliam/linux-vfio.git next > head: ea870730d83fc13a5fa2bd0e175176d7ac8a400a > commit: 343b7258687ecfbb363bfda8833a7cf641aac524 [33/38] PCI: Add > 'override_only' field to struct pci_device_id >

Re: [Intel-gfx] [PATCH rdma-next v4 2/3] lib/scatterlist: Fix wrong update of orig_nents

2021-08-24 Thread Jason Gunthorpe
On Tue, Aug 24, 2021 at 05:25:30PM +0300, Maor Gottlieb wrote: > @@ -514,11 +531,13 @@ struct scatterlist > *sg_alloc_append_table_from_pages(struct sg_table *sgt, > offset = 0; > cur_page = j; > } > - sgt->nents += added_nents; > + sgt_append->sgt.nents

Re: [Intel-gfx] [PATCH rdma-next v3 2/3] lib/scatterlist: Fix wrong update of orig_nents

2021-08-23 Thread Jason Gunthorpe
On Mon, Aug 23, 2021 at 02:09:37PM +0300, Maor Gottlieb wrote: > > On 8/20/2021 6:54 PM, Jason Gunthorpe wrote: > > On Thu, Jul 29, 2021 at 12:39:12PM +0300, Leon Romanovsky wrote: > > > > > +/** > > > + * __sg_free_table - Free a previously mapped sg table

Re: [Intel-gfx] [PATCH rdma-next v3 2/3] lib/scatterlist: Fix wrong update of orig_nents

2021-08-20 Thread Jason Gunthorpe
On Thu, Jul 29, 2021 at 12:39:12PM +0300, Leon Romanovsky wrote: > +/** > + * __sg_free_table - Free a previously mapped sg table > + * @table: The sg table header to use > + * @max_ents:The maximum number of entries per single scatterlist > + * @total_ents: The total number of

Re: [Intel-gfx] [PATCH rdma-next v3 2/3] lib/scatterlist: Fix wrong update of orig_nents

2021-08-20 Thread Jason Gunthorpe
On Fri, Aug 20, 2021 at 12:54:25PM -0300, Jason Gunthorpe wrote: > On Thu, Jul 29, 2021 at 12:39:12PM +0300, Leon Romanovsky wrote: > > > +/** > > + * __sg_free_table - Free a previously mapped sg table > > + * @table: The sg table header to use > > + * @max

Re: [Intel-gfx] [PATCH rdma-next v3 0/3] SG fix together with update to RDMA umem

2021-08-20 Thread Jason Gunthorpe
On Thu, Jul 29, 2021 at 12:39:10PM +0300, Leon Romanovsky wrote: > From: Leon Romanovsky > > Changelog: > v3: > * Rewrote to new API suggestion > * Split for more patches > v2: https://lore.kernel.org/lkml/cover.1626605893.git.leo...@nvidia.com > * Changed implementation of first patch, based

Re: [Intel-gfx] [PATCH rdma-next v3 2/3] lib/scatterlist: Fix wrong update of orig_nents

2021-08-23 Thread Jason Gunthorpe
On Mon, Aug 23, 2021 at 04:45:45PM +0300, Maor Gottlieb wrote: > > On 8/23/2021 3:45 PM, Jason Gunthorpe wrote: > > On Mon, Aug 23, 2021 at 02:09:37PM +0300, Maor Gottlieb wrote: > > > On 8/20/2021 6:54 PM, Jason Gunthorpe wrote: > > > > On Thu, Jul 29, 2021 at

Re: [Intel-gfx] [PATCH rdma-next v4 2/3] lib/scatterlist: Fix wrong update of orig_nents

2021-08-25 Thread Jason Gunthorpe
On Wed, Aug 25, 2021 at 07:59:27AM +0300, Maor Gottlieb wrote: > > On 8/24/2021 10:12 PM, Jason Gunthorpe wrote: > > On Tue, Aug 24, 2021 at 05:25:30PM +0300, Maor Gottlieb wrote: > > > @@ -514,11 +531,13 @@ struct scatterlist > > > *sg_alloc_append_table_f

Re: [Intel-gfx] [PATCH v2 5/9] vfio/mdev: Consolidate all the device_api sysfs into the core code

2021-09-10 Thread Jason Gunthorpe
On Fri, Sep 10, 2021 at 01:10:46PM +0100, Christoph Hellwig wrote: > On Thu, Sep 09, 2021 at 04:38:45PM -0300, Jason Gunthorpe wrote: > > Every driver just emits a static string, simply feed it through the ops > > and provide a standard sysfs show function. > > Looks sensib

Re: [Intel-gfx] [PATCH v2 1/9] vfio/ccw: Use functions for alloc/free of the vfio_ccw_private

2021-09-14 Thread Jason Gunthorpe
On Tue, Sep 14, 2021 at 05:50:25PM +0200, Cornelia Huck wrote: > On Fri, Sep 10 2021, Christoph Hellwig wrote: > > > On Thu, Sep 09, 2021 at 04:38:41PM -0300, Jason Gunthorpe wrote: > >> + > >> + private = kzalloc(sizeof(*private), GFP_KERNEL |

Re: [Intel-gfx] [PATCH v2 0/9] Move vfio_ccw to the new mdev API

2021-09-17 Thread Jason Gunthorpe
On Fri, Sep 17, 2021 at 01:59:16PM +0200, Cornelia Huck wrote: > > ret = cio_cancel_halt_clear(sch, ); > > - > > if (ret == -EIO) { > > pr_err("vfio_ccw: could not quiesce subchannel > > 0.%x.%04x!\n", > >sch->schid.ssid,

Re: [Intel-gfx] [PATCH v2 0/9] Move vfio_ccw to the new mdev API

2021-09-14 Thread Jason Gunthorpe
On Mon, Sep 13, 2021 at 04:31:54PM -0400, Eric Farman wrote: > > I rebased it and fixed it up here: > > > > https://github.com/jgunthorpe/linux/tree/vfio_ccw > > > > Can you try again? > > That does address the crash, but then why is it processing a BROKEN > event? Seems problematic. The

Re: [Intel-gfx] [PATCH v2 0/9] Move vfio_ccw to the new mdev API

2021-09-13 Thread Jason Gunthorpe
On Mon, Sep 13, 2021 at 01:40:34PM -0400, Eric Farman wrote: > On Thu, 2021-09-09 at 16:38 -0300, Jason Gunthorpe wrote: > > This addresses Cornelia's remark on the earlier patch that ccw has a > > confusing lifecycle. While it doesn't seem like the original attempt > > was &

[Intel-gfx] [PATCH v2 6/9] vfio/mdev: Add mdev available instance checking to the core

2021-09-09 Thread Jason Gunthorpe
for their mtypes which is fixed at registration time. The core provides a standard sysfs attribute to return the available_instances. Signed-off-by: Jason Gunthorpe --- drivers/s390/cio/vfio_ccw_drv.c | 1 - drivers/s390/cio/vfio_ccw_ops.c | 26 ++- drivers/s390/cio

[Intel-gfx] [PATCH v2 1/9] vfio/ccw: Use functions for alloc/free of the vfio_ccw_private

2021-09-09 Thread Jason Gunthorpe
Makes the code easier to understand what is memory lifecycle and what is other stuff. Signed-off-by: Jason Gunthorpe --- drivers/s390/cio/vfio_ccw_drv.c | 137 ++-- 1 file changed, 78 insertions(+), 59 deletions(-) diff --git a/drivers/s390/cio/vfio_ccw_drv.c b

[Intel-gfx] [PATCH v2 4/9] vfio/ccw: Make the FSM complete and synchronize it to the mdev

2021-09-09 Thread Jason Gunthorpe
sch_shutdown() now simply tries to close and leaves the device BROKEN (though arguably the bus should take care to quiet down the subchannel HW during shutdown, not the drivers) Signed-off-by: Jason Gunthorpe --- drivers/s390/cio/vfio_ccw_drv.c | 74 ++-- drivers/s390/cio/vf

[Intel-gfx] [PATCH v2 8/9] vfio: Export vfio_device_try_get()

2021-09-09 Thread Jason Gunthorpe
vfio_ccw will need it. Signed-off-by: Jason Gunthorpe --- drivers/vfio/vfio.c | 3 ++- include/linux/vfio.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c index c3ca33e513c8e9..e78278a0b98a96 100644 --- a/drivers/vfio/vfio.c +++ b

[Intel-gfx] [PATCH v2 2/9] vfio/ccw: Pass vfio_ccw_private not mdev_device to various functions

2021-09-09 Thread Jason Gunthorpe
mdev_device should only be used in functions assigned to ops callbacks, interior functions should use the struct vfio_ccw_private instead of repeatedly trying to get it from the mdev. Signed-off-by: Jason Gunthorpe --- drivers/s390/cio/vfio_ccw_ops.c | 37 + 1

[Intel-gfx] [PATCH v2 9/9] vfio/ccw: Move the lifecycle of the struct vfio_ccw_private to the mdev

2021-09-09 Thread Jason Gunthorpe
is allocated/freed during probe/remove of the mdev like any other vfio_device struct. Signed-off-by: Jason Gunthorpe --- drivers/s390/cio/vfio_ccw_drv.c | 67 ++--- drivers/s390/cio/vfio_ccw_ops.c | 40 +++-- drivers/s390/cio/vfio_ccw_private.h | 23

[Intel-gfx] [PATCH v2 5/9] vfio/mdev: Consolidate all the device_api sysfs into the core code

2021-09-09 Thread Jason Gunthorpe
Every driver just emits a static string, simply feed it through the ops and provide a standard sysfs show function. Signed-off-by: Jason Gunthorpe --- drivers/gpu/drm/i915/gvt/kvmgt.c | 9 + drivers/s390/cio/vfio_ccw_ops.c | 9 + drivers/s390/crypto/vfio_ap_ops.c | 9

[Intel-gfx] [PATCH v2 3/9] vfio/ccw: Convert to use vfio_register_group_dev()

2021-09-09 Thread Jason Gunthorpe
and the mdev_device point at the private, and container_of is used to get it back from the vfio_device. Signed-off-by: Jason Gunthorpe --- drivers/s390/cio/vfio_ccw_drv.c | 21 -- drivers/s390/cio/vfio_ccw_ops.c | 107 +--- drivers/s390/cio/vfio_ccw_private.h | 5

[Intel-gfx] [PATCH v2 7/9] vfio/ccw: Remove private->mdev

2021-09-09 Thread Jason Gunthorpe
ed-off-by: Jason Gunthorpe --- drivers/s390/cio/vfio_ccw_drv.c | 6 ++-- drivers/s390/cio/vfio_ccw_fsm.c | 48 + drivers/s390/cio/vfio_ccw_ops.c | 16 -- drivers/s390/cio/vfio_ccw_private.h | 2 -- include/linux/mdev.h| 4 --- 5

  1   2   3   4   5   >