Re: [PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

2024-02-09 Thread Alex Williamson
On Fri, 9 Feb 2024 13:19:03 -0400 Jason Gunthorpe wrote: > On Fri, Feb 09, 2024 at 08:55:31AM -0700, Alex Williamson wrote: > > I think Kevin's point is also relative to this latter scenario, in the > > L1 instance of the nvgrace-gpu driver the mmap of the usemem BAR is > > cachable, but in the

Re: [PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

2024-02-09 Thread Jason Gunthorpe
On Fri, Feb 09, 2024 at 08:55:31AM -0700, Alex Williamson wrote: > I think Kevin's point is also relative to this latter scenario, in the > L1 instance of the nvgrace-gpu driver the mmap of the usemem BAR is > cachable, but in the L2 instance of the driver where we only use the > vfio-pci-core ops

Re: [PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

2024-02-09 Thread Alex Williamson
On Fri, 9 Feb 2024 09:20:22 + Ankit Agrawal wrote: > Thanks Kevin for the review. Comments inline. > > >> > >> Note that the usemem memory is added by the VM Nvidia device driver [5] > >> to the VM kernel as memblocks. Hence make the usable memory size > >> memblock > >> aligned. > > > >

Re: [PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

2024-02-09 Thread Ankit Agrawal
>> > >> > IMO, this seems like adding too much code to reduce the call length for a >> > very specific case. If there aren't any strong opinion on this, I'm >> > planning to >> > leave this code as it is. >> >> a slight difference. if mem_count==0 the result should always succeed >> no matter

Re: [PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

2024-02-09 Thread Ankit Agrawal
Thanks Kevin for the review. Comments inline. >> >> Note that the usemem memory is added by the VM Nvidia device driver [5] >> to the VM kernel as memblocks. Hence make the usable memory size >> memblock >> aligned. > > Is memblock size defined in spec or purely a guest implementation choice?

Re: [PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

2024-02-08 Thread Alex Williamson
On Thu, 8 Feb 2024 07:21:40 + "Tian, Kevin" wrote: > > From: Ankit Agrawal > > Sent: Thursday, February 8, 2024 3:13 PM > > >> > +    * Determine how many bytes to be actually read from the > > >> > device memory. > > >> > +    * Read request beyond the actual device memory size is > > >>

Re: [PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

2024-02-08 Thread Ankit Agrawal
>> >> >> >> If mem_count == 0, going through nvgrace_gpu_map_and_read() is not >> >> necessary. >> > >> > Harmless, other than the possibly unnecessary call through to >> > nvgrace_gpu_map_device_mem().  Maybe both >> nvgrace_gpu_map_and_read() >> > and nvgrace_gpu_map_and_write() could

RE: [PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

2024-02-07 Thread Tian, Kevin
> From: Ankit Agrawal > Sent: Thursday, February 8, 2024 3:13 PM > >> > +    * Determine how many bytes to be actually read from the > >> > device memory. > >> > +    * Read request beyond the actual device memory size is > >> > filled with ~0, > >> > +    * while those beyond the actual reported

RE: [PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

2024-02-07 Thread Tian, Kevin
> From: ank...@nvidia.com > Sent: Tuesday, February 6, 2024 7:01 AM > > Note that the usemem memory is added by the VM Nvidia device driver [5] > to the VM kernel as memblocks. Hence make the usable memory size > memblock > aligned. Is memblock size defined in spec or purely a guest

Re: [PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

2024-02-07 Thread Ankit Agrawal
>> > +/* Memory size expected as non cached and reserved by the VM driver >> > */ +#define RESMEM_SIZE 0x4000 >> > +#define MEMBLK_SIZE 0x2000 >> > + >> >> Maybe use SZ_* definitions in linux/size.h > > Good suggestion. Ack. >> >> Better move this part to the place between

Re: [PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

2024-02-07 Thread Ankit Agrawal
>> diff --git a/MAINTAINERS b/MAINTAINERS >> index 8999497011a2..529ec8966f58 100644 >> --- a/MAINTAINERS >> +++ b/MAINTAINERS >> @@ -23103,6 +23103,12 @@ L:   k...@vger.kernel.org >>  S:   Maintained >>  F:   drivers/vfio/platform/ >> >> +VFIO NVIDIA GRACE GPU DRIVER >> +M:   Ankit Agrawal >>

Re: [PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

2024-02-07 Thread Alex Williamson
On Tue, 6 Feb 2024 04:31:23 +0530 wrote: > From: Ankit Agrawal > > NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device > for the on-chip GPU that is the logical OS representation of the > internal proprietary chip-to-chip cache coherent interconnect. > > The device is peculiar

Re: [PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

2024-02-07 Thread Alex Williamson
On Thu, 8 Feb 2024 00:32:10 +0200 Zhi Wang wrote: > On Tue, 6 Feb 2024 04:31:23 +0530 > wrote: > > > From: Ankit Agrawal > > > > NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device > > for the on-chip GPU that is the logical OS representation of the > > internal proprietary

Re: [PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

2024-02-07 Thread Zhi Wang
On Tue, 6 Feb 2024 04:31:23 +0530 wrote: > From: Ankit Agrawal > > NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device > for the on-chip GPU that is the logical OS representation of the > internal proprietary chip-to-chip cache coherent interconnect. > > The device is peculiar

[PATCH v17 3/3] vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

2024-02-05 Thread ankita
From: Ankit Agrawal NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device for the on-chip GPU that is the logical OS representation of the internal proprietary chip-to-chip cache coherent interconnect. The device is peculiar compared to a real PCI device in that whilst there is a