Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

>Yes i need to document that some more in hmm.txt... Hi Jermone, thanks for the explanation. Can I suggest you update hmm.txt with what you sent out? > I am about to send RFC for nouveau, i am still working out some bugs. Great. I will keep an eye out for it. An example user of hmm will

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

>Yes i need to document that some more in hmm.txt... Hi Jermone, thanks for the explanation. Can I suggest you update hmm.txt with what you sent out? > I am about to send RFC for nouveau, i am still working out some bugs. Great. I will keep an eye out for it. An example user of hmm will

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 02/03/18 02:44 PM, Benjamin Herrenschmidt wrote: Allright, so, I think I have a plan to fix this, but it will take a little bit of time. Basically the idea is to have firmware pass to Linux a region that's known to not have anything in it that it can use for the vmalloc space rather than

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 02/03/18 02:44 PM, Benjamin Herrenschmidt wrote: Allright, so, I think I have a plan to fix this, but it will take a little bit of time. Basically the idea is to have firmware pass to Linux a region that's known to not have anything in it that it can use for the vmalloc space rather than

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, Mar 02, 2018 at 09:38:43PM +, Stephen Bates wrote: > > It seems people miss-understand HMM :( > > Hi Jerome > > Your unhappy face emoticon made me sad so I went off to (re)read up > on HMM. Along the way I came up with a couple of things. > > While hmm.txt is really nice to read

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, Mar 02, 2018 at 09:38:43PM +, Stephen Bates wrote: > > It seems people miss-understand HMM :( > > Hi Jerome > > Your unhappy face emoticon made me sad so I went off to (re)read up > on HMM. Along the way I came up with a couple of things. > > While hmm.txt is really nice to read

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, 2018-03-02 at 10:25 +1100, Benjamin Herrenschmidt wrote: > On Thu, 2018-03-01 at 16:19 -0700, Logan Gunthorpe wrote: > > > > On 01/03/18 04:00 PM, Benjamin Herrenschmidt wrote: > > > We use only 52 in practice but yes. > > > > > > > That's 64PB. If you use need > > > > a sparse vmemmap

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, 2018-03-02 at 10:25 +1100, Benjamin Herrenschmidt wrote: > On Thu, 2018-03-01 at 16:19 -0700, Logan Gunthorpe wrote: > > > > On 01/03/18 04:00 PM, Benjamin Herrenschmidt wrote: > > > We use only 52 in practice but yes. > > > > > > > That's 64PB. If you use need > > > > a sparse vmemmap

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

> It seems people miss-understand HMM :( Hi Jerome Your unhappy face emoticon made me sad so I went off to (re)read up on HMM. Along the way I came up with a couple of things. While hmm.txt is really nice to read it makes no mention of DEVICE_PRIVATE and DEVICE_PUBLIC. It also gives no

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

> It seems people miss-understand HMM :( Hi Jerome Your unhappy face emoticon made me sad so I went off to (re)read up on HMM. Along the way I came up with a couple of things. While hmm.txt is really nice to read it makes no mention of DEVICE_PRIVATE and DEVICE_PUBLIC. It also gives no

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, 2018-03-02 at 08:57 -0800, Linus Torvalds wrote: > On Fri, Mar 2, 2018 at 8:22 AM, Kani, Toshi wrote: > > > > FWIW, this thing is called MTRRs on x86, which are initialized by BIOS. > > No. > > Or rather, that's simply just another (small) part of it all - and an >

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, 2018-03-02 at 08:57 -0800, Linus Torvalds wrote: > On Fri, Mar 2, 2018 at 8:22 AM, Kani, Toshi wrote: > > > > FWIW, this thing is called MTRRs on x86, which are initialized by BIOS. > > No. > > Or rather, that's simply just another (small) part of it all - and an > architected and

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, Mar 2, 2018 at 8:57 AM, Linus Torvalds wrote: > > Like the page table caching entries, the memory type range registers > are really just "secondary information". They don't actually select > between PCIe and RAM, they just affect the behavior on top of that.

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, Mar 2, 2018 at 8:57 AM, Linus Torvalds wrote: > > Like the page table caching entries, the memory type range registers > are really just "secondary information". They don't actually select > between PCIe and RAM, they just affect the behavior on top of that. Side note: historically the

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, Mar 2, 2018 at 8:22 AM, Kani, Toshi wrote: > > FWIW, this thing is called MTRRs on x86, which are initialized by BIOS. No. Or rather, that's simply just another (small) part of it all - and an architected and documented one at that. Like the page table caching

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, Mar 2, 2018 at 8:22 AM, Kani, Toshi wrote: > > FWIW, this thing is called MTRRs on x86, which are initialized by BIOS. No. Or rather, that's simply just another (small) part of it all - and an architected and documented one at that. Like the page table caching entries, the memory type

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, 2018-03-02 at 09:34 +1100, Benjamin Herrenschmidt wrote: > On Thu, 2018-03-01 at 14:31 -0800, Linus Torvalds wrote: > > On Thu, Mar 1, 2018 at 2:06 PM, Benjamin Herrenschmidt > > wrote: > > > > > > Could be that x86 has the smarts to do the right thing, still trying to

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, 2018-03-02 at 09:34 +1100, Benjamin Herrenschmidt wrote: > On Thu, 2018-03-01 at 14:31 -0800, Linus Torvalds wrote: > > On Thu, Mar 1, 2018 at 2:06 PM, Benjamin Herrenschmidt > > wrote: > > > > > > Could be that x86 has the smarts to do the right thing, still trying to > > > untangle

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 04:26 PM, Benjamin Herrenschmidt wrote: The big problem is not the vmemmap, it's the linear mapping. Ah, yes, ok. Logan

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 04:26 PM, Benjamin Herrenschmidt wrote: The big problem is not the vmemmap, it's the linear mapping. Ah, yes, ok. Logan

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 16:19 -0700, Logan Gunthorpe wrote: (Switching back to my non-IBM address ...) > On 01/03/18 04:00 PM, Benjamin Herrenschmidt wrote: > > We use only 52 in practice but yes. > > > > > That's 64PB. If you use need > > > a sparse vmemmap for the entire space it will take

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 16:19 -0700, Logan Gunthorpe wrote: (Switching back to my non-IBM address ...) > On 01/03/18 04:00 PM, Benjamin Herrenschmidt wrote: > > We use only 52 in practice but yes. > > > > > That's 64PB. If you use need > > > a sparse vmemmap for the entire space it will take

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 16:19 -0700, Logan Gunthorpe wrote: > > On 01/03/18 04:00 PM, Benjamin Herrenschmidt wrote: > > We use only 52 in practice but yes. > > > > > That's 64PB. If you use need > > > a sparse vmemmap for the entire space it will take 16TB which leaves you > > > with 63.98PB of

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 16:19 -0700, Logan Gunthorpe wrote: > > On 01/03/18 04:00 PM, Benjamin Herrenschmidt wrote: > > We use only 52 in practice but yes. > > > > > That's 64PB. If you use need > > > a sparse vmemmap for the entire space it will take 16TB which leaves you > > > with 63.98PB of

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 04:00 PM, Benjamin Herrenschmidt wrote: We use only 52 in practice but yes. That's 64PB. If you use need a sparse vmemmap for the entire space it will take 16TB which leaves you with 63.98PB of address space left. (Similar calculations for other numbers of address bits.) We

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 04:00 PM, Benjamin Herrenschmidt wrote: We use only 52 in practice but yes. That's 64PB. If you use need a sparse vmemmap for the entire space it will take 16TB which leaves you with 63.98PB of address space left. (Similar calculations for other numbers of address bits.) We

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 14:57 -0700, Logan Gunthorpe wrote: > > On 01/03/18 02:45 PM, Logan Gunthorpe wrote: > > It handles it fine for many situations. But when you try to map > > something that is at the end of the physical address space then the > > spares-vmemmap needs virtual address space

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 14:57 -0700, Logan Gunthorpe wrote: > > On 01/03/18 02:45 PM, Logan Gunthorpe wrote: > > It handles it fine for many situations. But when you try to map > > something that is at the end of the physical address space then the > > spares-vmemmap needs virtual address space

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 14:31 -0800, Linus Torvalds wrote: > On Thu, Mar 1, 2018 at 2:06 PM, Benjamin Herrenschmidt > wrote: > > > > Could be that x86 has the smarts to do the right thing, still trying to > > untangle the code :-) > > Afaik, x86 will not cache PCI unless the

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 14:31 -0800, Linus Torvalds wrote: > On Thu, Mar 1, 2018 at 2:06 PM, Benjamin Herrenschmidt > wrote: > > > > Could be that x86 has the smarts to do the right thing, still trying to > > untangle the code :-) > > Afaik, x86 will not cache PCI unless the system is

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, Mar 1, 2018 at 2:06 PM, Benjamin Herrenschmidt wrote: > > Could be that x86 has the smarts to do the right thing, still trying to > untangle the code :-) Afaik, x86 will not cache PCI unless the system is misconfigured, and even then it's more likely to just raise a

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, Mar 1, 2018 at 2:06 PM, Benjamin Herrenschmidt wrote: > > Could be that x86 has the smarts to do the right thing, still trying to > untangle the code :-) Afaik, x86 will not cache PCI unless the system is misconfigured, and even then it's more likely to just raise a machine check

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 13:53 -0700, Jason Gunthorpe wrote: > On Fri, Mar 02, 2018 at 07:40:15AM +1100, Benjamin Herrenschmidt wrote: > > Also we need to be able to hard block MEMREMAP_WB mappings of non-RAM > > on ppc64 (maybe via an arch hook as it might depend on the processor > > family). Server

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 13:53 -0700, Jason Gunthorpe wrote: > On Fri, Mar 02, 2018 at 07:40:15AM +1100, Benjamin Herrenschmidt wrote: > > Also we need to be able to hard block MEMREMAP_WB mappings of non-RAM > > on ppc64 (maybe via an arch hook as it might depend on the processor > > family). Server

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 02:45 PM, Logan Gunthorpe wrote: It handles it fine for many situations. But when you try to map something that is at the end of the physical address space then the spares-vmemmap needs virtual address space that's the size of the physical address space divided by PAGE_SIZE which

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 02:45 PM, Logan Gunthorpe wrote: It handles it fine for many situations. But when you try to map something that is at the end of the physical address space then the spares-vmemmap needs virtual address space that's the size of the physical address space divided by PAGE_SIZE which

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 02:37 PM, Dan Williams wrote: Ah ok, I'd need to look at the details. I had been assuming that sparse-vmemmap could handle such a situation, but that could indeed be a broken assumption. It handles it fine for many situations. But when you try to map something that is at the end

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 02:37 PM, Dan Williams wrote: Ah ok, I'd need to look at the details. I had been assuming that sparse-vmemmap could handle such a situation, but that could indeed be a broken assumption. It handles it fine for many situations. But when you try to map something that is at the end

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

> The intention of HMM is to be useful for all device memory that wish > to have struct page for various reasons. Hi Jermone and thanks for your input! Understood. We have looked at HMM in the past and long term I definitely would like to consider how we can add P2P functionality to HMM for

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

> The intention of HMM is to be useful for all device memory that wish > to have struct page for various reasons. Hi Jermone and thanks for your input! Understood. We have looked at HMM in the past and long term I definitely would like to consider how we can add P2P functionality to HMM for

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, Mar 1, 2018 at 12:34 PM, Benjamin Herrenschmidt wrote: > On Thu, 2018-03-01 at 11:21 -0800, Dan Williams wrote: >> On Wed, Feb 28, 2018 at 7:56 PM, Benjamin Herrenschmidt >> wrote: >> > On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote:

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, Mar 1, 2018 at 12:34 PM, Benjamin Herrenschmidt wrote: > On Thu, 2018-03-01 at 11:21 -0800, Dan Williams wrote: >> On Wed, Feb 28, 2018 at 7:56 PM, Benjamin Herrenschmidt >> wrote: >> > On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: >> > > On Wed, 2018-02-28 at 16:39

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, Mar 01, 2018 at 02:15:01PM -0700, Logan Gunthorpe wrote: > > > On 01/03/18 02:10 PM, Jerome Glisse wrote: > > It seems people miss-understand HMM :( you do not have to use all of > > its features. If all you care about is having struct page then just > > use that for instance in your

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, Mar 01, 2018 at 02:15:01PM -0700, Logan Gunthorpe wrote: > > > On 01/03/18 02:10 PM, Jerome Glisse wrote: > > It seems people miss-understand HMM :( you do not have to use all of > > its features. If all you care about is having struct page then just > > use that for instance in your

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 02:18 PM, Jerome Glisse wrote: This is pretty easy to do with HMM: unsigned long hmm_page_to_phys_pfn(struct page *page) This is not useful unless you want to go through all the kernel paths we are using and replace page_to_phys() and friends with something else that calls an

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 02:18 PM, Jerome Glisse wrote: This is pretty easy to do with HMM: unsigned long hmm_page_to_phys_pfn(struct page *page) This is not useful unless you want to go through all the kernel paths we are using and replace page_to_phys() and friends with something else that calls an

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, Mar 01, 2018 at 02:11:34PM -0700, Logan Gunthorpe wrote: > > > On 01/03/18 02:03 PM, Benjamin Herrenschmidt wrote: > > However, what happens if anything calls page_address() on them ? Some > > DMA ops do that for example, or some devices might ... > > Although we could probably work

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, Mar 01, 2018 at 02:11:34PM -0700, Logan Gunthorpe wrote: > > > On 01/03/18 02:03 PM, Benjamin Herrenschmidt wrote: > > However, what happens if anything calls page_address() on them ? Some > > DMA ops do that for example, or some devices might ... > > Although we could probably work

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 02:10 PM, Jerome Glisse wrote: It seems people miss-understand HMM :( you do not have to use all of its features. If all you care about is having struct page then just use that for instance in your case only use those following 3 functions: hmm_devmem_add() or

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 02:10 PM, Jerome Glisse wrote: It seems people miss-understand HMM :( you do not have to use all of its features. If all you care about is having struct page then just use that for instance in your case only use those following 3 functions: hmm_devmem_add() or

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 02:03 PM, Benjamin Herrenschmidt wrote: However, what happens if anything calls page_address() on them ? Some DMA ops do that for example, or some devices might ... Although we could probably work around it with some pain, we rely on page_address() and virt_to_phys(), etc to

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 02:03 PM, Benjamin Herrenschmidt wrote: However, what happens if anything calls page_address() on them ? Some DMA ops do that for example, or some devices might ... Although we could probably work around it with some pain, we rely on page_address() and virt_to_phys(), etc to

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, Mar 01, 2018 at 02:03:26PM -0700, Logan Gunthorpe wrote: > > > On 01/03/18 01:55 PM, Jerome Glisse wrote: > > Well this again a new user of struct page for device memory just for > > one usecase. I wanted HMM to be more versatile so that it could be use > > for this kind of thing too. I

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, Mar 01, 2018 at 02:03:26PM -0700, Logan Gunthorpe wrote: > > > On 01/03/18 01:55 PM, Jerome Glisse wrote: > > Well this again a new user of struct page for device memory just for > > one usecase. I wanted HMM to be more versatile so that it could be use > > for this kind of thing too. I

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 11:21 -0800, Dan Williams wrote: > > > The devm_memremap_pages() infrastructure allows placing the memmap in > "System-RAM" even if the hotplugged range is in PCI space. So, even if > it is an issue on some configurations, it's just a simple adjustment > to where the memmap

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 11:21 -0800, Dan Williams wrote: > > > The devm_memremap_pages() infrastructure allows placing the memmap in > "System-RAM" even if the hotplugged range is in PCI space. So, even if > it is an issue on some configurations, it's just a simple adjustment > to where the memmap

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 01:55 PM, Jerome Glisse wrote: Well this again a new user of struct page for device memory just for one usecase. I wanted HMM to be more versatile so that it could be use for this kind of thing too. I guess the message didn't go through. I will take some cycles tomorrow to look

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 01:55 PM, Jerome Glisse wrote: Well this again a new user of struct page for device memory just for one usecase. I wanted HMM to be more versatile so that it could be use for this kind of thing too. I guess the message didn't go through. I will take some cycles tomorrow to look

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 01:53 PM, Jason Gunthorpe wrote: On Fri, Mar 02, 2018 at 07:40:15AM +1100, Benjamin Herrenschmidt wrote: Also we need to be able to hard block MEMREMAP_WB mappings of non-RAM on ppc64 (maybe via an arch hook as it might depend on the processor family). Server powerpc cannot do

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 01:53 PM, Jason Gunthorpe wrote: On Fri, Mar 02, 2018 at 07:40:15AM +1100, Benjamin Herrenschmidt wrote: Also we need to be able to hard block MEMREMAP_WB mappings of non-RAM on ppc64 (maybe via an arch hook as it might depend on the processor family). Server powerpc cannot do

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, Mar 02, 2018 at 07:29:55AM +1100, Benjamin Herrenschmidt wrote: > On Thu, 2018-03-01 at 11:04 -0700, Logan Gunthorpe wrote: > > > > On 28/02/18 08:56 PM, Benjamin Herrenschmidt wrote: > > > On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: > > > > The problem is that

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, Mar 02, 2018 at 07:29:55AM +1100, Benjamin Herrenschmidt wrote: > On Thu, 2018-03-01 at 11:04 -0700, Logan Gunthorpe wrote: > > > > On 28/02/18 08:56 PM, Benjamin Herrenschmidt wrote: > > > On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: > > > > The problem is that

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 01:29 PM, Benjamin Herrenschmidt wrote: Oliver can you look into this ? You sais the memory was effectively hotplug'ed into the system when creating the struct pages. That would mean to me that it's a) mapped (which for us is cachable, maybe x86 has tricks to avoid that) and b)

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 01:29 PM, Benjamin Herrenschmidt wrote: Oliver can you look into this ? You sais the memory was effectively hotplug'ed into the system when creating the struct pages. That would mean to me that it's a) mapped (which for us is cachable, maybe x86 has tricks to avoid that) and b)

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, Mar 02, 2018 at 07:40:15AM +1100, Benjamin Herrenschmidt wrote: > Also we need to be able to hard block MEMREMAP_WB mappings of non-RAM > on ppc64 (maybe via an arch hook as it might depend on the processor > family). Server powerpc cannot do cachable accesses on IO memory > (unless it's

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, Mar 02, 2018 at 07:40:15AM +1100, Benjamin Herrenschmidt wrote: > Also we need to be able to hard block MEMREMAP_WB mappings of non-RAM > on ppc64 (maybe via an arch hook as it might depend on the processor > family). Server powerpc cannot do cachable accesses on IO memory > (unless it's

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, 2018-03-02 at 07:34 +1100, Benjamin Herrenschmidt wrote: > > But what happens with that PCI memory ? Is it effectively turned into > nromal memory (ie, usable for normal allocations, potentially used to > populate user pages etc...) or is it kept aside ? (What I mean is is it added to

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, 2018-03-02 at 07:34 +1100, Benjamin Herrenschmidt wrote: > > But what happens with that PCI memory ? Is it effectively turned into > nromal memory (ie, usable for normal allocations, potentially used to > populate user pages etc...) or is it kept aside ? (What I mean is is it added to

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 11:21 -0800, Dan Williams wrote: > On Wed, Feb 28, 2018 at 7:56 PM, Benjamin Herrenschmidt > wrote: > > On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: > > > On Wed, 2018-02-28 at 16:39 -0700, Logan Gunthorpe wrote: > > > > Hi Everyone, > >

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 11:21 -0800, Dan Williams wrote: > On Wed, Feb 28, 2018 at 7:56 PM, Benjamin Herrenschmidt > wrote: > > On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: > > > On Wed, 2018-02-28 at 16:39 -0700, Logan Gunthorpe wrote: > > > > Hi Everyone, > > > > > > > > >

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 18:09 +, Stephen Bates wrote: > > > So Oliver (CC) was having issues getting any of that to work for us. > > > > > > The problem is that acccording to him (I didn't double check the latest > > > patches) you effectively hotplug the PCIe memory into the system when > > >

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 18:09 +, Stephen Bates wrote: > > > So Oliver (CC) was having issues getting any of that to work for us. > > > > > > The problem is that acccording to him (I didn't double check the latest > > > patches) you effectively hotplug the PCIe memory into the system when > > >

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 11:04 -0700, Logan Gunthorpe wrote: > > On 28/02/18 08:56 PM, Benjamin Herrenschmidt wrote: > > On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: > > > The problem is that acccording to him (I didn't double check the latest > > > patches) you effectively

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 11:04 -0700, Logan Gunthorpe wrote: > > On 28/02/18 08:56 PM, Benjamin Herrenschmidt wrote: > > On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: > > > The problem is that acccording to him (I didn't double check the latest > > > patches) you effectively

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 03:31 AM, Sagi Grimberg wrote: * We also reject using devices that employ 'dma_virt_ops' which should    fairly simply handle Jason's concerns that this work might break with    the HFI, QIB and rxe drivers that use the virtual ops to implement    their own special DMA operations.

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 03:31 AM, Sagi Grimberg wrote: * We also reject using devices that employ 'dma_virt_ops' which should    fairly simply handle Jason's concerns that this work might break with    the HFI, QIB and rxe drivers that use the virtual ops to implement    their own special DMA operations.

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 12:21 PM, Dan Williams wrote: Note: I think the above means it won't work behind a switch on x86 either, will it ? The devm_memremap_pages() infrastructure allows placing the memmap in "System-RAM" even if the hotplugged range is in PCI space. So, even if it is an issue on some

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 01/03/18 12:21 PM, Dan Williams wrote: Note: I think the above means it won't work behind a switch on x86 either, will it ? The devm_memremap_pages() infrastructure allows placing the memmap in "System-RAM" even if the hotplugged range is in PCI space. So, even if it is an issue on some

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Wed, Feb 28, 2018 at 7:56 PM, Benjamin Herrenschmidt wrote: > On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: >> On Wed, 2018-02-28 at 16:39 -0700, Logan Gunthorpe wrote: >> > Hi Everyone, >> >> >> So Oliver (CC) was having issues getting any of that to work

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Wed, Feb 28, 2018 at 7:56 PM, Benjamin Herrenschmidt wrote: > On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: >> On Wed, 2018-02-28 at 16:39 -0700, Logan Gunthorpe wrote: >> > Hi Everyone, >> >> >> So Oliver (CC) was having issues getting any of that to work for us. >> >> The

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

>> So Oliver (CC) was having issues getting any of that to work for us. >> >> The problem is that acccording to him (I didn't double check the latest >> patches) you effectively hotplug the PCIe memory into the system when >> creating struct pages. >> >> This cannot possibly work for us. First

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

>> So Oliver (CC) was having issues getting any of that to work for us. >> >> The problem is that acccording to him (I didn't double check the latest >> patches) you effectively hotplug the PCIe memory into the system when >> creating struct pages. >> >> This cannot possibly work for us. First

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 28/02/18 08:56 PM, Benjamin Herrenschmidt wrote: On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: The problem is that acccording to him (I didn't double check the latest patches) you effectively hotplug the PCIe memory into the system when creating struct pages. This

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On 28/02/18 08:56 PM, Benjamin Herrenschmidt wrote: On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: The problem is that acccording to him (I didn't double check the latest patches) you effectively hotplug the PCIe memory into the system when creating struct pages. This

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

Hi Everyone, Hi Logan, Here's v2 of our series to introduce P2P based copy offload to NVMe fabrics. This version has been rebased onto v4.16-rc3 which already includes Christoph's devpagemap work the previous version was based off as well as a couple of the cleanup patches that were in v1.

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

Hi Everyone, Hi Logan, Here's v2 of our series to introduce P2P based copy offload to NVMe fabrics. This version has been rebased onto v4.16-rc3 which already includes Christoph's devpagemap work the previous version was based off as well as a couple of the cleanup patches that were in v1.

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Wed, 2018-02-28 at 16:39 -0700, Logan Gunthorpe wrote: > Hi Everyone, So Oliver (CC) was having issues getting any of that to work for us. The problem is that acccording to him (I didn't double check the latest patches) you effectively hotplug the PCIe memory into the system when creating

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Wed, 2018-02-28 at 16:39 -0700, Logan Gunthorpe wrote: > Hi Everyone, So Oliver (CC) was having issues getting any of that to work for us. The problem is that acccording to him (I didn't double check the latest patches) you effectively hotplug the PCIe memory into the system when creating

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: > On Wed, 2018-02-28 at 16:39 -0700, Logan Gunthorpe wrote: > > Hi Everyone, > > > So Oliver (CC) was having issues getting any of that to work for us. > > The problem is that acccording to him (I didn't double check the latest >

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Thu, 2018-03-01 at 14:54 +1100, Benjamin Herrenschmidt wrote: > On Wed, 2018-02-28 at 16:39 -0700, Logan Gunthorpe wrote: > > Hi Everyone, > > > So Oliver (CC) was having issues getting any of that to work for us. > > The problem is that acccording to him (I didn't double check the latest >