Re: Interacting with coherent memory on external devices

2015-05-28 Thread Paul E. McKenney
On Thu, May 14, 2015 at 05:51:19PM +1000, Benjamin Herrenschmidt wrote: > On Thu, 2015-05-14 at 09:39 +0200, Vlastimil Babka wrote: > > On 05/14/2015 01:38 AM, Benjamin Herrenschmidt wrote: > > > On Wed, 2015-05-13 at 16:10 +0200, Vlastimil Babka wrote: > > >> Sorry for reviving oldish thread... >

Re: Interacting with coherent memory on external devices

2015-05-28 Thread Paul E. McKenney
On Thu, May 14, 2015 at 05:51:19PM +1000, Benjamin Herrenschmidt wrote: On Thu, 2015-05-14 at 09:39 +0200, Vlastimil Babka wrote: On 05/14/2015 01:38 AM, Benjamin Herrenschmidt wrote: On Wed, 2015-05-13 at 16:10 +0200, Vlastimil Babka wrote: Sorry for reviving oldish thread... Well,

Re: Interacting with coherent memory on external devices

2015-05-14 Thread Benjamin Herrenschmidt
On Thu, 2015-05-14 at 09:39 +0200, Vlastimil Babka wrote: > On 05/14/2015 01:38 AM, Benjamin Herrenschmidt wrote: > > On Wed, 2015-05-13 at 16:10 +0200, Vlastimil Babka wrote: > >> Sorry for reviving oldish thread... > > > > Well, that's actually appreciated since this is constructive discussion >

Re: Interacting with coherent memory on external devices

2015-05-14 Thread Vlastimil Babka
On 05/14/2015 01:38 AM, Benjamin Herrenschmidt wrote: On Wed, 2015-05-13 at 16:10 +0200, Vlastimil Babka wrote: Sorry for reviving oldish thread... Well, that's actually appreciated since this is constructive discussion of the kind I was hoping to trigger initially :-) I'll look at I hoped

Re: Interacting with coherent memory on external devices

2015-05-14 Thread Vlastimil Babka
On 05/14/2015 01:38 AM, Benjamin Herrenschmidt wrote: On Wed, 2015-05-13 at 16:10 +0200, Vlastimil Babka wrote: Sorry for reviving oldish thread... Well, that's actually appreciated since this is constructive discussion of the kind I was hoping to trigger initially :-) I'll look at I hoped

Re: Interacting with coherent memory on external devices

2015-05-14 Thread Benjamin Herrenschmidt
On Thu, 2015-05-14 at 09:39 +0200, Vlastimil Babka wrote: On 05/14/2015 01:38 AM, Benjamin Herrenschmidt wrote: On Wed, 2015-05-13 at 16:10 +0200, Vlastimil Babka wrote: Sorry for reviving oldish thread... Well, that's actually appreciated since this is constructive discussion of the

Re: Interacting with coherent memory on external devices

2015-05-13 Thread Benjamin Herrenschmidt
On Wed, 2015-05-13 at 16:10 +0200, Vlastimil Babka wrote: > Sorry for reviving oldish thread... Well, that's actually appreciated since this is constructive discussion of the kind I was hoping to trigger initially :-) I'll look at ZONE_MOVABLE, I wasn't aware of its existence. Don't we still

Re: Interacting with coherent memory on external devices

2015-05-13 Thread Vlastimil Babka
Sorry for reviving oldish thread... On 04/28/2015 01:54 AM, Benjamin Herrenschmidt wrote: On Mon, 2015-04-27 at 11:48 -0500, Christoph Lameter wrote: On Mon, 27 Apr 2015, Rik van Riel wrote: Why would we want to avoid the sane approach that makes this thing work with the fewest required

Re: Interacting with coherent memory on external devices

2015-05-13 Thread Vlastimil Babka
Sorry for reviving oldish thread... On 04/28/2015 01:54 AM, Benjamin Herrenschmidt wrote: On Mon, 2015-04-27 at 11:48 -0500, Christoph Lameter wrote: On Mon, 27 Apr 2015, Rik van Riel wrote: Why would we want to avoid the sane approach that makes this thing work with the fewest required

Re: Interacting with coherent memory on external devices

2015-05-13 Thread Benjamin Herrenschmidt
On Wed, 2015-05-13 at 16:10 +0200, Vlastimil Babka wrote: Sorry for reviving oldish thread... Well, that's actually appreciated since this is constructive discussion of the kind I was hoping to trigger initially :-) I'll look at ZONE_MOVABLE, I wasn't aware of its existence. Don't we still have

Re: Interacting with coherent memory on external devices

2015-04-28 Thread Jerome Glisse
On Tue, Apr 28, 2015 at 09:18:55AM -0500, Christoph Lameter wrote: > On Mon, 27 Apr 2015, Jerome Glisse wrote: > > > > is the mechanism that DAX relies on in the VM. > > > > Which would require fare more changes than you seem to think. First using > > MIXED|PFNMAP means we loose any kind of

Re: Interacting with coherent memory on external devices

2015-04-28 Thread Christoph Lameter
On Mon, 27 Apr 2015, Jerome Glisse wrote: > > is the mechanism that DAX relies on in the VM. > > Which would require fare more changes than you seem to think. First using > MIXED|PFNMAP means we loose any kind of memory accounting and forget about > memcg too. Seconds it means we would need to

Re: Interacting with coherent memory on external devices

2015-04-28 Thread Christoph Lameter
On Mon, 27 Apr 2015, Jerome Glisse wrote: is the mechanism that DAX relies on in the VM. Which would require fare more changes than you seem to think. First using MIXED|PFNMAP means we loose any kind of memory accounting and forget about memcg too. Seconds it means we would need to set

Re: Interacting with coherent memory on external devices

2015-04-28 Thread Jerome Glisse
On Tue, Apr 28, 2015 at 09:18:55AM -0500, Christoph Lameter wrote: On Mon, 27 Apr 2015, Jerome Glisse wrote: is the mechanism that DAX relies on in the VM. Which would require fare more changes than you seem to think. First using MIXED|PFNMAP means we loose any kind of memory

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Benjamin Herrenschmidt
On Mon, 2015-04-27 at 11:48 -0500, Christoph Lameter wrote: > On Mon, 27 Apr 2015, Rik van Riel wrote: > > > Why would we want to avoid the sane approach that makes this thing > > work with the fewest required changes to core code? > > Becaus new ZONEs are a pretty invasive change to the memory

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Jerome Glisse
On Mon, Apr 27, 2015 at 02:26:04PM -0500, Christoph Lameter wrote: > On Mon, 27 Apr 2015, Jerome Glisse wrote: > > > > We can drop the DAX name and just talk about mapping to external memory if > > > that confuses the issue. > > > > DAX is for direct access block layer (X is for the cool name

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Rik van Riel
On 04/27/2015 03:26 PM, Christoph Lameter wrote: > DAX is about directly accessing memory. It is made for the purpose of > serving as a block device for a filesystem right now but it can easily be > used as a way to map any external memory into a processes space using the > abstraction of a block

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Christoph Lameter
On Mon, 27 Apr 2015, Jerome Glisse wrote: > > We can drop the DAX name and just talk about mapping to external memory if > > that confuses the issue. > > DAX is for direct access block layer (X is for the cool name factor) > there is zero code inside DAX that would be usefull to us. Because it >

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Jerome Glisse
On Mon, Apr 27, 2015 at 11:51:51AM -0500, Christoph Lameter wrote: > On Mon, 27 Apr 2015, Jerome Glisse wrote: > > > > Well lets avoid that. Access to device memory comparable to what the > > > drivers do today by establishing page table mappings or a generalization > > > of DAX approaches would

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Christoph Lameter
On Mon, 27 Apr 2015, Jerome Glisse wrote: > > Well lets avoid that. Access to device memory comparable to what the > > drivers do today by establishing page table mappings or a generalization > > of DAX approaches would be the most straightforward way of implementing it > > and would build based

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Christoph Lameter
On Mon, 27 Apr 2015, Rik van Riel wrote: > Why would we want to avoid the sane approach that makes this thing > work with the fewest required changes to core code? Becaus new ZONEs are a pretty invasive change to the memory management and because there are other ways to handle references to

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Jerome Glisse
On Mon, Apr 27, 2015 at 11:17:43AM -0500, Christoph Lameter wrote: > On Mon, 27 Apr 2015, Jerome Glisse wrote: > > > > Improvements to the general code would be preferred instead of > > > having specialized solutions for a particular hardware alone. If the > > > general code can then handle the

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Christoph Lameter
On Mon, 27 Apr 2015, Paul E. McKenney wrote: > I would instead look on this as a way to try out use of hardware migration > hints, which could lead to hardware vendors providing similar hints for > node-to-node migrations. At that time, the benefits could be provided > all the functionality

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Rik van Riel
On 04/27/2015 12:17 PM, Christoph Lameter wrote: > On Mon, 27 Apr 2015, Jerome Glisse wrote: > >>> Improvements to the general code would be preferred instead of >>> having specialized solutions for a particular hardware alone. If the >>> general code can then handle the special coprocessor

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Christoph Lameter
On Mon, 27 Apr 2015, Jerome Glisse wrote: > > Improvements to the general code would be preferred instead of > > having specialized solutions for a particular hardware alone. If the > > general code can then handle the special coprocessor situation then we > > avoid a lot of code development. >

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Paul E. McKenney
On Mon, Apr 27, 2015 at 10:08:29AM -0500, Christoph Lameter wrote: > On Sat, 25 Apr 2015, Paul E. McKenney wrote: > > > Would you have a URL or other pointer to this code? > > linux/mm/migrate.c Ah, I thought you were calling out something not yet in mainline. > > > > Without modifying a

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Jerome Glisse
On Mon, Apr 27, 2015 at 10:08:29AM -0500, Christoph Lameter wrote: > On Sat, 25 Apr 2015, Paul E. McKenney wrote: > > > Would you have a URL or other pointer to this code? > > linux/mm/migrate.c > > > > > Without modifying a single line of mm code, the only way to do this is > > > > to > > > >

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Christoph Lameter
On Sat, 25 Apr 2015, Paul E. McKenney wrote: > Would you have a URL or other pointer to this code? linux/mm/migrate.c > > > Without modifying a single line of mm code, the only way to do this is to > > > either unmap from the cpu page table the range being migrated or to > > > mprotect > > >

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Rik van Riel
On 04/27/2015 03:26 PM, Christoph Lameter wrote: DAX is about directly accessing memory. It is made for the purpose of serving as a block device for a filesystem right now but it can easily be used as a way to map any external memory into a processes space using the abstraction of a block

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Jerome Glisse
On Mon, Apr 27, 2015 at 02:26:04PM -0500, Christoph Lameter wrote: On Mon, 27 Apr 2015, Jerome Glisse wrote: We can drop the DAX name and just talk about mapping to external memory if that confuses the issue. DAX is for direct access block layer (X is for the cool name factor) there

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Benjamin Herrenschmidt
On Mon, 2015-04-27 at 11:48 -0500, Christoph Lameter wrote: On Mon, 27 Apr 2015, Rik van Riel wrote: Why would we want to avoid the sane approach that makes this thing work with the fewest required changes to core code? Becaus new ZONEs are a pretty invasive change to the memory

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Christoph Lameter
On Sat, 25 Apr 2015, Paul E. McKenney wrote: Would you have a URL or other pointer to this code? linux/mm/migrate.c Without modifying a single line of mm code, the only way to do this is to either unmap from the cpu page table the range being migrated or to mprotect it in some

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Jerome Glisse
On Mon, Apr 27, 2015 at 10:08:29AM -0500, Christoph Lameter wrote: On Sat, 25 Apr 2015, Paul E. McKenney wrote: Would you have a URL or other pointer to this code? linux/mm/migrate.c Without modifying a single line of mm code, the only way to do this is to either unmap from

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Christoph Lameter
On Mon, 27 Apr 2015, Paul E. McKenney wrote: I would instead look on this as a way to try out use of hardware migration hints, which could lead to hardware vendors providing similar hints for node-to-node migrations. At that time, the benefits could be provided all the functionality relying

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Christoph Lameter
On Mon, 27 Apr 2015, Jerome Glisse wrote: Well lets avoid that. Access to device memory comparable to what the drivers do today by establishing page table mappings or a generalization of DAX approaches would be the most straightforward way of implementing it and would build based on

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Paul E. McKenney
On Mon, Apr 27, 2015 at 10:08:29AM -0500, Christoph Lameter wrote: On Sat, 25 Apr 2015, Paul E. McKenney wrote: Would you have a URL or other pointer to this code? linux/mm/migrate.c Ah, I thought you were calling out something not yet in mainline. Without modifying a single line of

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Rik van Riel
On 04/27/2015 12:17 PM, Christoph Lameter wrote: On Mon, 27 Apr 2015, Jerome Glisse wrote: Improvements to the general code would be preferred instead of having specialized solutions for a particular hardware alone. If the general code can then handle the special coprocessor situation then

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Christoph Lameter
On Mon, 27 Apr 2015, Rik van Riel wrote: Why would we want to avoid the sane approach that makes this thing work with the fewest required changes to core code? Becaus new ZONEs are a pretty invasive change to the memory management and because there are other ways to handle references to

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Christoph Lameter
On Mon, 27 Apr 2015, Jerome Glisse wrote: Improvements to the general code would be preferred instead of having specialized solutions for a particular hardware alone. If the general code can then handle the special coprocessor situation then we avoid a lot of code development. I think

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Jerome Glisse
On Mon, Apr 27, 2015 at 11:17:43AM -0500, Christoph Lameter wrote: On Mon, 27 Apr 2015, Jerome Glisse wrote: Improvements to the general code would be preferred instead of having specialized solutions for a particular hardware alone. If the general code can then handle the special

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Jerome Glisse
On Mon, Apr 27, 2015 at 11:51:51AM -0500, Christoph Lameter wrote: On Mon, 27 Apr 2015, Jerome Glisse wrote: Well lets avoid that. Access to device memory comparable to what the drivers do today by establishing page table mappings or a generalization of DAX approaches would be the most

Re: Interacting with coherent memory on external devices

2015-04-27 Thread Christoph Lameter
On Mon, 27 Apr 2015, Jerome Glisse wrote: We can drop the DAX name and just talk about mapping to external memory if that confuses the issue. DAX is for direct access block layer (X is for the cool name factor) there is zero code inside DAX that would be usefull to us. Because it is all

Re: Interacting with coherent memory on external devices

2015-04-25 Thread Paul E. McKenney
On Sat, Apr 25, 2015 at 01:32:39PM +1000, Benjamin Herrenschmidt wrote: > On Fri, 2015-04-24 at 22:32 -0400, Rik van Riel wrote: > > > The result would be that the kernel would allocate only > > migratable > > > pages within the CCAD device's memory, and even then only if > > >

Re: Interacting with coherent memory on external devices

2015-04-25 Thread Paul E. McKenney
On Fri, Apr 24, 2015 at 10:49:28AM -0500, Christoph Lameter wrote: > On Fri, 24 Apr 2015, Paul E. McKenney wrote: > > > can deliver, but where the cost of full-fledge hand tuning cannot be > > justified. > > > > You seem to believe that this latter category is the empty set, which > > I must

Re: Interacting with coherent memory on external devices

2015-04-25 Thread Paul E. McKenney
On Fri, Apr 24, 2015 at 03:00:18PM -0500, Christoph Lameter wrote: > On Fri, 24 Apr 2015, Jerome Glisse wrote: > > > > Still no answer as to why is that not possible with the current scheme? > > > You keep on talking about pointers and I keep on responding that this is a > > > matter of making

Re: Interacting with coherent memory on external devices

2015-04-25 Thread Paul E. McKenney
On Fri, Apr 24, 2015 at 11:09:36AM -0400, Jerome Glisse wrote: > On Fri, Apr 24, 2015 at 07:57:38AM -0700, Paul E. McKenney wrote: > > On Fri, Apr 24, 2015 at 09:12:07AM -0500, Christoph Lameter wrote: > > > On Thu, 23 Apr 2015, Paul E. McKenney wrote: > > > > > > > > > > > DAX > > > > > > > >

Re: Interacting with coherent memory on external devices

2015-04-25 Thread Paul E. McKenney
On Fri, Apr 24, 2015 at 11:09:36AM -0400, Jerome Glisse wrote: On Fri, Apr 24, 2015 at 07:57:38AM -0700, Paul E. McKenney wrote: On Fri, Apr 24, 2015 at 09:12:07AM -0500, Christoph Lameter wrote: On Thu, 23 Apr 2015, Paul E. McKenney wrote: DAX DAX is a mechanism

Re: Interacting with coherent memory on external devices

2015-04-25 Thread Paul E. McKenney
On Fri, Apr 24, 2015 at 03:00:18PM -0500, Christoph Lameter wrote: On Fri, 24 Apr 2015, Jerome Glisse wrote: Still no answer as to why is that not possible with the current scheme? You keep on talking about pointers and I keep on responding that this is a matter of making the address

Re: Interacting with coherent memory on external devices

2015-04-25 Thread Paul E. McKenney
On Fri, Apr 24, 2015 at 10:49:28AM -0500, Christoph Lameter wrote: On Fri, 24 Apr 2015, Paul E. McKenney wrote: can deliver, but where the cost of full-fledge hand tuning cannot be justified. You seem to believe that this latter category is the empty set, which I must confess does

Re: Interacting with coherent memory on external devices

2015-04-25 Thread Paul E. McKenney
On Sat, Apr 25, 2015 at 01:32:39PM +1000, Benjamin Herrenschmidt wrote: On Fri, 2015-04-24 at 22:32 -0400, Rik van Riel wrote: The result would be that the kernel would allocate only migratable pages within the CCAD device's memory, and even then only if memory was

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Benjamin Herrenschmidt
On Fri, 2015-04-24 at 22:32 -0400, Rik van Riel wrote: > > The result would be that the kernel would allocate only > migratable > > pages within the CCAD device's memory, and even then only if > > memory was otherwise exhausted. > > Does it make sense to allocate the device's

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Rik van Riel
On 04/21/2015 05:44 PM, Paul E. McKenney wrote: > AUTONUMA > > The Linux kernel's autonuma facility supports migrating both > memory and processes to promote NUMA memory locality. It was > accepted into 3.13 and is available in RHEL 7.0 and SLES 12. > It is enabled by

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Benjamin Herrenschmidt
On Fri, 2015-04-24 at 11:58 -0500, Christoph Lameter wrote: > On Fri, 24 Apr 2015, Jerome Glisse wrote: > > > > What exactly is the more advanced version's benefit? What are the features > > > that the other platforms do not provide? > > > > Transparent access to device memory from the CPU, you

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Jerome Glisse
On Fri, Apr 24, 2015 at 03:00:18PM -0500, Christoph Lameter wrote: > On Fri, 24 Apr 2015, Jerome Glisse wrote: > > > > Still no answer as to why is that not possible with the current scheme? > > > You keep on talking about pointers and I keep on responding that this is a > > > matter of making

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Fri, 24 Apr 2015, Jerome Glisse wrote: > > Still no answer as to why is that not possible with the current scheme? > > You keep on talking about pointers and I keep on responding that this is a > > matter of making the address space compatible on both sides. > > So if do that in a naive way,

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Jerome Glisse
On Fri, Apr 24, 2015 at 01:56:45PM -0500, Christoph Lameter wrote: > On Fri, 24 Apr 2015, Jerome Glisse wrote: > > > > Right this is how things work and you could improve on that. Stay with the > > > scheme. Why would that not work if you map things the same way in both > > > environments if both

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Fri, 24 Apr 2015, Jerome Glisse wrote: > > Right this is how things work and you could improve on that. Stay with the > > scheme. Why would that not work if you map things the same way in both > > environments if both accellerator and host processor can acceess each > > others memory? > >

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Oded Gabbay
On 04/23/2015 07:22 PM, Jerome Glisse wrote: On Thu, Apr 23, 2015 at 09:20:55AM -0500, Christoph Lameter wrote: On Thu, 23 Apr 2015, Benjamin Herrenschmidt wrote: There are hooks in glibc where you can replace the memory management of the apps if you want that. We don't control the app.

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Jerome Glisse
On Fri, Apr 24, 2015 at 11:58:39AM -0500, Christoph Lameter wrote: > On Fri, 24 Apr 2015, Jerome Glisse wrote: > > > > What exactly is the more advanced version's benefit? What are the features > > > that the other platforms do not provide? > > > > Transparent access to device memory from the

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Fri, 24 Apr 2015, Jerome Glisse wrote: > > What exactly is the more advanced version's benefit? What are the features > > that the other platforms do not provide? > > Transparent access to device memory from the CPU, you can map any of the GPU > memory inside the CPU and have the whole cache

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Jerome Glisse
On Fri, Apr 24, 2015 at 11:03:52AM -0500, Christoph Lameter wrote: > On Fri, 24 Apr 2015, Jerome Glisse wrote: > > > On Fri, Apr 24, 2015 at 09:29:12AM -0500, Christoph Lameter wrote: > > > On Thu, 23 Apr 2015, Jerome Glisse wrote: > > > > > > > No this not have been solve properly. Today

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Rik van Riel
On 04/24/2015 10:30 AM, Christoph Lameter wrote: > On Thu, 23 Apr 2015, Paul E. McKenney wrote: > >> If by "entire industry" you mean everyone who might want to use hardware >> acceleration, for example, including mechanical computer-aided design, >> I am skeptical. > > The industry designs GPUs

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Rik van Riel
On 04/24/2015 11:49 AM, Christoph Lameter wrote: > On Fri, 24 Apr 2015, Paul E. McKenney wrote: > >> can deliver, but where the cost of full-fledge hand tuning cannot be >> justified. >> >> You seem to believe that this latter category is the empty set, which >> I must confess does greatly

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Fri, 24 Apr 2015, Jerome Glisse wrote: > On Fri, Apr 24, 2015 at 09:29:12AM -0500, Christoph Lameter wrote: > > On Thu, 23 Apr 2015, Jerome Glisse wrote: > > > > > No this not have been solve properly. Today solution is doing an explicit > > > copy and again and again when complex data struct

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Jerome Glisse
On Fri, Apr 24, 2015 at 09:30:40AM -0500, Christoph Lameter wrote: > On Thu, 23 Apr 2015, Paul E. McKenney wrote: > > > If by "entire industry" you mean everyone who might want to use hardware > > acceleration, for example, including mechanical computer-aided design, > > I am skeptical. > > The

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Rik van Riel
On 04/24/2015 10:01 AM, Christoph Lameter wrote: > On Thu, 23 Apr 2015, Paul E. McKenney wrote: > >>> As far as I know Jerome is talkeing about HPC loads and high performance >>> GPU processing. This is the same use case. >> >> The difference is sensitivity to latency. You have latency-sensitive

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Fri, 24 Apr 2015, Paul E. McKenney wrote: > > DAX is a mechanism to access memory not managed by the kernel and is the > > successor to XIP. It just happens to be needed for persistent memory. > > Fundamentally any driver can provide an MMAPPed interface to allow access > > to a devices

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Fri, 24 Apr 2015, Paul E. McKenney wrote: > can deliver, but where the cost of full-fledge hand tuning cannot be > justified. > > You seem to believe that this latter category is the empty set, which > I must confess does greatly surprise me. If there are already compromises are being made

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Jerome Glisse
On Fri, Apr 24, 2015 at 07:57:38AM -0700, Paul E. McKenney wrote: > On Fri, Apr 24, 2015 at 09:12:07AM -0500, Christoph Lameter wrote: > > On Thu, 23 Apr 2015, Paul E. McKenney wrote: > > > > > > > > DAX > > > > > > DAX is a mechanism for providing direct-memory access to > > > high-speed

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Jerome Glisse
On Fri, Apr 24, 2015 at 09:29:12AM -0500, Christoph Lameter wrote: > On Thu, 23 Apr 2015, Jerome Glisse wrote: > > > No this not have been solve properly. Today solution is doing an explicit > > copy and again and again when complex data struct are involve (list, tree, > > ...) this is extremly

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Paul E. McKenney
On Fri, Apr 24, 2015 at 09:12:07AM -0500, Christoph Lameter wrote: > On Thu, 23 Apr 2015, Paul E. McKenney wrote: > > > > > DAX > > > > DAX is a mechanism for providing direct-memory access to > > high-speed non-volatile (AKA "persistent") memory. Good > > introductions to DAX may be

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Paul E. McKenney
On Fri, Apr 24, 2015 at 09:30:40AM -0500, Christoph Lameter wrote: > On Thu, 23 Apr 2015, Paul E. McKenney wrote: > > > If by "entire industry" you mean everyone who might want to use hardware > > acceleration, for example, including mechanical computer-aided design, > > I am skeptical. > > The

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Thu, 23 Apr 2015, Paul E. McKenney wrote: > If by "entire industry" you mean everyone who might want to use hardware > acceleration, for example, including mechanical computer-aided design, > I am skeptical. The industry designs GPUs with super fast special ram and accellerators with special

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Thu, 23 Apr 2015, Jerome Glisse wrote: > No this not have been solve properly. Today solution is doing an explicit > copy and again and again when complex data struct are involve (list, tree, > ...) this is extremly tedious and hard to debug. So today solution often > restrict themself to easy

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Paul E. McKenney
On Fri, Apr 24, 2015 at 09:01:47AM -0500, Christoph Lameter wrote: > On Thu, 23 Apr 2015, Paul E. McKenney wrote: > > > > As far as I know Jerome is talkeing about HPC loads and high performance > > > GPU processing. This is the same use case. > > > > The difference is sensitivity to latency.

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Thu, 23 Apr 2015, Paul E. McKenney wrote: > > DAX > > DAX is a mechanism for providing direct-memory access to > high-speed non-volatile (AKA "persistent") memory. Good > introductions to DAX may be found in the following LWN > articles: DAX is a mechanism to access

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Thu, 23 Apr 2015, Jerome Glisse wrote: > The numa code we have today for CPU case exist because it does make > a difference but you keep trying to restrict GPU user to a workload > that is specific. Go talk to people doing physic, biology, data > mining, CAD most of them do not care about

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Thu, 23 Apr 2015, Austin S Hemmelgarn wrote: Looking at this whole conversation, all I see is two different views on how to present the asymmetric multiprocessing arrangements that have become commonplace in today's systems to userspace. Your model favors performance, while CAPI favors

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Thu, 23 Apr 2015, Paul E. McKenney wrote: > > As far as I know Jerome is talkeing about HPC loads and high performance > > GPU processing. This is the same use case. > > The difference is sensitivity to latency. You have latency-sensitive > HPC workloads, and Jerome is talking about HPC

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Thu, 23 Apr 2015, Austin S Hemmelgarn wrote: Looking at this whole conversation, all I see is two different views on how to present the asymmetric multiprocessing arrangements that have become commonplace in today's systems to userspace. Your model favors performance, while CAPI favors

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Thu, 23 Apr 2015, Paul E. McKenney wrote: If by entire industry you mean everyone who might want to use hardware acceleration, for example, including mechanical computer-aided design, I am skeptical. The industry designs GPUs with super fast special ram and accellerators with special ram

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Paul E. McKenney
On Fri, Apr 24, 2015 at 09:01:47AM -0500, Christoph Lameter wrote: On Thu, 23 Apr 2015, Paul E. McKenney wrote: As far as I know Jerome is talkeing about HPC loads and high performance GPU processing. This is the same use case. The difference is sensitivity to latency. You have

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Thu, 23 Apr 2015, Paul E. McKenney wrote: As far as I know Jerome is talkeing about HPC loads and high performance GPU processing. This is the same use case. The difference is sensitivity to latency. You have latency-sensitive HPC workloads, and Jerome is talking about HPC workloads

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Thu, 23 Apr 2015, Jerome Glisse wrote: The numa code we have today for CPU case exist because it does make a difference but you keep trying to restrict GPU user to a workload that is specific. Go talk to people doing physic, biology, data mining, CAD most of them do not care about latency.

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Thu, 23 Apr 2015, Jerome Glisse wrote: No this not have been solve properly. Today solution is doing an explicit copy and again and again when complex data struct are involve (list, tree, ...) this is extremly tedious and hard to debug. So today solution often restrict themself to easy

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Thu, 23 Apr 2015, Paul E. McKenney wrote: DAX DAX is a mechanism for providing direct-memory access to high-speed non-volatile (AKA persistent) memory. Good introductions to DAX may be found in the following LWN articles: DAX is a mechanism to access memory not

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Oded Gabbay
On 04/23/2015 07:22 PM, Jerome Glisse wrote: On Thu, Apr 23, 2015 at 09:20:55AM -0500, Christoph Lameter wrote: On Thu, 23 Apr 2015, Benjamin Herrenschmidt wrote: There are hooks in glibc where you can replace the memory management of the apps if you want that. We don't control the app.

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Paul E. McKenney
On Fri, Apr 24, 2015 at 09:30:40AM -0500, Christoph Lameter wrote: On Thu, 23 Apr 2015, Paul E. McKenney wrote: If by entire industry you mean everyone who might want to use hardware acceleration, for example, including mechanical computer-aided design, I am skeptical. The industry

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Benjamin Herrenschmidt
On Fri, 2015-04-24 at 11:58 -0500, Christoph Lameter wrote: On Fri, 24 Apr 2015, Jerome Glisse wrote: What exactly is the more advanced version's benefit? What are the features that the other platforms do not provide? Transparent access to device memory from the CPU, you can map any

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Fri, 24 Apr 2015, Paul E. McKenney wrote: DAX is a mechanism to access memory not managed by the kernel and is the successor to XIP. It just happens to be needed for persistent memory. Fundamentally any driver can provide an MMAPPed interface to allow access to a devices memory. I

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Fri, 24 Apr 2015, Jerome Glisse wrote: On Fri, Apr 24, 2015 at 09:29:12AM -0500, Christoph Lameter wrote: On Thu, 23 Apr 2015, Jerome Glisse wrote: No this not have been solve properly. Today solution is doing an explicit copy and again and again when complex data struct are involve

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Rik van Riel
On 04/24/2015 11:49 AM, Christoph Lameter wrote: On Fri, 24 Apr 2015, Paul E. McKenney wrote: can deliver, but where the cost of full-fledge hand tuning cannot be justified. You seem to believe that this latter category is the empty set, which I must confess does greatly surprise me. If

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Jerome Glisse
On Fri, Apr 24, 2015 at 09:30:40AM -0500, Christoph Lameter wrote: On Thu, 23 Apr 2015, Paul E. McKenney wrote: If by entire industry you mean everyone who might want to use hardware acceleration, for example, including mechanical computer-aided design, I am skeptical. The industry

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Jerome Glisse
On Fri, Apr 24, 2015 at 09:29:12AM -0500, Christoph Lameter wrote: On Thu, 23 Apr 2015, Jerome Glisse wrote: No this not have been solve properly. Today solution is doing an explicit copy and again and again when complex data struct are involve (list, tree, ...) this is extremly tedious

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Jerome Glisse
On Fri, Apr 24, 2015 at 07:57:38AM -0700, Paul E. McKenney wrote: On Fri, Apr 24, 2015 at 09:12:07AM -0500, Christoph Lameter wrote: On Thu, 23 Apr 2015, Paul E. McKenney wrote: DAX DAX is a mechanism for providing direct-memory access to high-speed non-volatile (AKA

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Paul E. McKenney
On Fri, Apr 24, 2015 at 09:12:07AM -0500, Christoph Lameter wrote: On Thu, 23 Apr 2015, Paul E. McKenney wrote: DAX DAX is a mechanism for providing direct-memory access to high-speed non-volatile (AKA persistent) memory. Good introductions to DAX may be found in the

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Rik van Riel
On 04/24/2015 10:01 AM, Christoph Lameter wrote: On Thu, 23 Apr 2015, Paul E. McKenney wrote: As far as I know Jerome is talkeing about HPC loads and high performance GPU processing. This is the same use case. The difference is sensitivity to latency. You have latency-sensitive HPC

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Christoph Lameter
On Fri, 24 Apr 2015, Paul E. McKenney wrote: can deliver, but where the cost of full-fledge hand tuning cannot be justified. You seem to believe that this latter category is the empty set, which I must confess does greatly surprise me. If there are already compromises are being made then

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Rik van Riel
On 04/21/2015 05:44 PM, Paul E. McKenney wrote: AUTONUMA The Linux kernel's autonuma facility supports migrating both memory and processes to promote NUMA memory locality. It was accepted into 3.13 and is available in RHEL 7.0 and SLES 12. It is enabled by the

Re: Interacting with coherent memory on external devices

2015-04-24 Thread Benjamin Herrenschmidt
On Fri, 2015-04-24 at 22:32 -0400, Rik van Riel wrote: The result would be that the kernel would allocate only migratable pages within the CCAD device's memory, and even then only if memory was otherwise exhausted. Does it make sense to allocate the device's page tables

  1   2   >