Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Barry Smith
> On Jun 3, 2015, at 10:35 PM, Jed Brown wrote: > > Barry Smith writes: > >> It is OUR job as PETSc developers to hide that complexity from the >> "most people" who would be driven away from HPC because of it. > > Absolutely. So now the question becomes "what benefit can this have, > pred

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Barry Smith
> On Jun 3, 2015, at 10:28 PM, Jed Brown wrote: > > Barry Smith writes: >> Sure but the super high-end (DOE LCF centers) focus allows (actually >> loves) the need for "super brittle" stuff. > > Job security. > >> It is not the bread and butter of PETSc but if they (silly ASCR) are >> wi

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Jed Brown
Barry Smith writes: > It is OUR job as PETSc developers to hide that complexity from the > "most people" who would be driven away from HPC because of it. Absolutely. So now the question becomes "what benefit can this have, predicated on not letting the complexity bleed onto the user. >

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Jed Brown
Barry Smith writes: > Sure but the super high-end (DOE LCF centers) focus allows (actually > loves) the need for "super brittle" stuff. Job security. > It is not the bread and butter of PETSc but if they (silly ASCR) are > willing to foot our bills to do our bread and butter by panderi

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Barry Smith
> On Jun 3, 2015, at 10:08 PM, Jed Brown wrote: > > Barry Smith writes: >> Even if it "helps" in only 30 percent of applications that is still >> a good thing (and a great thing politically). Then it becomes an >> issue of education and proper profiling tools to tell people for >> their app

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Jed Brown
Jeff Hammond writes: > On Wed, Jun 3, 2015 at 9:58 PM, Jed Brown wrote: >> Jeff Hammond writes: >>> The beauty of git/github is one can make branches to try out anything >>> they want even if Jed thinks that he knows better than Intel how to >>> write system software for Intel's hardware. >> >>

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Barry Smith
> On Jun 3, 2015, at 10:04 PM, Jeff Hammond wrote: > > If everyone would just indent with tabs, we could just set the indent > spacing with our editors ;-) Ah, heresy, kill him! > > On Wed, Jun 3, 2015 at 10:01 PM, Barry Smith wrote: >> >>> On Jun 3, 2015, at 9:58 PM, Jeff Hammon

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Jed Brown
Barry Smith writes: > Even if it "helps" in only 30 percent of applications that is still > a good thing (and a great thing politically). Then it becomes an > issue of education and proper profiling tools to tell people for > their apps that it won't work; so the other 70% is not "confused

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Jeff Hammond
If everyone would just indent with tabs, we could just set the indent spacing with our editors ;-) On Wed, Jun 3, 2015 at 10:01 PM, Barry Smith wrote: > >> On Jun 3, 2015, at 9:58 PM, Jeff Hammond wrote: >> >> http://git.mpich.org/mpich.git/blob/HEAD:/src/mpi/init/init.c >> https://github.com/op

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Jeff Hammond
On Wed, Jun 3, 2015 at 9:58 PM, Jed Brown wrote: > Jeff Hammond writes: >> The beauty of git/github is one can make branches to try out anything >> they want even if Jed thinks that he knows better than Intel how to >> write system software for Intel's hardware. > > I'm objecting to the interface

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Barry Smith
> On Jun 3, 2015, at 9:58 PM, Jeff Hammond wrote: > > http://git.mpich.org/mpich.git/blob/HEAD:/src/mpi/init/init.c > https://github.com/open-mpi/ompi/blob/master/ompi/mpi/c/init.c As I said, super insane :-) Barry I'm just having fun here; I do believe that 2 is the ultimate correct i

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Barry Smith
> On Jun 3, 2015, at 9:51 PM, Jed Brown wrote: > > Barry Smith writes: >> Perhaps, and this is just nonsense off the top of my head, if you >> had some measure of the importance of a vector (or matrix; I would >> start with vectors for simplicity and since we have more of them) >> based on

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Jeff Hammond
http://git.mpich.org/mpich.git/blob/HEAD:/src/mpi/init/init.c https://github.com/open-mpi/ompi/blob/master/ompi/mpi/c/init.c Jeff On Wed, Jun 3, 2015 at 9:43 PM, Barry Smith wrote: > > Jeff, > >Ahh, from this page, it is definitively clear that the Intel people have > their heads totally

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Jed Brown
Jeff Hammond writes: > The beauty of git/github is one can make branches to try out anything > they want even if Jed thinks that he knows better than Intel how to > write system software for Intel's hardware. I'm objecting to the interface. I think that if they try to get memkind merged into the

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Jed Brown
Barry Smith writes: > Perhaps, and this is just nonsense off the top of my head, if you > had some measure of the importance of a vector (or matrix; I would > start with vectors for simplicity and since we have more of them) > based on how often it's values would be "accessed". So a vector

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Barry Smith
Jeff, Ahh, from this page, it is definitively clear that the Intel people have their heads totally up their asses formatted source code with astyle --style=linux --indent=spaces=4 -y -S when everyone knows that any indent that is not 2 characters is totally insane :-) Barry > On Ju

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Jed Brown
Richard Mills writes: >> > but it screws up memkind's partitioning of the heap (it won't be aware >> > that the pages have been moved). >> >> Then memkind is stupid or the kernel isn't exposing the correct >> information to memkind. Tell them to not be lazy and do it right. >> > > I believe that

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Jeff Hammond
>> but it screws up memkind's partitioning of the heap (it won't be aware >> that the pages have been moved). > > Then memkind is stupid or the kernel isn't exposing the correct > information to memkind. Tell them to not be lazy and do it right. The beauty of git/github is one can make branches t

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Richard Mills
On Wed, Jun 3, 2015 at 7:28 PM, Jed Brown wrote: > Barry Smith writes: > > Richard has access to the hardware > > Is this true? Or he will have hardware "soon"? > Yes, I finally have access to hardware. It's a bit hard to get time on, and it's flaky because it is from the initial tape-in, b

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Barry Smith
> On Jun 3, 2015, at 9:28 PM, Jed Brown wrote: > > Barry Smith writes: >> Richard has access to the hardware > > Is this true? Or he will have hardware "soon"? > >> and is not going to "lie to us" that "oh it helps so much" because >> he knows that you will test it yourself and see that h

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Jed Brown
Barry Smith writes: > Richard has access to the hardware Is this true? Or he will have hardware "soon"? > and is not going to "lie to us" that "oh it helps so much" because > he knows that you will test it yourself and see that he is lying. I'm not at all worried about him lying, but I'

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Barry Smith
To follow up on this, going back to my "advise object" to malloc being a living object as opposed to just some flags. In the case where different vectors may have very different "importances" at different times in the runtime of the simulation one could "switch" some vectors from using slow

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Barry Smith
> On Jun 3, 2015, at 8:55 PM, Richard Mills wrote: > > Ha, yes. I'll try this out, but I do wonder what people's thoughts are on > the best way to "tag" an object like a Vec or Mat for some particular > treatment of its placement in memory. Does doing this at the level of a Mat > or Vec (e.

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Richard Mills
On Wed, Jun 3, 2015 at 6:54 PM, Jed Brown wrote: > Richard Mills writes: > > > On Wed, Jun 3, 2015 at 6:04 PM, Jed Brown wrote: > > > >> Have you heard anything back about whether move_pages() will work? > >> > > > > move_pages() will work to move pages between MCDRAM and DRAM right > > now, >

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Barry Smith
> On Jun 3, 2015, at 9:00 PM, Jed Brown wrote: > > Barry Smith writes: >> Yes, Jed has already transformed himself into a cranky old conservative >> PETSc developer > > Is disinclination to spend effort on something with negative expected > value "conservative"? > > Actually, it's almost the

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Jed Brown
Barry Smith writes: > Yes, Jed has already transformed himself into a cranky old conservative PETSc > developer Is disinclination to spend effort on something with negative expected value "conservative"? Actually, it's almost the definition. But if you spend time on legitimately high-risk thin

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Richard Mills
Ha, yes. I'll try this out, but I do wonder what people's thoughts are on the best way to "tag" an object like a Vec or Mat for some particular treatment of its placement in memory. Does doing this at the level of a Mat or Vec (e.g., VecSetAdvMallocCtx() ) sound appropriate? We could actually ma

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Jed Brown
Richard Mills writes: > On Wed, Jun 3, 2015 at 6:04 PM, Jed Brown wrote: > >> Have you heard anything back about whether move_pages() will work? >> > > move_pages() will work to move pages between MCDRAM and DRAM right > now, Great! > but it screws up memkind's partitioning of the heap (it wo

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Barry Smith
The beauty of git/bitbucket is one can make branches to try out anything they want even if some cranky old conservative PETSc developer thinks it is worse then consorting with the devil. As I said before I think that "additional argument" to advised_malloc should be a living object which

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Richard Mills
On Wed, Jun 3, 2015 at 6:04 PM, Jed Brown wrote: > Richard Mills writes: > > > It's been a while, but I'd like to pick up this discussion of adding a > > context to memory allocations again. > > Have you heard anything back about whether move_pages() will work? > move_pages() will work to move

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Jed Brown
Richard Mills writes: > It's been a while, but I'd like to pick up this discussion of adding a > context to memory allocations again. Have you heard anything back about whether move_pages() will work? > hbw_preferred_str = (char *)memkind_malloc(MEMKIND_HBW_PREFERRED, size); How much would you

Re: [petsc-dev] Adding support memkind allocators in PETSc

2015-06-03 Thread Richard Mills
Hi Folks, It's been a while, but I'd like to pick up this discussion of adding a context to memory allocations again. The immediate motivation I have is that I'd like to support use of the memkind library (https://github.com/memkind/memkind), though adding a context to PetscMallocN() (or making s