> On Mar 9, 2017, at 7:38 PM, Barry Smith <[email protected]> wrote: > > >> On Mar 9, 2017, at 7:18 PM, Richard Mills <[email protected]> wrote: >> >> Hi Barry, >> >> I like the sound of this, but I think we'd need to be careful about not >> messing up data alignment if we do this. If we want a malloc that is going >> to let us put the start of an array on, say, a 64 byte alignment boundary, >> then we need to not mess that up by putting this integer value there. > > As I said the extra space is 64 bit. Now if you want 128 bit alignment we > could put a 128 bit. > >> We could pad with an extra 64 bytes internally, though that may be getting >> too wasteful. I don't know how to get a malloc that gives us a starting >> address that is 64 bits *before* an alignment boundary (so that the memory >> the user sees from the malloc call indeed starts at the boundary), but maybe >> that's doable... > > What alignment boundaries are useful for Intel processes? 64 yup, 128, 256, > 512 ? Does higher values provide better performance for SIMD etc? > >> >> If the goal is to simply deal with allocations to high bandwidth memory on >> KNL, the memkind-provided free() will do the right thing with allocations in >> DRAM or MCDRAM. > > Hmm, Hong, how come we don't use this? I didn't realize it worked this way. > This would shut Jed up immediately.
This is what I have already mentioned in our discussions under the pull request. If we have smarter APIs in the future (and I believe we will), things will be much easier. However, hbw_free() in memkind does not work this way yet. If I do the allocation with normal malloc() and free it with hbw_free(), I will get an SEGV error. If I pair hbw_malloc() with free(), I will also get an SEGV error. What is interesting is that hbw_malloc() in memkind can automatically change its allocator to use DRAM if there is insufficient MCDRAM. So I would hope for hbw_free() could change to normal free() when handling data allocated by normal malloc(). Hong (Mr.) > > Sadly, I fear the answer is we don't use memkind because it sucks :-) Calm > down Jeff, I didn't insult your mother. > > > > Barry > >> But, as you say, there are issues in other cases, like with -malloc_debug. >> >> --Richard >> >> On Thu, Mar 9, 2017 at 4:19 PM, Barry Smith <[email protected]> wrote: >> >> Using different mallocs for different objects/arrays in PETSc is very iffy >> because each free() has to match the malloc used for that memory. This is >> even true with just -malloc_debug in that certain initialization functions >> in PETSc need to use the raw malloc() because we cannot be sure if the >> (*PetscTrMalloc)() has been set yet and the raw free() that comes at >> PetscFinalize() time needs to be matched with it. >> >> Why not have PetscMalloc() ALWAYS allocate an extra 64 bit space at the >> beginning and put in an integer indicating the malloc family that has been >> used to get the space. PetscFree() would use this integer to determine the >> correct free() to use. A mechanism to register new malloc families could be >> easily done, for example >> >> PetscMallocRegister(malloc,realloc,free,&basicmalloc); >> >> PetscMallocRegister(PetscMallocDebug,PetscReallocDebug,PetscFreeDebug,&debugmalloc); >> >> PetscMallocRegister(PetscMallocHBW,PetscReallocHBW,PetscFreeHBW,&hbwmalloc); >> >> To change the malloc used you would do PetscMallocPush(debugmalloc); >> PetscMalloc(....); .... PetscMallocPop(); Note that you can register >> additional malloc families at any time (it doesn't have to be as soon as the >> program starts up). >> >> What is wrong with the model and why shouldn't we use it? >> >> Barry >> >> Notes: >> >> It is easy to implement, so that is not a reason. >> >> The extra memory usage is trivial. >> >> The mapping from integer to malloc() or free() would be a bounds check and >> then accessing the function pointer from a little array so pretty cheap. >> >> if certain mallocs are missing (like PetscMallocHBW) the hbwmalloc variable >> could be set to the basicmalloc value (or some other) so one would not need >> to ifdef if if () code deciding which malloc to use in many places. >> >> It seems so simple something must be fundamentally flawed with it. Even with >> just PetscTrMallocDefault() and PetscMallocAlign() I feel like implementing >> it. >> >> >> >> >> >> >
