> On Mar 9, 2017, at 8:29 PM, Zhang, Hong <[email protected]> wrote: > > >> On Mar 9, 2017, at 7:38 PM, Barry Smith <[email protected]> wrote: >> >> >>> On Mar 9, 2017, at 7:18 PM, Richard Mills <[email protected]> wrote: >>> >>> Hi Barry, >>> >>> I like the sound of this, but I think we'd need to be careful about not >>> messing up data alignment if we do this. If we want a malloc that is going >>> to let us put the start of an array on, say, a 64 byte alignment boundary, >>> then we need to not mess that up by putting this integer value there. >> >> As I said the extra space is 64 bit. Now if you want 128 bit alignment we >> could put a 128 bit. >> >>> We could pad with an extra 64 bytes internally, though that may be getting >>> too wasteful. I don't know how to get a malloc that gives us a starting >>> address that is 64 bits *before* an alignment boundary (so that the memory >>> the user sees from the malloc call indeed starts at the boundary), but >>> maybe that's doable... >> >> What alignment boundaries are useful for Intel processes? 64 yup, 128, 256, >> 512 ? Does higher values provide better performance for SIMD etc? >> >>> >>> If the goal is to simply deal with allocations to high bandwidth memory on >>> KNL, the memkind-provided free() will do the right thing with allocations >>> in DRAM or MCDRAM. >> >> Hmm, Hong, how come we don't use this? I didn't realize it worked this way. >> This would shut Jed up immediately. > > This is what I have already mentioned in our discussions under the pull > request. If we have smarter APIs in the future (and I believe we will), > things will be much easier. > > However, hbw_free() in memkind does not work this way yet. If I do the > allocation with normal malloc() and free it with hbw_free(), I will get an > SEGV error. If I pair hbw_malloc() with free(), I will also get an SEGV > error.
Yes, but why are you using the hwb_ calls instead of using memkind calls? There could be good reasons I am not suggesting calling the memkind ... > > What is interesting is that hbw_malloc() in memkind can automatically change > its allocator to use DRAM if there is insufficient MCDRAM. So I would hope > for hbw_free() could change to normal free() when handling data allocated by > normal malloc(). > > Hong (Mr.) > >> >> Sadly, I fear the answer is we don't use memkind because it sucks :-) Calm >> down Jeff, I didn't insult your mother. >> >> >> >> Barry >> >>> But, as you say, there are issues in other cases, like with -malloc_debug. >>> >>> --Richard >>> >>> On Thu, Mar 9, 2017 at 4:19 PM, Barry Smith <[email protected]> wrote: >>> >>> Using different mallocs for different objects/arrays in PETSc is very iffy >>> because each free() has to match the malloc used for that memory. This is >>> even true with just -malloc_debug in that certain initialization functions >>> in PETSc need to use the raw malloc() because we cannot be sure if the >>> (*PetscTrMalloc)() has been set yet and the raw free() that comes at >>> PetscFinalize() time needs to be matched with it. >>> >>> Why not have PetscMalloc() ALWAYS allocate an extra 64 bit space at the >>> beginning and put in an integer indicating the malloc family that has been >>> used to get the space. PetscFree() would use this integer to determine the >>> correct free() to use. A mechanism to register new malloc families could be >>> easily done, for example >>> >>> PetscMallocRegister(malloc,realloc,free,&basicmalloc); >>> >>> PetscMallocRegister(PetscMallocDebug,PetscReallocDebug,PetscFreeDebug,&debugmalloc); >>> >>> PetscMallocRegister(PetscMallocHBW,PetscReallocHBW,PetscFreeHBW,&hbwmalloc); >>> >>> To change the malloc used you would do PetscMallocPush(debugmalloc); >>> PetscMalloc(....); .... PetscMallocPop(); Note that you can register >>> additional malloc families at any time (it doesn't have to be as soon as >>> the program starts up). >>> >>> What is wrong with the model and why shouldn't we use it? >>> >>> Barry >>> >>> Notes: >>> >>> It is easy to implement, so that is not a reason. >>> >>> The extra memory usage is trivial. >>> >>> The mapping from integer to malloc() or free() would be a bounds check and >>> then accessing the function pointer from a little array so pretty cheap. >>> >>> if certain mallocs are missing (like PetscMallocHBW) the hbwmalloc variable >>> could be set to the basicmalloc value (or some other) so one would not need >>> to ifdef if if () code deciding which malloc to use in many places. >>> >>> It seems so simple something must be fundamentally flawed with it. Even >>> with just PetscTrMallocDefault() and PetscMallocAlign() I feel like >>> implementing it. >>> >>> >>> >>> >>> >>> >> >
