> On Mar 9, 2017, at 7:18 PM, Richard Mills <[email protected]> wrote: > > Hi Barry, > > I like the sound of this, but I think we'd need to be careful about not > messing up data alignment if we do this. If we want a malloc that is going > to let us put the start of an array on, say, a 64 byte alignment boundary, > then we need to not mess that up by putting this integer value there.
As I said the extra space is 64 bit. Now if you want 128 bit alignment we could put a 128 bit. > We could pad with an extra 64 bytes internally, though that may be getting > too wasteful. I don't know how to get a malloc that gives us a starting > address that is 64 bits *before* an alignment boundary (so that the memory > the user sees from the malloc call indeed starts at the boundary), but maybe > that's doable... What alignment boundaries are useful for Intel processes? 64 yup, 128, 256, 512 ? Does higher values provide better performance for SIMD etc? > > If the goal is to simply deal with allocations to high bandwidth memory on > KNL, the memkind-provided free() will do the right thing with allocations in > DRAM or MCDRAM. Hmm, Hong, how come we don't use this? I didn't realize it worked this way. This would shut Jed up immediately. Sadly, I fear the answer is we don't use memkind because it sucks :-) Calm down Jeff, I didn't insult your mother. Barry > But, as you say, there are issues in other cases, like with -malloc_debug. > > --Richard > > On Thu, Mar 9, 2017 at 4:19 PM, Barry Smith <[email protected]> wrote: > > Using different mallocs for different objects/arrays in PETSc is very iffy > because each free() has to match the malloc used for that memory. This is > even true with just -malloc_debug in that certain initialization functions in > PETSc need to use the raw malloc() because we cannot be sure if the > (*PetscTrMalloc)() has been set yet and the raw free() that comes at > PetscFinalize() time needs to be matched with it. > > Why not have PetscMalloc() ALWAYS allocate an extra 64 bit space at the > beginning and put in an integer indicating the malloc family that has been > used to get the space. PetscFree() would use this integer to determine the > correct free() to use. A mechanism to register new malloc families could be > easily done, for example > > PetscMallocRegister(malloc,realloc,free,&basicmalloc); > > PetscMallocRegister(PetscMallocDebug,PetscReallocDebug,PetscFreeDebug,&debugmalloc); > > PetscMallocRegister(PetscMallocHBW,PetscReallocHBW,PetscFreeHBW,&hbwmalloc); > > To change the malloc used you would do PetscMallocPush(debugmalloc); > PetscMalloc(....); .... PetscMallocPop(); Note that you can register > additional malloc families at any time (it doesn't have to be as soon as the > program starts up). > > What is wrong with the model and why shouldn't we use it? > > Barry > > Notes: > > It is easy to implement, so that is not a reason. > > The extra memory usage is trivial. > > The mapping from integer to malloc() or free() would be a bounds check and > then accessing the function pointer from a little array so pretty cheap. > > if certain mallocs are missing (like PetscMallocHBW) the hbwmalloc variable > could be set to the basicmalloc value (or some other) so one would not need > to ifdef if if () code deciding which malloc to use in many places. > > It seems so simple something must be fundamentally flawed with it. Even with > just PetscTrMallocDefault() and PetscMallocAlign() I feel like implementing > it. > > > > > >
