The beauty of git/bitbucket is one can make branches to try out anything they want even if some cranky old conservative PETSc developer thinks it is worse then consorting with the devil.
As I said before I think that "additional argument" to advised_malloc should be a living object which one can change over time as opposed to just a "flag" type argument that only effects the malloc at malloc time. Of course the "living part" can be implemented later. Barry Yes, Jed has already transformed himself into a cranky old conservative PETSc developer > On Jun 3, 2015, at 7:33 PM, Richard Mills <[email protected]> wrote: > > Hi Folks, > > It's been a while, but I'd like to pick up this discussion of adding a > context to memory allocations again. > > The immediate motivation I have is that I'd like to support use of the > memkind library (https://github.com/memkind/memkind), though adding a context > to PetscMallocN() (or making some other interface, say PetscAdvMalloc() or > whatever) could have much broader utility than simply memkind support (which > Jed doesn't like anyway, and I share some of his concerns). For the sake of > having a concrete example, I'll discuss memkind here. > > Memkind's memkind_malloc() works like malloc() but takes a memkind_t argument > to specify some desired property of the memory being allocated. For example, > > hugetlb_str = (char *)memkind_malloc(MEMKIND_HUGETLB, size); > > returns a pointer to memory allocated using huge pages, and > > hbw_preferred_str = (char *)memkind_malloc(MEMKIND_HBW_PREFERRED, size); > > allocates memory from a high-bandwidth region if it's available and elsewhere > if not (specifying MEMKIND_HBW will insist on the allocation coming from > high-bandwidth memory, failing if it's not available). > > It should be straightforward to add a variant of PetscMalloc() that accepts a > context: I'll call this PetscAdvMalloc(), for now, though we can come up with > a better name later. This will allow passing on the memkind_t via this > context to the underlying memkind allocator, and we can have some mechanism > to set a default context (in the case of Memkind, this is likely > MEMKIND_DEFAULT) that gets used when plain PetscMalloc() gets called. > > Of course, we'll need some way to ensure that the "advanced malloc" gets used > to allocated the critical data structures. As a low-level way to start, it > may make sense to simply add a way to stash a context in Vec and Mat objects. > Maybe have VecSetAdvMallocCtx(), and if that context gets set, then > PetscAdvMalloc() is used for the allocations associated with the contents of > that object. It would probably be better to eventually have a higher-level > way to do this, e.g., support standard settings in the options database that > PETSc uses to construct the appropriate arguments to underlying allocators > that are supported, but I think just adding a way to set this context > directly is an appropriate first step. > > Does this sound like a reasonable thing for me to prototype, or are others > thinking something very different? Please let me know. I'm getting more > access to early systems I can experiment on, and I'd really like to move > forward on trying things with high bandwidth memory (imperfect as our APIs > for using it are). > > Best regards, > Richard > > > On Wed, Apr 29, 2015 at 11:10 PM, Richard Mills <[email protected]> wrote: > On Wed, Apr 29, 2015 at 1:28 PM, Barry Smith <[email protected]> wrote: > > Forget about the issue of "changing" PetscMallocN() or adding a new > interface instead, that is a minor syntax and annoyance issue: > > The question is "is it worth exploring adding a context for certain memory > allocations that would allow us to "do" various things to the memory and > "indicate" properties of the memory"? I think, though I agree with Jed that > it could be fraught with difficulties, that is is worthwhile playing around > with this. > > Barry > > > I vote "yes". One might want to, say > > * Give hints via something like madvise() on how/when the memory might be > accessed. > * Specify a preferred "kind" of memory (and behavior if the preferred kind is > not available, or perhaps even specify a priority on how hard to try to get > the preferred memory kind) > * Specify something like a preference to interleave allocation blocks between > different kinds of memory > > I'm sure we can come up with plenty of other possibilities, some of which > might actually be useful, many of which will be useful only for very > contrived cases, and some that are not useful today but may become useful as > memory systems evolve. > > --Richard >
