>> 1. If tcmalloc et al are brittle, do we understand why is it so? Is it >> an inherent problem or just a lame implementation?
It wasn't itself tcmalloc that was brittle, it was Mac's equivalent of LD_PRELOAD. >> 2. Douglas' patch stores the pointer to allocator function in a global >> variable. That breaks the model of separate contexts (i.e. that several >> instances of 0mq can exist in a single process without misinteracting >> with each other). We have to think of something more resilient. > > Perhaps the allocator should be embedded into the context. You could even > create separate contexts if you wish to have different allocators. Exactly. That eliminates the global variable nicely. But you've have to add a context parameter to the zmq_msg_t functions. >> 3. There are at least two use cases here. AFAIU, Douglas want to >> custom-allocate the message bodies while Chunk want to custom-allocate >> everything. We should think a bit more about whether one use case >> subsumes the other or whether they are two separate use cases with two >> separate solutions. > > Maybe you could provide two args to the allocator when allocating memory: the > size in bytes and a hint. This hint could be used by the allocator to decide > upon it: use different memory pools for different parts of the program, etc. > Then, you could use the allocator for everything in ZeroMQ itself and in > programs using ZeroMQ, assuming ZeroMQ will have a default implementation of > the allocator. > > If we go this route, it could also be the chance to use a couple of macros > for interfacing with the allocator: > > ZMQ_ALLOC(size, hint); > ZMQ_REALLOC(old_size, new_size, hint); > ZMQ_FREE(size, hint); > > The point is that the macros can (maybe only in debugging mode) decorate the > call by passing the file name and line number to the allocator. I have found > this invaluable when debugging memory problems. > > Unluckily, it has been too long since the last time I did something like > this; I would have to force myself to get back up to speed in order to > contribute with more than simple ideas. The basic API in my previous patch would be a good starting point for this — it defined those macros, like you suggest. You provide a custom allocator as a single function: void * (*alloc_func)(void *user_data, void *old_ptr, size_t old_size, size_t new_size); Depending on which parameters you pass in, this mimics malloc, realloc, and free. This API is the same one they use in the Lua interpreter, and I implemented the same thing in the Avro C bindings, too. It seems to work pretty well. The main issue with this API is that you have to keep track of the allocated size of everything you allocate. For most fixed-size types, that's not an issue, since you can just use sizeof when you free, just like when you malloc. For variable-sized buffers, though, you usually need an extra field to store the allocated size. Another issue is that the zmq_msg_t API is all pure C, whereas most of the rest of the library is C++, so we'll have to overload the definitions of new and delete to really make this pervasive. I'm not sure about the hint parameter you mention — can you explain what you mean by that? I had a user_data parameter in the allocation function, but this was passed in by the user of the library. Your macros look like the hint would come from within 0mq...so would we have some well-defined hints that tell the allocator which part of the library the memory is for? –doug
PGP.sig
Description: This is a digitally signed message part
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
