On Thu, Feb 18, 2010 at 2:43 PM, Koen Deforche <[email protected]> wrote: > Hey, > > 2010/2/18 mobi phil <[email protected]>: >> I hope you do not feel polluted by my emails :) >> >> In all C++ projects (and not only C++) I faced sooner or later memory >> fragmentation. Given the specificity of WT application, I think one >> could do some minimal trick to reduce significantly memory >> fragmentation, improving thus performance of the application. The >> linux libc does not have the best possible allocator, especially for >> small objects, and applications can have their memory very fast >> defragmented, given that C++ allocates often lots of small objects. >> Especially in multi threaded applications this can be a serious >> performance penalty. >> >> What is specific for Wt is that when a session is terminated, all the >> object tree is deleted. Now if one would allocate with malloc() an >> area of memory and would manage itself that memory and would free() >> that area after the session of memory, the total defragmentation would >> be zero. What is important, is that you need only basic memory >> management above the area allocated with malloc. Implementation of the >> new operator for size X would return a pointer to current p, and >> current will become current+X. On delete you would not do anything. >> This game could be generalized in case that the same thread supports >> several sessions, or several sessions are served by several threads: >> >> 1. there would be a global map session id to area, that would help new >> to go to the right memory area. Each thread would have such a map, so >> it would not be necessary to do extra locking. >> >> 2. If a session uses two threads then locking during allocation could >> be also avoided, as each thread would have a map as above, the only >> question is how to go to the right area based on the thread ID. >> >> One could limit thus the memory usage of a session. If a user would >> use to many sessions, you either allocate new blocks for it, or raise >> error "dear user, you are too expensive for me :)" >> >> Obviously this would not fit for long sessions, or where the memory >> consumption of a session is considerably high. > > We did consider this, and I think that you are right in the sense that > Wt does not do anything (except for a memory pool for event signals) > to prevent this. > > I think there are even other interesting variations on this idea: for > example, in Linux it is possible to protect memory access within a > process (mprotect(2)), and you could make sure that a session remains > isolated from another session memory-wise, when allocating memory > block-wise per session as you suggest. > > However, I do think we will need to support proper deletion of memory > within sessions (as a library author we cannot assume too much of what > a user will do), and thus we would need to implement a proper memory > allocator (which is not a fun thing to do). This is the reason why we > currently have not done any work on this and are rather seeing if this > becomes a real issue.
Actually I prefer Google's memory allocator, it reduces fragmentation and is a great deal faster then the system allocator on either linux or windows. Just google: Google tcmalloc ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ witty-interest mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/witty-interest
