On Friday 19 February 2010 03:25:42 OvermindDL1 wrote: > On Thu, Feb 18, 2010 at 2:43 PM, Koen Deforche <[email protected]> wrote: > > Hey, > > > > 2010/2/18 mobi phil <[email protected]>: > >> I hope you do not feel polluted by my emails :) > >> > >> In all C++ projects (and not only C++) I faced sooner or later memory > >> fragmentation. Given the specificity of WT application, I think one > >> could do some minimal trick to reduce significantly memory > >> fragmentation, improving thus performance of the application. The > >> linux libc does not have the best possible allocator, especially for > >> small objects, and applications can have their memory very fast > >> defragmented, given that C++ allocates often lots of small objects. > >> Especially in multi threaded applications this can be a serious > >> performance penalty. > >> > >> What is specific for Wt is that when a session is terminated, all the > >> object tree is deleted. Now if one would allocate with malloc() an > >> area of memory and would manage itself that memory and would free() > >> that area after the session of memory, the total defragmentation would > >> be zero. What is important, is that you need only basic memory > >> management above the area allocated with malloc. Implementation of the > >> new operator for size X would return a pointer to current p, and > >> current will become current+X. On delete you would not do anything. > >> This game could be generalized in case that the same thread supports > >> several sessions, or several sessions are served by several threads: > >> > >> 1. there would be a global map session id to area, that would help new > >> to go to the right memory area. Each thread would have such a map, so > >> it would not be necessary to do extra locking. > >> > >> 2. If a session uses two threads then locking during allocation could > >> be also avoided, as each thread would have a map as above, the only > >> question is how to go to the right area based on the thread ID. > >> > >> One could limit thus the memory usage of a session. If a user would > >> use to many sessions, you either allocate new blocks for it, or raise > >> error "dear user, you are too expensive for me :)" > >> > >> Obviously this would not fit for long sessions, or where the memory > >> consumption of a session is considerably high. > > > > We did consider this, and I think that you are right in the sense that > > Wt does not do anything (except for a memory pool for event signals) > > to prevent this. > > > > I think there are even other interesting variations on this idea: for > > example, in Linux it is possible to protect memory access within a > > process (mprotect(2)), and you could make sure that a session remains > > isolated from another session memory-wise, when allocating memory > > block-wise per session as you suggest. > > > > However, I do think we will need to support proper deletion of memory > > within sessions (as a library author we cannot assume too much of what > > a user will do), and thus we would need to implement a proper memory > > allocator (which is not a fun thing to do). This is the reason why we > > currently have not done any work on this and are rather seeing if this > > becomes a real issue. > > Actually I prefer Google's memory allocator, it reduces fragmentation > and is a great deal faster then the system allocator on either linux > or windows. Just google: Google tcmalloc
Can you just link it in when compiling Wt? ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ witty-interest mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/witty-interest
