On Sun, 18 Jan 2009, Rainer Gerhards wrote: > On Sun, 2009-01-18 at 04:33 -0800, [email protected] wrote: >> On Sun, 18 Jan 2009, Rainer Gerhards wrote: >> >>> On Sat, 2009-01-17 at 16:55 -0800, [email protected] wrote: >>> >>>> >>>> what I think that you are seeing is that the default rsyslog memory queue >>>> only uses as much ram as it needs to hold the data (even though it's >>>> described as a array it seems to grow dynamicly, I'm not sure about it >>>> shrinking) >>> >>> If you use "fixedarray" mode, the pointer array is allocated statically, >>> no matter how many messages are in the queue. HOWEVER, this is only the >>> pointers, so quite few memory. Actual messages are dynamically allocated >>> and freed when processed - in any mode. >> >> that makes sense. It would be interesting to see what would happen to the >> enqueue/dequeue timings if the message memory was staticly allocated >> >> from what I remember seeing of the memory footprint it does appear as if >> you allocate the max size for the message each time, not the minimum sized >> needed to hold the message >> > yes, that's right. This is done to prevent an additional copy to clean > things up (realloc might work, too) and memory fragmentation. The later > is really nasty, I've seen that some memory areas remain allocated for > quite some while due to fragmentation. > >> if that shows a noticable difference it may be worth allocating the memory >> in chunks substantially larger than a single message > > That's a good suggestion. The basic classes are able to trim strings. It > may be worth putting a config option into it. The current approach works > well for small queues, but obviously does provide sub-optimal > performance as soon as the queues grow considerably. So it may even make > sense to start trimming messages only after a certain amount of messages > are in-queue.
I'm not sure that we're saying the same thing. let me try again. what I was thinking was that instead of allocating memory for one message at a time, initially allocate memory for 100 messages, then if this needs to be extended increase the allocation by 50-100%. this minimizes the number of allocations needed and the fragmentation of system memory. just like the fixed-array queue option is significantly faster than the linked list queue option (I assume from a combination of having to chase pointers and allocate/deallocate memory), there may be similar benifits from doing the same thing for the message content itself. David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
