Sorry for the late reply, but I was really busy with my article last week. Let me provide some rough numbers. For usual syslog traffic (< 100 bytes message size), 768 bytes is a good size to be expected for use by each message. If you have larger traffic, add the full average size to that picture (e.g. average size is 200 bytes, the typical size of a message object is around 968 bytes, NOT 868 - this has to do with some internal optimization).
Then, simply multiply the configured size of the queue with that number. For example, you have a 2,000,000 message max configured, so 2,000,000 * 768 which gives you ~ 1.4 GB message store if the queue is full. The same applies to the action queues, here we have 200,000 entries per queue, that would sum up to ~140 MB per queue. Multiply that with the number of action queues. If, for example, you have 20 action queues, that would sum up to a total memory requirement of 20*.14+1.4 = 4.2 GB of total queue memory. Of course, this is an extreme number, and typically no system will have all action queues totally filled up. For omfile, with background writing (always enabled in ZIP mode), we use double-buffering with the configured buffers sizes (256KB being the default). So add 0.5 MB per open dynafile in this case. Multiply that by the maximum number of dynafiles you permit to be kept open. Add these numbers for each dynafile action. That gives you a (very theoretical) upper bound of memory use for the output system. Limiting the memory use is so simple that it probably is easy to overlook: just reduce the maximum numbers - that's why they exist ;) Depending on the input source, rsyslog then tries to slow down too-fast senders if it can't keep up. This works pretty well with TCP and not at all with UDP. Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Aaron Wiebe > Sent: Wednesday, October 28, 2009 4:14 PM > To: rsyslog-users > Subject: [rsyslog] Queue sizing > > Greetings, > > I'm trying to get a good handle on queue sizing and memory usage. In > a higher-traffic environment, I have a dedicated rsyslog machine with > a large amount of ram. I want to try and find the right balance > between large queues for burst traffic, but I also want to make sure > that the queue sizes are small enough that we don't run out of memory > and crash. I'm using 4.5.5 currently. > > I'm currently using disk-backed memory queues, configured in > fixedarray, like so: > > $MainMsgQueueSize 2000000 > $MainMsgQueueMaxFileSize 50g > $MainMsgQueueSaveOnShutdown on > $MainMsgQueueHighWaterMark 15000000 > $MainMsgQueueLowWaterMark 10000000 > $MainMsgQueueType FixedArray > $MainMsgQueueWorkerThreads 2 > > $ActionQueueSize 200000 > $ActionQueueMaxFileSize 50g > $ActionQueueSaveOnShutdown on > $ActionQueueHighWaterMark 1500000 > $ActionQueueLowWaterMark 1000000 > $ActionQueueType FixedArray > $ActionQueueWorkerThreads 1 > > > I usually have upwards of a dozen action queue definitions, and my > omfile outputs are based on hostname of the incoming connection. Is > there some way I can calculate maximum memory use by the queuing > system, or force the system to a set memory maximum where I -know- > that it will simply start pushing back on the incoming connection > instead of adding more data to the system? > > Thoughts? > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
