oh, one thing I forgot: you mentioned disk queues in some mail. You can not use disk-assisted queues if you totally insist on ordering, because there is some inherent re-ordering possible when starting up disk mode. So you need to use pure disk queues or pure memory queues.
Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Marc Schiffbauer > Sent: Thursday, November 04, 2010 11:52 PM > To: rsyslog-users > Subject: Re: [rsyslog] One Queue for multiple Actions in 4.6.4? > > > > > > > About up to 3000 Messages/s > > To be more precise: That will be the expected peak. So the average will > be > much lower. > > And we have decent hardware to do this. > > > > > at that data rate you will have a very hard time doing things the way > > you > > are trying. > > > > at the database level, doing 3000 inserts/sec requires _very_ high- > end > > hardware. > > rsyslog supports inserting multiple messages in one command, > > and > > with that you could probably handle 3000 messages/sec on very low-end > > hardware (because rsyslog could insert messages in batches of 100 or > > more > > if needed). unfortunantly, when you enable this, you don't maintain > > the > > message order. > > > > at the rsyslog level, doing disk based queues with everything tuned > to > > come as close to minimizing the chance of data loss is very hard to > do > > at > > that traffic level. I ran tests last year on a 8 core box with 64G of > > ram > > and a fusion-io SSD pci card, and depending on the filesystem I used, > > I > > was able to get from 2K to 8K messages/sec where I wasn't trying to > do > > anything more than write the message to a log file. > > > > Thanks for the numbers. > > > you are going to have to think very hard about how critical it really > > is > > for you to maintain the same message order and what type of > > reliability > > you really need for your messages. > > > > it's counter-intuitive, but it may be that you end up with better > > overall > > reliability with a much faster configuration that has worse > > 'worst-case' > > data loss, but is fast enough that under normal conditions everything > > is > > processed really fast than you would under a configuration that is > > much > > slower all the time, but has a better worst-case data loss. > > Yes, I will keep that in mind, thanks. > > > > > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
