On Thu, 4 Nov 2010, Marc Schiffbauer wrote:
On Thu, 4 Nov 2010, Marc Schiffbauer wrote:
Will the global MainMsgQueue maintain message order?
I believe that it will, but keep in mind that the settings needed to
try
and maintain message order are very different than the settings that
you
need to use to get best performance.
what sort of message rate are you talking about using?
About up to 3000 Messages/s
at that data rate you will have a very hard time doing things the way you
are trying.
at the database level, doing 3000 inserts/sec requires _very_ high-end
hardware. rsyslog supports inserting multiple messages in one command, and
with that you could probably handle 3000 messages/sec on very low-end
hardware (because rsyslog could insert messages in batches of 100 or more
if needed). unfortunantly, when you enable this, you don't maintain the
message order.
at the rsyslog level, doing disk based queues with everything tuned to
come as close to minimizing the chance of data loss is very hard to do at
that traffic level. I ran tests last year on a 8 core box with 64G of ram
and a fusion-io SSD pci card, and depending on the filesystem I used, I
was able to get from 2K to 8K messages/sec where I wasn't trying to do
anything more than write the message to a log file.
you are going to have to think very hard about how critical it really is
for you to maintain the same message order and what type of reliability
you really need for your messages.
it's counter-intuitive, but it may be that you end up with better overall
reliability with a much faster configuration that has worse 'worst-case'
data loss, but is fast enough that under normal conditions everything is
processed really fast than you would under a configuration that is much
slower all the time, but has a better worst-case data loss.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
