How fat a pipe do you have? that sounds like it's pushing the limit for 10G
interfaces.
Also, remember that using multiple threads on the input module only helps if you
have multiple connections to you. A given connection can only really be handled
by one thread (if you think about it, the possibility that different threads
could get different parts of the same message when reading from one connection
would be a nightmare to deal with)
I would guess that you can probably handle this load with a handful of threads
(low to mid single digits), but at those volumes, you are going to be running
into all sorts of bottlenecks. Just getting the OS TCP stack to keep up at 10Gb
is rough, and you then have to do something with the log messages. It takes
quite an impressive I/O system to be able to write data at that speed, and if
you are sending the data out at that speed, I hope it's through another
interface.
Around the late v5 and early v6 days I was doing testing that was pushing ~400K
logs/sec at 1/4KB per message (256 bytes) and I found that Gig-E was my limiting
factor, so going 1.5x faster with messages 16x-8000x larger on a pipe only 10x
faster seems like it would be pushing it. If you are doing compression you may
be able to get away with it, but it will take significantly more CPU, and again,
you have to be able to output the logs.
I'm pretty sure that you're going to need to scale horizontally to do this. What
is it that's driving you to try scaling vertically?
David Lang
On Tue, 10 Mar 2015, Boylan, James wrote:
We are running 100,000 to 600,000 messages per second of messages ranging from
4KB to 2MB in size. (obviously done using imptcp)
I wasn't aware of the potential batching issue. That is extremely good to know.
-- James
--- Sent from my mobile phone ---
----- Reply message -----
From: "David Lang" <[email protected]>
To: "rsyslog-users" <[email protected]>
Subject: [rsyslog] Question about maximum queue.workerthreads count
Date: Tue, Mar 10, 2015 5:30 PM
On Tue, 10 Mar 2015, Boylan, James wrote:
Greetings everyone!
Is there a hardcoded maximum number of worker threads for queues and threads
for imtcp/imudp? Also what are the concerns with using a higher number of
threads for handling of these.
There is probably a max number of threads, but it is almost certinly much higher
than you should need to go before the limiting factor will be the speed of your
connection. a couple thread can handle a saturated gig-E network (at least
without encryption or compression), at ~400k logs/second
The problem you run into with too many threads is that message batching doesn't
take place, each thread grabs one message and puts it on the queue. Doing this
one message at a time results in a lot of locking overhead and can end up
drastically slowing down all work. The threads that try to retrieve the messages
from the queue end up having trouble getting the lock to be able to do work
because they are competing against the thread trying to add messages. The
batching process drastically reduces the amount of time the main queue is locked
to process a given set of messages.
We've seen cases where people set the number of threads in the 4-8 range and
found they were getting much worse throughput than when they had it set to 1.
This is really something that you need to measure, both under full load and
under partial load
I find myself in a position to have to scale central logging servers
vertically and I'm attempting to locate the risks and issues of increasing
these numbers.
what sort of log numbers and traffic volume are you looking at?
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
