I had a short outage last night that built a backlog of a few tens of millions
of logs in a DA queue. I expected it to take a while to process this pending
data, but what I did not expect is to find that while it is replaying pending
data, it isn't processing new data at the normal rate
I know that we don't send old data before new data.
What I was seeing is that if I don't process the queued data (due to a missing
qi file), I was receiving and processing messages at ~200K/min. but if it was
also replaying the queued data, it was only receiving and processing new
messages via TCP at ~55K/min.
the worker thread for this queue was at 96% utilization or so. I tried setting
the workerthread count to 2, thinking that this would let me have one thread
processing the DA queue and the other handle new logs, but this did not appear
to help much (the second thread was created, but only using ~6% cpu and the rate
of processing new logs did not improve noticably)
would it make sense to have (at least as an option) a separate thread for
replaying logs from the DA queue, and that thread would not interact with the
in-memory queue at all (or use a temporary in-memory queue)?
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
