On Wed, 11 May 2016, Rainer Gerhards wrote:
This actually sounds like a bug. If DA is started, an *additional*
thread is created for the DA queue (actually a full queue instance,
but let's simplify it to a new thread). This thread is supposed to run
in *parallel* the the other threads that already exists (or more
precisely: to those threads that belong to the memory queue object,
which is fairly decoupled). So if the mem queue blocks, something is
definitely wrong.
Ok, that doesn't quite match my understanding. I thought that it pulled data
from the DA queue if the mem queue was empty or something like that. What you
say is what I was asking for below. My reason for making it optional was in case
there would be problems with multiple threads delivering to the same
destination.
As a side note, this also means that newer and older data is mixed,
because the two queues run concurrently. We changed that in v5 (IIRC),
because otherwise we would actually need to block the mem queue until
the DA queue is finished, which in practice meant we never recovered).
yes.
So it is now interesting to find out under which circumstances (in the
most simple setup) this problem occurs.
well, my setup is pretty simple (although not the most simple):
ction(type="omfwd" Target="10.2.192.22" Port="514" Protocol="tcp"
template="structured_forwarding" name="relay_cross_dc" queue.type="FixedArray"
queue.dequeuebatchsize = "1000" queue.size="500000" queue.filename="x_dc_queue"
action.resumeRetryCount="-1" queue.maxfilesize="256m" queue.saveonshutdown="on")
David Lang
Rainer
2016-05-11 16:21 GMT+02:00 David Lang <[email protected]>:
I had a short outage last night that built a backlog of a few tens of
millions of logs in a DA queue. I expected it to take a while to process
this pending data, but what I did not expect is to find that while it is
replaying pending data, it isn't processing new data at the normal rate
I know that we don't send old data before new data.
What I was seeing is that if I don't process the queued data (due to a
missing qi file), I was receiving and processing messages at ~200K/min. but
if it was also replaying the queued data, it was only receiving and
processing new messages via TCP at ~55K/min.
the worker thread for this queue was at 96% utilization or so. I tried
setting the workerthread count to 2, thinking that this would let me have
one thread processing the DA queue and the other handle new logs, but this
did not appear to help much (the second thread was created, but only using
~6% cpu and the rate of processing new logs did not improve noticably)
would it make sense to have (at least as an option) a separate thread for
replaying logs from the DA queue, and that thread would not interact with
the in-memory queue at all (or use a temporary in-memory queue)?
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
