You are using it correctly, it sounded from your prior post that the disk queues
were getting large while you were trying to figure out how to dequeue things
from them faster.
David Lang
On Wed, 4 Nov 2015, Joe Blow wrote:
I probably should have clarified. So let's say i'm dealing with a
catastrophic failure (action queue can't send remotely, either to ES, or
another forwarder, or whatever). Let's say that the problem then gets
fixed, but now i have many millions of logs from the backlog (let's say 2
hours worth of logs, at 30k EPS).
So now, the box that's supposed to catch the logs from the action queue
(whatever that may be) is back up, and the logs can now be sent to the
system. Regular in memory queues can now be dequeued properly, but now
there is a massive backlog to deal with. The in memory queues seem to be
working fine now, but they're not chipping away at the DA backlog.
Does this my ask make sense now? Essentially, this is an "after outage"
situation. Upstream is fixed, and now the DA queues need to be emptied,
but they're DOG slow.
Radu - My checkpoint interval is set at 100k. Are you suggesting this be
lowered? raised?
Hope this helps describe the problem a bit better.
Cheers,
JB
On Wed, Nov 4, 2015 at 9:58 AM, David Lang <[email protected]> wrote:
I believe that those only have an effect if you have configured syncing on.
disk assist performance is known to be poor (it's really designed as a
last-resort option that you should not end up usiing in normal operation).
There has been some talk about revamping the queue mechanism, including
disk queues and their performance, but that's a major project.
However, if you have new data going into the disk queues, you need to
troubleshoot what's wrong that's preventing the logs from being delivered
and causing them to back up. If you don't solve that problem, the speed you
can extract them from the disk queue is meaningless as you will just keep
adding to the queue and not pull from it.
Logs only go to the disk queue when the memory queue is too full and more
logs are arriving. Logs get delived from the memory queue first, and only
after it's empty do logs get pulled from the disk queue.
David Lang
On Wed, 4 Nov 2015, Radu Gheorghe wrote:
Hi Joe,
I think the way to change it is via queue.checkpointinterval (and
maybe also set queue.syncqueuefiles to "off"). It's also possible that
there are more settings under
http://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html
Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Wed, Nov 4, 2015 at 4:40 PM, Joe Blow <[email protected]> wrote:
Hey folks,
I've got a big disk assisted queue with many million logs in the backlog,
and the feed is still coming in. I see that the queue is releasing
messages, but at a very slow rate (sometimes as slow as 50/second).
Is there any way to change the bulk size for the DA portion of the queue?
What configs settings might help to deal with the backlog better? Right
now my dequeuebatch size is quite large (50k) as the feed is quite large,
and is still flowing in.
Any ideas?
Cheers,
JB
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
