If you cannot loose any logs, then when you run out of disk space and memory
queue space your systems will stop working and you won't even be able to login
to them (because doing so attempts to create a log entry)
Also, using Disk Assisted Queues means that you have some log entries on disk
and others in memory. When you restart rsyslog, the ones in memory are going to
be lost (because they can't be written to disk)
So the biggest thing you need to do is to look at where the logs are going and
try to make that fast enough to keep up. If you are delivering logs to logstash,
what is logstash doing with them (sending them to ElasticSearch would be my
guess, but are they manipulated first or sent elsewhere as well?) Rsyslog may be
able to deliver directly to those destinations, bypassing the bottleneck of
logstash
Yes, there are ways to get rsyslog to read the queue files, I'd have to hunt in
the archives, but IIRC there is a utility that will create the qi files so that
rsyslog will notice them the next time it starts. I'd have to hunt the list
archives to find references to how to do it.
David Lang
On Wed, 8 Jul 2015, Nicolas Guyomar wrote:
Hi everyone,
Unfortunately I cannot loose any log, because they all are access.log with
the same severity so I can't juste say "discard the lower level"
Let's say my problem could be solved by upgrading to the latest Rsyslog
stable version (planned for this summer), can I "replay" the log flushed
into queue file in my work directory ?
When my rsyslog V5 instance is stucked with its ActionQueueMaxDiskSpace
reached, restarting has no effect. I'd like to maybe save the queue file to
some other directory, and copy old queue file in the work directory so that
Rsyslog send them to logstash.
On 30 June 2015 at 11:41, Radu Gheorghe <[email protected]> wrote:
Hi Nicolas,
Unfortunately, I'm not aware of any specific issue here. But there are some
options regarding discarding messages when the queue exceeds a certain size
(look for DiscardMark and DiscardSeverity):
http://www.rsyslog.com/doc/v5-stable/configuration/action/index.html#action-queue-specific-configuration-statements
Maybe you can find a workaround that way (I assume you can discard
everything after you hit a certain limit).
Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Tue, Jun 30, 2015 at 12:32 PM, Nicolas Guyomar <
[email protected]
wrote:
Hi,
Upgrade to V8 is planned for this year, but in the meantime I thought I
could find a way to maybe discard messages when rsyslog has no more space
left in its work directory (which see.
Losing some messages during burst period is acceptable for us, but being
forced to manually delete and restart is more complicated.
I hoped the problem was some sort of misconfiguration on my side, or
maybe
a know issue using omrelp with logstash relp input.
On 30 June 2015 at 09:55, Radu Gheorghe <[email protected]>
wrote:
Hi Nicolas,
I have some vague memories about nasty bugs in disk-assisted queues
that
were fixed in the last few years. RELP modules surely have changed as
well.
Can you try with the latest stable (8.10 I think) and see if it helps?
Even
if it doesn't, I'm pretty sure the fix will come in the 8.x branch
because
it sounds pretty serious.
Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Tue, Jun 30, 2015 at 10:42 AM, Nicolas Guyomar <
[email protected]
wrote:
Hi All,
I've got a simple question on disk assisted queue behaviour, it could
be
trivial, but I can't find an answer on the internet.
I'm using rsyslog V5 to forward nginx access log to some logstash
instances
using omrelp
Sometimes, because of activity burst, rsyslog flush onto disk 200 1Mo
files
(which is the expected behaviour), but then stays stuck, no more
messages
are sent to logstash.
I have to delete state files as well as queue files so that rsyslog
start
sending messages to logstash again.
Restarting rsyslog without deleting those files has no effect.
Here is my rsyslog config in case I missed something
$RuleSet nginxRuleSet
$RulesetCreateMainQueue on
$WorkDirectory /tmp
$ActionQueueFileName queue-nginx
$ActionQueueMaxDiskSpace 200m
$ActionQueueSaveOnShutdown on
$ActionQueueType LinkedList
$ActionQueueSize 540000
$ActionResumeRetryCount -1
*.* :omrelp:<%= @lblog %>:5001;erableTmpl
& ~
$InputUDPServerBindRuleset nginxRuleSet
$UDPServerRun 515
Is it a known behaviour ?
Thank you for any help one could provide
Nicolas
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
