Hey all!
While trying to setup queueing like we want it to I wonder if we've
found a bug or most likely doing something wrong. We've set limits very
low just to make sure we fill the queue fast for testing. We're running
8.24.0-1 from Debian 9 Stretch and this is our config with comments
removed:
module(load="imuxsock"
SysSock.RateLimit.Interval="0"
)
module(load="imklog")
module(load="builtin:omfile"
dirCreateMode="0755"
fileCreateMode="0644"
fileGroup="adm"
fileOwner="root"
)
global(
workDirectory="/var/spool/rsyslog"
MaxMessageSize="64k"
preserveFQDN="on"
action.reportSuspensionContinuation="on"
)
if $fromhost-ip == "127.0.0.1" then {
action(
type="omfile"
name="omfile-/var/log/messages"
File="/var/log/messages"
)
}
module(load="imtcp")
module(load="imudp")
module(load="impstats"
format="json-elasticsearch"
interval="5"
)
module(load="omprog")
if $fromhost-ip == "127.0.0.1" and $programname == "rsyslogd-pstats" then {
action(
name="action-omprog-impstats"
type="omprog"
binary="/local/nrpe/libexec/check_rsyslogd.pl --write"
)
}
input(type="imtcp"
port="514"
)
action(type="omfwd"
name="om-logstash"
target="127.0.0.1"
port="2514"
protocol="tcp"
template="RSYSLOG_ForwardFormat"
action.resumeRetryCount="1"
action.resumeInterval="5"
resendlastmsgonreconnect="on"
queue.timeoutEnqueue="2000"
queue.spoolDirectory="/local/rsyslog/spool"
queue.saveonshutdown="on"
queue.type="LinkedList"
queue.filename="om_logstash"
queue.maxfilesize="10M"
queue.maxdiskspace="20M"
queue.checkpointinterval="4000"
queue.size="2048"
)
Nothing is listening on TCP/2514 so the queues fill up really quick:
# du -sch /local/rsyslog/spool/*
9.6M /local/rsyslog/spool/om_logstash.00000069
9.6M /local/rsyslog/spool/om_logstash.00000070
4.0K /local/rsyslog/spool/om_logstash.qi
20M total
(Logs have been trimmed for space saving):
2018-11-28T11:48:13.570023+01:00 { "name": "omfile-\/var\/log\/messages", "origin": "core.action", "processed": 63,
"failed": 0, "suspended": 0, "suspended!duration": 0, "resumed": 0 }
2018-11-28T11:48:13.570066+01:00 { "name": "om-logstash", "origin": "core.action", "processed": 494967,
"failed": 0, "suspended": 2, "suspended!duration": 25, "resumed": 0 }
2018-11-28T11:48:13.570073+01:00 { "name": "imtcp(514)", "origin": "imtcp",
"submitted": 513265 }
2018-11-28T11:48:13.570140+01:00 { "name": "om-logstash queue[DA]", "origin": "core.queue", "size": 6434, "enqueued":
12858, "full": 11, "discarded!full": 8, "discarded!nf": 0, "maxqsize": 6434 }
2018-11-28T11:48:13.570155+01:00 { "name": "om-logstash queue", "origin": "core.queue", "size": 2048, "enqueued":
494967, "full": 18, "discarded!full": 10, "discarded!nf": 0, "maxqsize": 2048 }
2018-11-28T11:48:13.570168+01:00 { "name": "main Q", "origin": "core.queue", "size": 18376, "enqueued": 513339,
"full": 0, "discarded!full": 0, "discarded!nf": 0, "maxqsize": 70020 }
2018-11-28T11:48:18.575464+01:00 { "name": "omfile-\/var\/log\/messages", "origin": "core.action", "processed": 63,
"failed": 0, "suspended": 0, "suspended!duration": 0, "resumed": 0 }
2018-11-28T11:48:18.575507+01:00 { "name": "om-logstash", "origin": "core.action", "processed": 494972,
"failed": 0, "suspended": 2, "suspended!duration": 25, "resumed": 0 }
2018-11-28T11:48:18.575544+01:00 { "name": "imtcp(514)", "origin": "imtcp",
"submitted": 564894 }
2018-11-28T11:48:18.575615+01:00 { "name": "om-logstash queue[DA]", "origin": "core.queue", "size": 6434, "enqueued":
12860, "full": 13, "discarded!full": 10, "discarded!nf": 0, "maxqsize": 6434 }
2018-11-28T11:48:18.575641+01:00 { "name": "om-logstash queue", "origin": "core.queue", "size": 2048, "enqueued":
494972, "full": 23, "discarded!full": 14, "discarded!nf": 0, "maxqsize": 2048 }
2018-11-28T11:48:18.575659+01:00 { "name": "main Q", "origin": "core.queue", "size": 70012, "enqueued": 564981,
"full": 0, "discarded!full": 0, "discarded!nf": 0, "maxqsize": 70020 }
2018-11-28T11:48:20.028339+01:00 : action 'om-logstash' suspended, next retry
is Wed Nov 28 11:48:25 2018 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
2018-11-28T11:48:21.116309+01:00 : action 'om-logstash' suspended, next retry
is Wed Nov 28 11:48:26 2018 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
But here something happens
# du -sch /local/rsyslog/spool/*
9.6M /local/rsyslog/spool/om_logstash.00000070
104K /local/rsyslog/spool/om_logstash.00000071
4.0K /local/rsyslog/spool/om_logstash.qi
9.7M total
Looks like the queue is emptied because the size of "om-logstash
queue[DA]" is 0.
2018-11-28T11:48:23.580929+01:00 { "name": "omfile-\/var\/log\/messages", "origin": "core.action", "processed": 92,
"failed": 0, "suspended": 0, "suspended!duration": 0, "resumed": 0 }
2018-11-28T11:48:23.580957+01:00 { "name": "om-logstash", "origin": "core.action", "processed": 671188,
"failed": 0, "suspended": 2, "suspended!duration": 35, "resumed": 0 }
2018-11-28T11:48:23.580965+01:00 { "name": "imtcp(514)", "origin": "imtcp",
"submitted": 671096 }
2018-11-28T11:48:23.581049+01:00 { "name": "om-logstash queue[DA]", "origin": "core.queue", "size": 0, "enqueued":
12928, "full": 14, "discarded!full": 11, "discarded!nf": 0, "maxqsize": 6477 }
2018-11-28T11:48:23.581080+01:00 { "name": "om-logstash queue", "origin": "core.queue", "size": 0, "enqueued": 671195,
"full": 25, "discarded!full": 15, "discarded!nf": 0, "maxqsize": 2048 }
2018-11-28T11:48:23.581106+01:00 { "name": "main Q", "origin": "core.queue", "size": 18, "enqueued": 671213,
"full": 0, "discarded!full": 0, "discarded!nf": 0, "maxqsize": 70020 }
2018-11-28T11:48:28.587074+01:00 { "name": "omfile-\/var\/log\/messages", "origin": "core.action", "processed": 104,
"failed": 0, "suspended": 0, "suspended!duration": 0, "resumed": 0 }
2018-11-28T11:48:28.587108+01:00 { "name": "om-logstash", "origin": "core.action", "processed": 758684,
"failed": 0, "suspended": 2, "suspended!duration": 35, "resumed": 0 }
2018-11-28T11:48:28.587116+01:00 { "name": "imtcp(514)", "origin": "imtcp",
"submitted": 828582 }
2018-11-28T11:48:28.587167+01:00 { "name": "om-logstash queue[DA]", "origin": "core.queue", "size": 12103, "enqueued":
25033, "full": 16, "discarded!full": 12, "discarded!nf": 0, "maxqsize": 12103 }
2018-11-28T11:48:28.587182+01:00 { "name": "om-logstash queue", "origin": "core.queue", "size": 2048, "enqueued":
758684, "full": 28, "discarded!full": 16, "discarded!nf": 0, "maxqsize": 2048 }
2018-11-28T11:48:28.587195+01:00 { "name": "main Q", "origin": "core.queue", "size": 70011, "enqueued": 828694,
"full": 0, "discarded!full": 0, "discarded!nf": 0, "maxqsize": 70020 }
I'm not sure what we've done wrong/missunderstood or if this really is a
bug. Why would rsyslog empty the queue?
We're trying to solve the problem of full queues, which will happen, and
what we want is for the rsyslog infront of logstash to start dropping
incoming messages not to push back to our central logserver.
This is our setup:
server 1 -> TCP -> central syslog server ( imtcp -> Action Queue ->
Action DA Queue -> omfwd TCP ) -> event server ( imtcp -> Action Queue
-> Action DA Queue -> omfwd TCP ) -> Logstash -> Elasticsearch.
On "central syslog server" and "event server" we want imtcp to just drop
the logs when one action queue is full. Is this possible at all or are
we trying to do something impossible?
BR,
- Simon
____________________________________
Simon Lundström
Section for Infrastructure
IT Services
Stockholm University
SE-106 91 Stockholm, Sweden
www.su.se/english/staff-info/it
https://www.su.se/english/about-this-website/privacy-policy
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
