On Sat, 28 Sep 2013, Rainer Gerhards wrote:
On Sat, Sep 28, 2013 at 6:15 AM, David Lang <[email protected]> wrote:
I've got a box that's getting a fair number of errors on sending UDP
messages. I don't understand how that can generate failures.
this is a pretty simple config, local logs get some metadata added to them
and sent off, logs from remote systems are forwarded to a list of
destiantions. no tests other than the one fromhost list.
mmhhh.. The only idea I currently have is that something actually goes
wrong during the sendto() call. An other option would be failing name
resultion, but given it is using IP addresses...
I suggest running rsyslog in debug mode. omfwd emits error messages to the
debug log if something goes wrong (but does not do so in regular mode, as
this could lead to an enourmous amount of messages and even a DoS loop if
they are tried to being forwarded).
is there any way to just put the one module in debug mode? this is a fairly high
throughput system and the errors are rare enough that it will probably take a
while to hit them.
2013-09-25T09:35:54.854363-07:00 oprdelica300 rsyslogd-pstats: @cee:
{"name":"main
Q","size":12,"enqueued":730769821,"full":0,"discarded.full":0,"discarded.nf":0,"maxqsize":12848}
2013-09-25T09:40:54.937589-07:00 oprdelica300 rsyslogd-pstats: @cee:
{"name":"main
Q","size":20,"enqueued":732940254,"full":0,"discarded.full":0,"discarded.nf":0,"maxqsize":12848}
5 min, 2m logs, nowhere close to a peak time (and no failures during this
time, so it's got to be load related)
David Lang
Rainer
first the pstats output:
2013-09-25T09:40:54.937548-07:**00 oprdelica300 rsyslogd-pstats: @cee:
{"name":"imuxsock","submitted"**:22321,"ratelimit.discarded":**
0,"ratelimit.numratelimiters":**454}
2013-09-25T09:40:54.937557-07:**00 oprdelica300 rsyslogd-pstats: @cee:
{"name":"action 1","processed":732940242,"**failed":0}
2013-09-25T09:40:54.937560-07:**00 oprdelica300 rsyslogd-pstats: @cee:
{"name":"action 2","processed":29017,"failed":**0}
2013-09-25T09:40:54.937563-07:**00 oprdelica300 rsyslogd-pstats: @cee:
{"name":"action 3","processed":29017,"failed":**0}
2013-09-25T09:40:54.937565-07:**00 oprdelica300 rsyslogd-pstats: @cee:
{"name":"action 4","processed":732911217,"**failed":576051}
2013-09-25T09:40:54.937567-07:**00 oprdelica300 rsyslogd-pstats: @cee:
{"name":"action 5","processed":732911217,"**failed":576051}
2013-09-25T09:40:54.937569-07:**00 oprdelica300 rsyslogd-pstats: @cee:
{"name":"action 6","processed":732911217,"**failed":918875}
2013-09-25T09:40:54.937571-07:**00 oprdelica300 rsyslogd-pstats: @cee:
{"name":"action 7","processed":732911217,"**failed":716816}
2013-09-25T09:40:54.937573-07:**00 oprdelica300 rsyslogd-pstats: @cee:
{"name":"action 8","processed":732911217,"**failed":588857}
2013-09-25T09:40:54.937576-07:**00 oprdelica300 rsyslogd-pstats: @cee:
{"name":"imtcp(514)","**submitted":166158}
2013-09-25T09:40:54.937579-07:**00 oprdelica300 rsyslogd-pstats: @cee:
{"name":"imudp(*:514)","**submitted":732745067}
2013-09-25T09:40:54.937585-07:**00 oprdelica300 rsyslogd-pstats: @cee:
{"name":"resource-usage","**utime":20429131299,"stime":**
32610228491,"maxrss":62380,"**minflt":248393,"majflt":0,"**
inblock":0,"oublock":35056,"**nvcsw":899367593,"nivcsw":**28274351}
2013-09-25T09:40:54.937589-07:**00 oprdelica300 rsyslogd-pstats: @cee:
{"name":"main Q","size":20,"enqueued":**732940254,"full":0,"discarded.**
full":0,"discarded.nf":0,"**maxqsize":12848}
now the config (IP addresses changed
module(load="impstats" interval="300" facility="5" severity="6"
format="cee" log.syslog="on" )
$MaxMessageSize 8192
$MainMsgQueueSize 500000
module(load="imuxsock" SysSock.Annotate="on" SysSock.ParseTrusted="on")
module(load="imklog")
module(load="imtcp" MaxSessions="1000")
input(type="imtcp" port="514")
module(load="imudp" TimeRequery="100" SchedulingPolicy="fifo"
SchedulingPriority="3" batchSize="20")
input(type="imudp" port="514")
module(load="mmjsonparse")
action(type="mmjsonparse")
if $fromhost-ip == "127.0.0.1" then {
set $!trusted!origserver = $fromhost-ip;
set $!trusted!edge!time = $timegenerated;
set $!trusted!edge!relay = $$myhostname;
set $!trusted!edge!input = $inputname;
#if this is a local log, send it to an edge relay.
/var/log/local-messages
@10.1.0.1
stop
}
$template std,"%timereported% %hostname% %syslogtag%%$!msg%\n"
#/var/log/messages;std
@10.15.1.85
@10.15.1.86
@10.15.1.97
@10.15.1.98
action(type="omfwd" Target="10.15.1.40" Port="514" Protocol="udp"
RebindInterval="1000")
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
