Hi,
We recently upgraded from RHEL5 to RHEL7.1 (kernel
3.10.0-229.11.1.el7.x86_64) on new hardware for our rsyslog collector. Since
the upgrade to RHEL7.1, rsyslog is randomly losing log messages - most
messages are written to file but some are not. A tcpdump on RHEL7.1 shows
the messages are being received over the network, but the rsyslog software
never writes the data out to a file.
The old RHEL5 server is still online and I've been testing with a log source
that can send data to both servers at the same time. We've tried both TCP
and UDP and varying ports with different rulesets. The RHEL5 server logs all
the data as expected, while the new RHEL7.1 server fails to write out
messages randomly. Both servers are running the same version of rsyslog -
v8.8.0. I've tried up to v8.12 on the RHEL7.1 box with the same result. The
rsyslog configurations on both servers are exactly the same. Both servers
are on the same VLAN. tcpdump shows the same exact raw output for the
packets that go missing. The only variable between the two setups is the OS
(and volume - the old box is only collecting data from this one source while
the new one is processing some thousands msgs/sec. But the old box didn't
lose messages in this fashion when it previously had the same workload).
The log message loss occurs randomly. I have not been able to identify a
pattern or specific event that causes the logging to stop or start again.
I've also confirmed that we're losing log messages from at least two log
sources, which means we are likely missing data from others.
I've tried setting $DynaFileCacheSize to 150, changing the number of worker
threads for the ruleset, and modifying rcvbufSize for UDP. I've also
performed to following kernel tuning to troubleshoot, to no effect:
net.core.netdev_max_backlog = 10000
net.core.rmem_default = 838608
net.core.rmem_max = 33554432
net.core.wmem_default = 838608
net.core.wmem_max = 33554432
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.udp_mem = 65536 131072 262144
net.ipv4.udp_rmem_min = 4096
Relevant parts of my syslog config below. Any ideas? I'm stuck.
$RepeatedMsgReduction off
$MaxMessageSize 128k
$EscapeControlCharactersOnReceive off
$DynaFileCacheSize 150
module(load="imudp" threads="2" batchSize="64" timeRequery="8"
schedulingPolicy="fifo" schedulingPriority="10")
module(load="imtcp" MaxSessions="10000")
ruleset(name="ruleset1" queue.type="LinkedList" queue.size="750000"
queue.workerthreads="4") {
# at very top of ruleset - IPs obscured for privacy
if $fromhost-ip == 'x.x.2.75'
or $fromhost-ip == 'x.x.2.85'
then {
/tmp/debug.log;RSYSLOG_DebugFormat
stop
}
#... other rules below
} # end ruleset
input(type="imudp" port="10516" ruleset="MailRuleSet" rcvbufSize="6m")
input(type="imtcp" port="10516" ruleset="MailRuleSet")
Dan Woodruff
University IT Security and Policy
University of Rochester
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
