Right after the queues filled up, it stoped sending logs to the second log server too.
2012/5/22 Juan Jose Pavlik <[email protected]> > More info: > > -The syslog server is running from a storage server > -At May 20 23:30:52 one of our biggest dom0 went down, killing about 20-25 > virtual servers. > -Most of our servers run OpenSuSE. > > > 2012/5/22 Juan Jose Pavlik <[email protected]> > >> Hi, im running this rsyslog version: >> >> bigbrother:/var/log/servidores/filomena # rsyslogd -v >> rsyslogd 5.8.5, compiled with: >> FEATURE_REGEXP: Yes >> FEATURE_LARGEFILE: No >> GSSAPI Kerberos 5 support: Yes >> FEATURE_DEBUG (debug build, slow code): No >> 32bit Atomic operations supported: Yes >> 64bit Atomic operations supported: No >> Runtime Instrumentation (slow code): No >> >> in my centralized logging server and im getting some troubles i'd really >> love to figure out. I've around 170 servers/switches/otherthings logging on >> this server, most of them just send auth.* logs, some apaches sending the >> access and error logs, and switches sending warns and errors. Sometimes the >> rsyslog queues get complettly filled up and it stops writing logs to disk, >> this is the exact logs of what happened: >> >> May 20 23:30:52 bigbrother rsyslogd-pstats: main Q: size=1 >> enqueued=6018511 full=0 maxqsize=1607 >> May 20 23:31:12 bigbrother rsyslogd-pstats: imuxsock: submitted=3897 >> ratelimit.discarded=0 ratelimit.numratelimiters=1505 >> May 20 23:31:12 bigbrother rsyslogd-pstats: main Q: size=83 >> enqueued=6018951 full=0 maxqsize=1607 >> May 20 23:31:32 bigbrother rsyslogd-pstats: imuxsock: submitted=3897 >> ratelimit.discarded=0 ratelimit.numratelimiters=1505 >> May 20 23:31:32 bigbrother rsyslogd-pstats: main Q: size=140 >> enqueued=6019008 full=0 maxqsize=1607 >> May 20 23:31:52 bigbrother rsyslogd-pstats: imuxsock: submitted=3897 >> ratelimit.discarded=0 ratelimit.numratelimiters=1505 >> May 20 23:31:52 bigbrother rsyslogd-pstats: main Q: size=146 >> enqueued=6019046 full=0 maxqsize=1607 >> May 20 23:32:12 bigbrother rsyslogd-pstats: imuxsock: submitted=3897 >> ratelimit.discarded=0 ratelimit.numratelimiters=1505 >> May 20 23:32:12 bigbrother rsyslogd-pstats: main Q: size=169 >> enqueued=6019101 full=0 maxqsize=1607 >> May 20 23:32:32 bigbrother rsyslogd-pstats: imuxsock: submitted=3897 >> ratelimit.discarded=0 ratelimit.numratelimiters=1505 >> May 20 23:32:32 bigbrother rsyslogd-pstats: main Q: size=333 >> enqueued=6019265 full=0 maxqsize=1607 >> May 20 23:32:52 bigbrother rsyslogd-pstats: imuxsock: submitted=3897 >> ratelimit.discarded=0 ratelimit.numratelimiters=1505 >> May 20 23:32:52 bigbrother rsyslogd-pstats: main Q: size=431 >> enqueued=6019395 full=0 maxqsize=1607 >> May 20 23:33:12 bigbrother rsyslogd-pstats: imuxsock: submitted=3897 >> ratelimit.discarded=0 ratelimit.numratelimiters=1505 >> May 20 23:33:12 bigbrother rsyslogd-pstats: main Q: size=469 >> enqueued=6019433 full=0 maxqsize=1607 >> May 20 23:33:32 bigbrother rsyslogd-pstats: imuxsock: submitted=3897 >> ratelimit.discarded=0 ratelimit.numratelimiters=1505 >> May 20 23:33:32 bigbrother rsyslogd-pstats: main Q: size=574 >> enqueued=6019538 full=0 maxqsize=1607 >> May 20 23:33:52 bigbrother rsyslogd-pstats: imuxsock: submitted=3897 >> ratelimit.discarded=0 ratelimit.numratelimiters=1505 >> May 20 23:33:52 bigbrother rsyslogd-pstats: main Q: size=603 >> enqueued=6019567 full=0 maxqsize=1607 >> May 20 23:34:12 bigbrother rsyslogd-pstats: imuxsock: submitted=3897 >> ratelimit.discarded=0 ratelimit.numratelimiters=1505 >> May 20 23:34:12 bigbrother rsyslogd-pstats: main Q: size=654 >> enqueued=6019650 full=0 maxqsize=1607 >> May 20 23:34:32 bigbrother rsyslogd-pstats: imuxsock: submitted=3897 >> ratelimit.discarded=0 ratelimit.numratelimiters=1505 >> May 20 23:34:32 bigbrother rsyslogd-pstats: main Q: >> *size=687*enqueued=6019683 full=0 maxqsize=1607 >> May 20 23:34:52 bigbrother rsyslogd-pstats: imuxsock: submitted=3897 >> ratelimit.discarded=0 ratelimit.numratelimiters=1505 >> May 20 23:34:52 bigbrother rsyslogd-pstats: main Q: >> *size=721*enqueued=6019717 full=0 maxqsize=1607 >> May 20 23:48:13 bigbrother rsyslogd-pstats: main Q: >> *size=5150*enqueued=6024786 full=0 maxqsize=5150 >> May 20 23:48:33 bigbrother rsyslogd-pstats: imuxsock: submitted=3897 >> ratelimit.discarded=0 ratelimit.numratelimiters=1505 >> May 21 00:43:21 bigbrother rsyslogd-pstats: imuxsock: submitted=3898 >> ratelimit.discarded=0 ratelimit.numratelimiters=1506 >> May 21 00:43:21 bigbrother rsyslogd-pstats: main Q: >> *size=9986*enqueued=6033582 full=1596 maxqsize=10000 >> May 21 00:51:20 bigbrother rsyslogd-pstats: main Q: >> *size=10000*enqueued=6034304 full=2086 maxqsize=10000 >> >> >> Once *size* reaches 10000 (the default max as far as i know) things get >> complicated, rsyslog starts to drop logs and misbehave. The rsyslog >> configuration write a per host files into /var/log/servidores/, it also >> sends some logs to another rsyslog server and a postgress database running >> in another server. 2 weeks ago, i disabled sending logs to the postgress >> databse, because i had this same problem and we lost too many hours of >> logs. Most of the servers are sending logs by TCP and a few servers and >> other devices use UDP. >> >> Is there a way i can avoid this problem? should i increase the mainqueue >> size? use other queues? Any help will be great. Thanks >> >> -- >> Pavlik Salles Juan José >> Prosecretaría de Informática - UNC >> Área Redes y Servidores >> >> > > > -- > Pavlik Salles Juan José > Prosecretaría de Informática - UNC > Área Redes y Servidores > > -- Pavlik Salles Juan José Prosecretaría de Informática - UNC Área Redes y Servidores _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards
