Rainer, I switched to 5.7.10 - and the problem was gone. No loss of log data any longer. Thanks for your hint. Another observation: The CPU load has decreased in 5.7.10 compared to 5.6.2. I just read Piavlo's posting about "high cpu spkies by rsyslog which barely does anything", who is on CentOS 5.5, too. May be it's an option for him to check 5.7.10?
Regards, Roland > -----Ursprüngliche Nachricht----- > Von: [email protected] [mailto:rsyslog- > [email protected]] Im Auftrag von Rainer Gerhards > Gesendet: Dienstag, 5. April 2011 15:51 > An: rsyslog-users > Betreff: Re: [rsyslog] Missing log entries from PostgreSQL backend > > > -----Original Message----- > > From: [email protected] [mailto:rsyslog- > > [email protected]] On Behalf Of Roland Kamke > > Sent: Tuesday, April 05, 2011 3:34 PM > > To: [email protected] > > Subject: [rsyslog] Missing log entries from PostgreSQL backend > > > > A newbie question: > > > > With a simple C program I create 8000 syslog messages to rsyslog and > wait > > then a minute before another 8000 messages will be created. This is the > code > > to create the messages, counter is 8000: > > ------- snip ------- > > for (iy=0; iy<counter; iy++) > > syslog(LOG_LOCAL3 | LOG_INFO, "C test program %d%s, %d", > > getpid(), count % 3 == 0 ? " (magic string)" : "", count++); > > ------- snap ------- > > > > This is a small snippet from the configuration: > > ------- snip ------- > > *.info;mail.none;authpriv.none;cron.none > > /var/log/messages > > > > if $msg contains 'C test program' then > > :ompgsql:localhost,secret,secret,secret; > > ------- snap ------- > > After the first 8000 messages are created I would expect to find 8000 > lines > in > > the log file and 8000 entries in the PostgreSQL database for these > messages. > > > > But: I clean up file system and data base, start rsyslogd and my test > program. > > After a short while I can find all messages in the log file, but only > 7984 > in the > > data base. This behaviour can be reproduced at will, in any attempt 16 > lines > > were missing. > > That's interesting. I unfortunately have no postgres test environment at > the > moment, so I cannot test this myself. > > > > > Tracing the database I saw that there were only 7984 all inserts into > the > log > > table. > > > > I ran rsyslogd with -dn option set. The log file created contains these > > lines: > > ------- snip ------- > > 6009.255950000:414d6940: action 9 queue: entry added, size now log 60, > phys > > 60 entries > > 6009.255964000:414d6940: action 9 queue: entry added, size now log 61, > phys > > 61 entries > > 6009.255978000:414d6940: action 9 queue: entry added, size now log 62, > phys > > 62 entries > > 6009.255992000:414d6940: action 9 queue: entry added, size now log 63, > phys > > 63 entries > > 6009.256012000:414d6940: action 9 queue: entry added, size now log 64, > phys > > 64 entries > > 6009.643753000:414d6940: action 9 queue: entry added, size now log 49, > phys > > 65 entries > > 6009.643770000:414d6940: action 9 queue: entry added, size now log 50, > phys > > 66 entries > > 6009.643784000:414d6940: action 9 queue: entry added, size now log 51, > phys > > 67 entries > > 6009.643797000:414d6940: action 9 queue: entry added, size now log 52, > phys > > 68 entries > > 6009.643811000:414d6940: action 9 queue: entry added, size now log 53, > phys > > 69 entries > > 6009.643825000:414d6940: action 9 queue: entry added, size now log 54, > phys > > 70 entries > > 6009.643838000:414d6940: action 9 queue: entry added, size now log 55, > phys > > 71 entries > > 6009.643852000:414d6940: action 9 queue: entry added, size now log 56, > phys > > 72 entries > > 6009.643865000:414d6940: action 9 queue: entry added, size now log 57, > phys > > 73 entries > > 6009.643879000:414d6940: action 9 queue: entry added, size now log 58, > phys > > 74 entries > > 6009.643892000:414d6940: action 9 queue: entry added, size now log 59, > phys > > 75 entries > > ------- snap ------- > > Ooops... phys increases nicely (64 -> 65), but there is a drop in the > log > value > > (64 -> 49), with the difference between 65 and 49 being 16... > > That means a batch of messages has been processed. In that case, the > logical > (log) queue size is reduced, but the messages remain inside the physical > queue as some cleanup condition is not yet met. > > > > > The messages with the numbers 17..24 and 26..32 (iy in the code) are > > missing. > > > > I built rsyslog from the source rpm archive rsyslog-5.6.2-1.ep.src.rpm > using > > these steps: > > > > ./configure --prefix=/opt/rsyslog --enable-silent-rules > --enable-dependency- > > tracking --enable-shared --enable-regexp --enable-zlib --enable-gssapi- > krb5 > - > > -enable-pthreads --enable-klog --enable-inet --enable-debug --enable- > rtinst > > --enable-memcheck --enable-diagtools --enable-pgsql --enable-snmp -- > > enable-gnutls --enable-mail --enable-imfile --enable-imptcp --enable- > > omprog --enable-omudpspoof --enable-omstdout --enable-omruleset make > > make install > > > > Is there a way to get all messages in the database (as they appear in > the > log > > file)? > > There have been a couple of fixes recently. Even though I am not sure they > help with the issue you see, it would be best if you could re-try with > 5.7.10. > > If that doesn't help, a complete debug log would be useful. Also, can you > reproduce the problem if you only submit, say, 100 messages? If not, when > does it occur? > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
