> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Roland Kamke > Sent: Tuesday, April 05, 2011 3:34 PM > To: [email protected] > Subject: [rsyslog] Missing log entries from PostgreSQL backend > > A newbie question: > > With a simple C program I create 8000 syslog messages to rsyslog and wait > then a minute before another 8000 messages will be created. This is the code > to create the messages, counter is 8000: > ------- snip ------- > for (iy=0; iy<counter; iy++) > syslog(LOG_LOCAL3 | LOG_INFO, "C test program %d%s, %d", > getpid(), count % 3 == 0 ? " (magic string)" : "", count++); > ------- snap ------- > > This is a small snippet from the configuration: > ------- snip ------- > *.info;mail.none;authpriv.none;cron.none > /var/log/messages > > if $msg contains 'C test program' then > :ompgsql:localhost,secret,secret,secret; > ------- snap ------- > After the first 8000 messages are created I would expect to find 8000 lines in > the log file and 8000 entries in the PostgreSQL database for these messages. > > But: I clean up file system and data base, start rsyslogd and my test program. > After a short while I can find all messages in the log file, but only 7984 in the > data base. This behaviour can be reproduced at will, in any attempt 16 lines > were missing.
That's interesting. I unfortunately have no postgres test environment at the moment, so I cannot test this myself. > > Tracing the database I saw that there were only 7984 all inserts into the log > table. > > I ran rsyslogd with -dn option set. The log file created contains these > lines: > ------- snip ------- > 6009.255950000:414d6940: action 9 queue: entry added, size now log 60, phys > 60 entries > 6009.255964000:414d6940: action 9 queue: entry added, size now log 61, phys > 61 entries > 6009.255978000:414d6940: action 9 queue: entry added, size now log 62, phys > 62 entries > 6009.255992000:414d6940: action 9 queue: entry added, size now log 63, phys > 63 entries > 6009.256012000:414d6940: action 9 queue: entry added, size now log 64, phys > 64 entries > 6009.643753000:414d6940: action 9 queue: entry added, size now log 49, phys > 65 entries > 6009.643770000:414d6940: action 9 queue: entry added, size now log 50, phys > 66 entries > 6009.643784000:414d6940: action 9 queue: entry added, size now log 51, phys > 67 entries > 6009.643797000:414d6940: action 9 queue: entry added, size now log 52, phys > 68 entries > 6009.643811000:414d6940: action 9 queue: entry added, size now log 53, phys > 69 entries > 6009.643825000:414d6940: action 9 queue: entry added, size now log 54, phys > 70 entries > 6009.643838000:414d6940: action 9 queue: entry added, size now log 55, phys > 71 entries > 6009.643852000:414d6940: action 9 queue: entry added, size now log 56, phys > 72 entries > 6009.643865000:414d6940: action 9 queue: entry added, size now log 57, phys > 73 entries > 6009.643879000:414d6940: action 9 queue: entry added, size now log 58, phys > 74 entries > 6009.643892000:414d6940: action 9 queue: entry added, size now log 59, phys > 75 entries > ------- snap ------- > Ooops... phys increases nicely (64 -> 65), but there is a drop in the log value > (64 -> 49), with the difference between 65 and 49 being 16... That means a batch of messages has been processed. In that case, the logical (log) queue size is reduced, but the messages remain inside the physical queue as some cleanup condition is not yet met. > > The messages with the numbers 17..24 and 26..32 (iy in the code) are > missing. > > I built rsyslog from the source rpm archive rsyslog-5.6.2-1.ep.src.rpm using > these steps: > > ./configure --prefix=/opt/rsyslog --enable-silent-rules --enable-dependency- > tracking --enable-shared --enable-regexp --enable-zlib --enable-gssapi-krb5 - > -enable-pthreads --enable-klog --enable-inet --enable-debug --enable-rtinst > --enable-memcheck --enable-diagtools --enable-pgsql --enable-snmp -- > enable-gnutls --enable-mail --enable-imfile --enable-imptcp --enable- > omprog --enable-omudpspoof --enable-omstdout --enable-omruleset make > make install > > Is there a way to get all messages in the database (as they appear in the log > file)? There have been a couple of fixes recently. Even though I am not sure they help with the issue you see, it would be best if you could re-try with 5.7.10. If that doesn't help, a complete debug log would be useful. Also, can you reproduce the problem if you only submit, say, 100 messages? If not, when does it occur? Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
