> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Roland Kamke > Sent: Thursday, April 07, 2011 8:23 AM > To: rsyslog-users > Subject: Re: [rsyslog] Missing log entries from PostgreSQL backend > > Rainer, I switched to 5.7.10 - and the problem was gone. No loss of log > data any longer. Thanks for your hint.
Excellent! > Another observation: The CPU load has decreased in 5.7.10 compared to > 5.6.2. I just read Piavlo's posting about "high cpu spkies by rsyslog > which barely does anything", who is on CentOS 5.5, too. May be it's an > option for him to check 5.7.10? That's a good idea. In general, newer version are faster than previous ones in the current series, because I have a focus on performance (even as a side-focus when the primary one is different). Rainer > > Regards, > Roland > > > -----Ursprüngliche Nachricht----- > > Von: [email protected] [mailto:rsyslog- > > [email protected]] Im Auftrag von Rainer Gerhards > > Gesendet: Dienstag, 5. April 2011 15:51 > > An: rsyslog-users > > Betreff: Re: [rsyslog] Missing log entries from PostgreSQL backend > > > > > -----Original Message----- > > > From: [email protected] [mailto:rsyslog- > > > [email protected]] On Behalf Of Roland Kamke > > > Sent: Tuesday, April 05, 2011 3:34 PM > > > To: [email protected] > > > Subject: [rsyslog] Missing log entries from PostgreSQL backend > > > > > > A newbie question: > > > > > > With a simple C program I create 8000 syslog messages to rsyslog > and > > wait > > > then a minute before another 8000 messages will be created. This is > the > > code > > > to create the messages, counter is 8000: > > > ------- snip ------- > > > for (iy=0; iy<counter; iy++) > > > syslog(LOG_LOCAL3 | LOG_INFO, "C test program %d%s, %d", > > > getpid(), count % 3 == 0 ? " (magic string)" : "", > count++); > > > ------- snap ------- > > > > > > This is a small snippet from the configuration: > > > ------- snip ------- > > > *.info;mail.none;authpriv.none;cron.none > > > /var/log/messages > > > > > > if $msg contains 'C test program' then > > > :ompgsql:localhost,secret,secret,secret; > > > ------- snap ------- > > > After the first 8000 messages are created I would expect to find > 8000 > > lines > > in > > > the log file and 8000 entries in the PostgreSQL database for these > > messages. > > > > > > But: I clean up file system and data base, start rsyslogd and my > test > > program. > > > After a short while I can find all messages in the log file, but > only > > 7984 > > in the > > > data base. This behaviour can be reproduced at will, in any attempt > 16 > > lines > > > were missing. > > > > That's interesting. I unfortunately have no postgres test environment > at > > the > > moment, so I cannot test this myself. > > > > > > > > Tracing the database I saw that there were only 7984 all inserts > into > > the > > log > > > table. > > > > > > I ran rsyslogd with -dn option set. The log file created contains > these > > > lines: > > > ------- snip ------- > > > 6009.255950000:414d6940: action 9 queue: entry added, size now log > 60, > > phys > > > 60 entries > > > 6009.255964000:414d6940: action 9 queue: entry added, size now log > 61, > > phys > > > 61 entries > > > 6009.255978000:414d6940: action 9 queue: entry added, size now log > 62, > > phys > > > 62 entries > > > 6009.255992000:414d6940: action 9 queue: entry added, size now log > 63, > > phys > > > 63 entries > > > 6009.256012000:414d6940: action 9 queue: entry added, size now log > 64, > > phys > > > 64 entries > > > 6009.643753000:414d6940: action 9 queue: entry added, size now log > 49, > > phys > > > 65 entries > > > 6009.643770000:414d6940: action 9 queue: entry added, size now log > 50, > > phys > > > 66 entries > > > 6009.643784000:414d6940: action 9 queue: entry added, size now log > 51, > > phys > > > 67 entries > > > 6009.643797000:414d6940: action 9 queue: entry added, size now log > 52, > > phys > > > 68 entries > > > 6009.643811000:414d6940: action 9 queue: entry added, size now log > 53, > > phys > > > 69 entries > > > 6009.643825000:414d6940: action 9 queue: entry added, size now log > 54, > > phys > > > 70 entries > > > 6009.643838000:414d6940: action 9 queue: entry added, size now log > 55, > > phys > > > 71 entries > > > 6009.643852000:414d6940: action 9 queue: entry added, size now log > 56, > > phys > > > 72 entries > > > 6009.643865000:414d6940: action 9 queue: entry added, size now log > 57, > > phys > > > 73 entries > > > 6009.643879000:414d6940: action 9 queue: entry added, size now log > 58, > > phys > > > 74 entries > > > 6009.643892000:414d6940: action 9 queue: entry added, size now log > 59, > > phys > > > 75 entries > > > ------- snap ------- > > > Ooops... phys increases nicely (64 -> 65), but there is a drop in > the > > log > > value > > > (64 -> 49), with the difference between 65 and 49 being 16... > > > > That means a batch of messages has been processed. In that case, the > > logical > > (log) queue size is reduced, but the messages remain inside the > physical > > queue as some cleanup condition is not yet met. > > > > > > > > The messages with the numbers 17..24 and 26..32 (iy in the code) > are > > > missing. > > > > > > I built rsyslog from the source rpm archive rsyslog-5.6.2- > 1.ep.src.rpm > > using > > > these steps: > > > > > > ./configure --prefix=/opt/rsyslog --enable-silent-rules > > --enable-dependency- > > > tracking --enable-shared --enable-regexp --enable-zlib --enable- > gssapi- > > krb5 > > - > > > -enable-pthreads --enable-klog --enable-inet --enable-debug -- > enable- > > rtinst > > > --enable-memcheck --enable-diagtools --enable-pgsql --enable-snmp - > - > > > enable-gnutls --enable-mail --enable-imfile --enable-imptcp -- > enable- > > > omprog --enable-omudpspoof --enable-omstdout --enable-omruleset > make > > > make install > > > > > > Is there a way to get all messages in the database (as they appear > in > > the > > log > > > file)? > > > > There have been a couple of fixes recently. Even though I am not sure > they > > help with the issue you see, it would be best if you could re-try > with > > 5.7.10. > > > > If that doesn't help, a complete debug log would be useful. Also, can > you > > reproduce the problem if you only submit, say, 100 messages? If not, > when > > does it occur? > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
