I'm back as said previously :) Well, I did the test launching rsyslogd like this:
rsyslogd -4 -dn and also like this: rsyslogd -dn I stopped and started again rsyslogd several times like that. I let it run several minutes each time, redirecting all the debug output to a file. First observation is that with or without the -4 flag, rsyslogd always listen on IPV6 sockets and IPV4 sockets. Second strange observation is that, in debug mode, rsyslog never loose the socket and always listen on it. After several minutes each time I restarted rsyslogd, the result of the netstat -ann | grep 514 is below and always the same (yes in debug mode, I ran it to listen with only one IMRELP server on port 514, just to send it maximum traffic and overload the only one imrelp instance): tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN tcp 776 0 192.168.11.247:514 10.92.99.106:55737 ESTABLISHED tcp 43217 0 192.168.11.247:514 192.168.73.113:38545 ESTABLISHED tcp 953 0 192.168.11.247:514 10.92.99.120:43479 ESTABLISHED tcp 580 0 192.168.11.247:514 10.92.199.103:48163 ESTABLISHED tcp 29984 0 192.168.11.247:514 10.92.99.100:37820 ESTABLISHED tcp 32108 0 192.168.11.247:514 10.92.99.102:58746 ESTABLISHED tcp 18763 0 192.168.11.247:514 192.168.73.114:38080 ESTABLISHED tcp 293 0 192.168.11.247:514 10.92.99.104:53676 ESTABLISHED tcp 35392 0 192.168.11.247:514 10.92.99.101:35081 ESTABLISHED tcp 572 0 192.168.11.247:514 10.92.99.103:49504 ESTABLISHED tcp 137 0 192.168.11.247:514 10.92.199.101:37153 ESTABLISHED tcp 33995 0 192.168.11.247:514 10.92.199.102:46932 ESTABLISHED tcp 48976 0 192.168.11.247:514 192.168.73.112:52707 ESTABLISHED tcp 137 0 192.168.11.247:514 10.92.199.100:44631 ESTABLISHED tcp 0 0 :::514 :::* LISTEN I'm a litlle bit disapointed as in debug mode it is running like a charm. I could keep it running with the -dn flag in production if it never loose the socket... (just jocking :-D) Also doing a tcpdump on port 514 on the receiver never makes rsyslog using 100 or 140% of CPU. _CONCLUSION: IN DEBUG MODE I CANNOT REPRODUCE ALL ISSUES I'VE WITHOUT DEBUGGING MODE._ _FYI, here below is the /etc/rsyslog.conf file of any of my rsyslog SENDERS (I hope my boss will not kill me, but i don't see anything secret in this config file). At the bottom, you can see that we use a forward rule to store directly in the database of one rsyslog server because sender and receiver are in the same network, and you can see a second forward rule to send the traffic via omrelp to the second rsyslog server when it is not in the same network (this sender is in New York, we have the receiver in New York also, and another receiver in Geneva)_: # rsyslog v5 configuration file # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html #### MODULES #### $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $SystemLogRateLimitInterval 0 $ModLoad imklog.so # provides kernel logging support (previously done by rklogd) #$ModLoad immark.so # provides --MARK-- message capability # Provides UDP syslog reception #$ModLoad imudp.so #$UDPServerRun 514 # Provides TCP syslog reception #$ModLoad imtcp.so #$InputTCPServerRun 514 # Provides RELP syslog output $ModLoad omrelp # Provide PostgreSLQ syslog output $ModLoad ompgsql # Set maximum message size to 8 kbytes $MaxMessageSize 8192 #### GLOBAL DIRECTIVES #### # Use default timestamp format #$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $ActionFileDefaultTemplate RSYSLOG_FileFormat $ActionForwardDefaultTemplate RSYSLOG_FileFormat # File syncing capability is disabled by default. This feature is usually not required, # not useful and an extreme performance hit #$ActionFileEnableSync on # Include all config files in /etc/rsyslog.d/ $IncludeConfig /etc/rsyslog.d/*.conf #### RULES #### # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg :omusrmsg:* # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # ### begin forwarding rule ### # The statement between the begin ... end define a SINGLE forwarding # rule. They belong together, do NOT split them. If you create multiple # forwarding rules, duplicate the whole block! # Remote Logging (we use TCP for reliable delivery) # # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. #$WorkDirectory /var/spppl/rsyslog # where to place spool files #$ActionQueueFileName fwdRule1 # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinite retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host:514 # ### end of the forwarding rule ### # Added by NHN on 07/03/2012 # Send all to the log parser server via RELP or/and OMPGSQL if in the same network # Log maillog messages to the PostgreSQL database AND to /var/log/maillog # Postfix template $template mail_pgsql,"INSERT INTO SystemEvents(Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values (E'%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-rfc3339%', '%timegenerated:::date-rfc3339%', %iut%, '%syslogtag%')", SQL $WorkDirectory /var/spool/rsyslog # forwarding rule 1 $ActionQueueType LinkedList $ActionQueueFileName srvrfwdNADC $ActionQueueMaxDiskSpace 8G $ActionResumeRetryCount -1 $ActionResumeInterval 2 $ActionQueueSaveOnShutdown on #$ActionExecOnlyEveryNthTime 16 #Unusable with pgsql module $ActionExecOnlyEveryNthTimeTimeout 10 MAIL.* :OMPGSQL:10.92.99.105,MAILLOG2,MAILLOG,MAILLOG;MAIL_PGSQL #*.* :omrelp:(z9)10.92.99.105:514 ##*.* @@10.92.99.105:514 ##*.* @10.92.99.105:514 # end forwarding rule 1 # forwarding rule 2 $ActionQueueType LinkedList $ActionQueueFileName srvrfwdGVA $ActionQueueMaxDiskSpace 8G $ActionResumeRetryCount -1 $ActionResumeInterval 2 $ActionQueueSaveOnShutdown on #$ActionExecOnlyEveryNthTime 16 #Unusable with pgsql module $ActionExecOnlyEveryNthTimeTimeout 10 #mail.* :ompgsql:192.168.11.247,maillog2,maillog,maillog;mail_pgsql *.* :OMRELP:(Z9)192.168.11.247:514 #*.* @@192.168.11.247:514 ##*.* @192.168.11.247:514 # end forwarding rule 2 Best regards, Nicolas - United Nations International Computing Center Geneva Rainer Gerhards <[email protected]> a écrit : > On Tue, 2013-03-12 at 16:35 +0100, Nicolas HAHN wrote: >> Hello Rainer, >> >> We run it only with -d. >> I'll try to run it with -dn as you suggest, and I'll come back to >> the mailing list :) >> >> Then, I'll try after that to run it under valgrind as per your >> second suggestion. >> >> Also, I've seen that when I stop rsyslog using the famous well known >> command service rsyslog stop (version 7.2.6-2 now), it seems it's >> not properly closing the database connections because from time to >> time, I've this SQL error in the pg_log directory: >> >> LOG: unexpected EOF on client connection with an open transaction >> >> I should probably fill a bug request, no? >> > not sure - this could probably be related. IF something blocks the > output thread, and rsyslog is terminated, and the block remains for > longer than the timeout period you have configured, rsyslogd core > cancels the thread. That could lead to what you see and would be > perfectly fine (of course, except for the question why the output is > blocking, but that's probably a different story...). > > Rainer >> I come back to you at least with the output of rsyslog -dn >> >> Thanks a lot for your suggestions Rainer. >> >> Best regards, >> Nicolas >> - >> United Nations International Computing Center >> Geneva >> >> Rainer Gerhards <[email protected]> a écrit : >> >> > On Tue, 2013-03-12 at 14:57 +0100, Nicolas HAHN wrote: >> >> Hello David, >> >> >> >> As written at the top of my e-mail, I confirm SELINUX is disabled >> >> and there is no firewall rules preventing the traffic to come. >> >> Furthermore, it works like a charm on the current rsyslog production >> >> server we try to replace, and which is using rsyslog 4.8.x on RHEL 5. >> >> >> >> When syslog stops, nothing shows in the debug logs as like rsyslog >> >> "loose" some IMRELP servers, it also looses its debugging ability. >> >> That's what I wrote in a previous e-mail: it display tons of things >> >> on the screen when ran with the -d flag, then it stops displaying >> >> and give you back the hand on the shell. But we can still see >> >> rsyslog in the process table running with its -d flag... >> >> >> > Do you run it with just -d or -dn? Without -n (As in -dn), it >> > auto-backgrounds and this means you usually lose the debug information. >> > >> > Also, I would suggest to run it under valgrind control, as this can >> > bring up any misadressing problems. That would be along the lines of >> > >> > $ valgrind rsyslogd -dn ...other options.... >> > >> > Rainer >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> > POST if you DON'T LIKE THAT. >> > >> >> >> ---------------------------------------------------------------- >> This message was sent using IMP, the Internet Messaging Program. >> >> Hello Rainer, >> >> We run it only with -d. >> I'll try to run it with -dn as you suggest, and I'll come back to the >> mailing list :) >> >> Then, I'll try after that to run it under valgrind as per your second >> suggestion. >> >> Also, I've seen that when I stop rsyslog using the famous well known >> command service rsyslog stop (version 7.2.6-2 now), it seems it's not >> properly closing the database connections because from time to time, >> I've this SQL error in the pg_log directory: >> >> LOG: unexpected EOF on client connection with an open transaction >> >> I should probably fill a bug request, no? >> >> I come back to you at least with the output of rsyslog -dn >> >> Thanks a lot for your suggestions Rainer. >> >> Best regards, >> Nicolas >> - >> United Nations International Computing Center >> Geneva >> >> >> Rainer Gerhards <[email protected]> a écrit : >> >> > On Tue, 2013-03-12 at 14:57 +0100, Nicolas HAHN wrote: >> >> Hello David, >> >> >> >> As written at the top of my e-mail, I confirm SELINUX is disabled >> >> and there is no firewall rules preventing the traffic to come. >> >> Furthermore, it works like a charm on the current rsyslog >> production >> >> server we try to replace, and which is using rsyslog 4.8.x on RHEL >> 5. >> >> >> >> When syslog stops, nothing shows in the debug logs as like rsyslog >> >> "loose" some IMRELP servers, it also looses its debugging ability. >> >> That's what I wrote in a previous e-mail: it display tons of things >> >> on the screen when ran with the -d flag, then it stops displaying >> >> and give you back the hand on the shell. But we can still see >> >> rsyslog in the process table running with its -d flag... >> >> >> > Do you run it with just -d or -dn? Without -n (As in -dn), it >> > auto-backgrounds and this means you usually lose the debug >> information. >> > >> > Also, I would suggest to run it under valgrind control, as this can >> > bring up any misadressing problems. That would be along the lines of >> > >> > $ valgrind rsyslogd -dn ...other options.... >> > >> > Rainer >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> > POST if you DON'T LIKE THAT. >> > >> >> >> ---------------------------------------------------------------- >> This message was sent using IMP, the Internet Messaging Program. >> >> > > ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
