Ok, at least we know we have a new issue. Can you please provide me the complete debug log. You can mail it to me privately (do NOT mail to the list except if you want to broadcast the contents to the world ;)).
Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Andreas Grosse > Sent: Thursday, September 08, 2011 2:32 PM > To: rsyslog-users > Subject: Re: [rsyslog] netstream errors when logging with a high message > rate > > Hi Rainer, > thanks for the quick response. I updated to the latest version 5.8.5 and still > have the same error messages. Two examples for this happening can be > seen here: > > Error message on console/logfile: > rsyslogd: netstream session 0x4100cab0 will be closed due to error > > Debug output of the thread managing this netstream when the error > occured: > 4442.720793561:40976b70: netstream 0x4100cab0 with new data > 4442.720833000:40976b70: error during recv on NSD 0x413d1ae8: Connection > reset by peer > 4442.720840716:40976b70: gtlsRcv return. nsd 0x41079868, iRet -2165, > lenRcvBuf 0, ptrRcvBuf 0 > 4442.720848795:40976b70: Called LogError, msg: netstream session > 0x4100cab0 will be closed due to error > > 4442.720863318:40976b70: MsgSetTAG in: len 14, pszBuf: rsyslogd-2165: > 4442.720869899:40976b70: MsgSetTAG exit: pMsg->iLenTAG 14, pMsg- > >TAG.szBuf: rsyslogd-2165: > 4442.720884625:40976b70: main Q: entry added, size now log 6340, phys 6372 > entries > 4442.720893681:40976b70: main Q: EnqueueMsg advised worker start > 4442.720920645:40976b70: --------<NSDSEL_PTCP> calling select, active fds > (max 19): 14 15 19 > > Second occurence, different netstream: > rsyslogd: netstream session 0x8640fc0 will be closed due to error > > 4438.033367497:40976b70: main Q: entry added, size now log 6809, phys 6841 > entries > 4438.033388122:40976b70: main Q: entry added, size now log 6810, phys 6842 > entries > 4438.033396264:40976b70: main Q: entry added, size now log 6811, phys 6843 > entries > 4438.033404087:40976b70: main Q: entry added, size now log 6812, phys 6844 > entries > 4438.033412176:40976b70: main Q: entry added, size now log 6813, phys 6845 > entries > 4438.033420108:40976b70: main Q: entry added, size now log 6814, phys 6846 > entries > 4438.033427975:40976b70: main Q: entry added, size now log 6815, phys 6847 > entries > 4438.033436020:40976b70: main Q: entry added, size now log 6816, phys 6848 > entries > 4438.033443886:40976b70: main Q: entry added, size now log 6817, phys 6849 > entries > 4438.033451823:40976b70: main Q: entry added, size now log 6818, phys 6850 > entries > 4438.033459819:40976b70: main Q: entry added, size now log 6819, phys 6851 > entries > 4438.033467757:40976b70: main Q: entry added, size now log 6820, phys 6852 > entries > 4438.033475728:40976b70: main Q: entry added, size now log 6821, phys 6853 > entries > 4438.033483613:40976b70: main Q: entry added, size now log 6822, phys 6854 > entries > 4438.033491751:40976b70: main Q: entry added, size now log 6823, phys 6855 > entries > 4438.033499760:40976b70: main Q: entry added, size now log 6824, phys 6856 > entries > 4438.033507589:40976b70: main Q: entry added, size now log 6825, phys 6857 > entries > 4438.033515679:40976b70: main Q: entry added, size now log 6826, phys 6858 > entries > 4438.033523711:40976b70: main Q: entry added, size now log 6827, phys 6859 > entries > 4438.033531526:40976b70: main Q: entry added, size now log 6828, phys 6860 > entries > 4438.033539457:40976b70: main Q: entry added, size now log 6829, phys 6861 > entries > 4438.033547373:40976b70: main Q: entry added, size now log 6830, phys 6862 > entries > 4438.033555150:40976b70: main Q: entry added, size now log 6831, phys 6863 > entries > 4438.033563201:40976b70: main Q: entry added, size now log 6832, phys 6864 > entries > 4438.033571121:40976b70: main Q: entry added, size now log 6833, phys 6865 > entries > 4438.033578976:40976b70: main Q: entry added, size now log 6834, phys 6866 > entries > 4438.033586987:40976b70: main Q: entry added, size now log 6835, phys 6867 > entries > 4438.033594920:40976b70: main Q: entry added, size now log 6836, phys 6868 > entries > 4438.033602860:40976b70: main Q: entry added, size now log 6837, phys 6869 > entries > 4438.033610750:40976b70: main Q: entry added, size now log 6838, phys 6870 > entries > 4438.033618581:40976b70: main Q: entry added, size now log 6839, phys 6871 > entries > 4438.033626573:40976b70: main Q: entry added, size now log 6840, phys 6872 > entries > 4438.033635275:40976b70: main Q: entry added, size now log 6841, phys 6873 > entries > 4438.033644155:40976b70: main Q: entry added, size now log 6842, phys 6874 > entries > 4438.033652187:40976b70: main Q: entry added, size now log 6843, phys 6875 > entries > 4438.033660012:40976b70: main Q: entry added, size now log 6844, phys 6876 > entries > 4438.033667977:40976b70: main Q: entry added, size now log 6845, phys 6877 > entries > 4438.033675943:40976b70: main Q: entry added, size now log 6846, phys 6878 > entries > 4438.033683820:40976b70: main Q: entry added, size now log 6847, phys 6879 > entries > 4438.033691877:40976b70: main Q: entry added, size now log 6848, phys 6880 > entries > 4438.033699825:40976b70: main Q: entry added, size now log 6849, phys 6881 > entries > 4438.033707743:40976b70: main Q: entry added, size now log 6850, phys 6882 > entries > 4438.033716401:40976b70: main Q: MultiEnqObj advised worker start > 4438.033735637:40976b70: --------<NSDSEL_PTCP> calling select, active fds > (max 19): 14 15 16 18 19 > 4438.033772666:40976b70: netstream 0x8640fc0 with new data > 4438.033792922:40976b70: error during recv on NSD 0x8612ca0: Connection > reset by peer > 4438.033800723:40976b70: gtlsRcv return. nsd 0x863fd58, iRet -2165, > lenRcvBuf 0, ptrRcvBuf 0 > 4438.033808563:40976b70: Called LogError, msg: netstream session 0x8640fc0 > will be closed due to error > > 4438.033821043:40976b70: MsgSetTAG in: len 14, pszBuf: rsyslogd-2165: > 4438.033827708:40976b70: MsgSetTAG exit: pMsg->iLenTAG 14, pMsg- > >TAG.szBuf: rsyslogd-2165: > 4438.033850214:40976b70: main Q: entry added, size now log 6851, phys 6883 > entries > 4438.033859170:40976b70: main Q: EnqueueMsg advised worker start > 4438.033881307:40976b70: --------<NSDSEL_PTCP> calling select, active fds > (max 19): 14 15 18 19 > > Because of the gtlsRcv I assume this happens on the outgoing connection > since this is the only encrypted communication channel, but I am not sure > even this assumption is correct. > If you have anything you want me to provide or try, I'd be happy to do so. > Many thanks in advance! > Best regards, > Andreas > > > Rainer Gerhards <[email protected]> [08:09:11 13:11] wrote: > > You should update to the latest stable version and see if the problem > > persists. > > > > Rainer > > > -----Original Message----- > > > From: [email protected] [mailto:rsyslog- > > > [email protected]] On Behalf Of Andreas Grosse > > > Sent: Thursday, September 08, 2011 12:11 PM > > > To: [email protected] > > > Subject: [rsyslog] netstream errors when logging with a high message > > > rate > > > > > > Hi, > > > I am running rsyslog 5.8.3 as a central log collector which then > > > sends the log messages to an archive using tls encryption. When the > > > log message rate increases, I start seeing log messages like this: > > > > > > rsyslog: netstream session 0x8b05ef0 will be closed due to error > > > [try > > > http://www.rsyslog.com/e/2165 ] > > > > > > The logging does not stop and the tcp connection to the remote > > > archive does not break, though - it just starts spewing out these > > > messages up to two times per minute. > > > I enabled additional debugging, and the logfile contained this: > > > > > > 9275.065615635:40976b70: netstream 0x41000c50 with new data > > > 9275.065635173:40976b70: error during recv on NSD 0x41000b88: > > > Connection reset by peer > > > 9275.065641798:40976b70: gtlsRcv return. nsd 0x413fff98, iRet -2165, > > > lenRcvBuf 0, ptrRcvBuf 0 > > > 9275.065648064:40976b70: Called LogError, msg: netstream session > > > 0x41000c50 will be closed due to error > > > > > > 9275.065675043:40976b70: main Q: entry added, size now log 6919, > > > phys > > > 6951 entries > > > 9275.065682225:40976b70: main Q: EnqueueMsg advised worker start > > > 9275.065707944:40976b70: --------<NSDSEL_PTCP> calling select, > > > active fds (max 19): 14 15 16 19 > > > > > > On the receiving end I get no notification of an error happening at > > > all. Following is the configuration the I use: > > > > > > $MaxMessageSize 64k > > > $RepeatedMsgReduction off > > > $EscapeControlCharactersOnReceive off > > > $WorkDirectory /var/rsyslog # default location for work (spool) files > > > > > > $ModLoad imtcp > > > $ModLoad imudp > > > $ModLoad imptcp > > > $ModLoad omuxsock > > > $ModLoad impstats > > > > > > $InputPTCPServerListenIP 127.0.0.1 > > > $InputPTCPServerRun 10100 > > > > > > $PStatsInterval 300 > > > > > > # log local syslog messages back to syslog-ng $OMUxSockSocket > > > /dev/tosyslog if $programname startswith 'rsyslog' then :omuxsock: > > > if $programname startswith 'rsyslog' then ~ > > > > > > $ActionQueueType LinkedList > > > $ActionQueueFileName srvrfwd > > > $ActionResumeRetryCount -1 > > > $ActionQueueSaveOnShutdown on > > > $ActionQueueMaxDiskSpace 819200 > > > > > > $DefaultNetstreamDriver gtls > > > > > > $DefaultNetstreamDriverCAFile /etc/ca/cacert.pem > > > $DefaultNetstreamDriverCertFile /etc/client.pem > > > $DefaultNetstreamDriverKeyFile /etc/client.key > > > > > > $ActionSendStreamDriverMode 1 > > > $ActionSendStreamDriverAuthMode x509/certvalid > > > > > > $InputTCPServerStreamDriverMode 0 > > > $InputTCPServerRun 10101 > > > $UDPServerRun 10101 > > > > > > *.* @@(o,z0)loghost:5077;RSYSLOG_SyslogProtocol23Format > > > > > > > > > If you need more data from the debug log, just ask. This is also > > > easy to reproduce, therefore I am able to try some things if you > > > come up with suggestions what happens there and how to get rid of > > > those error messages. Thank you for your help! > > > > > > Best regards, > > > Andreas Grosse > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
