Hi David, The central servers logged many messages like,
<43>1 2018-05-14T17:35:49.117264+02:00 be-s0873-c2a rsyslogd-2079 - - - too many tcp sessions - dropping incoming request [try http://www.rsyslog.com/e/2079 ] I increased these values to : $InputTCPMaxSessions 2000 $InputTCPMaxListeners 2000 $MainMsgQueueSize 100000 The central syslog server has 22Gb RAM free and runs Solaris. I have some 700 boxes with a mixture of TCP and UDP logging, and each server logs around 10 messages a second. Are the above TCP settings appropriate, and are there other network related settings I can mess with? Thanks, Sophie. > -----Original Message----- > From: David Lang [mailto:[email protected]] > Sent: Thursday, May 17, 2018 7:43 PM > To: LOEWENTHAL Sophie > Cc: rsyslog-users > Subject: RE: [rsyslog] action 'action 0' resumed (module 'builtin:omfwd') > [v8.24.0 > try http://www.rsyslog.com/e/2359 ] > > On Thu, 17 May 2018, [email protected] wrote: > > > Thanks David for pointing this out. > > > > > > /etc/rsyslog.d/ contains three files > > # cat central_syslog.conf > > *.* @@(o)syslog1:514 > > *.* @@(o)syslog2:514 > > # cat central_syslog.conf.19042 > > *.* @syslog1 > > *.* @syslog2 > > # cat listen.conf > > $SystemLogSocketName /run/systemd/journal/syslog > > > > Does rsyslog read all files in rsyslog.d or only the *.conf files? Maybe the > central_syslog.conf.19042 old config file is tripping this up. > > it reads whatever is defined in the include statement. In this case the *.conf > files > > so action0 and action1 are defined in the central_syslog.conf file as being > reliabl delivery to syslog1 and syslog2, this means that if those destinations > can't keep up, it will pause (suspending delivery to those destinations) and > then when it can send messages again, it will resume delivery. > > So it looks like your central syslog servers aren't keeping up with the load, > or > there are network issues reaching them. > > the next step would be to start looking at what's happening on those central > syslog servers > > David Lang > > > > > > >> -----Original Message----- > >> From: David Lang [mailto:[email protected]] > >> Sent: Wednesday, May 16, 2018 10:22 PM > >> To: sophie.loewenthal--- via rsyslog > >> Cc: LOEWENTHAL Sophie > >> Subject: Re: [rsyslog] action 'action 0' resumed (module 'builtin:omfwd') > [v8.24.0 > >> try http://www.rsyslog.com/e/2359 ] > >> > >> The question is why action 0 is getting suspended > >> > >> what is in /etc/rsyslog.d/* those files are being included ahead of the > >> actions > >> in rsyslog.conf, so action 0 would be in an included file. > >> > >> David Lang > >> > >> On Wed, 16 May 2018, sophie.loewenthal--- via rsyslog wrote: > >> > >>> Date: Wed, 16 May 2018 08:30:08 +0000 > >>> From: sophie.loewenthal--- via rsyslog <[email protected]> > >>> To: sophie.loewenthal--- via rsyslog <[email protected]> > >>> Cc: "[email protected]" > >>> <[email protected]> > >>> Subject: [rsyslog] action 'action 0' resumed (module 'builtin:omfwd') > [v8.24.0 > >>> try http://www.rsyslog.com/e/2359 ] > >>> > >>> Hi everybody, > >>> > >>> Our RHEL 7 servers were patched over the weekend 7.4 to 7.5. Since then > I've > >> had these messages in the logs from rsyslog. Since then we have 38000 of > the > >> 'action' messages since 23 April. Although rsyslog was updated, this looks > like a > >> minor revision. Downgrading to rsyslog-8.24.0-12.el7.x86_64 from 8.24.0- > >> 16.el7.x86_64 suppressed the message. > >>> > >>> Rather than downgrade, I'd prefer to correct my configuration. > >>> > >>> Can anyone see what produced the message? What should I look at > changing? > >> If not, how may I suppress the message? > >>> > >>> Messages were: > >>> <46>1 2018-05-15T09:30:01+02:00 be-AAAA-11 rsyslogd - - - action 'action > 0' > >> resumed (module 'builtin:omfwd') [v8.24.0 try > http://www.rsyslog.com/e/2359 ] > >>> <46>1 2018-05-15T09:30:01+02:00 be-AAAA-11 rsyslogd - - - action 'action > 1' > >> resumed (module 'builtin:omfwd') [v8.24.0 try > http://www.rsyslog.com/e/2359 ] > >>> <46>1 2018-05-15T09:30:01+02:00 be-AAAA-11 rsyslogd - - - action 'action > 1' > >> resumed (module 'builtin:omfwd') [v8.24.0 try > http://www.rsyslog.com/e/2359 ] > >>> > >>> > >>> The /etc/rsyslog.conf is > >>> $ModLoad imuxsock > >>> $ModLoad imjournal > >>> $WorkDirectory /var/lib/rsyslog > >>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > >>> $IncludeConfig /etc/rsyslog.d/*.conf > >>> $OmitLocalLogging on > >>> $IMJournalStateFile imjournal.state > >>> *.info;mail.none;authpriv.none;cron.none /var/log/messages > >>> authpriv.* /var/log/secure > >>> mail.* -/var/log/maillog > >>> cron.* /var/log/cron > >>> *.emerg :omusrmsg:* > >>> uucp,news.crit /var/log/spooler > >>> local7.* /var/log/boot.log > >>> > >>> > >>> > >>> Best wishes, > >>> Sophie > >>> > >>> Team mailbox : [email protected] > >>> or direct [email protected] > >>> > >>> > >>> > >>> This message and any attachments (the "message") is > >>> intended solely for the intended addressees and is confidential. > >>> If you receive this message in error,or are not the intended recipient(s), > >>> please delete it and any copies from your systems and immediately notify > >>> the sender. Any unauthorized view, use that does not comply with its > purpose, > >>> dissemination or disclosure, either whole or partial, is prohibited. > >>> Since the > >> internet > >>> cannot guarantee the integrity of this message which may not be reliable, > BNP > >> PARIBAS > >>> (and its subsidiaries) shall not be liable for the message if modified, > >>> changed > or > >> falsified. > >>> Do not print this message unless it is necessary, consider the > >>> environment. > >>> > >>> -------------------------------------------------------------------------------------------- > --- > >> ----------------------------------- > >>> > >>> Ce message et toutes les pieces jointes (ci-apres le "message") > >>> sont etablis a l'intention exclusive de ses destinataires et sont > >>> confidentiels. > >>> Si vous recevez ce message par erreur ou s'il ne vous est pas destine, > >>> merci de le detruire ainsi que toute copie de votre systeme et d'en > >>> avertir > >>> immediatement l'expediteur. Toute lecture non autorisee, toute utilisation > de > >>> ce message qui n'est pas conforme a sa destination, toute diffusion ou > toute > >>> publication, totale ou partielle, est interdite. L'Internet ne permettant > >>> pas > >> d'assurer > >>> l'integrite de ce message electronique susceptible d'alteration, BNP > >>> Paribas > >>> (et ses filiales) decline(nt) toute responsabilite au titre de ce message > >>> dans > >> l'hypothese > >>> ou il aurait ete modifie, deforme ou falsifie. > >>> N'imprimez ce message que si necessaire, pensez a l'environnement. > >>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >>> of > >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > >> LIKE THAT. > >>> > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
