config looks good on quick look, so we really need to see impstats. Set it 10 second interval, then create the problem situation.
doc on impstats: https://www.rsyslog.com/doc/v8-stable/configuration/modules/impstats.html Rainer El vie., 8 mar. 2019 a las 9:50, João Pereira (<[email protected]>) escribió: > Hi Rainer, > > Sure. > > The forwarding service (the first one) just compress and sends data to the > other. > /etc/rsyslog.d/31-forwarding.conf > > if $syslogfacility-text == 'local6' or $syslogfacility-text == 'local5' > then { > > action( > type="omfwd" > Target="example.com" > ZipLevel="9" > compression.mode="stream:always" > compression.stream.flushOnTXEnd="off" > TCP_Framing="octet-counted" > Port="514" > Protocol="tcp" > ) > > stop > } > > > The others have several rules like: > > /etc/rsyslog.d/31-second_service.conf > > template(name="providerIndexTemplate" type="string" > string="%PROGRAMNAME%-%TIMESTAMP:::date-year%%TIMESTAMP:::date-month%%TIMESTAMP:::date-day%") > > template(name="providerPEIndexTemplate" type="string" > string="fastly_pe-%TIMESTAMP:::date-year%%TIMESTAMP:::date-month%%TIMESTAMP:::date-day%") > > if $syslogfacility-text == 'local0' then { > action(type="mmnormalize" > rulebase="/etc/rsyslog.d/33-second_service.rb") > > set $!severity_code = $syslogseverity; > set $!severity = $syslogseverity-text; > > if $parsesuccess != "OK" then { > set $!timestamp = exec_template("timeStampGenerator"); > set $!fac = $syslogfacility; > set $!host = $hostname; > > action( > type="omelasticsearch" > template="all-json" # we use the template defined > earlier. > searchIndex="providerPEIndexTemplate" > dynSearchIndex="on" > searchType="providerSyslog" # we specify a static > string. > dynSearchType="off" > server="127.0.0.1" > serverport="9200" > bulkmode="on" # use the bulk API > queue.size="60000" > queue.workerthreads="2" > queue.dequeuebatchsize="2000" > ) > } else { > > action( > type="omelasticsearch" > template="all-json" # we use the template defined > earlier. > searchIndex="providerIndexTemplate" > dynSearchIndex="on" > searchType="providerSyslog" # we specify a static > string. > dynSearchType="off" > server="127.0.0.1" > serverport="9200" > bulkmode="on" # use the bulk API > queue.size="600000" > queue.workerthreads="2" > queue.dequeuebatchsize="2000" > queue.timeoutEnqueue="0" > ) > > } > stop > } > > The /etc/rsyslog.conf file is the same on both services: > > $ModLoad imuxsock # provides support for local system logging > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > module(load="omprog") > module(load="imptcp" threads="8") > input(type="imptcp" port="514" compression.mode="stream:always" > KeepAlive="on") > input(type="imptcp" port="515") > > main_queue ( > queue.type="fixedArray" > queue.size="250000" > queue.dequeueBatchSize="4096" > queue.workerThreads="4" > queue.workerThreadMinimumMessages="60000" > queue.discardSeverity="6" > queue.timeoutEnqueue="0" > ) > > $MaxOpenFiles 40000 > $RepeatedMsgReduction off > $EscapeControlCharactersOnReceive off > $MaxMessageSize 124k > $FileOwner syslog > $FileGroup adm > $FileCreateMode 0640 > $DirCreateMode 0755 > $Umask 0022 > $PrivDropToUser syslog > $PrivDropToGroup syslog > $WorkDirectory /var/spool/rsyslog > $IncludeConfig /etc/rsyslog.d/*.conf > > Regarding the imstats, I'm having some problem understanding them. There > is a place where I can have all explained ? > > Thanks a lot > > On Thu, Mar 7, 2019 at 5:52 PM Rainer Gerhards <[email protected]> > wrote: > >> We need at least the config of the initial instane. I guess buffering is >> not properly set up. May also be a volume problem, what impstats will show. >> >> Rainer >> >> João Pereira <[email protected]> schrieb am Do., 7. März 2019, >> 16:21: >> >>> Hi all, >>> >>> We are facing an issue with rsyslog and we cannot find what is happening >>> behind. >>> >>> We're using rsyslog to receive logs from one of our providers, the >>> problem >>> is that the provider stops sending logs (during aprox 10m) when it >>> detects >>> the receiver is down meaning that every time we restart rsyslog server we >>> loose logs for ~10m. >>> >>> As we cannot control what the provider does, we came up with the idea of >>> having two rsyslog services on our machines. The first would only receive >>> the logs sent by our provider and forward them to the other rsyslog >>> service, the latests being responsible for parsing the logs and send it >>> to >>> elasticsearch. This would allow us to change the configuration on the >>> second service (which are changes mostly on parsing rules) without having >>> to restart the forwarding service that contacts with our provider. >>> >>> That way we would be able to fool our provider because the forwarding >>> service would always be available, this sounded good on paper but when we >>> put it in production we realised that when we restart the second service >>> the first hangs (stops working for a while) and the failure is detected >>> by >>> our provider which stops sending logs. >>> >>> Is there any way to improve this setup ? Can we make the forwarding >>> service >>> to not hang ? Why rsyslog has this behaviour ? >>> >>> Thanks in advance >>> >>> -- >>> >>> João Pereira >>> >>> <https://www.marfeel.com> >>> >>> <https://www.marfeel.com/> >>> [image: Inline images 4] >>> < >>> https://atenea.marfeel.com/atn/marfeel-business/what-it-means-to-be-a-google-certified-publishing-partner >>> > >>> [image: Inline images 3] >>> < >>> https://atenea.marfeel.com/atn/marfeel-business/what-it-means-to-be-a-facebook-instant-articles-partner >>> > >>> >>> >>> Avda. Josep Tarradellas 20-30, 6th Floor >>> >>> 08029 Barcelona, Spain >>> >>> ES: (34) 93 178 59 50 >>> <%2834%29%2093%20178%2059%2050%20%C2%A0ext.%20107> >>> US: (1) 917-341-2540 <%281%29%20917-341-2540%20ext.%20107> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >> >> > > -- > > João Pereira > > <https://www.marfeel.com> > > <https://www.marfeel.com/> > [image: Inline images 4] > <https://atenea.marfeel.com/atn/marfeel-business/what-it-means-to-be-a-google-certified-publishing-partner> > [image: Inline images 3] > <https://atenea.marfeel.com/atn/marfeel-business/what-it-means-to-be-a-facebook-instant-articles-partner> > > > > Avda. Josep Tarradellas 20-30, 6th Floor > > 08029 Barcelona, Spain > > ES: (34) 93 178 59 50 > <%2834%29%2093%20178%2059%2050%20%C2%A0ext.%20107> > US: (1) 917-341-2540 <%281%29%20917-341-2540%20ext.%20107> >
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
