Now I see on the server status for rsyslog this (the messages are from the hour I restarted the rsyslog server):
Jan 09 10:10:43 logcenter rsyslogd[9133]: imjournal: begin to drop messages due to rate-limiting Jan 09 10:25:01 logcenter rsyslogd[9133]: imjournal: 1000128 messages lost due to rate-limiting What I'm doing wrong? Is there any way to recover those messages without having to manually rotate then from the clients? On 9/1/20 12:04, Daniel Rubio via rsyslog wrote: > I've got 2 problems :) > > > 1.- Yesterday, In one of our rsyslog servers, we started to see massive > too many open files messages: > > Jan 8 21:05:39 logcenter rsyslogd: file > '/logs/2020/frontal/web/01/apacheaccess_08_S1.log': open error: Too many > open files [v8.24.0-41.el7_7.2 try http://www.rsyslog.com/e/2433 ] > Jan 8 21:05:40 logcenter rsyslogd: file > '/logs/2020/munihosting/web/01/apacheaccess_08_S1.log': open error: Too > many open files [v8.24.0-41.el7_7.2 try http://www.rsyslog.com/e/2433 ] > ... > > This morning I restarted the rsyslog server and now it seems to be all > ok but... How can I raise this max value up? Is the default config for > the operating system (CentOs 7.6)? or there is something we have to > configure in rsyslog config? > > > The second problem is related with this one... > > > 2.-On my rsyslog clients I have this config for the output: > > > ruleset(name="sendToLogserver") { > action( type="omfwd" target="logcenter.intranet.dtgna" port="514" > protocol="tcp" queue.type="LinkedList" queue.size="500000" > queue.filename="q_sendToLogserver" queue.highwatermark="290000" queue.lowwa > termark="50000" queue.maxdiskspace="2g" queue.saveonshutdown="on" > action.resumeRetryCount="-1" action.resumeInterval="20") > action( type="omfwd" target="logcenter02.intranet.dtgna" port="514" > protocol="tcp" queue.type="LinkedList" queue.size="500000" > queue.filename="q_sendToLogserver02" queue.highwatermark="290000" queue.l > owwatermark="50000" queue.maxdiskspace="2g" queue.saveonshutdown="on" > action.resumeRetryCount="-1" action.resumeInterval="20" > action.execOnlyWhenPreviousIsSuspended="on") > stop > } > > Yesterday, the problem with the open files started about 21:05, and > there is a problem with this client file (server appsact01) > > Client config: > input(type="imfile" file="/logs/wildfly10/server.log" > tag="wildfly_serverlog" ruleset="sendToLogserver" reopenontruncate="on") > > In the rsyslog server, the last modification time and line are: > > [root@logcenter ~]# tail -1 > /logs/2020/appsact01/appssrv/01/wildfly_serverlog_08_S1.log > 2020-01-08 21:20:38,138 FINE [groovy.sql.Sql] (default task-48) select > distinct * from XXX | [] > > But in the server, the last line for the file yesterday was: > [root@appsactio01 ~]# tail -1 /logs/wildfly10/server.log.2020-01-08 > 2020-01-08 23:43:17,289 FINE [groovy.sql.Sql] (default task-64) select > distinct * fromXXX WHERE codi = '02' | [] > > Is not the same :(( > > This line is neither in the file created after I restarted the service, > which starts about 00:42 in the night... > > [root@logcenter ~]# more > /logs/2020/appsact01/appssrv/01/wildfly_serverlog_09_S1.log > 2020-01-09 00:42:34,093 FINE [groovy.sql.Sql] > (sidng_scheduler_Worker-2) select view_name from user_views > > While in the server, the first line for today's server.log file is: > > [root@appsact01 ~]# more /logs/wildfly10/server.log > 2020-01-09 00:13:11,796 FINE [groovy.sql.Sql] (default task-40) select > distinct * from V_XXX codi = '02' | [] > > Also different :(( > > > The questions are: > > -If the primary rsyslog server was giving those open files errors, the > client shouldn't have to start send the messages to the secondary server? > -What happened to the disappeared messages? how we could trace it? it's > a config problem? > -During the problem, the client server.log rotated, I suppose that there > wouldn't have to be a problem because those lines where in the rsyslog > queue, isn't it? > > I'm very worried about those lines lost, It happended to various log > files :( > > PS:Rsyslog version in server and clients is the last red-hat official > update, 8.24.0-41.el7_7.2 > -- signatura * Daniel Rubio Rodríguez* Sistemes | Infraestructures Informàtiques Tecnologies de la informació i les Comunicacions (TIC) Diputació de Tarragona Tel. 977 296 635 | [email protected] Passeig Sant Antoni, 100 | 43003 Tarragona www.dipta.cat /Aquest missatge s’adreça exclusivament a qui va destinat i pot contenir informació privilegiada o confidencial i dades de caràcter personal, la difusió de les quals és regulada per la LOPD i la LSSI. Si no sou la persona destinatària indicada (o la responsable de lliurar-lo a qui va destinat), no heu de copiar aquest missatge ni lliurar-lo a tercers per cap concepte. Si heu rebut aquest missatge per error o l’heu aconseguit per altres mitjans, us demanem que ens ho comuniqueu immediatament per aquesta mateixa via i l’elimineu irreversiblement./ /Abans d’imprimir aquest missatge, assegureu-vos que és realment necessari i penseu en el medi ambient./ _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
