On Thu, 9 Jan 2020, Daniel Rubio via rsyslog wrote:
Now I see on the server status for rsyslog this (the messages are from
the hour I restarted the rsyslog server):
Jan 09 10:10:43 logcenter rsyslogd[9133]: imjournal: begin to drop
messages due to rate-limiting
Jan 09 10:25:01 logcenter rsyslogd[9133]: imjournal: 1000128 messages
lost due to rate-limiting
What I'm doing wrong? Is there any way to recover those messages without
having to manually rotate then from the clients?
see the rate limiting settings for imjournal.
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html
I'm not sure what you would need to do to retrieve the lost messages. telling
rsyslog to retrieve all journald messages, including re-sending old ones would
probably do it. but that will happen every time you restart rsyslog (at some
point you will want to tell rsyslog to only retrieve new ones, but when you
restart rsyslog to implement the new configs, there will be a race condition
that will cause you to miss some logs at that point.
David Lang
On 9/1/20 12:04, Daniel Rubio via rsyslog wrote:
I've got 2 problems :)
1.- Yesterday, In one of our rsyslog servers, we started to see massive
too many open files messages:
Jan 8 21:05:39 logcenter rsyslogd: file
'/logs/2020/frontal/web/01/apacheaccess_08_S1.log': open error: Too many
open files [v8.24.0-41.el7_7.2 try http://www.rsyslog.com/e/2433 ]
Jan 8 21:05:40 logcenter rsyslogd: file
'/logs/2020/munihosting/web/01/apacheaccess_08_S1.log': open error: Too
many open files [v8.24.0-41.el7_7.2 try http://www.rsyslog.com/e/2433 ]
...
This morning I restarted the rsyslog server and now it seems to be all
ok but... How can I raise this max value up? Is the default config for
the operating system (CentOs 7.6)? or there is something we have to
configure in rsyslog config?
The second problem is related with this one...
2.-On my rsyslog clients I have this config for the output:
ruleset(name="sendToLogserver") {
action( type="omfwd" target="logcenter.intranet.dtgna" port="514"
protocol="tcp" queue.type="LinkedList" queue.size="500000"
queue.filename="q_sendToLogserver" queue.highwatermark="290000" queue.lowwa
termark="50000" queue.maxdiskspace="2g" queue.saveonshutdown="on"
action.resumeRetryCount="-1" action.resumeInterval="20")
action( type="omfwd" target="logcenter02.intranet.dtgna" port="514"
protocol="tcp" queue.type="LinkedList" queue.size="500000"
queue.filename="q_sendToLogserver02" queue.highwatermark="290000" queue.l
owwatermark="50000" queue.maxdiskspace="2g" queue.saveonshutdown="on"
action.resumeRetryCount="-1" action.resumeInterval="20"
action.execOnlyWhenPreviousIsSuspended="on")
stop
}
Yesterday, the problem with the open files started about 21:05, and
there is a problem with this client file (server appsact01)
Client config:
input(type="imfile" file="/logs/wildfly10/server.log"
tag="wildfly_serverlog" ruleset="sendToLogserver" reopenontruncate="on")
In the rsyslog server, the last modification time and line are:
[root@logcenter ~]# tail -1
/logs/2020/appsact01/appssrv/01/wildfly_serverlog_08_S1.log
2020-01-08 21:20:38,138 FINE [groovy.sql.Sql] (default task-48) select
distinct * from XXX | []
But in the server, the last line for the file yesterday was:
[root@appsactio01 ~]# tail -1 /logs/wildfly10/server.log.2020-01-08
2020-01-08 23:43:17,289 FINE [groovy.sql.Sql] (default task-64) select
distinct * fromXXX WHERE codi = '02' | []
Is not the same :((
This line is neither in the file created after I restarted the service,
which starts about 00:42 in the night...
[root@logcenter ~]# more
/logs/2020/appsact01/appssrv/01/wildfly_serverlog_09_S1.log
2020-01-09 00:42:34,093 FINE [groovy.sql.Sql]
(sidng_scheduler_Worker-2) select view_name from user_views
While in the server, the first line for today's server.log file is:
[root@appsact01 ~]# more /logs/wildfly10/server.log
2020-01-09 00:13:11,796 FINE [groovy.sql.Sql] (default task-40) select
distinct * from V_XXX codi = '02' | []
Also different :((
The questions are:
-If the primary rsyslog server was giving those open files errors, the
client shouldn't have to start send the messages to the secondary server?
-What happened to the disappeared messages? how we could trace it? it's
a config problem?
-During the problem, the client server.log rotated, I suppose that there
wouldn't have to be a problem because those lines where in the rsyslog
queue, isn't it?
I'm very worried about those lines lost, It happended to various log
files :(
PS:Rsyslog version in server and clients is the last red-hat official
update, 8.24.0-41.el7_7.2
--
signatura
*
Daniel Rubio Rodríguez*
Sistemes | Infraestructures Informàtiques
Tecnologies de la informació i les Comunicacions (TIC)
Diputació de Tarragona
Tel. 977 296 635 | [email protected]
Passeig Sant Antoni, 100 | 43003 Tarragona
www.dipta.cat
/Aquest missatge s’adreça exclusivament a qui va destinat i pot contenir
informació privilegiada o confidencial i dades de caràcter personal, la
difusió de les quals és regulada per la LOPD i la LSSI. Si no sou la
persona destinatària indicada (o la responsable de lliurar-lo a qui va
destinat), no heu de copiar aquest missatge ni lliurar-lo a tercers per
cap concepte. Si heu rebut aquest missatge per error o l’heu aconseguit
per altres mitjans, us demanem que ens ho comuniqueu immediatament per
aquesta mateixa via i l’elimineu irreversiblement./
/Abans d’imprimir aquest missatge, assegureu-vos que és realment
necessari i penseu en el medi ambient./
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
