does rsyslogd -N1 report any issues?
David Lang
On Mon, 12 Dec 2022, helices wrote:
I just now restarted again, like this:
# systemctl restart rsyslog
We'll see overnight if that does the trick.
# date; grep -v "^\(#\|\s*$\)" /etc/rsyslog.conf ;date
Mon Dec 12 13:56:12 CST 2022
module(load="imjournal" Ratelimit.Burst="30000" Ratelimit.Interval="1000"
StateFile="imjournal.state")
module(load="imklog")
module(load="immark")
module(load="impstats" interval="600" severity="7")
syslog.=debug /var/log/rsyslog-stats
module(load="imtcp")
input(type="imtcp" port="514")
module(load="imudp")
input(type="imudp" port="514")
module(load="ommysql.so")
global(workDirectory="/var/lib/rsyslog")
authpriv.none;cron.none;*.info;mail.none /var/log/messages
authpriv.* /var/log/secure
cron.* /var/log/cron
*.emerg :omusrmsg:*
ftp.* /var/log/vsftpd.log
local7.* /var/log/boot.log
mail.* /var/log/maillog
uucp,news.crit /var/log/spooler
$ActionName Ftp
$ActionQueueFileName dbFtpQueue # Set file name, also enables disk mode
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
$ActionQueueType LinkedList # Use asynchronous processing
$ActionResumeRetryCount -1 # Infinite retries on insert failure
ftp.*
:ommysql:10.199.5.177,vsftplog,hermesvsftplog,_____
$ActionName Sftp
$ActionQueueFileName dbSftpQueue # Set file name, also enables disk mode
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
$ActionQueueType LinkedList # Use asynchronous processing
$ActionResumeRetryCount -1 # Infinite retries on insert failure
authpriv.*
:ommysql:10.199.5.177,sftplogDB,hermesvsftplog,_____
$ActionName Admin
$ActionQueueFileName ZenossQueue # Set file name, also enables disk mode
$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown
$ActionQueueType LinkedList # Use asynchronous processing
$ActionResumeRetryCount -1 # Infinite retries on insert failure
*.* @@10.199.1.160
Mon Dec 12 13:56:12 CST 2022
On Mon, Dec 12, 2022 at 1:34 PM David Lang <[email protected]> wrote:
did you do a full restart after making the change? can you show the full
config?
the messages you are showing are saying taht the config line you show
isn't
being used.
David Lang
On Mon, 12 Dec 2022, helices via rsyslog wrote:
Date: Mon, 12 Dec 2022 12:39:54 -0600
From: helices via rsyslog <[email protected]>
To: Rainer Gerhards <[email protected]>
Cc: helices <[email protected]>,
rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Rsyslogd/ommysql.so: Not writing to DB
intermittently
We're still missing something:
module(load="imjournal" Ratelimit.Burst="30000" Ratelimit.Interval="1000"
StateFile="imjournal.state")
2022-12-12T00:53:14.001626-06:00 hermes rsyslogd[1536]:
rsyslogd[internal_messages]: 1728 messages lost due to rate-limiting (500
allowed within 5 seconds)
2022-12-12T00:53:20.004006-06:00 hermes rsyslogd[1536]:
rsyslogd[internal_messages]: 1818 messages lost due to rate-limiting (500
allowed within 5 seconds)
2022-12-12T00:53:26.003870-06:00 hermes rsyslogd[1536]:
rsyslogd[internal_messages]: 1794 messages lost due to rate-limiting (500
allowed within 5 seconds)
2022-12-12T00:53:32.005388-06:00 hermes rsyslogd[1536]:
rsyslogd[internal_messages]: 1797 messages lost due to rate-limiting (500
allowed within 5 seconds)
2022-12-12T00:53:38.001367-06:00 hermes rsyslogd[1536]:
rsyslogd[internal_messages]: 1812 messages lost due to rate-limiting (500
allowed within 5 seconds)
2022-12-12T00:53:44.006085-06:00 hermes rsyslogd[1536]:
rsyslogd[internal_messages]: 1791 messages lost due to rate-limiting (500
allowed within 5 seconds)
2022-12-12T00:53:50.005487-06:00 hermes rsyslogd[1536]:
rsyslogd[internal_messages]: 1797 messages lost due to rate-limiting (500
allowed within 5 seconds)
2022-12-12T00:53:56.001546-06:00 hermes rsyslogd[1536]:
rsyslogd[internal_messages]: 1808 messages lost due to rate-limiting (500
allowed within 5 seconds)
2022-12-12T00:54:02.007743-06:00 hermes rsyslogd[1536]:
rsyslogd[internal_messages]: 1759 messages lost due to rate-limiting (500
allowed within 5 seconds)
What are we missing?
Please, advise. Thank you.
On Fri, Dec 9, 2022 at 8:49 AM Rainer Gerhards <[email protected]
wrote:
you set the interval, but not ratelimit.burst
doc:
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html
Rainer
El mar, 6 dic 2022 a las 15:16, helices via rsyslog
(<[email protected]>) escribió:
David,
What am I doing wrong?
module(load="imjournal" Ratelimit.Interval="10000"
StateFile="imjournal.state")
2022-12-06T07:19:26.004772-06:00 hermes rsyslogd[29735]:
rsyslogd[internal_messages]: 1755 messages lost due to rate-limiting
(500
allowed within 5 seconds)
Please, advise. Thank you.
~ Mike
On Thu, Dec 1, 2022 at 3:12 PM David Lang <[email protected]> wrote:
On Thu, 1 Dec 2022, helices wrote:
[1] What is "action() syntax?" Which lines ought to be converted?
How?
https://www.rsyslog.com/doc/master/configuration/basic_structure.html#statement-types
instead of
@@10.0.0.1
you would do
action(type="omfwd" target="10.0.0.1" port="514" protocol="tcp")
for this trivial example, the earlier syntax makes more sense, but
when
you have
more complex things (like the queues that you have), adding them all
into
the
action makes it clearer exactly what is happening
so you currently have
$ActionName Admin
$ActionQueueDequeueSlowdown 1000 # How long (in microseconds)
dequeueing
should be delayed
$ActionQueueFileName ZenossQueue # Set file name, also enables
disk
mode
$ActionQueueSaveOnShutdown on # Save messages to disk on
shutdown
$ActionQueueType LinkedList # Use asynchronous processing
$ActionResumeRetryCount -1 # Infinite retries on insert
failure
*.* @@10.199.1.160
This would be
action(name="Admin" type="omfwd" target="10.199.1.160"
protocol="tcp"
queue.filename="ZenossQueue" queue.saveonshutdown="on"
queue.type="linkedlist"
resumeretrycount="-1" queue.dequeueslowdown="1000")
this makes it very clear that all these parameters apply only to
this
action
(which is what the old syntax does, but it's less obvious to people
that
it only
applies to the next action)
[2] Where is the "pause" you mention? I don't recognize that.
$ActionQueueDequeueSlowdown 1000 # How long (in microseconds)
dequeueing
This tells rsyslog to pause after each batch of messages before
processing
the
next batch.
[3] impstats? Permanently? Only for this debugging?
I like to have it on permanently, but especially for debugging it
provides
a lot
of useful info
[4] How to modify imjournal rate limits?
see
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html
[5] RSYSLOG_DebugFormat? I found this:
https://www.rsyslog.com/doc/v8-stable/configuration/templates.html
- Is
that example proper by itself? Where does this template go? How
can I
specify the file and location for debugging?
as I said below
ftp.* /var/log/ftp;RSYSLOG_DebugFormat (legacy format, add
template="RSYSLOG_DebugFormat" to that action() format)
If there are URLs to inform me, I appreciate your direction.
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html
https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommysql.html
https://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html
https://www.rsyslog.com/doc/master/configuration/actions.html
https://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html
https://www.rsyslog.com/doc/v8-stable/configuration/modules/omfile.html
feel free to keep asking questions.
David Lang
~ Mike
On Thu, Dec 1, 2022 at 1:33 PM David Lang <[email protected]> wrote:
it would be useful to convert to the action() syntax as it makes
it
clearer
what's happening.
Why are you pausing between writing logs? (this could be why you
are
dropping
logs)
given the number of queues and actions, look at configuring
impstats so
that you
can see the number of messages in the queues, number processed,
etc.
imjournal defaults to some fairly aggressive rate limiting, I
find
that
I
always
need to drastically increase the limits.
writing logs using the RSYSLOG_DebugFormat is adding the template
to the
file
ftp.* /var/log/ftp;RSYSLOG_DebugFormat (legacy format, add
template="RSYSLOG_DebugFormat" to that action() format)
the debug format is large, but you really need to see the message
that's
failing
to figure out why it's failing. The MySQL logs may give you
better
info
on
that.
David Lang
On Thu, 1 Dec 2022, helices wrote:
Date: Thu, 1 Dec 2022 13:26:47 -0600
From: helices <[email protected]>
To: David Lang <[email protected]>
Cc: helices via rsyslog <[email protected]>
Subject: Re: [rsyslog] Rsyslogd/ommysql.so: Not writing to DB
intermittently
Thank you.
[1] rsyslog.conf
# date; grep -v "^\(#\|\s*$\)" /etc/rsyslog.conf ;date
Thu Dec 1 13:19:34 CST 2022
module(load="imjournal" StateFile="imjournal.state")
module(load="imklog")
module(load="immark")
module(load="impstats" interval="600" severity="7")
syslog.=debug /var/log/rsyslog-stats
module(load="imtcp")
input(type="imtcp" port="514")
module(load="imudp")
input(type="imudp" port="514")
module(load="ommysql.so")
global(workDirectory="/var/lib/rsyslog")
authpriv.none;cron.none;*.info;mail.none /var/log/messages
authpriv.* /var/log/secure
cron.* /var/log/cron
*.emerg :omusrmsg:*
ftp.* /var/log/vsftpd.log
local7.* /var/log/boot.log
mail.* /var/log/maillog
uucp,news.crit /var/log/spooler
$ActionName Ftp
$ActionQueueDequeueSlowdown 1000 # How long (in microseconds)
dequeueing
should be delayed
$ActionQueueFileName dbFtpQueue # Set file name, also enables
disk
mode
$ActionQueueSaveOnShutdown on # Save messages to disk on
shutdown
$ActionQueueType LinkedList # Use asynchronous processing
$ActionResumeRetryCount -1 # Infinite retries on insert
failure
ftp.*
:ommysql:10.199.5.177,vsftplog,hermesvsftplog,_____
$ActionName Sftp
$ActionQueueDequeueSlowdown 1000 # How long (in microseconds)
dequeueing
should be delayed
$ActionQueueFileName dbSftpQueue # Set file name, also enables
disk
mode
$ActionQueueSaveOnShutdown on # Save messages to disk on
shutdown
$ActionQueueType LinkedList # Use asynchronous processing
$ActionResumeRetryCount -1 # Infinite retries on insert
failure
authpriv.*
:ommysql:10.199.5.177,sftplogDB,hermesvsftplog,_____
$ActionName Admin
$ActionQueueDequeueSlowdown 1000 # How long (in microseconds)
dequeueing
should be delayed
$ActionQueueFileName ZenossQueue # Set file name, also enables
disk
mode
$ActionQueueSaveOnShutdown on # Save messages to disk on
shutdown
$ActionQueueType LinkedList # Use asynchronous processing
$ActionResumeRetryCount -1 # Infinite retries on insert
failure
*.* @@10.199.1.160
Thu Dec 1 13:19:34 CST 2022
[2] How do we "log the message with the template
RSYSLOG_DebugFormat
to a
file?" How much disk space is needed? This problem appears to
have
started
recently, and appears to happen once or twice per day, without a
common
time.
[3] I didn't notice the rate-limiting until now. It is not
uncommon.
How
can we avoid losing so many messages?
~ Mike
On Thu, Dec 1, 2022 at 1:05 PM David Lang <[email protected]>
wrote:
please post your full config.
It would also help to log the message with the template
RSYSLOG_DebugFormat to a
file and find the log entry that is failing to insert.
my guess is that the quotes in the message are confusing mysql
note that rate limiting is throwing away messages because you
are
trying
to
process them too fast.
David Lang
On Thu, 1 Dec 2022, helices via rsyslog wrote:
Date: Thu, 1 Dec 2022 10:08:01 -0600
From: helices via rsyslog <[email protected]>
To: rsyslog-users <[email protected]>
Cc: helices <[email protected]>
Subject: [rsyslog] Rsyslogd/ommysql.so: Not writing to DB
intermittently
# date; /bin/yum list rsyslog rsyslog-mysql ;date
Thu Dec 1 09:47:18 CST 2022
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: download.cf.centos.org
* epel: mirror.genesisadaptive.com
* extras: download.cf.centos.org
* remi-php56: mirror.pit.teraswitch.com
* remi-safe: mirror.pit.teraswitch.com
* updates: download.cf.centos.org
Installed Packages
rsyslog.x86_64
8.2210.0-1.el7
@rsyslog_v8
rsyslog-mysql.x86_64
8.2210.0-1.el7
@rsyslog_v8
Thu Dec 1 09:47:19 CST 2022
Sample of numerous error messages (/var/log/messages):
rsyslogd[17344]: ommysql: db error (1172): Result consisted of
more
than
one row [v8.2210.0]
rsyslogd[17344]: The error statement was: insert into
SystemEvents
(Message, Facility, FromHost, Priority, DeviceReportedTime,
ReceivedAt,
InfoUnitID, SysLogTag) values ('close
"/incoming/wood.pgez.scen.11302022.sa.pgp" bytes read 0
written
2603
[postauth]', 10, 'hermes', 6, '20221201081257',
'20221201081257', 1,
'sshd[19654]:') [v8.2210.0 try https://www.rsyslog.com/e/2218
]
rsyslogd[17344]: rsyslogd[internal_messages]: 215 messages
lost
due
to
rate-limiting (500 allowed within 5 seconds)
rsyslogd[17344]: action 'Sftp' (module 'ommysql.so') message
lost,
could
not be processed. Check for additional error messages before
this
one.
[v8.2210.0 try https://www.rsyslog.com/e/2218 ]
We have been writing all data from Internet file transfers to
a
Mysql
table
for years. Recently, we began seeing intermittent errors like
those
above.
What is happening here?
What can we do to fix this problem?
Please, advise. Thank you.
~ Mike
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
