On Wed, 29 May 2019, John Jasen wrote:
a) I'll note the simple reversion to 1904 restores functionality. I'll also
note 1905 works for about half an hour, then just ... stops.
b) impstats stopped when rsyslog stopped logging.
If you set it to log to a file, it's logs will be independent of the regular log
stream and be written even if the main queue ends up getting blocked.
The other thing that I will do is to define a ruleset with it's own queue for
the impstats (again, making it independent of the main queue)
note: I am not seeing where the ruleset Network is ever used
David Lang
c) #darn. -o was added in 1905.
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see
http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
$ModLoad impstats
$PStatInterval 300
$PStatSeverity 7
$MaxOpenFiles 524288
$MaxMessageSize 131072
# The imjournal module bellow is now used as a message source instead of
imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
module(load="imjournal"
ratelimit.interval="0"
ratelimit.burst="0"
StateFile="imjournal.state"
WorkAroundJournalBug="on")
#main.queue tuning
main_queue(
queue.size="1000000"
queue.workerthreads="6"
)
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files. SSD backed mountpoint
$WorkDirectory /var/local/spool/rsyslog
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
# byhostname to also store local in directories on SSD by hostname.
*.info;mail.none;authpriv.none;cron.none /var/log/messages
& ?ByHostname
# The authpriv file has restricted access.
authpriv.* /var/log/secure
& ?ByHostname
# Log all the mail messages in one place.
mail.* -/var/log/maillog
& ?ByHostname
# Log cron stuff
cron.* /var/log/cron
& ?ByHostname
# Everybody gets emergency messages
*.emerg :omusrmsg:*
& ?ByHostname
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
& ?ByHostname
# Save boot messages also to boot.log
local7.* /var/log/boot.log
& ?ByHostname
syslog.=debug
/var/log/rsyslog-stats
& ?ByHostname
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* @@my-log-host:1516
# ### end of the forwarding rule ###
rsyslog.d/auditd-imfile.conf
$ModLoad imfile
$Template
AuditByHostname,"/var/local/log/raw/%HOSTNAME%/%HOSTNAME%.log.%$YEAR%%
$MONTH%%$DAY%"
ruleset(name="ToDisk"){
action(name="Local" type="omfile" dynafile="AuditByHostname" )
action(name="Remote" type="omfwd" Target="x.y.z.a" Port="1516"
Protocol="tcp" queue.type="linkedlist"
queue.filename="remote_fwd_zama1-audit" action.resumeRetryCount="-1"
queue.saveOnShutdown="on" )
stop
}
input(type="imfile" ruleset="ToDisk"
File="/var/log/audit/audit.log"
Tag="auditd"
)
rsyslog.d/listen.conf
# nccs remote logging capabilities test
module(load="imptcp" Threads="16" ProcessOnPoller="off")
module(load="imudp" threads="12" BatchSize="128")
module(load="imtcp" MaxSessions="16384" StreamDriver.Name="gtls"
StreamDriver.Mode="1" StreamDriver.AuthMode="x509/name
")
module(load="imrelp")
$Template
ByHostname,"/var/local/log/raw/%HOSTNAME%/%HOSTNAME%.log.%$YEAR%%$MONTH%%$DAY%"
ruleset(name="Network"){
action(name="ToLocalDisk" type="omfile" dynafile="ByHostname"
DynaFileCacheSize="131072" ioBufferSize="64k" )
if ($syslogtag startswith "pf:") then {
action(name="ToRemotePF" type="omfwd" Target="x.y.z.a" Port="1517"
Protocol="tcp" queue.type="linkedlist"
queue.size="1000000" queue.workerthreads="6"
queue.filename="remote_fwd_pf" action.resumeRetryCount="-1"
queue.saveOnShutdown="on" )
stop
} else if ($syslogtag startswith "httpd-modsec-audit-json") then {
action(name="ToRemoteJSON" type="omfwd" Target="x.y.z.a" Port="1518"
Protocol="tcp" queue.type="linkedlist" queue.size="1000000"
queue.workerthreads="6" queue.filename="remotejson_fwd"
action.resumeRetryCount="-1" queue.saveOnShutdown="on" )
stop
} else if ($syslogtag startswith "audispd") then {
action(name="ToRemoteaudisp" type="omfwd" Target="x.y.z.a"
Port="1519" Protocol="tcp" queue.type="linkedlist" queue.size="1000000"
queue.workerthreads="6" queue.filename="remoteaudisp_fwd"
action.resumeRetryCount="-1" queue.saveOnShutdown="on" )
stop
} else if ($syslogtag startswith "auditd") then {
action(name="ToRemoteauditd" type="omfwd" Target="x.y.z.a"
Port="1519" Protocol="tcp" queue.type="linkedlist" queue.size="1000000"
queue.workerthreads="6" queue.filename="zamls1audit_fwd"
action.resumeRetryCount="-1" queue.saveOnShutdown="on" )
stop
} else {
action(name="ToRemote" type="omfwd" Target="x.y.z.a" Port="1516"
Protocol="tcp" queue.type="linkedlist"
queue.size="1000000" queue.workerthreads="6" queue.filename="remote_fwd"
action.resumeRetryCount="-1" queue.saveOnShutdown="on" )
stop
}
}
On Thu, May 30, 2019 at 1:41 PM David Lang <[email protected]> wrote:
do you have impstats configured? that would help identify what's going
wrong.
Can you share your config? without that we don't have any place to start.
I have not seen other reports of problems, which isn't saying you didn't
run
into something, just that it's not something common enough to have already
been
reported.
David Lang
On Wed, 29 May 2019, John Jasen via rsyslog wrote:
Yesterday evening, we rolled rsyslog 8.1905 into production on our log
servers. In general, they process on the order of 300-400k
messages/minute.
Twice, under rsyslog 8.1905, it stopped processing everything (local logs
to /var/log, remote logs to $LOGDIR, forwarding to log analysis). We
reverted back to 8.1904, and it has been stable at 400+k messages/minute
for over 12 hours now.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
