There is not much in configuration:
- impstats module configuration
- one imtcp input (one udp input - not used atm)
- main queue configuration
- two omfwd:tcp forwards with DA queues
- local file store
There is some msg regex parsing and manipulation which might look strange
and complicated, and maybe could be simplified one day.
How could/might the multithreading of main queue help to improve
performance?
Thought the rsyslog's 'main queue preprocessor' will spread the messages
from imtcp input between two main queue threads proportionally. (according
the data flow diagram on
https://www.rsyslog.com/doc/v8-stable/whitepapers/queues_analogy.html).
Also thinking that the main queue thread itself is responsible for the
message parsing and manipulation.
Attached is the most of the configuration. It is divided into more files,
thus merged them in order into one file.
Peter
On Thu, Nov 28, 2019 at 2:52 PM David Lang <[email protected]> wrote:
> first off, adding additional threads is probably not going to help, and
> can
> actually hurt (locking contention between the threads)
>
> I would look at increasing the batch size before adding additional threads.
>
> It's hard to know what's going on without seeing your config.
>
> David Lang
>
> On Thu, 28 Nov 2019, Peter Viskup via rsyslog wrote:
>
> > Date: Thu, 28 Nov 2019 09:22:43 +0100
> > From: Peter Viskup via rsyslog <[email protected]>
> > To: rsyslog-users <[email protected]>
> > Cc: Peter Viskup <[email protected]>
> > Subject: [rsyslog] MainQ workerthreads not effective
> >
> > Experiencing high load on some rsyslog instances.
> > Status of threads showed the mainQ thread consumed 50-100% CPU.
> > Change of queue.workerthreads to 2 enabled the second workerthread, but
> > this does not consume any CPU.
> > How are the workerthreads for main queue loaded?
> >
> > Running on Debian 10 with rsyslog 8.1901.0-1.
> >
> > Top output:
> >
> > top - 08:12:50 up 6 days, 17:12, 2 users, load average: 1.25, 0.50,
> 0.46
> > Threads: 14 total, 0 running, 14 sleeping, 0 stopped, 0 zombie
> > %Cpu(s): 14.0 us, 2.9 sy, 0.0 ni, 81.5 id, 0.2 wa, 0.0 hi, 1.4 si,
> > 0.0 st
> > MiB Mem : 7973.2 total, 119.5 free, 341.9 used, 7511.9
> buff/cache
> > MiB Swap: 952.0 total, 825.7 free, 126.2 used. 7273.8 avail
> Mem
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
> >
> > *30437 lognet 20 0 815648 28612 6320 S 66.1 0.4 0:46.40
> > rs:main Q:Reg*30438 lognet 20 0 815648 28612 6320 S 7.0 0.4
> > 0:05.58 rs:ESP02 queue:
> > 30439 lognet 20 0 815648 28612 6320 S 6.3 0.4 0:04.58
> > rs:SIEMEP1 queu
> > 28705 lognet 20 0 815648 28612 6320 S 6.0 0.4 0:04.31
> > in:imtcp
> > 30440 lognet 20 0 815648 28612 6320 S 1.0 0.4 0:00.40
> > rs:net-all.log
> > 30461 lognet 20 0 815648 28612 6320 S 1.0 0.4 0:00.56
> > rs:ESP01-IPS qu
> > .....
> > *30507 lognet 20 0 815648 28612 6320 S 0.0 0.4 0:00.83
> > rs:main Q:Reg*
> >
> > Peter
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> >
>
module(load="impstats"
interval="15"
severity="7"
log.syslog="off"
# need to turn log stream logging off!
log.file="/var/spool/rsyslog/rsyslog.stats")
global(workDirectory="/var/spool/rsyslog")
global(action.reportSuspensionContinuation="on")
global(processInternalMessages="off")
global(abortOnUncleanConfig="on")
$EscapeControlCharactersOnReceive off
$Umask 0027
$FileCreateMode 0640
$DirCreateMode 0750
$CreateDirs off
module(load="imudp")
module(load="imtcp")
# set local variable .localip
$IncludeConfig /etc/rsyslog.d/host/global/includes/localip.inc
$template
getOrigIpOnly,"%hostname:R,ERE,1,BLANK:(^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})$--end%"
$template
getOrigHostOnly,"%hostname:R,ERE,1,BLANK:(^[a-zA-Z]+[a-zA-Z0-9-]+[a-zA-Z0-9.-]*)$--end%"
$template
getOrigHost,"%hostname:R,ERE,1,BLANK:(^[a-zA-Z]+[a-zA-Z0-9-]+[a-zA-Z0-9.-]*)-([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})$--end%"
$template
getOrigIp,"%hostname:R,ERE,2,BLANK:(^[a-zA-Z]+[a-zA-Z0-9-]+[a-zA-Z0-9.-]*)-([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})$--end%"
$template getShorthost,"%fromhost:R,ERE,1,BLANK:(^[a-zA-Z]+[a-zA-Z0-9-]*)--end%"
template(name="getLocalip" type="string" string="%$.localip%")
template(name="getHostwithip" type="string" string="%$.host%-%$.ip%")
template(name="getFromhostip" type="string" string="%fromhost-ip%")
if ( $hostname contains '-' ) then {
set $.origip=exec_template("getOrigIp");
set $.orighost=exec_template("getOrigHost");
if ( $.orighost == "" and $.origip == "" ) then {
set $.orighost=exec_template("getOrigHostOnly");
set $.origip=exec_template("getOrigIpOnly");
}
}
else {
set $.orighost=exec_template("getOrigHostOnly");
set $.origip=exec_template("getOrigIpOnly");
}
if ( not ( $.origip == "" or $.orighost == "" ) and not ( $.origip ==
'127.0.0.1' or $.orighost == "localhost" )) then {
set $.ip=$.origip;
set $.host=$.orighost;
}
else {
if ( $.orighost == "" or $.orighost == "localhost" ) then {
if not ( $.origip == '127.0.0.1' or $.origip == "" ) then {
set $.ip=$.origip;
}
else if not ( $fromhost-ip == '127.0.0.1' ) then {
set $.ip=$fromhost-ip;
}
else {
set $.ip=$.localip;
}
if ( $.origip == $fromhost-ip ) then {
if not ( $fromhost-ip == '127.0.0.1' ) then {
set $.host=exec_template("getShorthost");
}
else {
set $.host=$$myhostname;
}
}
}
if ( $.origip == "" or $.origip == '127.0.0.1' ) then {
# if fromhost startswith orighost
# use origip=fromhost-ip or preserve orighost
if ( $fromhost startswith $.orighost and not ( $.orighost == "" )) then {
set $.ip=$fromhost-ip;
set $.host=$.orighost;
}
else if not ( $fromhost-ip == '127.0.0.1' ) then {
if not ( $.orighost == "" or $.orighost == "localhost" ) then {
set $.host=$.orighost;
}
else {
set $.host=exec_template("getShorthost");
}
}
}
if (( $fromhost == $fromhost-ip or $hostname == $fromhost-ip ) and not (
$fromhost-ip == '127.0.0.1' )) then {
set $.ip=$fromhost-ip;
}
else if (( $hostname == '127.0.0.1' or $hostname == "" ) and $fromhost-ip ==
'127.0.0.1' ) then {
set $.ip=$.localip;
set $.host=$$myhostname;
}
else if (( $hostname == '127.0.0.1' or $hostname == "" ) and not (
$fromhost-ip == '127.0.0.1' )) then {
set $.ip=$fromhost-ip;
}
else if ( $fromhost-ip == '127.0.0.1' ) then {
set $.ip=$.localip;
set $.host=$hostname;
}
#else {
# set $.ip=$fromhost-ip;
#}
}
if not ( $.host == "" or $.ip == "" ) then {
set $.hostwithip=exec_template("getHostwithip");
set $.iporhost=$.ip;
}
else if not ( $.host == "" ) then {
set $.hostwithip=$.host;
set $.iporhost=$.host;
}
else if not ( $.ip == "" ) then {
set $.hostwithip=$.ip;
set $.iporhost=$.ip;
}
$template
getSDOrigTemplate,"%structured-data:R,ERE,1,BLANK:(\[.*\])\[syslogTimes@29171
.*\]$--end%"
$template SDAddSTTemplate,"%structured-data%[syslogTimes@29171
%$.localip%=\"%timegenerated:::date-rfc3339%\"]"
$template setSTTemplate,"[syslogTimes@29171
%$.localip%=\"%timegenerated:::date-rfc3339%\"]"
$template addSTTemplate,"%$.sdstorigstring%
%$.localip%=\"%timegenerated:::date-rfc3339%\"]"
$template
getSTOrigTemplate,"%structured-data:R,ERE,1,DFLT:.*(\[syslogTimes@29171(
[0-9.]+=\"[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(.[0-9]{3,9})?[+-][0-9]{2}:[0-9]{2}\")+)--end%"
if ( $structured-data == "-" ) then {
set $.sdorigstring="";
set $.sdststring=exec_template("setSTTemplate");
set $.sdsiemstring="-";
}
else {
if ( $structured-data contains "[syslogTimes@29171 " ) then {
set $.sdstorigstring=exec_template("getSTOrigTemplate");
set $.sdststring=exec_template("addSTTemplate");
set $.sdorigstring=exec_template("getSDOrigTemplate");
if ( $.sdorigstring == "" ) then {
set $.sdsiemstring="-";
}
unset $.sdstorigstring;
}
else {
set $.sdstring=exec_template("SDAddSTTemplate");
set $.sdststring=exec_template("setSTTemplate");
set $.sdorigstring=$structured-data;
set $.sdsiemstring=$structured-data;
}
}
### some templates here ###
global(maxMessageSize="2K")
$PrivDropToUser lognet
$PrivDropToGroup lognet
$IncludeConfig /etc/rsyslog.d/host/net/forwards/1*.conf
action(type="omfile" file="/var/log/net-all.log"
template="fileStoreSDTemplateOld" asyncWriting="on" ioBufferSize="128K")
# action(type="omfile" file="/var/log/net-dbg.log"
template="RSYSLOG_DebugFormat" asyncWriting="on" ioBufferSize="128K")
$IncludeConfig /etc/rsyslog.d/host/net/forwards/9*.conf
main_queue(
queue.filename="mainq"
queue.size="10000"
queue.maxDiskSpace="1G"
queue.highWatermark="8000"
queue.lowWatermark="7000"
queue.discardSeverity="7"
queue.syncqueuefiles="off"
queue.checkpointInterval="10"
queue.type="LinkedList"
queue.timeoutshutdown="2000"
queue.maxFileSize="5m"
queue.saveOnShutdown="on"
queue.workerthreads="2"
)
$IncludeConfig /etc/rsyslog.d/host/net/listeners/*.conf
### forwards ###
action(type="omfwd" protocol="tcp" target="1.1.2.3" port="9514"
name="ESP01-IPS"
queue.FileName="fq_esp01_ips"
queue.spoolDirectory="/var/spool/rsyslog"
queue.size="1000000"
queue.MaxDiskSpace="32000m"
queue.Type="LinkedList"
queue.HighWaterMark="300000"
queue.LowWaterMark="200000"
template="relay2ForwardTemplate"
)
else
action(type="omfwd" protocol="tcp" target="1.1.2.4" port="1514"
name="ESP02"
queue.FileName="fq_esp02"
queue.spoolDirectory="/var/spool/rsyslog"
queue.size="10000000"
queue.MaxDiskSpace="5000m"
queue.Type="LinkedList"
queue.HighWaterMark="500000"
queue.LowWaterMark="400000"
template="relay2ForwardTemplate"
)
global(
DefaultNetstreamDriverCertFile="/etc/rsyslog.d/host/tls/host-net-cert.pem"
DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/host/tls/host-net-key.pem"
)
global(
DefaultNetstreamDriver="gtls"
DefaultNetstreamDriverCAFile="/etc/rsyslog.d/global/tls/ca.pem"
)
action(type="omfwd" protocol="tcp" target="1.1.2.7" port="6514"
name="SIEMEP1"
queue.FileName="fq_siemep1"
StreamDriver="gtls"
StreamDriverMode="1"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="siem-ep1"
queue.spoolDirectory="/var/spool/rsyslog"
queue.size="10000000"
queue.MaxDiskSpace="5000m"
queue.Type="LinkedList"
queue.HighWaterMark="500000"
queue.LowWaterMark="400000"
template="relay3ForwardTemplate"
)
### listeners ###
input(type="imtcp" port="1514")
input(type="imudp" port="1514")_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
