On Wed, 18 Jan 2017, mostolog--- via rsyslog wrote:
what happens after that point?
atop VSIZE=842 / RSIZE=512 remain stuck for rsyslogd, no matter if there's
not traffic anymore.
vsize of 842 is not unreasonable, that sounds like it may be back down to normal
ranges.
does the maxrss keep climbing when you get another burst of data?
But overall, it does look like there is a memory leak somewhere, can you
run a copy of rsyslog somewhere that will allow you to tinker with the
config significantly? change the output to go to a file instead of ES
(using the same template that you are using in ES would be good), and then
see what happens. If the memory leak stops, it's an omelasticsearch issue,
if not, we can try tinkering with the other actions and see what makes a
difference.
Going to do that, as this is actually the testing env.
good. I suspect that without the omelasticsearch, things will be fast enough
that you don't have the queue buildup.
one interesting thing was that at lower traffic rates, it looked like the maxrss
didn't move at all. It would be worth checking on this, does a long stream of
low traffic have the same effect on maxrss as a short stream of high traffic
(same number of log messages in both cases)
Thanks as usual.
module(load="impstats" log.file="/data/stats.log")
syslog.=debug /data/rsyslog-stats
global(
MaxMessageSize="32k"
workDirectory="/data"
parser.escapeControlCharactersOnReceive="off"
)
module(load="imrelp")
input(
port="20514"
type="imrelp"
name="imrelp"
ruleset="relp"
)
template(name="json" type="string" string="%$!%\n")
template(name="index" type="string" string="%$.index%@%$.interval%")
template(name="type" type="string" string="%$.type%")
template(name="ts" type="string" string="%timestamp:::date-rfc3339%")
module(load="mmjsonparse")
module(load="mmnormalize")
module(load="omelasticsearch")
ruleset(
name="relp"
queue.filename="relp"
queue.maxdiskspace="1G"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
just a note, I've found that fixedarray is slightly faster than linkedlist
(which makes sense as it doesn't have to allocate/free memory as much), the
starting size will be a little higher, but under load it should be about the
same.
) {
action(
name="json"
cookie=""
type="mmjsonparse"
)
if $parsesuccess == "FAIL" then {
call error
stop
}
action(
name="norm"
type="mmnormalize"
variable="$!msg"
rulebase="/etc/rsyslog.d/rsyslog.rb"
)
$IncludeConfig /etc/rsyslog.d/apps/conf/1*.conf
$IncludeConfig /etc/rsyslog.d/apps/conf/2*.conf
#there are no 1* neither 2* files
# Set default index and type
set $.index="unknown";
set $.type="unknown";
#defaults
set $.interval=$$now & ":" & $$hour;
if $!app != $!app then {
call unknown
stop
}
$IncludeConfig /etc/rsyslog.d/apps/conf/3*.conf
#a few files like
#else if $!app == "myapp" then {
# set $.index="account-app@" & $$now;
# set $.type="logs";
# call geoip
#}
call clean
set $!host_forwarded=$hostname;
set $!host_received=$$myhostname;
set $!time_received=$timegenerated;
set $@timestamp=exec_template("ts");
action(
name="elastic"
action.resumeRetryCount="-1"
action.reportsuspension="on"
type="omelasticsearch"
server="server"
serverport="9200"
searchIndex="index"
dynSearchIndex="on"
searchType="type"
dynSearchType="on"
template="json"
)
}
$IncludeConfig /etc/rsyslog.d/apps/conf/4*.conf
#a few files like:
#ruleset(name="geoip"){
# if $!ip != "" then {
# set $!geo="true";
# unset $!ip;
# }
#}
module(load="builtin:omfile")
you don't want to load the modules multiple times, so it's best not to have this
in an included file. It also tends to make people think that the load (and any
parameters to the load) only apply to the configs in this file.
ruleset(name="error"){
action(
name="error"
type="omfile"
file="/data/rsyslog-errors.log"
)
}
ruleset(name="unknown"){
action(
name="unk"
type="omfile"
file="/data/rsyslog-unknown.log"
)
}
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
