Hi,
So we have a somewhat complex environment involving rules for various
application logs and elasticsearch output generated by Puppet, and some
other parts involving regex extractions.
Our staging environment runs 8.10, and there have been a small number of
segfaults. Maybe one a month.
In the dev environment, I made some changes and now get a segfault
usually within a few seconds of starting rsyslog, even when upgrading to
8.13.
Enabling debug logging and capturing the last part of the output before
the crash, here's what I got:
1205.991657460:main Q:Reg/w0 : eval expr 0x7fccf4f6dfc0, return
datatype 'S'
1205.991665047:main Q:Reg/w0 : SET !vip_name =
1205.991673224:main Q:Reg/w0 : function 'exec_template' (id:12,
params:1)
1205.991685453:main Q:Reg/w0 : string 'extractvip'
1205.991699401:main Q:Reg/w0 : END SET
1205.991707006:main Q:Reg/w0 : eval expr 0x7fccf4f6e190, type 'F[70]'
1205.991710003:main Q:Reg/w0 : rainerscript: executing function id 12
I need to be careful about how much of our environment details I post
here, but here's the code where this comes from:
template(name="extractvip" type="string" string="%syslogtag:14:$%")
set $!vip_name = exec_template("extractvip");
Note that this is code that did NOT change. It's been in our
environment for probably 2 years. I repeated it 3 times, with the same
result. rsyslog crashed with a message like this being at the end of
the line.
I then removed the file with this command (it was included) and ran it
again twice, getting this both times:
2271.865308042:main Q:Reg/w0 : eval expr 0x7f9fe74869d0, return
datatype 'N'
2271.865310862:main Q:Reg/w0 : if condition result is 1
2271.865313611:main Q:Reg/w0 : SET .endpoint =
2271.865320618:main Q:Reg/w0 : function 're_extract' (id:8, params:5)
2271.865332270:main Q:Reg/w0 : var 'msg'
2271.865343109:main Q:Reg/w0 : string '[redacted, VIP F5
partition name]\/([a-zA-Z0-9.]+)'
2271.865492825:main Q:Reg/w0 : 0
2271.865505204:main Q:Reg/w0 : 1
2271.865516558:main Q:Reg/w0 : string ''
2271.865530362:main Q:Reg/w0 : END SET
2271.865538150:main Q:Reg/w0 : eval expr 0x7f9fe74067d0, type 'F[70]'
2271.865541189:main Q:Reg/w0 : rainerscript: executing function id 8
2271.865544581:main Q:Reg/w0 : eval expr 0x7f9fe7408da0, type 'V[86]'
2271.865548299:main Q:Reg/w0 : rainerscript: var 1: '[redacted, access
log line for a VIP]'
2271.865551401:main Q:Reg/w0 : eval expr 0x7f9fe7408da0, return
datatype 'S'
2271.865554481:main Q:Reg/w0 : eval expr 0x7f9fe7408e20, type 'N[78]'
2271.865561495:main Q:Reg/w0 : eval expr 0x7f9fe7408e20, return
datatype 'N'
2271.865564783:main Q:Reg/w0 : eval expr 0x7f9fe7409010, type 'N[78]'
2271.865567815:main Q:Reg/w0 : eval expr 0x7f9fe7409010, return
datatype 'N'
The line that does that is just setting $.endpoint to an re_extract of
$msg. This also did not change.
Both of the things though are included *after* things that did change.
I removed this file also and rsyslog has been running ever since.
What did change is some elasticsearch queues, from direct to disk
assisted and some other minor structure changes.
Any pointers to finding the source of this segfault? I can try to run
it under gdb if that would help, but I'm not great with that.
Thanks!
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
