On Wed, 7 Oct 2015, Micah Yoder wrote:
Hi,
So we have a somewhat complex environment involving rules for various
application logs and elasticsearch output generated by Puppet, and some
other parts involving regex extractions.
Our staging environment runs 8.10, and there have been a small number of
segfaults. Maybe one a month.
It looks like json-c is not really thread-safe. specifically when using anything
that outputs a json structure. Running into problems requires that you have
multiple threads working on the same message, but if you have an action queue
for your elasticsearch output, that would create the conditions that can trigger
this.
There is some additional locking that I think got added just before the release
of 8.13 that reduced this window, but didn't eliminate it.
I've re-written my config file to do all the string creation before calling any
ruleses/actions that have queues
so instead of
$template foo,"%$!%\n"
action(template="foo" queue.type="fixedarray")
I do something like
$template foo,"%$!%\n"
$template bar,"%$,foo%"
set $.bar = exec_template("foo");
action(template="bar" queue.type="fixedarray")
by setting simple variables to the results of the templates before any action
queues are invoked, all the complex json manipulation is done in the main worker
thread.
This is a work-around for the bug, Rainer is working on figuring out the proper
fix. With 8.12 I was seeing this more, 8.13 significantly reduced the window for
the problem. It's possible that my work-around isn't eliminating the problem
entirely either, just reducing the window further, but I've now gone two weeks
without a segfault on a busy server (several thousand logs/sec) that was running
into problems daily for a bit.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
