what does your config look like?
David Lang
On Thu, 8 Oct 2015, Micah Yoder wrote:
Date: Thu, 8 Oct 2015 09:22:23 -0500
From: Micah Yoder <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: [email protected]
Subject: Re: [rsyslog] segfault problems
well..... I updated our spare staging server, which gets virtually no
logs but has the same config that's been working fairly well for a
while, from 8.10 to 8.13 and it's been crashing all night, being
restarted by Puppet.
rs:main Q:Reg[14469]: segfault at 7fc31c023000 ip 00007fc34d85cf2c sp
00007fc3418412d0 error 6 in rsyslogd[7fc34d838000+8d000]
Error 6 seems to be no such device or address.
I think we have a problem here .....
On 10/7/15 1:29 PM, Ciprian Hacman wrote:
I strongly recommend using 8.13 if you push logs to ES. There are many
useful patches since 8.10 (some of them might not be in the changelog
though).
Ciprian
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Wed, Oct 7, 2015 at 9:17 PM, Micah Yoder <[email protected]>
wrote:
On 10/7/15 12:11 PM, David Lang wrote:
It looks like json-c is not really thread-safe. specifically when using
anything that outputs a json structure. Running into problems requires
that you have multiple threads working on the same message, but if you
have an action queue for your elasticsearch output, that would create
the conditions that can trigger this.
That may be part of it, thanks. I get this in dmesg on the staging
server (8.10):
rs:action-app_e[45178] general protection ip:7fcc56c91633
sp:7fcc45bf89d8 error:0 in libjson-c.so.2.0.1[7fcc56c8a000+a000]
rs:action-app_e[3213]: segfault at 0 ip 00007f538356e633 sp
00007f53715f19d8 error 6 in libjson-c.so.2.0.1[7f5383567000+a000]
With the changes I made in the dev server (8.13), here are the dmesg
outputs:
rs:main Q:Reg[26553]: segfault at 7f0d7009a000 ip 00007f0d9a667f2c sp
00007f0d7fbfd2d0 error 6 in rsyslogd[7f0d9a643000+8d000]
rs:main Q:Reg[26835]: segfault at 7fccc4073000 ip 00007fccf3c67f2c sp
00007fcce1bf82d0 error 6 in rsyslogd[7fccf3c43000+8d000]
rs:main Q:Reg[27566]: segfault at 7f471404a000 ip 00007f473c5e2f2c sp
00007f472b1fc2d0 error 6 in rsyslogd[7f473c5be000+8d000]
rs:main Q:Reg[27855]: segfault at 7fa58002d000 ip 00007fa5a79c1f2c sp
00007fa5957f72d0 error 6 in rsyslogd[7fa5a799d000+8d000]
rs:main Q:Reg[28169]: segfault at 0 ip (null) sp 00007f53b11f73d8 error 14
rs:main Q:Reg[31317]: segfault at 0 ip (null) sp 00007f9fd4df63d8 error 14
No json-c there.
Maybe we should just update to 8.13 in staging, putting my other changes
off for a bit. We do kind of need them though - currently messages are
being dropped, we think because the ElasticSearch actions are direct
queues instead of linked lists, and they are causing bottlenecks
elsewhere in the system.
Will discuss with my team, thanks!
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
