I have some gdb output following the instructions at the bottom of http://www.rsyslog.com/doc/v8-stable/troubleshooting/troubleshoot.html
Is there someone I can send this to? Thanks! On 10/8/15 9:22 AM, Micah Yoder wrote: > well..... I updated our spare staging server, which gets virtually no > logs but has the same config that's been working fairly well for a > while, from 8.10 to 8.13 and it's been crashing all night, being > restarted by Puppet. > > rs:main Q:Reg[14469]: segfault at 7fc31c023000 ip 00007fc34d85cf2c sp > 00007fc3418412d0 error 6 in rsyslogd[7fc34d838000+8d000] > > Error 6 seems to be no such device or address. > > I think we have a problem here ..... > > On 10/7/15 1:29 PM, Ciprian Hacman wrote: >> I strongly recommend using 8.13 if you push logs to ES. There are many >> useful patches since 8.10 (some of them might not be in the changelog >> though). >> >> Ciprian >> -- >> Performance Monitoring * Log Analytics * Search Analytics >> Solr & Elasticsearch Support * http://sematext.com/ >> >> On Wed, Oct 7, 2015 at 9:17 PM, Micah Yoder <[email protected]> >> wrote: >> >>> On 10/7/15 12:11 PM, David Lang wrote: >>>> It looks like json-c is not really thread-safe. specifically when using >>>> anything that outputs a json structure. Running into problems requires >>>> that you have multiple threads working on the same message, but if you >>>> have an action queue for your elasticsearch output, that would create >>>> the conditions that can trigger this. >>> >>> That may be part of it, thanks. I get this in dmesg on the staging >>> server (8.10): >>> >>> rs:action-app_e[45178] general protection ip:7fcc56c91633 >>> sp:7fcc45bf89d8 error:0 in libjson-c.so.2.0.1[7fcc56c8a000+a000] >>> rs:action-app_e[3213]: segfault at 0 ip 00007f538356e633 sp >>> 00007f53715f19d8 error 6 in libjson-c.so.2.0.1[7f5383567000+a000] >>> >>> With the changes I made in the dev server (8.13), here are the dmesg >>> outputs: >>> >>> rs:main Q:Reg[26553]: segfault at 7f0d7009a000 ip 00007f0d9a667f2c sp >>> 00007f0d7fbfd2d0 error 6 in rsyslogd[7f0d9a643000+8d000] >>> rs:main Q:Reg[26835]: segfault at 7fccc4073000 ip 00007fccf3c67f2c sp >>> 00007fcce1bf82d0 error 6 in rsyslogd[7fccf3c43000+8d000] >>> rs:main Q:Reg[27566]: segfault at 7f471404a000 ip 00007f473c5e2f2c sp >>> 00007f472b1fc2d0 error 6 in rsyslogd[7f473c5be000+8d000] >>> rs:main Q:Reg[27855]: segfault at 7fa58002d000 ip 00007fa5a79c1f2c sp >>> 00007fa5957f72d0 error 6 in rsyslogd[7fa5a799d000+8d000] >>> rs:main Q:Reg[28169]: segfault at 0 ip (null) sp 00007f53b11f73d8 error 14 >>> rs:main Q:Reg[31317]: segfault at 0 ip (null) sp 00007f9fd4df63d8 error 14 >>> >>> No json-c there. >>> >>> Maybe we should just update to 8.13 in staging, putting my other changes >>> off for a bit. We do kind of need them though - currently messages are >>> being dropped, we think because the ElasticSearch actions are direct >>> queues instead of linked lists, and they are causing bottlenecks >>> elsewhere in the system. >>> >>> Will discuss with my team, thanks! >>> >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
