Can you create an issue here? https://github.com/rsyslog/rsyslog/issues
If you have some easier way to reproduce, it would help also. Ciprian -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Thu, Oct 8, 2015 at 8:07 PM, Micah Yoder <[email protected]> wrote: > I have some gdb output following the instructions at the bottom of > http://www.rsyslog.com/doc/v8-stable/troubleshooting/troubleshoot.html > > Is there someone I can send this to? > > Thanks! > > On 10/8/15 9:22 AM, Micah Yoder wrote: > > well..... I updated our spare staging server, which gets virtually no > > logs but has the same config that's been working fairly well for a > > while, from 8.10 to 8.13 and it's been crashing all night, being > > restarted by Puppet. > > > > rs:main Q:Reg[14469]: segfault at 7fc31c023000 ip 00007fc34d85cf2c sp > > 00007fc3418412d0 error 6 in rsyslogd[7fc34d838000+8d000] > > > > Error 6 seems to be no such device or address. > > > > I think we have a problem here ..... > > > > On 10/7/15 1:29 PM, Ciprian Hacman wrote: > >> I strongly recommend using 8.13 if you push logs to ES. There are many > >> useful patches since 8.10 (some of them might not be in the changelog > >> though). > >> > >> Ciprian > >> -- > >> Performance Monitoring * Log Analytics * Search Analytics > >> Solr & Elasticsearch Support * http://sematext.com/ > >> > >> On Wed, Oct 7, 2015 at 9:17 PM, Micah Yoder <[email protected]> > >> wrote: > >> > >>> On 10/7/15 12:11 PM, David Lang wrote: > >>>> It looks like json-c is not really thread-safe. specifically when > using > >>>> anything that outputs a json structure. Running into problems requires > >>>> that you have multiple threads working on the same message, but if you > >>>> have an action queue for your elasticsearch output, that would create > >>>> the conditions that can trigger this. > >>> > >>> That may be part of it, thanks. I get this in dmesg on the staging > >>> server (8.10): > >>> > >>> rs:action-app_e[45178] general protection ip:7fcc56c91633 > >>> sp:7fcc45bf89d8 error:0 in libjson-c.so.2.0.1[7fcc56c8a000+a000] > >>> rs:action-app_e[3213]: segfault at 0 ip 00007f538356e633 sp > >>> 00007f53715f19d8 error 6 in libjson-c.so.2.0.1[7f5383567000+a000] > >>> > >>> With the changes I made in the dev server (8.13), here are the dmesg > >>> outputs: > >>> > >>> rs:main Q:Reg[26553]: segfault at 7f0d7009a000 ip 00007f0d9a667f2c sp > >>> 00007f0d7fbfd2d0 error 6 in rsyslogd[7f0d9a643000+8d000] > >>> rs:main Q:Reg[26835]: segfault at 7fccc4073000 ip 00007fccf3c67f2c sp > >>> 00007fcce1bf82d0 error 6 in rsyslogd[7fccf3c43000+8d000] > >>> rs:main Q:Reg[27566]: segfault at 7f471404a000 ip 00007f473c5e2f2c sp > >>> 00007f472b1fc2d0 error 6 in rsyslogd[7f473c5be000+8d000] > >>> rs:main Q:Reg[27855]: segfault at 7fa58002d000 ip 00007fa5a79c1f2c sp > >>> 00007fa5957f72d0 error 6 in rsyslogd[7fa5a799d000+8d000] > >>> rs:main Q:Reg[28169]: segfault at 0 ip (null) sp 00007f53b11f73d8 > error 14 > >>> rs:main Q:Reg[31317]: segfault at 0 ip (null) sp 00007f9fd4df63d8 > error 14 > >>> > >>> No json-c there. > >>> > >>> Maybe we should just update to 8.13 in staging, putting my other > changes > >>> off for a bit. We do kind of need them though - currently messages are > >>> being dropped, we think because the ElasticSearch actions are direct > >>> queues instead of linked lists, and they are causing bottlenecks > >>> elsewhere in the system. > >>> > >>> Will discuss with my team, thanks! > >>> > >>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you DON'T LIKE THAT. > >> > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
