well..... I updated our spare staging server, which gets virtually no logs but has the same config that's been working fairly well for a while, from 8.10 to 8.13 and it's been crashing all night, being restarted by Puppet.
rs:main Q:Reg[14469]: segfault at 7fc31c023000 ip 00007fc34d85cf2c sp 00007fc3418412d0 error 6 in rsyslogd[7fc34d838000+8d000] Error 6 seems to be no such device or address. I think we have a problem here ..... On 10/7/15 1:29 PM, Ciprian Hacman wrote: > I strongly recommend using 8.13 if you push logs to ES. There are many > useful patches since 8.10 (some of them might not be in the changelog > though). > > Ciprian > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > On Wed, Oct 7, 2015 at 9:17 PM, Micah Yoder <[email protected]> > wrote: > >> On 10/7/15 12:11 PM, David Lang wrote: >>> It looks like json-c is not really thread-safe. specifically when using >>> anything that outputs a json structure. Running into problems requires >>> that you have multiple threads working on the same message, but if you >>> have an action queue for your elasticsearch output, that would create >>> the conditions that can trigger this. >> >> That may be part of it, thanks. I get this in dmesg on the staging >> server (8.10): >> >> rs:action-app_e[45178] general protection ip:7fcc56c91633 >> sp:7fcc45bf89d8 error:0 in libjson-c.so.2.0.1[7fcc56c8a000+a000] >> rs:action-app_e[3213]: segfault at 0 ip 00007f538356e633 sp >> 00007f53715f19d8 error 6 in libjson-c.so.2.0.1[7f5383567000+a000] >> >> With the changes I made in the dev server (8.13), here are the dmesg >> outputs: >> >> rs:main Q:Reg[26553]: segfault at 7f0d7009a000 ip 00007f0d9a667f2c sp >> 00007f0d7fbfd2d0 error 6 in rsyslogd[7f0d9a643000+8d000] >> rs:main Q:Reg[26835]: segfault at 7fccc4073000 ip 00007fccf3c67f2c sp >> 00007fcce1bf82d0 error 6 in rsyslogd[7fccf3c43000+8d000] >> rs:main Q:Reg[27566]: segfault at 7f471404a000 ip 00007f473c5e2f2c sp >> 00007f472b1fc2d0 error 6 in rsyslogd[7f473c5be000+8d000] >> rs:main Q:Reg[27855]: segfault at 7fa58002d000 ip 00007fa5a79c1f2c sp >> 00007fa5957f72d0 error 6 in rsyslogd[7fa5a799d000+8d000] >> rs:main Q:Reg[28169]: segfault at 0 ip (null) sp 00007f53b11f73d8 error 14 >> rs:main Q:Reg[31317]: segfault at 0 ip (null) sp 00007f9fd4df63d8 error 14 >> >> No json-c there. >> >> Maybe we should just update to 8.13 in staging, putting my other changes >> off for a bit. We do kind of need them though - currently messages are >> being dropped, we think because the ElasticSearch actions are direct >> queues instead of linked lists, and they are causing bottlenecks >> elsewhere in the system. >> >> Will discuss with my team, thanks! >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
