On 10/7/15 12:11 PM, David Lang wrote: > It looks like json-c is not really thread-safe. specifically when using > anything that outputs a json structure. Running into problems requires > that you have multiple threads working on the same message, but if you > have an action queue for your elasticsearch output, that would create > the conditions that can trigger this.
That may be part of it, thanks. I get this in dmesg on the staging server (8.10): rs:action-app_e[45178] general protection ip:7fcc56c91633 sp:7fcc45bf89d8 error:0 in libjson-c.so.2.0.1[7fcc56c8a000+a000] rs:action-app_e[3213]: segfault at 0 ip 00007f538356e633 sp 00007f53715f19d8 error 6 in libjson-c.so.2.0.1[7f5383567000+a000] With the changes I made in the dev server (8.13), here are the dmesg outputs: rs:main Q:Reg[26553]: segfault at 7f0d7009a000 ip 00007f0d9a667f2c sp 00007f0d7fbfd2d0 error 6 in rsyslogd[7f0d9a643000+8d000] rs:main Q:Reg[26835]: segfault at 7fccc4073000 ip 00007fccf3c67f2c sp 00007fcce1bf82d0 error 6 in rsyslogd[7fccf3c43000+8d000] rs:main Q:Reg[27566]: segfault at 7f471404a000 ip 00007f473c5e2f2c sp 00007f472b1fc2d0 error 6 in rsyslogd[7f473c5be000+8d000] rs:main Q:Reg[27855]: segfault at 7fa58002d000 ip 00007fa5a79c1f2c sp 00007fa5957f72d0 error 6 in rsyslogd[7fa5a799d000+8d000] rs:main Q:Reg[28169]: segfault at 0 ip (null) sp 00007f53b11f73d8 error 14 rs:main Q:Reg[31317]: segfault at 0 ip (null) sp 00007f9fd4df63d8 error 14 No json-c there. Maybe we should just update to 8.13 in staging, putting my other changes off for a bit. We do kind of need them though - currently messages are being dropped, we think because the ElasticSearch actions are direct queues instead of linked lists, and they are causing bottlenecks elsewhere in the system. Will discuss with my team, thanks! _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
