Just did. https://github.com/rsyslog/rsyslog/issues/550
Pastebin with gdb info linked therein.
Unfortunately I'm not sure I can get it to a small reproducible config.
David, re: config:
It is a hierarchy of files generated by Puppet to put in several
different elasticsearch rules and application log rules. If you need
the whole thing, I'll have to discuss that with folks here first.
(There could be a case for hiring Adiscon here....)
Here is one of our elastic files:
### Send ssh logs to ElasticSearch!
template(name="idxperday_ssh" type="list") {
constant(value="ssh-")
property(name="timereported" dateFormat="rfc3339" position.from="1"
position.to="4")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="6"
position.to="7")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="9"
position.to="10")
}
if $!event.tags[0] == "ssh" then {
action(type="omelasticsearch" name="action-elastic_ssh"
server="[hostname redacted]" searchIndex="idxperday_ssh"
dynSearchIndex="on" searchType="events" bulkmode="on"
queue.type="linkedlist" queue.filename="ssh" queue.size="200000"
queue.spoolDirectory="/var/log-queues" queue.maxdiskspace="100M"
action.resumeretrycount="-1" queue.timeoutenqueue="0"
queue.dequeuebatchsize="1000" template="all")
}
There are several others like this, and several other things.
You had mentioned separating the templates from the actions, but this
current Puppet generated hierarchy makes that somewhat difficult. In
any case I think that was for the json-c bug, which is something
different I think.
Let me know if the pastebin gdb info makes any sense.....
Thanks!
On 10/8/15 12:10 PM, Ciprian Hacman wrote:
> Can you create an issue here?
> https://github.com/rsyslog/rsyslog/issues
>
> If you have some easier way to reproduce, it would help also.
>
> Ciprian
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
> On Thu, Oct 8, 2015 at 8:07 PM, Micah Yoder <[email protected]>
> wrote:
>
>> I have some gdb output following the instructions at the bottom of
>> http://www.rsyslog.com/doc/v8-stable/troubleshooting/troubleshoot.html
>>
>> Is there someone I can send this to?
>>
>> Thanks!
>>
>> On 10/8/15 9:22 AM, Micah Yoder wrote:
>>> well..... I updated our spare staging server, which gets virtually no
>>> logs but has the same config that's been working fairly well for a
>>> while, from 8.10 to 8.13 and it's been crashing all night, being
>>> restarted by Puppet.
>>>
>>> rs:main Q:Reg[14469]: segfault at 7fc31c023000 ip 00007fc34d85cf2c sp
>>> 00007fc3418412d0 error 6 in rsyslogd[7fc34d838000+8d000]
>>>
>>> Error 6 seems to be no such device or address.
>>>
>>> I think we have a problem here .....
>>>
>>> On 10/7/15 1:29 PM, Ciprian Hacman wrote:
>>>> I strongly recommend using 8.13 if you push logs to ES. There are many
>>>> useful patches since 8.10 (some of them might not be in the changelog
>>>> though).
>>>>
>>>> Ciprian
>>>> --
>>>> Performance Monitoring * Log Analytics * Search Analytics
>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>
>>>> On Wed, Oct 7, 2015 at 9:17 PM, Micah Yoder <[email protected]>
>>>> wrote:
>>>>
>>>>> On 10/7/15 12:11 PM, David Lang wrote:
>>>>>> It looks like json-c is not really thread-safe. specifically when
>> using
>>>>>> anything that outputs a json structure. Running into problems requires
>>>>>> that you have multiple threads working on the same message, but if you
>>>>>> have an action queue for your elasticsearch output, that would create
>>>>>> the conditions that can trigger this.
>>>>>
>>>>> That may be part of it, thanks. I get this in dmesg on the staging
>>>>> server (8.10):
>>>>>
>>>>> rs:action-app_e[45178] general protection ip:7fcc56c91633
>>>>> sp:7fcc45bf89d8 error:0 in libjson-c.so.2.0.1[7fcc56c8a000+a000]
>>>>> rs:action-app_e[3213]: segfault at 0 ip 00007f538356e633 sp
>>>>> 00007f53715f19d8 error 6 in libjson-c.so.2.0.1[7f5383567000+a000]
>>>>>
>>>>> With the changes I made in the dev server (8.13), here are the dmesg
>>>>> outputs:
>>>>>
>>>>> rs:main Q:Reg[26553]: segfault at 7f0d7009a000 ip 00007f0d9a667f2c sp
>>>>> 00007f0d7fbfd2d0 error 6 in rsyslogd[7f0d9a643000+8d000]
>>>>> rs:main Q:Reg[26835]: segfault at 7fccc4073000 ip 00007fccf3c67f2c sp
>>>>> 00007fcce1bf82d0 error 6 in rsyslogd[7fccf3c43000+8d000]
>>>>> rs:main Q:Reg[27566]: segfault at 7f471404a000 ip 00007f473c5e2f2c sp
>>>>> 00007f472b1fc2d0 error 6 in rsyslogd[7f473c5be000+8d000]
>>>>> rs:main Q:Reg[27855]: segfault at 7fa58002d000 ip 00007fa5a79c1f2c sp
>>>>> 00007fa5957f72d0 error 6 in rsyslogd[7fa5a799d000+8d000]
>>>>> rs:main Q:Reg[28169]: segfault at 0 ip (null) sp 00007f53b11f73d8
>> error 14
>>>>> rs:main Q:Reg[31317]: segfault at 0 ip (null) sp 00007f9fd4df63d8
>> error 14
>>>>>
>>>>> No json-c there.
>>>>>
>>>>> Maybe we should just update to 8.13 in staging, putting my other
>> changes
>>>>> off for a bit. We do kind of need them though - currently messages are
>>>>> being dropped, we think because the ElasticSearch actions are direct
>>>>> queues instead of linked lists, and they are causing bottlenecks
>>>>> elsewhere in the system.
>>>>>
>>>>> Will discuss with my team, thanks!
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you DON'T LIKE THAT.
>>>>
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
