On Thu, 8 Oct 2015, Micah Yoder wrote:
Just did. https://github.com/rsyslog/rsyslog/issues/550
Pastebin with gdb info linked therein.
Unfortunately I'm not sure I can get it to a small reproducible config.
David, re: config:
It is a hierarchy of files generated by Puppet to put in several
different elasticsearch rules and application log rules. If you need
the whole thing, I'll have to discuss that with folks here first.
(There could be a case for hiring Adiscon here....)
for the record, I'm not an Adiscon employee.
Here is one of our elastic files:
### Send ssh logs to ElasticSearch!
template(name="idxperday_ssh" type="list") {
constant(value="ssh-")
property(name="timereported" dateFormat="rfc3339" position.from="1"
position.to="4")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="6"
position.to="7")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="9"
position.to="10")
}
if $!event.tags[0] == "ssh" then {
action(type="omelasticsearch" name="action-elastic_ssh"
server="[hostname redacted]" searchIndex="idxperday_ssh"
dynSearchIndex="on" searchType="events" bulkmode="on"
queue.type="linkedlist" queue.filename="ssh" queue.size="200000"
queue.spoolDirectory="/var/log-queues" queue.maxdiskspace="100M"
action.resumeretrycount="-1" queue.timeoutenqueue="0"
queue.dequeuebatchsize="1000" template="all")
}
There are several others like this, and several other things.
You had mentioned separating the templates from the actions, but this
current Puppet generated hierarchy makes that somewhat difficult. In
any case I think that was for the json-c bug, which is something
different I think.
json-c has problems that cause internal datastructures to be corrupted, the
resulting corruption causes segfaults at a later time, which may or may not end
up being identified in the failure as json-c related.
even using puppet, I think you can separate out the template generation
worst case, create a separate include directory that only includes the templates
and the code that evaluates the templates into variables, then your actions in
the main include directory can use those.
But if your various ES things mostly use the template "all" (which is what I
would expect), then you may be able to do something much simpler than you are
thinking. In my case it turned out that I only had 3 templates that ended up
getting used in all the different rules.
the only segfault problems we know of in 8.13 are related to json-c
David Lang
Let me know if the pastebin gdb info makes any sense.....
Thanks!
On 10/8/15 12:10 PM, Ciprian Hacman wrote:
Can you create an issue here?
https://github.com/rsyslog/rsyslog/issues
If you have some easier way to reproduce, it would help also.
Ciprian
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Thu, Oct 8, 2015 at 8:07 PM, Micah Yoder <[email protected]>
wrote:
I have some gdb output following the instructions at the bottom of
http://www.rsyslog.com/doc/v8-stable/troubleshooting/troubleshoot.html
Is there someone I can send this to?
Thanks!
On 10/8/15 9:22 AM, Micah Yoder wrote:
well..... I updated our spare staging server, which gets virtually no
logs but has the same config that's been working fairly well for a
while, from 8.10 to 8.13 and it's been crashing all night, being
restarted by Puppet.
rs:main Q:Reg[14469]: segfault at 7fc31c023000 ip 00007fc34d85cf2c sp
00007fc3418412d0 error 6 in rsyslogd[7fc34d838000+8d000]
Error 6 seems to be no such device or address.
I think we have a problem here .....
On 10/7/15 1:29 PM, Ciprian Hacman wrote:
I strongly recommend using 8.13 if you push logs to ES. There are many
useful patches since 8.10 (some of them might not be in the changelog
though).
Ciprian
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Wed, Oct 7, 2015 at 9:17 PM, Micah Yoder <[email protected]>
wrote:
On 10/7/15 12:11 PM, David Lang wrote:
It looks like json-c is not really thread-safe. specifically when
using
anything that outputs a json structure. Running into problems requires
that you have multiple threads working on the same message, but if you
have an action queue for your elasticsearch output, that would create
the conditions that can trigger this.
That may be part of it, thanks. I get this in dmesg on the staging
server (8.10):
rs:action-app_e[45178] general protection ip:7fcc56c91633
sp:7fcc45bf89d8 error:0 in libjson-c.so.2.0.1[7fcc56c8a000+a000]
rs:action-app_e[3213]: segfault at 0 ip 00007f538356e633 sp
00007f53715f19d8 error 6 in libjson-c.so.2.0.1[7f5383567000+a000]
With the changes I made in the dev server (8.13), here are the dmesg
outputs:
rs:main Q:Reg[26553]: segfault at 7f0d7009a000 ip 00007f0d9a667f2c sp
00007f0d7fbfd2d0 error 6 in rsyslogd[7f0d9a643000+8d000]
rs:main Q:Reg[26835]: segfault at 7fccc4073000 ip 00007fccf3c67f2c sp
00007fcce1bf82d0 error 6 in rsyslogd[7fccf3c43000+8d000]
rs:main Q:Reg[27566]: segfault at 7f471404a000 ip 00007f473c5e2f2c sp
00007f472b1fc2d0 error 6 in rsyslogd[7f473c5be000+8d000]
rs:main Q:Reg[27855]: segfault at 7fa58002d000 ip 00007fa5a79c1f2c sp
00007fa5957f72d0 error 6 in rsyslogd[7fa5a799d000+8d000]
rs:main Q:Reg[28169]: segfault at 0 ip (null) sp 00007f53b11f73d8
error 14
rs:main Q:Reg[31317]: segfault at 0 ip (null) sp 00007f9fd4df63d8
error 14
No json-c there.
Maybe we should just update to 8.13 in staging, putting my other
changes
off for a bit. We do kind of need them though - currently messages are
being dropped, we think because the ElasticSearch actions are direct
queues instead of linked lists, and they are causing bottlenecks
elsewhere in the system.
Will discuss with my team, thanks!
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
