Re: [rsyslog] Logstash vs. omelasticsearch
On Fri, 16 Dec 2016, Micah Yoder wrote: On 11/21/2016 05:21 PM, David Lang wrote: On Mon, 21 Nov 2016, Micah Yoder wrote: The other reason I preferred Logstash was the configuration format was a bit more user-friendly than some of the equivalent rsyslog rules. can you provide some more info about the issues you had? Hi David, sorry I was going to reply but didn't right away and got behind! Actually it's been over a year and I don't remember all the specifics. Part of it (most of it probably) were segfaults with mmnormalize and/or mmjsonparse. And they've probably been fixed by now. that was the timeframe when we were battling the json-c issues, so that was probably the cause of the grief there. But since we want maximum stability for our other log messages, we knew then that we wanted to separate this out from our main rsyslog process. The alternatives would have been a secondary rsyslog process or logstash. a separate queue inside rsyslog gives you almost the same separation. I just liked the way logstash config file works a bit more than how you set up rsyslog for this sort of thing. This is what I was trying to probe for. What is it that you liked better about logstash, is it something that we can adapt, or address the problem in a different way that makes it even easier, or ... There were some performance concerns, but logstash is keeping up fine and server load is low. Would I switch back to rsyslog for this processing? In this particular application probably not, because we don't really want to touch it again! :p Would I consider rsyslog in the future for something similar? Probably. Looks like it's come a long way. Especially with the ERK conversations. I like what I'm seeing. Main things are great documentation and easy to read config files. Progress could be made on both Maybe I could jump in on some of the documentation at some point. I once wrote an rsyslog+elasticsearch tutorial that got reposted a couple places (Rackspace dev blog and Puppet blog). It's ancient now though. the documentation can be improved, we know that, so any help there is gratefully accepted :-) in terms of the config file, what is it that you see as being hard to read? (Assuming that the config file is written using the v6+ config syntax, which was created because the adaptation of the legacy config was getting _really_ ugly) I might consider jumping in the code if it were written in modern C++ instead of C. I'm a bit baffled why C is still used, but that will probably get me flamed to a crisp here! :p the project has been around a while, and started life as a fork of a C program, so there's just never been a push (or the manpower) to re-write it in a different language. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Logstash vs. omelasticsearch
On 11/21/2016 05:21 PM, David Lang wrote: On Mon, 21 Nov 2016, Micah Yoder wrote: The other reason I preferred Logstash was the configuration format was a bit more user-friendly than some of the equivalent rsyslog rules. can you provide some more info about the issues you had? Hi David, sorry I was going to reply but didn't right away and got behind! Actually it's been over a year and I don't remember all the specifics. Part of it (most of it probably) were segfaults with mmnormalize and/or mmjsonparse. And they've probably been fixed by now. But since we want maximum stability for our other log messages, we knew then that we wanted to separate this out from our main rsyslog process. The alternatives would have been a secondary rsyslog process or logstash. I just liked the way logstash config file works a bit more than how you set up rsyslog for this sort of thing. There were some performance concerns, but logstash is keeping up fine and server load is low. Would I switch back to rsyslog for this processing? In this particular application probably not, because we don't really want to touch it again! :p Would I consider rsyslog in the future for something similar? Probably. Looks like it's come a long way. Especially with the ERK conversations. I like what I'm seeing. Main things are great documentation and easy to read config files. Progress could be made on both Maybe I could jump in on some of the documentation at some point. I once wrote an rsyslog+elasticsearch tutorial that got reposted a couple places (Rackspace dev blog and Puppet blog). It's ancient now though. I might consider jumping in the code if it were written in modern C++ instead of C. I'm a bit baffled why C is still used, but that will probably get me flamed to a crisp here! :p ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] rsyslogd: gnutls returned error on handshake: A TLS packet with unexpected length was received.
We have the same log messages and are using it in the same way. We have a story in the backlog to investigate, but haven't got to it yet. It doesn't seem to be losing messages. On 12/13/2016 03:31 PM, yingchun cai via rsyslog wrote: Hi, All I use rsyslog-gnutls-8.23.0-1.el6.x86_64rsyslog-8.23.0-1.el6.x86_64and configured with TLS connection. I configured with imptcp as non-tls connection for one port and configure imtcp as tls connection on another port. However, I do get this error message in the log: "rsyslogd: gnutls returned error on handshake: A TLS packet with unexpected length was received. [v8.23.0 try http://www.rsyslog.com/e/2083 ]". Here is the configuration I have for rsyslog:module(load="imudp")input(type="imudp" port="514")module(load="imptcp")input(type="imptcp" port="514") # Provides TCP syslog reception# for parameters see http://www.rsyslog.com/doc/imtcp.htmlmodule(load="imtcp"streamdriver.mode="1"streamdriver.authmode="x509/name"PermittedPeer="*")input(type="imtcp" port="2514" name="tcp-tls") $DefaultNetstreamDriver gtls$DefaultNetstreamDriverCAFile /opt/sec/certs/$DefaultNetstreamDriverCertFile /opt/sec/certs/app_cert.pem$DefaultNetstreamDriverKeyFile /opt/sec/keys/app.key$ActionSendStreamDriverAuthMode x509/name$ActionSendStreamDriverPermittedPeer * Anyone has any idea why I got this error? thanks Yingchun ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Are we building an ERK stack?
This is exactly why we have $. variables as well as $! variables. They work exactly the same, but by convention, $! variables are where you put things that you are going to want to send elsewhere, and $. variables are where you put things that you need to create for your internal logic, templates, etc but don't want to send to the destinatino as part of your log content if you get something that you don't want to send, you can unset $!foo; to remove it from the $! set of data. I didn't know that (if ever read, I forgot). I'll document that on filters.rst :P Still, I'm having some issues with @timestamp. I'll let you know if we found any problem. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] liblognorm segfault ?
Hi
Having more problems with liblognorm. Let me now if I should open an issue.
echo "a" | /usr/lib/lognorm/lognormalizer -r a.rb
Segmentation fault (core dumped)
File:
version=2
#foo
type=@rfc3164pri:<%priority:number%>
type=@rfc3164header:%date:date-rfc3164% %hostname:word%
type=@rfc3164tag:%syslogtag:char-to{"extradata":":"}%:
type=@rfc3164:%.:@rfc3164pri%%.:@rfc3164header% %.:@rfc3164tag%
type=@rfc3164:%.:@rfc3164header% %.:@rfc3164tag%
type=@syslog:%.:@rfc3164%
#bar
#
#it complains liblognorm error: rulebase file a.rb[23]: invalid
record type detected: ']%'
#if written this way:
# {"type":"rest","name":"message"}
#]%
# Theres a blank line at the end of file
rule=:%[
{"type":"@syslog","name":"a"},
{"type":"rest","name":"message"}]%
Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Are we building an ERK stack?
On Thu, 15 Dec 2016, mostolog--- via rsyslog wrote: Solved using json template (code blindness). Is there any way to set fields and use them (like @timestamp) but not indexing them on elastic? (hidden fields) Just tried with @timestamp, but it's being indexed :( This is exactly why we have $. variables as well as $! variables. They work exactly the same, but by convention, $! variables are where you put things that you are going to want to send elsewhere, and $. variables are where you put things that you need to create for your internal logic, templates, etc but don't want to send to the destinatino as part of your log content if you get something that you don't want to send, you can unset $!foo; to remove it from the $! set of data. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
