Hi,
Hi Radu,
I think using JASON at the logtash is not a solution for me, as we cannot
use Ruby code as filter in the JASON. We have to place a lot of ruby code
inside the Ruby filters. I just want to quote an answer -see below, which I
got from logtash ML here which describe what we want to achieve with Ruby
filters in Logtash (from a guy from Microsoft) -- in order to balance the
performance we have Rsyslog with Logtash:
1. Yes, you can imbed a ruby filter
2. Actually, logstash has a small buffer in the pipeline.rb but does
not persist to disk. Some of us have drastically increased this buffer to
great benefit when processing large volumes of logs or dealing with bursts
and output limitations (like rabbitmq).
a. W.r.t. performance, it really depends on what your filters do. We
have around 1000 lines of filter logic, but it has a lot of if/elseif
ordering so that it escapes as soon as possible. We also do large
dictionary lookups (40k entries) and are able to get around 250 msg/sec per
CPU core with this "big" config. We scale out indexers as needed.
You can use also the "translate" plugin in order to simplify the code.
Now how can we do the same in Rsyslog?
We use Redis and RabbitMQ for this work. With a custom script, we can
extend messages easily. Unfortunately Rsyslog has not RabbitMQ module,
we use Logstash as intermediary.
--
Dle
Rainer: Performance is not an issue for Rsyslog but including support for
languages like Ruby might not be possible for Rsyslog.
Thanks
On Fri, Apr 11, 2014 at 11:28 AM, Radu Gheorghe
<[email protected]>wrote:
Hi,
Rainer - here's where Grok seems to live:
https://github.com/jordansissel/grok
It is indeed a library, and there are also other implementations (I know of
one in Ruby and one in Java).
Masoom - I think Richard and David already answered your questions about
what you can use in rsyslog. If that's not enough, please give some
examples of what kind of filtering you need to do.
Best regards,
Radu
On Fri, Apr 11, 2014 at 8:03 AM, Rainer Gerhards
<[email protected]>wrote:
I have yet to look into it, but maybe someone knows if grok is a
stand-alone component. If so, we could probably very easily make it
available in rsyslog via the new external message modification plugin
capability.
Anyone in the know (else I'll try to find out)?
Rainer
On Fri, Apr 11, 2014 at 5:51 AM, David Lang <[email protected]> wrote:
On Fri, 11 Apr 2014, masoom alam wrote:
Very detailed answer. Thanks!!!
Since it is related with both Rsyslog and Logtash, thats why I am
asking
here. After your kind guidance, Its now clear that we should use JASON
template in Rsyslog, and then use JASON in logtash. I did not find any
significant difference at the logtash end regarding Grok and JASON,
except
the word Jason in the filter instead of Grok, am I right? -- I mean as
for
as the syntax is concerned. For the execution it will definitely have
performance gains, as you suggested.
Another thing which I think I did not explain well in my email is that
we
are thinking to place some regex at the Rsyslog end too. Suppose we
have
200 filters defined in Logtash, so will happen that when a log entry
will
arrive at the Logtash, it will have to match it against all the 200
filters
-- worst case, and/or some thing matches earlier and we compose the
configuration file of Logtash in a way that it escapes. Any ideas how
to
optimize the log deep/fancy parsing at this end?
I don't know about the logstash side, but I suspect that you are
correct.
On the rsyslog side, the equivalent would be mmlognorm, and with it the
number of rules doesn't matter because they get compiled into a parse
tree,
you go through the log message once.
David Lang
Once Again thanks Radu. You are very helpful.
On Fri, Apr 11, 2014 at 12:36 AM, Radu Gheorghe
<[email protected]>wrote:
I've never actually tried this, but I think the best way for
performance
is
to send over TCP, but make the template a JSON with everything
rsyslog
can
parse (by default, stuff like severity, date, etc). On the Logstash
side,
you'll use the JSON that should parse much faster than grok can parse
syslog. After that, you'd set the rest of the Logstash filters you
want
to
use for fancy processing.
Also, sending over TCP allows you to use rsyslog for buffering, and
if
you're using in-memory queues (or disk-assisted, assuming those
rarely
spill out to disk), this means you'll avoid the I/O penalty of
writing
to
disks and having Logstash poll from disk periodically.
If you need help with any of those, please write here (or on the
Logstash
ML for the Logstash part, people are really helpful there).
Best regards,
Radu
On Thu, Apr 10, 2014 at 6:13 PM, masoom alam <[email protected]>
wrote:
Is it necessary to fill the templates inside rsyslog so that rsyslog
should
write each log source to a separate file for logtash - will be easy
for
it
for parsing? - also due to the reason logrtash has to catch
rsyslog? .
What
is the alternative if we are doing extensive parsing in logtash? -
simply
directing log on to a port and ask logtash to pick it up - match it
against
200 plugins?
from phone thus brief.
On Apr 10, 2014 5:06 PM, "Radu Gheorghe" <
[email protected]>
wrote:
Here's an article that explains how to configure squeeze
performance
from a
rsyslog>ES>Kibana setup, and the numbers I got (20-30K EPS on my
good-old
laptop): http://www.rsyslog.com/performance-tuning-elasticsearch/
You also have links there about other articles in this are (that
also
have
config snippets and explanations).
On Tue, Apr 8, 2014 at 11:34 PM, Josh Bitto <
[email protected]>
wrote:
If I'm reading this right your saying that you did
Rsyslog->Elasticsearch->gui?
I've tried installing the rpm on centos and it installs but
apparently
it
doesn't come with a config file and so the daemon starts it errors
out
in
the logs and just shuts down after that.
-----Original Message-----
From: [email protected] [mailto:
[email protected]] On Behalf Of Rick Brown
Sent: Tuesday, April 08, 2014 11:31 AM
To: rsyslog-users
Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana
server
Today I've setup my central rsyslog server to replay the logs via
omudpspoof to a logstash server -> ES. It's already indexing
about
twice
as much as just rsyslog -> ES was using the recipe in the first
link
below,
and I haven't even begun to dig into the scads of plugins
available
for
logstash.
http://blog.sematext.com/2013/07/01/recipe-rsyslog-
elasticsearch-kibana/isagoodplace to start, although you can replace
the omelasticsearch OM
with omudpspoof if you want to do logstash.
http://cookbook.logstash.net/recipes/rsyslog-agent/ is a good
place
to
start with rsyslog -> logstash, although I did UDP instead of TCP,
and
used
the elasticsearch output module instead of stdout, which is
documented
here: http://cookbook.logstash.net/recipes/central-syslog/
Good luck to you! Those three links is basically all I needed,
and
should
set you down the right path, regardless of how your path differs
from
mine
;)
----- Original Message -----
From: "Orangepeel Beef" <[email protected]>
To: "rsyslog-users" <[email protected]>
Sent: Tuesday, April 8, 2014 2:17:42 PM
Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana
server
it works, but I find it overly complex for my environment. read:
I
don't need it ;) On Apr 8, 2014 11:13 AM, "Josh Bitto"
<[email protected]> wrote:
I have read about Redis as being the "broker" thoughts?
-----Original Message-----
From: [email protected] [mailto:
[email protected]] On Behalf Of Orangepeel Beef
Sent: Tuesday, April 08, 2014 11:11 AM
To: rsyslog-users
Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana
server
I use rsyslog to pipe into sec, and then use logstash file input
to
index.
could be done without SEC as well. I don't like delivering
syslog
right into logstash.
On Apr 8, 2014 11:09 AM, "Sphonic" <[email protected]>
wrote:
I use rsyslog to send all items to logstash which has a syslog
listener enabled.
Sent from my iPhone
On 8 Apr 2014, at 18:05, Josh Bitto <[email protected]>
wrote:
Hello Everyone,
I'm wanting to setup a syslog server that combines the three
programs
listed above with rsyslog. Has anyone had any success using
this?
I'm
running on a CentOS 6.5 and finding adequate instructions on
how
to not only setup all three PLUS rsyslog has been somewhat of a
challenge.
This issue that I run into is on how to get
logstash/elasticsearch and
kibana to talk with rsyslog. Halp meh! Please!
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhardsNOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
--
Rick Brown
Office of Information Technology
Georgia Institute of Technology
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL:
This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites
beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
