Re: [rsyslog] Are we building an ERK stack?

[mostolog--- via rsyslog]

This is exactly why we have $. variables as well as $! variables. They 
work exactly the same, but by convention, $! variables are where you 
put things that you are going to want to send elsewhere, and $. 
variables are where you put things that you need to create for your 
internal logic, templates, etc but don't want to send to the 
destinatino as part of your log content


if you get something that you don't want to send, you can unset $!foo; 
to remove it from the $! set of data.

I didn't know that (if ever read, I forgot).
I'll document that on filters.rst
:P

Still, I'm having some issues with @timestamp. I'll let you know if we 
found any problem.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

On Thu, 15 Dec 2016, mostolog--- via rsyslog wrote:


Solved using json template (code blindness).

Is there any way to set fields and use them (like @timestamp) but not 
indexing them on elastic? (hidden fields)


Just tried with @timestamp, but it's being indexed :(



This is exactly why we have $. variables as well as $! variables. They work 
exactly the same, but by convention, $! variables are where you put things that 
you are going to want to send elsewhere, and $. variables are where you put 
things that you need to create for your internal logic, templates, etc but don't 
want to send to the destinatino as part of your log content


if you get something that you don't want to send, you can unset $!foo; to remove 
it from the $! set of data.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Brian Knox via rsyslog]

I noticed looking through the code that it looks like the error file
routine in omelasticsearch is not tied into the stats system - we use
impstats to monitor our rsyslog pipelines, and having a counter for write
errors would be super useful.

 I've submitted a PR to add the counter:
https://github.com/rsyslog/rsyslog/pull/1331

Cheers,
Brian

On Thu, Dec 15, 2016 at 7:58 AM mostolog--- via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Solved using json template (code blindness).
>
> Is there any way to set fields and use them (like @timestamp) but not
> indexing them on elastic? (hidden fields)
>
> Just tried with @timestamp, but it's being indexed :(
>
>
>
> El 15/12/16 a las 12:32, mosto...@gmail.com escribió:
> > Hi
> >
> >
> > At this moment we are frowarding RELP messages to Elasticsearch using
> > omelasticsearch plugin, but sadly message appears as json instead of
> > storing each properties. eg: message is { "app": "app1"... instead of
> > indexed document having a app property.
> >
> > Should we specify an especial param on rsyslog, a setting on elastic...?
> >
> > Regards
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mostolog--- via rsyslog]

Solved using json template (code blindness).

Is there any way to set fields and use them (like @timestamp) but not 
indexing them on elastic? (hidden fields)


Just tried with @timestamp, but it's being indexed :(



El 15/12/16 a las 12:32, mosto...@gmail.com escribió:

Hi


At this moment we are frowarding RELP messages to Elasticsearch using 
omelasticsearch plugin, but sadly message appears as json instead of 
storing each properties. eg: message is { "app": "app1"... instead of 
indexed document having a app property.


Should we specify an especial param on rsyslog, a setting on elastic...?

Regards


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mostolog--- via rsyslog]

Hi


At this moment we are frowarding RELP messages to Elasticsearch using 
omelasticsearch plugin, but sadly message appears as json instead of 
storing each properties. eg: message is { "app": "app1"... instead of 
indexed document having a app property.


Should we specify an especial param on rsyslog, a setting on elastic...?

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Rainer Gerhards]

2016-12-02 10:31 GMT+01:00 David Lang :
> On Fri, 2 Dec 2016, Bob Gregory wrote:
>
>> I'm not sure that's true in the general case.
>>
>> Of the errors I've had with our elk stack, upward of 95% have been caused
>> by type errors (json field should be an int but is an object); some small
>> handful have failed because a message was truncated somewhere asking the
>> line; a smaller number have failed because somebody hand-crafted json and
>> forgot about a trailing comma or quote.
>> Overwhelmingly, the data aren't corrupted: they were invalid at source in
>> a
>> way that would still allow them to be read as plain Unicode strings.
>>
>> Obviously I accept that given enough data, I'll see more interesting
>> failure modes that need more thought, but reading from the errorfile and
>> pushing to a separate error index would work very well in our environment.
>
>
> I get _really_ nervous about even low probability failure modes in my
> failure paths. Murphy likes me too much :-)
>
> doing it your way, you still have the failedlog messages from your failure
> path that you will need to monitor, so you have reduced the scope of the
> problem, but still have the same basic problem.

FYI: the original intent of the error file was to provide errors in a
way that makes it easy to (semi?) automatically handle them via a
different procedure (which my re-inject them once the problem has been
solved).

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

On Fri, 2 Dec 2016, Bob Gregory wrote:


I'm not sure that's true in the general case.

Of the errors I've had with our elk stack, upward of 95% have been caused
by type errors (json field should be an int but is an object); some small
handful have failed because a message was truncated somewhere asking the
line; a smaller number have failed because somebody hand-crafted json and
forgot about a trailing comma or quote.
Overwhelmingly, the data aren't corrupted: they were invalid at source in a
way that would still allow them to be read as plain Unicode strings.

Obviously I accept that given enough data, I'll see more interesting
failure modes that need more thought, but reading from the errorfile and
pushing to a separate error index would work very well in our environment.


I get _really_ nervous about even low probability failure modes in my failure 
paths. Murphy likes me too much :-)


doing it your way, you still have the failedlog messages from your failure path 
that you will need to monitor, so you have reduced the scope of the problem, but 
still have the same basic problem.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Bob Gregory]

I'm not sure that's true in the general case.

Of the errors I've had with our elk stack, upward of 95% have been caused
by type errors (json field should be an int but is an object); some small
handful have failed because a message was truncated somewhere asking the
line; a smaller number have failed because somebody hand-crafted json and
forgot about a trailing comma or quote.
Overwhelmingly, the data aren't corrupted: they were invalid at source in a
way that would still allow them to be read as plain Unicode strings.

Obviously I accept that given enough data, I'll see more interesting
failure modes that need more thought, but reading from the errorfile and
pushing to a separate error index would work very well in our environment.

On Fri, 2 Dec 2016, 08:43 David Lang,  wrote:

On Fri, 2 Dec 2016, Bob Gregory wrote:

> You may well be able to insert the rejected log into a different index.
> Most of our failed logs are down to a mismatch between the mapping config
> and the fields in json logs.
>
> An error index that treats the whole message as a single blob should work
> fine.

what bytes would need to be escaped?

what if it's invalid unicode junk, etc.

almost by definition we are talking about corrupt data.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

On Fri, 2 Dec 2016, Bob Gregory wrote:


You may well be able to insert the rejected log into a different index.
Most of our failed logs are down to a mismatch between the mapping config
and the fields in json logs.

An error index that treats the whole message as a single blob should work
fine.


what bytes would need to be escaped?

what if it's invalid unicode junk, etc.

almost by definition we are talking about corrupt data.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Bob Gregory]

You may well be able to insert the rejected log into a different index.
Most of our failed logs are down to a mismatch between the mapping config
and the fields in json logs.

An error index that treats the whole message as a single blob should work
fine.

On Fri, 2 Dec 2016, 08:35 mosto...@gmail.com,  wrote:

> El 01/12/16 a las 23:08, David Lang escribió:
> > On Thu, 1 Dec 2016, mosto...@gmail.com wrote:
> >
> > I think that you are going to end up with some grief, if the message
> > could not be insterted into ES for some reason, I think the odds are
> > good that you will find that rawmsg can't be inserted either.
> After sending the email I though the same...
>
> > I would keep the errorfile as a file and look at it periodially. I
> > expect that when you first start things up, you will run into a number
> > of errors, but once you work your way though them, the error rate will
> > be low.
> >
> > Set your monitoring system to monitor the size of the errorfile, and
> > it it starts growing significantly, generate an alert.
> Would love to have a more unattended/XXth century way, if anyone knows.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

El 01/12/16 a las 23:08, David Lang escribió:

On Thu, 1 Dec 2016, mosto...@gmail.com wrote:

I think that you are going to end up with some grief, if the message 
could not be insterted into ES for some reason, I think the odds are 
good that you will find that rawmsg can't be inserted either.

After sending the email I though the same...

I would keep the errorfile as a file and look at it periodially. I 
expect that when you first start things up, you will run into a number 
of errors, but once you work your way though them, the error rate will 
be low.


Set your monitoring system to monitor the size of the errorfile, and 
it it starts growing significantly, generate an alert.

Would love to have a more unattended/XXth century way, if anyone knows.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

On Thu, 1 Dec 2016, mosto...@gmail.com wrote:


Hi Bob.

Today we finally found some time to have an eye on our 
rsyslog-normalizer-indexer which uses omelasticsearch


According to 
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omelasticsearch.html 
indexing parameter *errorfile* helps to store failed indexing attempts.


How do you handle those errors?
We are thinking on

* setting errorfile=file
* imfile ruleset=omelasticsearch
* elastic template like: {index="errors" msg="rawmsg" }, and keep an
  eye on that

What do you think?


I think that you are going to end up with some grief, if the message could not 
be insterted into ES for some reason, I think the odds are good that you will 
find that rawmsg can't be inserted either.


I would keep the errorfile as a file and look at it periodially. I expect that 
when you first start things up, you will run into a number of errors, but once 
you work your way though them, the error rate will be low.


Set your monitoring system to monitor the size of the errorfile, and it it 
starts growing significantly, generate an alert.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

Hi Bob.

Today we finally found some time to have an eye on our 
rsyslog-normalizer-indexer which uses omelasticsearch


According to 
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omelasticsearch.html 
indexing parameter *errorfile* helps to store failed indexing attempts.


How do you handle those errors?
We are thinking on

 * setting errorfile=file
 * imfile ruleset=omelasticsearch
 * elastic template like: {index="errors" msg="rawmsg" }, and keep an
   eye on that

What do you think?


El 24/11/16 a las 12:52, Bob Gregory escribió:

https://io.made.com/blog/rek-it/

I wrote this up earlier.

On Wed, 23 Nov 2016 at 19:38 mosto...@gmail.com  wrote:


Working, spamming mail list and writing on wiki at the same time. A
lovely afternoon...

Please, add your lines: https://github.com/rsyslog/rsyslog/wiki
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Brian Knox]

Seeing people referring to "REK" made me smile this morning. We've been
referring to our setup as REK for quit awhile -
http://www.meetup.com/RVA-Data-Hackers/events/214996202/

I don't really see a reason for redis in the equation unless it's already
part of your infrastructure though.  imkafka is potentially interesting
however. I wrote imczmq for rsyslog so I have a little experience writing
input plugins - if there are others with a need for a such a thing that
would  be a fun collaboration.

Brian

On Wed, Nov 23, 2016 at 6:52 AM Bob Gregory  wrote:

> There've been a few discussions over the last few days that are all
> pointing in the same direction:
>
> * Is it better to use Rsyslog's omelasticsearch rather than pushing to
> logstash?
> * Should we have a minimal log shipper component as distinct from rsyslog's
> processing capabilities?
> * Ought we to have an imhiredis module?
>
> Really what we're talking about is replacing Logstash (and the various
> beats) with rsyslog. I'm perfectly happy with that, Logstash is a
> resource-expensive and fickle beast that spoils my otherwise pristine log
> pipeline, but I do think the community ought to think about whether this is
> the direction they want to take.
>
> For my part, I'm quite happy to help build an imhiredis (and imkafka?)
> module but only if I can actually dogfood it, which means replacing
> Logstash in our own environment.
>
> For that, I'd like to see better support for GeoIP tagging, a Riemann
> output plugin, some better guidance on "failed message queues", etc. etc.
> etc.
>
> Are we jointly interested in building the REK stack and, if so, can we
> start to work out the feature set we're missing, and the documentation we'd
> need for this to work? I'm a little concerned that if we tackle the usecase
> piece-meal, we'll end up with lots of disjointed parts that don't really
> solve the problem: logstash is not an adequate logstash.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[chenlin rao]

https://github.com/rsyslog/rsyslog/pull/1099

2016-11-25 16:46 GMT+08:00 mosto...@gmail.com :

> Thanks!
>
> It's your mmdblookup opensourced?
>
>
> El 25/11/16 a las 03:46, chenlin rao escribió:
>
> re-upload an english version. The content was a little old though.
>>
>> 2016-11-23 22:39 GMT+08:00 mosto...@gmail.com :
>>
>> http://www.slideshare.net/chenryn/elk-stack-at-weibocom
>>>
>>> I NEED the english version :P
>>>
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

Thanks!

It's your mmdblookup opensourced?


El 25/11/16 a las 03:46, chenlin rao escribió:

re-upload an english version. The content was a little old though.

2016-11-23 22:39 GMT+08:00 mosto...@gmail.com :


http://www.slideshare.net/chenryn/elk-stack-at-weibocom

I NEED the english version :P

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[chenlin rao]

No, I don't detect them, just capture them with a special regexp because I
only need to process PHP slowlog. which memory addr appear in the beginning
`\[0x\w+\]`... The use case for slow functions stack without mem addrs can
be found at the slide 25 (pie charts for nested sub terms aggs).

2016-11-25 15:39 GMT+08:00 Rainer Gerhards :

> 2016-11-25 8:26 GMT+01:00 chenlin rao :
> > - rewrite most of mmgrok into mmnormalize+rainerscript. Except PHP
> slowlog
> > only. We want to translate the memory address of each line into "x",
> > but seems can't be done in rsyslog, so a mmexternal here.
>
> focussed question: how exactly do you detect memory address? I ask
> because there is mmanon, which does something similiar to IP
> addresses, and I *think* it could be extended to other objects if only
> we know pricesely what to look for and how to transform it.
>
> Rainer
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Rainer Gerhards]

2016-11-25 8:26 GMT+01:00 chenlin rao :
> - rewrite most of mmgrok into mmnormalize+rainerscript. Except PHP slowlog
> only. We want to translate the memory address of each line into "x",
> but seems can't be done in rsyslog, so a mmexternal here.

focussed question: how exactly do you detect memory address? I ask
because there is mmanon, which does something similiar to IP
addresses, and I *think* it could be extended to other objects if only
we know pricesely what to look for and how to transform it.

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[chenlin rao]

Yes, it's a slide nearly 1.5 years ago. After that, we:

- change to use omkafka + <https://github.com/childe/hangout> instead of
omelasticsearch directly. The reason I have said in another mail days ago.
- rewrite most of mmgrok into mmnormalize+rainerscript. Except PHP slowlog
only. We want to translate the memory address of each line into "x",
but seems can't be done in rsyslog, so a mmexternal here.
- try to use streaming compress with imptcp (between shipper and rsyslog
server), the bandwidth saved about 2/3. But discard msgs in peak every
night. So roll back.

No experiments about ES5 now. The author of hangout above told me ES5.0.0
has some terrible problems(
https://github.com/elastic/elasticsearch/issues/21612
https://github.com/elastic/elasticsearch/issues/21611), so waiting for
upgrade.

2016-11-25 14:58 GMT+08:00 David Lang <da...@lang.hm>:

> reading through the slides, a couple comments.
>
> I've found that queue type FixedArray is slightly (but measureably) faster
> than LinkedList
>
> I suspect that the problems you were running into with slide 52 were the
> json-c threading problems that have now been solved with libjsonfast
>
> I'd be very interested in seeing speed comparisons between lookuptable and
> your mmdblookup
>
> At your log volumes, I expect that creating a sting module (sm*, C version
> of a template definition) would make a noticable performance difference. We
> saw >10% when we changed the default templates to C definitions.
>
> It's a very useful slide deck. How has the 5.x version of ES changed
> things there.
>
> David Lang
>
> On Fri, 25 Nov 2016, chenlin rao wrote:
>
> Date: Fri, 25 Nov 2016 10:46:27 +0800
>> From: chenlin rao <rao.chen...@gmail.com>
>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Are we building an ERK stack?
>>
>>
>> re-upload an english version. The content was a little old though.
>>
>> 2016-11-23 22:39 GMT+08:00 mosto...@gmail.com <mosto...@gmail.com>:
>>
>>
>>> http://www.slideshare.net/chenryn/elk-stack-at-weibocom
>>>
>>> I NEED the english version :P
>>>
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

reading through the slides, a couple comments.

I've found that queue type FixedArray is slightly (but measureably) faster than 
LinkedList


I suspect that the problems you were running into with slide 52 were the json-c 
threading problems that have now been solved with libjsonfast


I'd be very interested in seeing speed comparisons between lookuptable and your 
mmdblookup


At your log volumes, I expect that creating a sting module (sm*, C version of a 
template definition) would make a noticable performance difference. We saw >10% 
when we changed the default templates to C definitions.


It's a very useful slide deck. How has the 5.x version of ES changed things 
there.


David Lang

On Fri, 25 Nov 2016, chenlin rao wrote:


Date: Fri, 25 Nov 2016 10:46:27 +0800
From: chenlin rao <rao.chen...@gmail.com>
Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
To: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Are we building an ERK stack?

re-upload an english version. The content was a little old though.

2016-11-23 22:39 GMT+08:00 mosto...@gmail.com <mosto...@gmail.com>:



http://www.slideshare.net/chenryn/elk-stack-at-weibocom

I NEED the english version :P

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Rainer Gerhards]

2016-11-25 0:45 GMT+01:00 matthew.gaetano :
> Which would be a lot easier to do if we didn't have to rely solely on message
> count to delimit time or percentages.

IMHO it would be an interesting experiment to create a queue mode
"redis" or "kafka". Given everything that's going on right now, there
almost for sure is not time to do the experiment, but that's something
a thought abot for a while (0mq might be an even better choice).

I just couldn't stand dispense that idea ;-)

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[chenlin rao]

re-upload an english version. The content was a little old though.

2016-11-23 22:39 GMT+08:00 mosto...@gmail.com :

>
> http://www.slideshare.net/chenryn/elk-stack-at-weibocom
>
> I NEED the english version :P
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[matthew.gaetano]

Which would be a lot easier to do if we didn't have to rely solely on message
count to delimit time or percentages.



-
~Regards

Matthew Gaetano
--
View this message in context: 
http://rsyslog-users.1305293.n2.nabble.com/Are-we-building-an-ERK-stack-tp7591564p7591674.html
Sent from the rsyslog-users mailing list archive at Nabble.com.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

On Thu, 24 Nov 2016, matthew.gaetano wrote:


Using the Rebind Interval and TCP load balancing has its limits. The higher
the volume and velocity the harder it becomes too balance overall. The
interval can not be too high as it risks overloading a single node in a
cluster. The interval can not be too low as it risks overhead with the
opening and closing of TCP connections.


in theory yes, in practice, I'm not so sure. I've had no problems using the 
rebind interval process at over 100K messages/sec load balanced across 20 
machines.


Now, I did this by running rsyslog on the destination machines and then having 
it deliver the messages to the local process that were the final destination. 
Rsyslog was easily able to receive and buffer the bursts.


I aim to have the rebind interval for N destinations be ~1/N to 1/2n seconds

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[matthew.gaetano]

Using the Rebind Interval and TCP load balancing has its limits. The higher
the volume and velocity the harder it becomes too balance overall. The
interval can not be too high as it risks overloading a single node in a
cluster. The interval can not be too low as it risks overhead with the
opening and closing of TCP connections.

Message based load balancing would present a more uniform spread amongst a
clustered destination. It would also mean not having to reset the TCP
connections as often, or at all. This is where message broker applications
like Redis or Kafka come into play.




-
~Regards

Matthew Gaetano
--
View this message in context: 
http://rsyslog-users.1305293.n2.nabble.com/Are-we-building-an-ERK-stack-tp7591564p7591672.html
Sent from the rsyslog-users mailing list archive at Nabble.com.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

El 24/11/16 a las 17:42, Rainer Gerhards escribió:

I added a project to rsyslog on github, where we can bind Issues to:
shouldn't that work for documentation? (as commented previously, I would 
love to have 1 repo!)



https://github.com/rsyslog/rsyslog/projects/1

I guess I must make the relationship, just let me know what you think
qualifies whenever you open something new.
Maybe you could match project to milestones, but I think projects are 
"wider".


BTW: With Bob's approval, I think a good application name can be TREK 
(trekkies could become angry :P)




Rainer

2016-11-24 13:27 GMT+01:00 Rainer Gerhards :

2016-11-24 12:52 GMT+01:00 Bob Gregory :

https://io.made.com/blog/rek-it/

I wrote this up earlier.

very good! Love to see the work coming in and participate in the effort!

Rainer

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

On Thu, 24 Nov 2016, mosto...@gmail.com wrote:

not really, but we haven't had anyone experiment with thousands of them, so 
it's possible, but unlikely that there would be a measureable slowdown as 
rsyslog finds the right one to use.


The bigger overhead is in interpreting the template, that's where 
simplifying it to be $! or $!foo would be a big win (or writing a string 
module)


Memory went above 5GB for our first dirty try (several rulesets, several 
queues...). I'll change that soon.


probably all the queues.

David Lang
kk
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Rainer Gerhards]

I added a project to rsyslog on github, where we can bind Issues to:

https://github.com/rsyslog/rsyslog/projects/1

I guess I must make the relationship, just let me know what you think
qualifies whenever you open something new.

Rainer

2016-11-24 13:27 GMT+01:00 Rainer Gerhards :
> 2016-11-24 12:52 GMT+01:00 Bob Gregory :
>> https://io.made.com/blog/rek-it/
>>
>> I wrote this up earlier.
>
> very good! Love to see the work coming in and participate in the effort!
>
> Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

or docker swarm mode :D


El 24/11/16 a las 16:22, David Lang escribió:

On Thu, 24 Nov 2016, mosto...@gmail.com wrote:

As we are concerned about high availability and load balancing, we 
plan to deploy multiple instances.


just a note that while rsyslog doesn't implement load balancing 
itself, it has features to support load balancing environments, so you 
pick the load balancer you want on the receiving end and have rsyslog 
disconnect every X messages to give the load balancer a chance to work.


I think this only works if you do IP based load balancing, rather than 
DNS based load balancing (especially as so many systems now run a 
caching DNS locally)


Personally, I use corosync (clusterlabs.org) but you can also use 
haproxy, lvs, or a commercial load balancer like f5


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

not really, but we haven't had anyone experiment with thousands of 
them, so it's possible, but unlikely that there would be a measureable 
slowdown as rsyslog finds the right one to use.


The bigger overhead is in interpreting the template, that's where 
simplifying it to be $! or $!foo would be a big win (or writing a 
string module)


Memory went above 5GB for our first dirty try (several rulesets, several 
queues...). I'll change that soon.


there isn't a good writeup, but if you read on how to use the maxmind 
database, the perl example has you create an array where the first 
element is the decimal equivalent of the first IP address that matches 
the data.


This is exactly the structure that a sparse array lookup table is 
intended for. I beleive there is a function that will take an IPv4 
address and return a decimal number (if not, we need to add one). Use 
that function to create a number, lookup the number in the lookup 
table, and have it return the data.


The second paragraph is correct, however I haven't used them yet in 
rsyslog. I'll document them then.


Thanks a lot, David, for your kind help, experienced comments and wise 
advice.

You deserve another prize ;)
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

On Thu, 24 Nov 2016, mosto...@gmail.com wrote:

As we are concerned about high availability and load balancing, we plan to 
deploy multiple instances.


just a note that while rsyslog doesn't implement load balancing itself, it has 
features to support load balancing environments, so you pick the load balancer 
you want on the receiving end and have rsyslog disconnect every X messages to 
give the load balancer a chance to work.


I think this only works if you do IP based load balancing, rather than DNS based 
load balancing (especially as so many systems now run a caching DNS locally)


Personally, I use corosync (clusterlabs.org) but you can also use haproxy, lvs, 
or a commercial load balancer like f5


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Rainer Gerhards]

2016-11-24 12:52 GMT+01:00 Bob Gregory :
> https://io.made.com/blog/rek-it/
>
> I wrote this up earlier.

very good! Love to see the work coming in and participate in the effort!

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

Doing the **same** here.

Currently I'm dealing with https://github.com/rsyslog/rsyslog/issues/625 
in order to have "one configuration file for each application", and 
copying them to rsyslog.d directory.


As we are concerned about high availability and load balancing, we plan 
to deploy multiple instances.
Still pending to decide if RELP->ES is done by the same rsyslog process 
or spplited in several stages.


Any discussion is much appreciated and highly valuable :)


El 24/11/16 a las 12:52, Bob Gregory escribió:

https://io.made.com/blog/rek-it/

I wrote this up earlier.

On Wed, 23 Nov 2016 at 19:38 mosto...@gmail.com  wrote:


Working, spamming mail list and writing on wiki at the same time. A
lovely afternoon...

Please, add your lines: https://github.com/rsyslog/rsyslog/wiki
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Bob Gregory]

https://io.made.com/blog/rek-it/

I wrote this up earlier.

On Wed, 23 Nov 2016 at 19:38 mosto...@gmail.com  wrote:

> Working, spamming mail list and writing on wiki at the same time. A
> lovely afternoon...
>
> Please, add your lines: https://github.com/rsyslog/rsyslog/wiki
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

Working, spamming mail list and writing on wiki at the same time. A 
lovely afternoon...


Please, add your lines: https://github.com/rsyslog/rsyslog/wiki
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

string cat is the simple part, the problem is then being able to treat the 
result as a real timestamp (including outputting it in different formats)


there was a small thread on this today.

David Lang

On Wed, 23 Nov 2016, Rainer Gerhards wrote:


Merge Looks like String concat, which I think we support (but I may be
wrong).

Sent from phone, thus brief.

Am 23.11.2016 19:51 schrieb "David Lang" :


On Wed, 23 Nov 2016, mosto...@gmail.com wrote:

you can combine variables to form a string that looks like a date in the

output, but you can't take arbitrary date parts in a log message and parse
them into a real timestamp field that would let you output it in different
formats.


back on my pipeline proposal, wouldn't this solve the issue?
   pipeline {
   input()
   processor() //extract %year%,%month%,%day%
   processor() //merge "%year%:%month%:%day%" as date type
property/field
   output()
   }



you don't need to invent pipelines and change how rsyslog processes
things, you need need to add the merge function.

The problem is the fact that there are so many ways timestamp data can be
scattered in a log message. take a look at the output of date --help and
look at all the formatting options. I guarantee that some log somewhere
will use every one of them.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

The problem is the fact that there are so many ways timestamp data can 
be scattered in a log message. take a look at the output of date 
--help and look at all the formatting options. I guarantee that some 
log somewhere will use every one of them.


IIRC, you had found a solution to this...
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Rainer Gerhards]

Merge Looks like String concat, which I think we support (but I may be
wrong).

Sent from phone, thus brief.

Am 23.11.2016 19:51 schrieb "David Lang" :

> On Wed, 23 Nov 2016, mosto...@gmail.com wrote:
>
> you can combine variables to form a string that looks like a date in the
>>> output, but you can't take arbitrary date parts in a log message and parse
>>> them into a real timestamp field that would let you output it in different
>>> formats.
>>>
>> back on my pipeline proposal, wouldn't this solve the issue?
>>pipeline {
>>input()
>>processor() //extract %year%,%month%,%day%
>>processor() //merge "%year%:%month%:%day%" as date type
>> property/field
>>output()
>>}
>>
>
> you don't need to invent pipelines and change how rsyslog processes
> things, you need need to add the merge function.
>
> The problem is the fact that there are so many ways timestamp data can be
> scattered in a log message. take a look at the output of date --help and
> look at all the formatting options. I guarantee that some log somewhere
> will use every one of them.
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

On Wed, 23 Nov 2016, mosto...@gmail.com wrote:

you can combine variables to form a string that looks like a date in the 
output, but you can't take arbitrary date parts in a log message and parse 
them into a real timestamp field that would let you output it in different 
formats.

back on my pipeline proposal, wouldn't this solve the issue?
   pipeline {
   input()
   processor() //extract %year%,%month%,%day%
   processor() //merge "%year%:%month%:%day%" as date type 
property/field

   output()
   }


you don't need to invent pipelines and change how rsyslog processes things, you 
need need to add the merge function.


The problem is the fact that there are so many ways timestamp data can be 
scattered in a log message. take a look at the output of date --help and look at 
all the formatting options. I guarantee that some log somewhere will use every 
one of them.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

you can combine variables to form a string that looks like a date in 
the output, but you can't take arbitrary date parts in a log message 
and parse them into a real timestamp field that would let you output 
it in different formats.

back on my pipeline proposal, wouldn't this solve the issue?
pipeline {
input()
processor() //extract %year%,%month%,%day%
processor() //merge "%year%:%month%:%day%" as date type 
property/field

output()
}

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

On Wed, 23 Nov 2016, mosto...@gmail.com wrote:


The ugly way to do this would be a series of

if $programname = "group/appX" then set $.owner = 
"\"group\":\"group\","\"unit\":\"unit\",\"app\":\"appX\",";


do having multiple templates affect performance? (what I really noticed it's 
they affect loading time!)


not really, but we haven't had anyone experiment with thousands of them, so it's 
possible, but unlikely that there would be a measureable slowdown as rsyslog 
finds the right one to use.


The bigger overhead is in interpreting the template, that's where simplifying it 
to be $! or $!foo would be a big win (or writing a string module)




a far more elegant way to do this would be to do a table lookup on the 
programname and have it return the string.
I have readed about how lookup tables can be used for geoIP. Could you 
provide a link to doc where there's an example?


there isn't a good writeup, but if you read on how to use the maxmind database, 
the perl example has you create an array where the first element is the decimal 
equivalent of the first IP address that matches the data.


This is exactly the structure that a sparse array lookup table is intended for. 
I beleive there is a function that will take an IPv4 address and return a 
decimal number (if not, we need to add one). Use that function to create a 
number, lookup the number in the lookup table, and have it return the data.


you can also simplify the template a bit. Instead of crafting the json in 
the template, create a variable that has what you want in/under it and 
output that variable. but compared to collapsing all the templates 
together, that's a minor change :-)



one variable for each file and one template which use it, isnt it?


$!foo!bar = "abc" maps to {"foo": { "bar": "abc" } } in json and if you put 
%$!foo% in a template, what you will get is '{ "bar": "abc" }'


so where you had group, unit, app, msg, and a couple other things, and then 
combined them with {}," into a json string, you could instead do


set $!foo!group = "A";
set $!foo!unit = "b";
set $!foo!msg = $!msg;
...

and then replace all that hard-to-read json construction in the template with 
$!foo



I question the value of having a separate sending queue for each app. I 
think it's better to send them in one combined firehose and split them on 
the receiving side. It makes it less disruptive when you find you want to 
change the groupings of things and all those queues on the sender can eat 
up a lot of ram.



Probably this is because i came from redis.


That's what I'm thinking. With logstash you are forced to use something external 
for queueing and lots of separate instances (and separate parser sets) or things 
just don't work well.


With rsyslog, the performance is 100-1000x as fast, and a lot of the stuff is 
built-in, so you don't need to split things up as much, and the reduction in the 
communications overhead adds to your wins.


Talking about elastic, probably ingest node would be the best option, while 
having index name as metadata.


There are two approaches, and I haven't tried them under fire on a ES cluster to 
know which is the best.


dedicate a node to ingest the data

spread the traffic across many different nodes and have a local copy of rsyslog 
receive the data and push it into the local ES instance.


I suspect that properly managed, a dedicated injest node will be a win.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

On Wed, 23 Nov 2016, mosto...@gmail.com wrote:

In order to improve the first draft of ERK project, I would like to get some 
feedback from you.


What features are you missing/you think it may be improved in rsyslog?

Please, try to be as more clear/self-explanatory/simple as you can for better 
understanding.


* logstash memory footprint is quite high compared to rsyslog,
  although both "doing the same".


that's not something to fix in rsyslog :-)


* rsyslog configuration can't be reloaded live


true


* dynamic variables (calculated on each message processing) aren't
  supported on templates


false. that's what templates do. You can use any variable in a template.


* combine multiple variables into one to build a "date" field isn't
  possible


you can combine variables to form a string that looks like a date in the output, 
but you can't take arbitrary date parts in a log message and parse them into a 
real timestamp field that would let you output it in different formats.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

The ugly way to do this would be a series of

if $programname = "group/appX" then set $.owner = 
"\"group\":\"group\","\"unit\":\"unit\",\"app\":\"appX\",";
do having multiple templates affect performance? (what I really noticed 
it's they affect loading time!)



a far more elegant way to do this would be to do a table lookup on the 
programname and have it return the string.
I have readed about how lookup tables can be used for geoIP. Could you 
provide a link to doc where there's an example?


you can also simplify the template a bit. Instead of crafting the json 
in the template, create a variable that has what you want in/under it 
and output that variable. but compared to collapsing all the templates 
together, that's a minor change :-)

one variable for each file and one template which use it, isnt it?

I question the value of having a separate sending queue for each app. 
I think it's better to send them in one combined firehose and split 
them on the receiving side. It makes it less disruptive when you find 
you want to change the groupings of things and all those queues on the 
sender can eat up a lot of ram.

Probably this is because i came from redis.
Talking about elastic, probably ingest node would be the best option, 
while having index name as metadata.


I'll have an eye on that too.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

Hi all

In order to improve the first draft of ERK project, I would like to get 
some feedback from you.


What features are you missing/you think it may be improved in rsyslog?

Please, try to be as more clear/self-explanatory/simple as you can for 
better understanding.


 * logstash memory footprint is quite high compared to rsyslog,
   although both "doing the same".
 * rsyslog configuration can't be reloaded live
 * dynamic variables (calculated on each message processing) aren't
   supported on templates
 * combine multiple variables into one to build a "date" field isn't
   possible

Regards

PS: those with deep knowledge, please, start thinking how you'll solve 
them...


El 23/11/16 a las 12:52, Bob Gregory escribió:

There've been a few discussions over the last few days that are all
pointing in the same direction:

* Is it better to use Rsyslog's omelasticsearch rather than pushing to
logstash?
* Should we have a minimal log shipper component as distinct from rsyslog's
processing capabilities?
* Ought we to have an imhiredis module?

Really what we're talking about is replacing Logstash (and the various
beats) with rsyslog. I'm perfectly happy with that, Logstash is a
resource-expensive and fickle beast that spoils my otherwise pristine log
pipeline, but I do think the community ought to think about whether this is
the direction they want to take.

For my part, I'm quite happy to help build an imhiredis (and imkafka?)
module but only if I can actually dogfood it, which means replacing
Logstash in our own environment.

For that, I'd like to see better support for GeoIP tagging, a Riemann
output plugin, some better guidance on "failed message queues", etc. etc.
etc.

Are we jointly interested in building the REK stack and, if so, can we
start to work out the feature set we're missing, and the documentation we'd
need for this to work? I'm a little concerned that if we tackle the usecase
piece-meal, we'll end up with lots of disjointed parts that don't really
solve the problem: logstash is not an adequate logstash.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

On Wed, 23 Nov 2016, Rainer Gerhards wrote:


wiki.rsyslog.com

But I think it has not been updated for a while. You may need an account,
if so, let me know.

I am not sure if the wiki is the best place to do it. We think about
retiring it for a while, it was mainly a spam dump...

While I haven't tried it, a GitHub wiki might be better, especially from a
visibility pov. I could enable it if there are no objections. I don't know
though how granular the GitHub premissons are.


anything that's on the old rsyslog wiki is rather out of date at this point. I 
think turning on the github wiki and trying it would be a good move.


David Lang


Rainer

Sent from phone, thus brief.

Am 23.11.2016 16:05 schrieb "mosto...@gmail.com" :


Of course it will help.

Let me take the risk: is there a rsyslog wiki where I could start
documenting what ERK should look like? :P

markdown is mandatory.


El 23/11/16 a las 15:57, Rainer Gerhards escribió:


Would the capability to add metadata in imfile input help? That would be
easy to add. If not, what would give you the Mets data?

Rainer

Sent from phone, thus brief.

Am 23.11.2016 15:51 schrieb "mosto...@gmail.com" :

there are probably ways to simplify the configs, 5K lines of configs seems



excessive :-) how much of this is rulebase config vs rsyslog config?

Each app generates app-access.log, app-tomcat.log, app-application.log

files. imfile allow me to add filename as metadata, but nothing more.
As each application belongs to a workgroup, part of an organizational
unit
and is running on some (multiple) hosts, at the end I have approximately
this for each app:

template(name="json_appX" type="list") {
 property(name="hostname")
 constant(value=" ")
 property(name="syslogtag")
 constant(value=" {")
constant(value="\"group\":\"group\","\"unit\":\"unit\",\"app
\":\"appX\",")
 constant(value="\",\"file\":\"")
 property(name="$!metadata!filename")
 constant(value="\",\"msg\":\"")
 property(name="msg" format="jsonr")
 constant(value="\"}")
}
ruleset(name="json_appX") {
 action(
 template="json_appX"
 type="omrelp"
 target="server"
 port="20514"
 action.resumeRetryCount="-1"
 action.reportSuspension="on"
 queue.maxdiskspace="5M"
 queue.type="LinkedList"
 queue.filename="appX.qi"
 queue.SaveOnShutdown="on"
 )
}
input(type="imfile" file="/logs/appX/access.log" tag="group/appX"
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
input(type="imfile" file="/logs/appX/tomcat.log" tag="group/appX"
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
input(type="imfile" file="/logs/appX/application.log" tag="group/appX"
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")


which becomes 5K lines of config file.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

___

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

On Wed, 23 Nov 2016, mosto...@gmail.com wrote:

there are probably ways to simplify the configs, 5K lines of configs seems 
excessive :-) how much of this is rulebase config vs rsyslog config?


Each app generates app-access.log, app-tomcat.log, app-application.log files. 
imfile allow me to add filename as metadata, but nothing more.
As each application belongs to a workgroup, part of an organizational unit 
and is running on some (multiple) hosts, at the end I have approximately this 
for each app:


template(name="json_appX" type="list") {
   property(name="hostname")
   constant(value=" ")
   property(name="syslogtag")
   constant(value=" {")
constant(value="\"group\":\"group\","\"unit\":\"unit\",\"app\":\"appX\",")
   constant(value="\",\"file\":\"")
   property(name="$!metadata!filename")
   constant(value="\",\"msg\":\"")
   property(name="msg" format="jsonr")
   constant(value="\"}")
}



ruleset(name="json_appX") {
   action(
   template="json_appX"
   type="omrelp"
   target="server"
   port="20514"
   action.resumeRetryCount="-1"
   action.reportSuspension="on"
   queue.maxdiskspace="5M"
   queue.type="LinkedList"
   queue.filename="appX.qi"
   queue.SaveOnShutdown="on"
   )
}


input(type="imfile" file="/logs/appX/access.log" tag="group/appX" 
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")


input(type="imfile" file="/logs/appX/tomcat.log" tag="group/appX" 
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")


input(type="imfile" file="/logs/appX/application.log" tag="group/appX" 
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")


you should be able to collapse all the different templates into one. Instead of 
hard-coding the group/unit/app in each template, you should have that be a 
variable that you set.


The ugly way to do this would be a series of

if $programname = "group/appX" then set $.owner = 
"\"group\":\"group\","\"unit\":\"unit\",\"app\":\"appX\",";

statements.

a far more elegant way to do this would be to do a table lookup on the 
programname and have it return the string.


you can also simplify the template a bit. Instead of crafting the json in the 
template, create a variable that has what you want in/under it and output that 
variable. but compared to collapsing all the templates together, that's a minor 
change :-)



I question the value of having a separate sending queue for each app. I think 
it's better to send them in one combined firehose and split them on the 
receiving side. It makes it less disruptive when you find you want to change the 
groupings of things and all those queues on the sender can eat up a lot of ram.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Rainer Gerhards]

wiki.rsyslog.com

But I think it has not been updated for a while. You may need an account,
if so, let me know.

I am not sure if the wiki is the best place to do it. We think about
retiring it for a while, it was mainly a spam dump...

While I haven't tried it, a GitHub wiki might be better, especially from a
visibility pov. I could enable it if there are no objections. I don't know
though how granular the GitHub premissons are.

Rainer

Sent from phone, thus brief.

Am 23.11.2016 16:05 schrieb "mosto...@gmail.com" :

> Of course it will help.
>
> Let me take the risk: is there a rsyslog wiki where I could start
> documenting what ERK should look like? :P
>
> markdown is mandatory.
>
>
> El 23/11/16 a las 15:57, Rainer Gerhards escribió:
>
>> Would the capability to add metadata in imfile input help? That would be
>> easy to add. If not, what would give you the Mets data?
>>
>> Rainer
>>
>> Sent from phone, thus brief.
>>
>> Am 23.11.2016 15:51 schrieb "mosto...@gmail.com" :
>>
>> there are probably ways to simplify the configs, 5K lines of configs seems
>>>
 excessive :-) how much of this is rulebase config vs rsyslog config?

 Each app generates app-access.log, app-tomcat.log, app-application.log
>>> files. imfile allow me to add filename as metadata, but nothing more.
>>> As each application belongs to a workgroup, part of an organizational
>>> unit
>>> and is running on some (multiple) hosts, at the end I have approximately
>>> this for each app:
>>>
>>> template(name="json_appX" type="list") {
>>>  property(name="hostname")
>>>  constant(value=" ")
>>>  property(name="syslogtag")
>>>  constant(value=" {")
>>> constant(value="\"group\":\"group\","\"unit\":\"unit\",\"app
>>> \":\"appX\",")
>>>  constant(value="\",\"file\":\"")
>>>  property(name="$!metadata!filename")
>>>  constant(value="\",\"msg\":\"")
>>>  property(name="msg" format="jsonr")
>>>  constant(value="\"}")
>>> }
>>> ruleset(name="json_appX") {
>>>  action(
>>>  template="json_appX"
>>>  type="omrelp"
>>>  target="server"
>>>  port="20514"
>>>  action.resumeRetryCount="-1"
>>>  action.reportSuspension="on"
>>>  queue.maxdiskspace="5M"
>>>  queue.type="LinkedList"
>>>  queue.filename="appX.qi"
>>>  queue.SaveOnShutdown="on"
>>>  )
>>> }
>>> input(type="imfile" file="/logs/appX/access.log" tag="group/appX"
>>> addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
>>> input(type="imfile" file="/logs/appX/tomcat.log" tag="group/appX"
>>> addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
>>> input(type="imfile" file="/logs/appX/application.log" tag="group/appX"
>>> addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
>>>
>>>
>>> which becomes 5K lines of config file.
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

Of course it will help.

Let me take the risk: is there a rsyslog wiki where I could start 
documenting what ERK should look like? :P


markdown is mandatory.


El 23/11/16 a las 15:57, Rainer Gerhards escribió:

Would the capability to add metadata in imfile input help? That would be
easy to add. If not, what would give you the Mets data?

Rainer

Sent from phone, thus brief.

Am 23.11.2016 15:51 schrieb "mosto...@gmail.com" :


there are probably ways to simplify the configs, 5K lines of configs seems

excessive :-) how much of this is rulebase config vs rsyslog config?


Each app generates app-access.log, app-tomcat.log, app-application.log
files. imfile allow me to add filename as metadata, but nothing more.
As each application belongs to a workgroup, part of an organizational unit
and is running on some (multiple) hosts, at the end I have approximately
this for each app:

template(name="json_appX" type="list") {
 property(name="hostname")
 constant(value=" ")
 property(name="syslogtag")
 constant(value=" {")
constant(value="\"group\":\"group\","\"unit\":\"unit\",\"app\":\"appX\",")
 constant(value="\",\"file\":\"")
 property(name="$!metadata!filename")
 constant(value="\",\"msg\":\"")
 property(name="msg" format="jsonr")
 constant(value="\"}")
}
ruleset(name="json_appX") {
 action(
 template="json_appX"
 type="omrelp"
 target="server"
 port="20514"
 action.resumeRetryCount="-1"
 action.reportSuspension="on"
 queue.maxdiskspace="5M"
 queue.type="LinkedList"
 queue.filename="appX.qi"
 queue.SaveOnShutdown="on"
 )
}
input(type="imfile" file="/logs/appX/access.log" tag="group/appX"
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
input(type="imfile" file="/logs/appX/tomcat.log" tag="group/appX"
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
input(type="imfile" file="/logs/appX/application.log" tag="group/appX"
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")


which becomes 5K lines of config file.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Rainer Gerhards]

Would the capability to add metadata in imfile input help? That would be
easy to add. If not, what would give you the Mets data?

Rainer

Sent from phone, thus brief.

Am 23.11.2016 15:51 schrieb "mosto...@gmail.com" :

>
> there are probably ways to simplify the configs, 5K lines of configs seems
>> excessive :-) how much of this is rulebase config vs rsyslog config?
>>
>
> Each app generates app-access.log, app-tomcat.log, app-application.log
> files. imfile allow me to add filename as metadata, but nothing more.
> As each application belongs to a workgroup, part of an organizational unit
> and is running on some (multiple) hosts, at the end I have approximately
> this for each app:
>
> template(name="json_appX" type="list") {
> property(name="hostname")
> constant(value=" ")
> property(name="syslogtag")
> constant(value=" {")
> constant(value="\"group\":\"group\","\"unit\":\"unit\",\"app\":\"appX\",")
> constant(value="\",\"file\":\"")
> property(name="$!metadata!filename")
> constant(value="\",\"msg\":\"")
> property(name="msg" format="jsonr")
> constant(value="\"}")
> }
> ruleset(name="json_appX") {
> action(
> template="json_appX"
> type="omrelp"
> target="server"
> port="20514"
> action.resumeRetryCount="-1"
> action.reportSuspension="on"
> queue.maxdiskspace="5M"
> queue.type="LinkedList"
> queue.filename="appX.qi"
> queue.SaveOnShutdown="on"
> )
> }
> input(type="imfile" file="/logs/appX/access.log" tag="group/appX"
> addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
> input(type="imfile" file="/logs/appX/tomcat.log" tag="group/appX"
> addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
> input(type="imfile" file="/logs/appX/application.log" tag="group/appX"
> addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
>
>
> which becomes 5K lines of config file.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

there are probably ways to simplify the configs, 5K lines of configs 
seems excessive :-) how much of this is rulebase config vs rsyslog config?


Each app generates app-access.log, app-tomcat.log, app-application.log 
files. imfile allow me to add filename as metadata, but nothing more.
As each application belongs to a workgroup, part of an organizational 
unit and is running on some (multiple) hosts, at the end I have 
approximately this for each app:


template(name="json_appX" type="list") {
property(name="hostname")
constant(value=" ")
property(name="syslogtag")
constant(value=" {")
constant(value="\"group\":\"group\","\"unit\":\"unit\",\"app\":\"appX\",")
constant(value="\",\"file\":\"")
property(name="$!metadata!filename")
constant(value="\",\"msg\":\"")
property(name="msg" format="jsonr")
constant(value="\"}")
}
ruleset(name="json_appX") {
action(
template="json_appX"
type="omrelp"
target="server"
port="20514"
action.resumeRetryCount="-1"
action.reportSuspension="on"
queue.maxdiskspace="5M"
queue.type="LinkedList"
queue.filename="appX.qi"
queue.SaveOnShutdown="on"
)
}
input(type="imfile" file="/logs/appX/access.log" tag="group/appX" 
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
input(type="imfile" file="/logs/appX/tomcat.log" tag="group/appX" 
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
input(type="imfile" file="/logs/appX/application.log" tag="group/appX" 
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")



which becomes 5K lines of config file.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

http://www.slideshare.net/chenryn/elk-stack-at-weibocom

I NEED the english version :P
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

On Wed, 23 Nov 2016, mosto...@gmail.com wrote:


Our current scenario (dockerized!):

  imfile_forwarder-->imrelp-->rsyslog-->redis-->logstash(grok+geoip)-->elastic

We are using redis as memory buffer and to split into multiple channels/lists 
(using dynakey ATM). We see kafka on the horizon.


We are also using several logstash containers to balance load, prevent single 
point of failure, etc.


What we're thinking after past days messages:

  imfile_forwarder-->imrelp-->rsyslog-->elastic

Having multiple rsyslog instances with simpler configs (instead of 5k lines 
with thousand of rulesets, templates and so), being able to geoip, reliable 
queues...


I wont dare to say it's time to review/refactor rsyslog, but 
maybe...https://www.youtube.com/watch?v=0O5h4enjrHw


there are probably ways to simplify the configs, 5K lines of configs seems 
excessive :-) how much of this is rulebase config vs rsyslog config?


Rsyslog is designed to be fast and supports a lot of threading options for speed 
(most defined implicitly by the creation of queues), so you should not need to 
have lots of different instances.


I've had single instances of rsyslog processing 100K messages/sec in real-world 
use, and people have benchmarked rsyslog with simple configs at over 1M 
messages/sec in a VM


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[chenlin rao]

ERK +1, I have published my experiement at
http://www.slideshare.net/chenryn/elk-stack-at-weibocom
rsyslog-imsock
  -> rsyslog-omfwd
-> rsyslog-imptcp
  ->
rsyslog-mmnormalize/rsyslog-mmgrok/rsyslog-mmdblookup/rsyslog-mmfields/rainerscripts...
-> rsyslog-omkafka -> kafka -> hangout -> es cluster
-> rsyslog-omprog -> python scripts -> zabbix

I had open my rsyslog-mmdblookup for geoip2 lookup, then david lang tell me
this can be done with lookup_table function. I think there should be a good
article about this great function and geoip lookup practice.

2016-11-23 21:20 GMT+08:00 Rainer Gerhards :

> 2016-11-23 13:19 GMT+01:00 mosto...@gmail.com :
> > +1
> >
> >
> > Our current scenario (dockerized!):
> >
> >
> > imfile_forwarder-->imrelp-->rsyslog-->redis-->logstash(
> grok+geoip)-->elastic
> >
> > We are using redis as memory buffer and to split into multiple
> > channels/lists (using dynakey ATM). We see kafka on the horizon.
> >
> > We are also using several logstash containers to balance load, prevent
> > single point of failure, etc.
> >
> > What we're thinking after past days messages:
> >
> >imfile_forwarder-->imrelp-->rsyslog-->elastic
> >
> > Having multiple rsyslog instances with simpler configs (instead of 5k
> lines
> > with thousand of rulesets, templates and so), being able to geoip,
> reliable
> > queues...
> >
> > I wont dare to say it's time to review/refactor rsyslog, but
> > maybe...https://www.youtube.com/watch?v=0O5h4enjrHw
> >
>
> refactoring per se is not a problem, we just need to keep it in
> managable pieces. We had big refactoring almost every year :-)
>
> Rainer
>
> >
> > El 23/11/16 a las 12:52, Bob Gregory escribió:
> >
> >> There've been a few discussions over the last few days that are all
> >> pointing in the same direction:
> >>
> >> * Is it better to use Rsyslog's omelasticsearch rather than pushing to
> >> logstash?
> >> * Should we have a minimal log shipper component as distinct from
> >> rsyslog's
> >> processing capabilities?
> >> * Ought we to have an imhiredis module?
> >>
> >> Really what we're talking about is replacing Logstash (and the various
> >> beats) with rsyslog. I'm perfectly happy with that, Logstash is a
> >> resource-expensive and fickle beast that spoils my otherwise pristine
> log
> >> pipeline, but I do think the community ought to think about whether this
> >> is
> >> the direction they want to take.
> >>
> >> For my part, I'm quite happy to help build an imhiredis (and imkafka?)
> >> module but only if I can actually dogfood it, which means replacing
> >> Logstash in our own environment.
> >>
> >> For that, I'd like to see better support for GeoIP tagging, a Riemann
> >> output plugin, some better guidance on "failed message queues", etc.
> etc.
> >> etc.
> >>
> >> Are we jointly interested in building the REK stack and, if so, can we
> >> start to work out the feature set we're missing, and the documentation
> >> we'd
> >> need for this to work? I'm a little concerned that if we tackle the
> >> usecase
> >> piece-meal, we'll end up with lots of disjointed parts that don't really
> >> solve the problem: logstash is not an adequate logstash.
> >> ___
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T
> >> LIKE THAT.
> >
> >
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> > LIKE THAT.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[David Lang]

On Wed, 23 Nov 2016, Bob Gregory wrote:


There've been a few discussions over the last few days that are all
pointing in the same direction:

* Is it better to use Rsyslog's omelasticsearch rather than pushing to
logstash?
* Should we have a minimal log shipper component as distinct from rsyslog's
processing capabilities?
* Ought we to have an imhiredis module?

Really what we're talking about is replacing Logstash (and the various
beats) with rsyslog. I'm perfectly happy with that, Logstash is a
resource-expensive and fickle beast that spoils my otherwise pristine log
pipeline, but I do think the community ought to think about whether this is
the direction they want to take.


if you look at the graphic on the main page of rsyslog.com you see that we have 
a very large number of inputs and outputs. We already have omelasticsearch, and 
onhiredis, adding an imhiredis just adds symetry to things and is not a large 
deviation


Rsyslog is a log processing engine that accepts logs from many sources and 
delivers them to many destinations, the more sources and destinations we support 
the better.



For my part, I'm quite happy to help build an imhiredis (and imkafka?)
module but only if I can actually dogfood it, which means replacing
Logstash in our own environment.


good, we are aiming to make that not only possible, but a generally accepted 
practice :-)



For that, I'd like to see better support for GeoIP tagging, a Riemann
output plugin, some better guidance on "failed message queues", etc. etc.
etc.


for GeoIP tagging, take a look at the table lookup capability. It was designed 
with the maxmind GeoIP database in mind.


what do you mena by  a Riemann output plugin


Are we jointly interested in building the REK stack and, if so, can we
start to work out the feature set we're missing, and the documentation we'd
need for this to work? I'm a little concerned that if we tackle the usecase
piece-meal, we'll end up with lots of disjointed parts that don't really
solve the problem: logstash is not an adequate logstash.


We are always interested in expanding rsyslog to fill in gaps in routing and 
formatting logs, we try to avoid getting involved in analyzing and summarizing 
logs (but do a bit of that), leaving that job for other tools.


Please do list the things you think are missing.

Documentation is always needed. Unfortunantly, too many of us deep in the guts 
of rsyslog are bad at writing docs.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Rainer Gerhards]

2016-11-23 13:19 GMT+01:00 mosto...@gmail.com :
> +1
>
>
> Our current scenario (dockerized!):
>
>
> imfile_forwarder-->imrelp-->rsyslog-->redis-->logstash(grok+geoip)-->elastic
>
> We are using redis as memory buffer and to split into multiple
> channels/lists (using dynakey ATM). We see kafka on the horizon.
>
> We are also using several logstash containers to balance load, prevent
> single point of failure, etc.
>
> What we're thinking after past days messages:
>
>imfile_forwarder-->imrelp-->rsyslog-->elastic
>
> Having multiple rsyslog instances with simpler configs (instead of 5k lines
> with thousand of rulesets, templates and so), being able to geoip, reliable
> queues...
>
> I wont dare to say it's time to review/refactor rsyslog, but
> maybe...https://www.youtube.com/watch?v=0O5h4enjrHw
>

refactoring per se is not a problem, we just need to keep it in
managable pieces. We had big refactoring almost every year :-)

Rainer

>
> El 23/11/16 a las 12:52, Bob Gregory escribió:
>
>> There've been a few discussions over the last few days that are all
>> pointing in the same direction:
>>
>> * Is it better to use Rsyslog's omelasticsearch rather than pushing to
>> logstash?
>> * Should we have a minimal log shipper component as distinct from
>> rsyslog's
>> processing capabilities?
>> * Ought we to have an imhiredis module?
>>
>> Really what we're talking about is replacing Logstash (and the various
>> beats) with rsyslog. I'm perfectly happy with that, Logstash is a
>> resource-expensive and fickle beast that spoils my otherwise pristine log
>> pipeline, but I do think the community ought to think about whether this
>> is
>> the direction they want to take.
>>
>> For my part, I'm quite happy to help build an imhiredis (and imkafka?)
>> module but only if I can actually dogfood it, which means replacing
>> Logstash in our own environment.
>>
>> For that, I'd like to see better support for GeoIP tagging, a Riemann
>> output plugin, some better guidance on "failed message queues", etc. etc.
>> etc.
>>
>> Are we jointly interested in building the REK stack and, if so, can we
>> start to work out the feature set we're missing, and the documentation
>> we'd
>> need for this to work? I'm a little concerned that if we tackle the
>> usecase
>> piece-meal, we'll end up with lots of disjointed parts that don't really
>> solve the problem: logstash is not an adequate logstash.
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Rainer Gerhards]

2016-11-23 12:52 GMT+01:00 Bob Gregory :
> There've been a few discussions over the last few days that are all
> pointing in the same direction:
>
> * Is it better to use Rsyslog's omelasticsearch rather than pushing to
> logstash?
> * Should we have a minimal log shipper component as distinct from rsyslog's
> processing capabilities?
> * Ought we to have an imhiredis module?
>
> Really what we're talking about is replacing Logstash (and the various
> beats) with rsyslog. I'm perfectly happy with that, Logstash is a
> resource-expensive and fickle beast that spoils my otherwise pristine log
> pipeline, but I do think the community ought to think about whether this is
> the direction they want to take.
>
> For my part, I'm quite happy to help build an imhiredis (and imkafka?)
> module but only if I can actually dogfood it, which means replacing
> Logstash in our own environment.
>
> For that, I'd like to see better support for GeoIP tagging, a Riemann
> output plugin, some better guidance on "failed message queues", etc. etc.
> etc.
>
> Are we jointly interested in building the REK stack and, if so, can we
> start to work out the feature set we're missing, and the documentation we'd
> need for this to work? I'm a little concerned that if we tackle the usecase
> piece-meal, we'll end up with lots of disjointed parts that don't really
> solve the problem: logstash is not an adequate logstash.

I am really extremely interested in this proposal and would appreciate
if we could go forward with it. Just let me explain my situation a
bit,which hopefully helps to understand how I act and what are my
limits. I don't like disappointed people, and so I think talking about
limits is essential to get to an agreement. Sorry that the posting is
a bit length!

I am with Adiscon, and Adiscon still sponsors most of the development
for rsyslog. Adiscon is a very small shop (less than 10 folks) and we
do have a big budget. That's fine with all of us, as we do not aim at
getting rich but aim at having a satisfactory and happy life, which is
unequal to being rich in our PoV ;) We still need to pay bills, and so
we a) sell closed-source Windows products and b) sell consulting and
support contracts.

Rsyslog revenue is small, it typically (barely) funds me and half a
support engineer. I put in quite a bit of my free time as I am
personally interested in this project. Besides rsyslog, I also have
some other appointments, for example I am currently working towards
two academic research projects, where one is targeted towards logging.

Development-wise, this boils down to me being the development
ressource, and often not at 100%. If we receive sponsored or custom
work, I can add development ressources inside Adiscon, so this
actually increases development capability.

More important is that Adiscon does not monetize rsyslog in any other
way: we do not sell appliances, we do not offer logging as a service
and we do not run a large network that we monitor with rsyslog. We
really do one thing (development and support for rsyslog) and we do
that thing well.

Among others, this means we do not have need for Kibana, redis, kafka,
... So we also do not use it. So we do not know it. And learning
*everything* just to develop rsyslog is out of reach giving the
ressources we have.

So far the reality check. The good news is the rsyslog community. It
may not be the fastest growing open source community on earth, but it
is very healthy and very knowledgable. And we have seen good, quality
growth especially in the past two years. We have a lot of different
talents, and we have folks that actually use all these subsystems that
Adiscon doesn't even know before someone asked a question.

As a community, I think we can make the ERK stack a reality. I am very
open to changing things, and rsyslog has been refactored more than
once since it's inception. Another round is not a problem.

If the community helps to shape what actually *needs* to be done
(leaving out the "nice to have" to go to a doable workload), and if
some folks inside the community help to implement it, I think we can
come very far, and can even do so quickly. What is now hopefully
obvious from my initial remarks is that I *alone* cannot do all of the
big hauling. But again, we had great contributions and we have great
contributors! So, yes we can ;-)

For example and to be honest, I frankly admit that I didn't know about
Riemann until 10 minutes ago. So developing any integration into it
will take a lot of time first learning and understanding how it works.
This usually is prohibitive expensive for me to do. If, however, we
have someone who already knows the ins and outs, we can either work
together on getting something done (with me doing the rsyslog bits),
or I can educate that person to know the bare minimum required to
integrate into rsyslog. Rsyslog integration is not very hard if you do
not insist on knowing every detail. And I can fine-tune it afterwards.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

+1


Our current scenario (dockerized!):

   imfile_forwarder-->imrelp-->rsyslog-->redis-->logstash(grok+geoip)-->elastic

We are using redis as memory buffer and to split into multiple 
channels/lists (using dynakey ATM). We see kafka on the horizon.


We are also using several logstash containers to balance load, prevent 
single point of failure, etc.


What we're thinking after past days messages:

   imfile_forwarder-->imrelp-->rsyslog-->elastic

Having multiple rsyslog instances with simpler configs (instead of 5k 
lines with thousand of rulesets, templates and so), being able to geoip, 
reliable queues...


I wont dare to say it's time to review/refactor rsyslog, but 
maybe...https://www.youtube.com/watch?v=0O5h4enjrHw



El 23/11/16 a las 12:52, Bob Gregory escribió:

There've been a few discussions over the last few days that are all
pointing in the same direction:

* Is it better to use Rsyslog's omelasticsearch rather than pushing to
logstash?
* Should we have a minimal log shipper component as distinct from rsyslog's
processing capabilities?
* Ought we to have an imhiredis module?

Really what we're talking about is replacing Logstash (and the various
beats) with rsyslog. I'm perfectly happy with that, Logstash is a
resource-expensive and fickle beast that spoils my otherwise pristine log
pipeline, but I do think the community ought to think about whether this is
the direction they want to take.

For my part, I'm quite happy to help build an imhiredis (and imkafka?)
module but only if I can actually dogfood it, which means replacing
Logstash in our own environment.

For that, I'd like to see better support for GeoIP tagging, a Riemann
output plugin, some better guidance on "failed message queues", etc. etc.
etc.

Are we jointly interested in building the REK stack and, if so, can we
start to work out the feature set we're missing, and the documentation we'd
need for this to work? I'm a little concerned that if we tackle the usecase
piece-meal, we'll end up with lots of disjointed parts that don't really
solve the problem: logstash is not an adequate logstash.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.