Re: [rsyslog] mmnormalize, mmjsonparse keyspace mapping

[David Lang]

Just a few notes on this,


$! in a template will include all variables under $!, so you don't have to go 
through listing them all one at a time. This also lets you add additional 
metadata (when did the relay receive the message, what IP was it from, which 
relay processed the message, etc) by just doing


set $!trusted!relay = $$myhostname;

or things like this. I like to put all the metadata added by rsyslog under 
$!trusted so that it isn't going to conflict with other data.



When you are load balancing, you probably want to configure the rebindinterval, 
this tells rsyslog to disconnect and reconnect every X messages. Load balancers 
can't move long-running connections from one destination to another, they can 
only move new connections. If everything is stable and running for a long time, 
and you add a destination serve to the pool, nothing will start sending to it 
until things disconnect. Rebind interval makes this happen as frequently as you 
need it to. I like to have it rebind every second or so under high load (with 
the thinking that systems should be able to handle a seconds worth of being 
overloaded and recover in a reasonable timeframe)


David Lang





On Fri, 8 Apr 2016, Matt Ford wrote:


Thanks for the help guys - I got my prototype working.  I wrote up
some stuff about the final design here
https://www.bashton.com/blog/2016/elk-on-ark/

On 7 April 2016 at 19:27, David Lang  wrote:

On Thu, 7 Apr 2016, Matt Ford wrote:


One thing that makes me nervous is the overhead of doing the
transformation via normalize on rsyslog.  Some the applications
generate a lot of logs per second.



mmnormalize is very fast, and it's getting faster as it transitions from
json-c to libfastjson and optimizations hit the library.

David Lang

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize, mmjsonparse keyspace mapping

[David Lang]

On Thu, 7 Apr 2016, Matt Ford wrote:


One thing that makes me nervous is the overhead of doing the
transformation via normalize on rsyslog.  Some the applications
generate a lot of logs per second.


mmnormalize is very fast, and it's getting faster as it transitions from json-c 
to libfastjson and optimizations hit the library.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize, mmjsonparse keyspace mapping

[singh.janmejay]

I have a convention of making loggers slap a special tag "obj@"
where  = a key based namespace.
So obj@w3 would have website logs, obj@oms would have
order-management-system logs etc.
I extract the ns, and generate objects in the form: {host: ...,
severity: , obj..x : ..., obj..y: ... }

So  allows users to not conflict across un-expected contexts.
Turns out, when users want to retrieve logs, in almost all scenarios
they want logs from a specific . Infact fields that have similar
names, have very different semantics across contexts, eg. status field
for web-server access logs is http-status, whereas in a stage-machine
log event is current-state, so pushing them in the same field makes
little sense.

On Thu, Apr 7, 2016 at 11:26 PM, Matt Ford  wrote:
> Hi, yes I've gotten fairly deep into ES.  We would like to be able to
> search over and perhaps even calculate based on fields with the same
> name and different types.  What's more we don't know when developers
> will add services and new json logs.  Without the rsyslog change - as
> far as I can see the only way to do this to have per app indexes (as
> per the tip at the very bottom of this page
> https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping.html).
>
> This is all cool stuff!  Please keep the ideas coming :-)
>
> One thing that makes me nervous is the overhead of doing the
> transformation via normalize on rsyslog.  Some the applications
> generate a lot of logs per second.
>
> On 7 April 2016 at 18:38, Dave Caplinger  
> wrote:
>> On Apr 7, 2016, at 12:04 PM, Matt Ford  wrote:
>>>
>>> Thanks for the help thus far I'm able to parse arbitrary json logs and
>>> get them into kafka very nicely.
>>> However, due to the many different systems in use there is key
>>> namespace clashes in the final destination (Elasticsearch)
>>>
>>> I have some JSON logs like this from one app
>>>
>>> { "login": 234343,... }
>>>
>>> and some JSON logs like this from another app
>>>
>>> { "login": "matt",... }
>>>
>>> Is it possible to parse and change the key space to look like this
>>>
>>> { "app1_login": 234343, "app1_XX:": }
>>> { "app2_login": "matt", "app2_XX:":...}
>>
>> I'm not sure how deep into ElasticSearch you've gotten, but it sounds like 
>> maybe you're seeing the result of automatic type mapping where the first 
>> field called "login" happens to be interpreted as a number, and later on a 
>> string value shows up and fails to be indexed because ElasticSearch now 
>> expects only numeric values.  You can solve this at ElasticSearch directly 
>> by having an explicit mapping (for example, "login" is a string), which in 
>> this case would force the numeric login value to be inserted as a string 
>> instead.
>>
>> (See: 
>> https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html 
>> )
>>
>> So you don't have to change the upstream JSON sources if you don't want to 
>> (though you certainly could do that instead).
>>
>> - Dave Caplinger
>>
>>
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
>> LIKE THAT.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize, mmjsonparse keyspace mapping

[Matt Ford]

Hi, yes I've gotten fairly deep into ES.  We would like to be able to
search over and perhaps even calculate based on fields with the same
name and different types.  What's more we don't know when developers
will add services and new json logs.  Without the rsyslog change - as
far as I can see the only way to do this to have per app indexes (as
per the tip at the very bottom of this page
https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping.html).

This is all cool stuff!  Please keep the ideas coming :-)

One thing that makes me nervous is the overhead of doing the
transformation via normalize on rsyslog.  Some the applications
generate a lot of logs per second.

On 7 April 2016 at 18:38, Dave Caplinger  wrote:
> On Apr 7, 2016, at 12:04 PM, Matt Ford  wrote:
>>
>> Thanks for the help thus far I'm able to parse arbitrary json logs and
>> get them into kafka very nicely.
>> However, due to the many different systems in use there is key
>> namespace clashes in the final destination (Elasticsearch)
>>
>> I have some JSON logs like this from one app
>>
>> { "login": 234343,... }
>>
>> and some JSON logs like this from another app
>>
>> { "login": "matt",... }
>>
>> Is it possible to parse and change the key space to look like this
>>
>> { "app1_login": 234343, "app1_XX:": }
>> { "app2_login": "matt", "app2_XX:":...}
>
> I'm not sure how deep into ElasticSearch you've gotten, but it sounds like 
> maybe you're seeing the result of automatic type mapping where the first 
> field called "login" happens to be interpreted as a number, and later on a 
> string value shows up and fails to be indexed because ElasticSearch now 
> expects only numeric values.  You can solve this at ElasticSearch directly by 
> having an explicit mapping (for example, "login" is a string), which in this 
> case would force the numeric login value to be inserted as a string instead.
>
> (See: 
> https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html )
>
> So you don't have to change the upstream JSON sources if you don't want to 
> (though you certainly could do that instead).
>
> - Dave Caplinger
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize, mmjsonparse keyspace mapping

[Dave Caplinger]

On Apr 7, 2016, at 12:04 PM, Matt Ford  wrote:
> 
> Thanks for the help thus far I'm able to parse arbitrary json logs and
> get them into kafka very nicely.
> However, due to the many different systems in use there is key
> namespace clashes in the final destination (Elasticsearch)
> 
> I have some JSON logs like this from one app
> 
> { "login": 234343,... }
> 
> and some JSON logs like this from another app
> 
> { "login": "matt",... }
> 
> Is it possible to parse and change the key space to look like this
> 
> { "app1_login": 234343, "app1_XX:": }
> { "app2_login": "matt", "app2_XX:":...}

I'm not sure how deep into ElasticSearch you've gotten, but it sounds like 
maybe you're seeing the result of automatic type mapping where the first field 
called "login" happens to be interpreted as a number, and later on a string 
value shows up and fails to be indexed because ElasticSearch now expects only 
numeric values.  You can solve this at ElasticSearch directly by having an 
explicit mapping (for example, "login" is a string), which in this case would 
force the numeric login value to be inserted as a string instead.

(See: 
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html )

So you don't have to change the upstream JSON sources if you don't want to 
(though you certainly could do that instead).

- Dave Caplinger


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize, mmjsonparse keyspace mapping

[David Lang]

On Thu, 7 Apr 2016, Matt Ford wrote:


Thanks for the help thus far I'm able to parse arbitrary json logs and
get them into kafka very nicely.
However, due to the many different systems in use there is key
namespace clashes in the final destination (Elasticsearch)

I have some JSON logs like this from one app

{ "login": 234343,... }

and some JSON logs like this from another app

{ "login": "matt",... }

Is it possible to parse and change the key space to look like this

{ "app1_login": 234343, "app1_XX:": }
{ "app2_login": "matt", "app2_XX:":...}

Anythoughts on that or an alternative approach greatly appreciated :-)


if (test for app1) then {
  set $!app1_login = $!login;
  unset $!login;
}

with mmnormalize, you can specify a namespace to put the resulting variables 
into, so instead of


{ "login": 234343,... }

you would do

{ "app1": { "login": 234343,... }}

In rsyslog, this would be referrred to as $!app1!login

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] mmnormalize, mmjsonparse keyspace mapping

[Matt Ford]

Thanks for the help thus far I'm able to parse arbitrary json logs and
get them into kafka very nicely.
However, due to the many different systems in use there is key
namespace clashes in the final destination (Elasticsearch)

I have some JSON logs like this from one app

{ "login": 234343,... }

and some JSON logs like this from another app

{ "login": "matt",... }

Is it possible to parse and change the key space to look like this

{ "app1_login": 234343, "app1_XX:": }
{ "app2_login": "matt", "app2_XX:":...}

Anythoughts on that or an alternative approach greatly appreciated :-)

Thanks,

Matt.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.