Re: [rsyslog] Send logs in JSON format to Elasticsearch

2017-02-23 Thread David Lang 

On Thu, 23 Feb 2017, Alec Swan via rsyslog wrote:


Liblognorm v2 supports json parser but it looks like it cannot be applied
to the entire log message, which is sad.


sure it can.

you can configure mmnormalize to pass $msg to the parser engine, $rawmsg, or 
$!whatever.


If your messages arrive in different formats, you can do a couple of things.

1. do a 2-stage parse where you first check the raw message, and if it's not 
parsed successfully, pass it to a ruleset that parses $msg


2. create a ruleset that parses the headers for the different formats

3. try to create a standard format that you then parse

I did a combination of these. I first did a mmnormalize call that checked if the 
message was raw json, if so I parsed it.


I then created a 'standard message' format that contained hostname and syslogtag 
as well as $msg (I really wanted to be able to include syslogtag in my parsing 
rules and sometimes hostname is misparsed if the input is malformed enough) as 
$.msg and passed that to mmnormalize with my main ruleset.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Send logs in JSON format to Elasticsearch

2017-02-23 Thread Alec Swan via rsyslog 
For some reason I stopped receiving emails from the mailing list even
though I can see them in the archive.

In response to Dave and Mostolog, each log line is a valid JSON and I would
really prefer to only rely on liblognorm for parsing and not depend on
mmjsonparse. Besides reluctance of installing and loading another library
(mmjsonparse) another reason for that is that I allow developers to provide
a liblognorm template for any type of log they would like to gather which I
then deploy. So, the contract between me and developers is just that
liblognorm template. With mmjsonparse approach, I will have to
conditionally load mmjsonparse if logs are already in JSON.

Liblognorm v2 supports json parser but it looks like it cannot be applied
to the entire log message, which is sad.

Is there any way to solve this without requiring installing a new library,
such as mmjsonparse?


Thanks,

Alec

On Wed, Feb 22, 2017 at 4:42 PM, Alec Swan  wrote:

> Hi there,
>
> We've been using librlognorm templates to parse log files and send their
> content to Elasticsearch as JSON. At this point we need to parse logs that
> are already in JSON format. So, I am wondering if there is a way to write a
> liblognorm template that would create a JSON object from each log line? Or
> is there a better way to populate $!all-json variable?
>
> Thanks,
>
> Alec
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog 8.25.0 (v8-stable) released

2017-02-23 Thread Thomas Deutschmann via rsyslog 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Hi,

the DOC tarball [1] is missing.


[1]
http://www.rsyslog.com/files/download/rsyslog/rsyslog-doc-8.25.0.tar.gz


- -- 
Regards,
Thomas
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0

iQJ8BAEBCgBmBQJYrxOtXxSAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQzM0M1ODQ4MkM0MDIyOTJEMkUzQzVDMDY5
NzA5RjkwQzNDOTZGRkM4AAoJEJcJ+Qw8lv/Ipv8P+wYxNkTSxUGzgf2RYzz5YlWv
ZhLG95W9MvXG4pV+pJMgwfP7wUCENn2Z297VFzb7rJttYIw35YiPykdqRe8GFoaC
8hyYGdMRcsMG4WnvfKX5fnmkCv/9MKBdrNDJSVni3YbjGXxzoBsDI1M+jPnXOpKD
36Ce3unmr87hgEoSN9Vj0bA2lBlOevjo85gJoK9iBR+8L1xDr08R4zIQ9G7ju1Fy
/rzjorvS57R1/X20pOWWGhsjpwOWI9WNPI7HCAt8ShU173w5HCkdRq0P0f18+fMW
HDvxJR2Sww3sYhC2N7oO5MNWIben8d7gnBeGTw23r7teQqAkoCtEZTOtSuCYPksH
mfybd9HWrCKP2a00PuH1ENLCu2zJ5hPbqY11SOIEBHDIBQKg8MIxENRG0CbmNPEt
XkPiu07nQOXTZAmkab1W6z6s+0bXhm93Wcq5RBfX0P/nrFDsCZGS7gFGhjJrv7+E
KrBN4KfjB0MZxkHIro8UeMJfno8Bby6uC1Mt7UUGCdaA79YMm3zjDbF+XmtvLu72
GUoz0dePZShiQorMvI3frTK7/EVZ6iDtXviKXw1bWE/wJjc05boYKqhDudD+cnw7
HUU4cQcwlugftGTTTlnwv3AF4NfHU1XGzR92/muRLmz0Jtdvz4j1f4LVe1wp9yad
dlnFEOZF4X6wh3xnMnEw
=ZGCy
-END PGP SIGNATURE-
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Using rsyslog to read docker log files

2017-02-23 Thread David Lang 
you can use mmnormalize and create a rulebase that uses the json type against 
rawmsg.


David Lang

On Thu, 23 Feb 2017, Chaloulos,   Klearchos (Nokia - GR/Athens) via rsyslog 
wrote:


Date: Thu, 23 Feb 2017 11:48:01 +
From: "Chaloulos,   Klearchos (Nokia - GR/Athens) via rsyslog"

To: rsyslog-users 
Cc: "Chaloulos,   Klearchos (Nokia - GR/Athens)"

Subject: [rsyslog] Using rsyslog to read docker log files

Hello,

I am investigating whether I can use rsyslog to read from docker log files. 
Docker log files display logs in the following format:

{"stream":"stdout","time":"2017-02-23T10:58:26.20407842Z","log":"The log 
message"}

So I used the mmjsonparse to parse the above message and get the "log" part. 
However the log messages I am working with are in fact json-formatted themselves, so the 
input looks like:

{"stream":"stdout","time":"2017-02-22T18:41:29.984845404Z","log":"{\"process\": \"myprocess\", \"output\": 
\"The log message output\", \"time\": \"2017-02-22T18:41:29.984186000Z\"}\n"}

So what I would like to do is to parse the "log" field above and extract the fields "process", 
"output" and "time", in order to have them available as properties. Is there a way to achieve this with 
rsyslog? It looks to me that I need to input the "log" string to mmjsonparse, but is this possible?

I tried a foreach loop like in /rsyslog/tests/testsuites/json_object_looping.conf, but it 
does not work as it expects "log" to be a json object, not a string.

Best regards,

Klearchos

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Send logs in JSON format to Elasticsearch

2017-02-23 Thread David Lang 
lognorm has the json type that you can use to extract json from the message, or 
if the message is all json you can use mmjsonparse.


if you aren't already using the version=2 features, you should really take a 
look at them, there is a lot of new capabilities in recent versions.


David Lang


El 23/02/17 a las 00:42, Alec Swan via rsyslog escribió:

Hi there,

We've been using librlognorm templates to parse log files and send their
content to Elasticsearch as JSON. At this point we need to parse logs that
are already in JSON format. So, I am wondering if there is a way to write a
liblognorm template that would create a JSON object from each log line? Or
is there a better way to populate $!all-json variable?

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Using rsyslog to read docker log files

2017-02-23 Thread Chaloulos, Klearchos (Nokia - GR/Athens) via rsyslog 
Hello,

I am investigating whether I can use rsyslog to read from docker log files. 
Docker log files display logs in the following format:

{"stream":"stdout","time":"2017-02-23T10:58:26.20407842Z","log":"The log 
message"}

So I used the mmjsonparse to parse the above message and get the "log" part. 
However the log messages I am working with are in fact json-formatted 
themselves, so the input looks like:

{"stream":"stdout","time":"2017-02-22T18:41:29.984845404Z","log":"{\"process\": 
\"myprocess\", \"output\": \"The log message output\", \"time\": 
\"2017-02-22T18:41:29.984186000Z\"}\n"}

So what I would like to do is to parse the "log" field above and extract the 
fields "process", "output" and "time", in order to have them available as 
properties. Is there a way to achieve this with rsyslog? It looks to me that I 
need to input the "log" string to mmjsonparse, but is this possible?

I tried a foreach loop like in 
/rsyslog/tests/testsuites/json_object_looping.conf, but it does not work as it 
expects "log" to be a json object, not a string.

Best regards,

Klearchos

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Send logs in JSON format to Elasticsearch

2017-02-23 Thread mostolog--- via rsyslog 

I dont know if this is what you are talking

http://www.rsyslog.com/doc/master/configuration/modules/mmjsonparse.html

BTW: whats your rsyslog omelasticsearch index rate?
Last test I made (with a basic configuration) we got ~10k/min only with 
latest ES version (perhaps its not fully compatible).


El 23/02/17 a las 00:42, Alec Swan via rsyslog escribió:

Hi there,

We've been using librlognorm templates to parse log files and send their
content to Elasticsearch as JSON. At this point we need to parse logs that
are already in JSON format. So, I am wondering if there is a way to write a
liblognorm template that would create a JSON object from each log line? Or
is there a better way to populate $!all-json variable?

Thanks,

Alec
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.