from:"mosto...@gmail.com"



a literal space is always more efficent than whitespace, only use 
whitespace if there can be more than one space, or tabs

Ok.


just a note, the new syntax is not always better than the old syntax

127.0.0.1 - - [17/Mar/2016:18:15:06 +0100] "GET /redacted HTTP/1.1" 
200 59506


type=@apache_common:%ip:ipv4% %ident:word% %user:word% 
[%date:char-to:]%] "%request:char-to:"%" %response:number% %bytes:rest%

Indeed. switched to old syntax and everything is working...¬¬

   type=@apache_common:%ip:ipv4% %ident:word% %user:word%
   [%date:char-to:]%] "%method:word%%-:whitespace%%request:char-to: %
   HTTP/%httpversion:float%" %response:number% %bytes:word%
   # ] this comment here fixes highlighting
   rule=access_common:%.:@apache_common%
   # .
   rule=access_combined:%.:@apache_common% %referrer:quoted-string%
   %useragent:quoted-string%
   # .


note that bytes really should be type number, but that requires a 
trailiing space right now.
Actually, as sometimes is "-", i must use word, which doesn't seem to 
have issues with SP/LF





  rule=access_combined:%[
   {"type":"@apache_common", "name":"."},
   {"type":"@apache_combined","name":"."}
  ]%


this is looking for one after the other, not either

you either use alternative or you have two different rule lines

I'm getting /invalid field type 'alternative'/ when using it. Any ideas?

   rule=test:%[
   {"type":"alternative","parser":[
   {"type":"literal","text":"-"},
   {"type":"word","name":"identd"}
]}
   ]%


when looking at the trace, everything before the "To normalize:" is 
probably not that useful (it's needed if you think the ruleset isn't 
being parsed correctly, but not to try and figure out why the log line 
isn't being parsed correctly)

Ok

it would be nice if -v only showed you the part we normally care 
about, there may be a way to get just this portion, but I don't know how
I didn't notice any difference between -v, -vv and -vvv, so perhaps it's 
a bug/not implemented/something to ask to @rgerhards


this looks like it's undoing things, it may be an artifact of using a 
custom type (misleading at best)


and we've undone averything.

No idea...does it make sense to declare "longer matching rules" first?
AKA: combined before common.




  normalized: '{ "originalmsg": "127.0.0.1 - - [17\/Mar\/2016:18:15:06
  +0100] \"GET \/redacted HTTP\/1.1\" 200 59506", "unparsed-data": "" }'
ok, now I understand this, it parsed the message with @apache_common 
and got to position 77 (the end of the message), but that wasn't the 
end of the rule, so the parsing failed, and it failed with nothing 
left to parse

Understood. Hope it won't happen again.

now we look at the second message (it helps understand this if you 
only look at one at a time, one rule and one log message)



  To normalize: '127.0.0.1 - - [17/Mar/2016:18:15:24 +0100] "OPTIONS /
did not find the field useragent, so backing up (probably end-of-line 
problem)

It was that, indeed.

Thanks for so long and instructive reply! ;)
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm vs grok




that is the same type of bug, just for another type.

just add a note that we need to allow end of line for all types, it's 
not limited to space.
I'm missing code commenting...probably I'm going to switch back to ~doc 
tasks :P


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm vs grok




{"type":"@apache" name="."} ?


actuall, %{"type":"@apache" name="."}%

This is one of the places where I like to use the older, more compact 
syntax :-)
Older/Compact doesn't seem to have an alternative, reason why I started 
using JSON syntax...right?


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm vs grok

I'm still trying to reproduce/understand what is happening and building 
a test case for the github issue if needed.


Consider the following HTTP access lines:

   127.0.0.1 - - [17/Mar/2016:18:15:06 +0100] "GET /redacted HTTP/1.1"
   200 59506
   127.0.0.1 - - [17/Mar/2016:18:15:24 +0100] "OPTIONS / HTTP/1.1" 403
   205 "-" "-"

And the following rule:

   # This is just access_log. Perhaps literal is more efficient than
   whitespace?
   type=@apache_common:%[
{"type":"ipv4", "name":"ip"},
{"type":"whitespace"},
{"type":"word", "name":"ident"},
{"type":"whitespace"},
{"type":"word", "name":"user"},
{"type":"literal", "text":" ["},
{"type":"char-to", "name":"date", "extradata":"]"},
{"type":"literal", "text":"] \""},
{"type":"word", "name":"method"},
{"type":"whitespace"},
{"type":"char-to", "name":"request", "extradata":" "},
{"type":"literal", "text":" HTTP/"},
{"type":"float", "name":"httpversion"},
{"type":"literal", "text":"\""},
{"type":"whitespace"},
{"type":"number", "name":"response"},
{"type":"whitespace"},
{"type":"word", "name":"bytes"}
   ]%

   #AFAIK this should accept null or apache combined log fields
   type=@apache_combined:-
   type=@apache_combined:%[
{"type":"whitespace"},
{"type":"quoted-string","name":"referrer"},
{"type":"whitespace"},
{"type":"quoted-string","name":"useragent"}
   ]%
   rule=access_combined:%[
{"type":"@apache_common", "name":"."}
   ]%

*As expected*, the first line matches and the second doesn't:

   { "bytes": "59506", "response": "200", "httpversion": "1.1",
   "request": "\/redacted", "method": "GET", "date":
   "17\/Mar\/2016:18:15:06 +0100", "user": "-", "ident": "-", "ip":
   "127.0.0.1" }
   { "originalmsg": "127.0.0.1 - - [17\/Mar\/2016:18:15:24 +0100]
   \"OPTIONS \/ HTTP\/1.1\" 403 205 \"-\" \"-", "unparsed-data": "
   \"-\" \"-" }

But if we try:

   rule=access_combined:%[
{"type":"@apache_common", "name":"."},
{"type":"@apache_combined","name":"."}
   ]%

Doesnt parse any of them properly, and we're getting:

   { "originalmsg": "127.0.0.1 - - [17\/Mar\/2016:18:15:06 +0100] \"GET
   \/redacted HTTP\/1.1\" 200 59506", "unparsed-data": "" }
   { "originalmsg": "127.0.0.1 - - [17\/Mar\/2016:18:15:24 +0100]
   \"OPTIONS \/ HTTP\/1.1\" 403 205 \"-\" \"-", "unparsed-data": "
   \"-\" \"-" }


Here's trace:

   liblognorm: loading rulebase file '/test/apps/10-apache.rb'
   liblognorm: rulebase version is 2

   liblognorm: read rulebase line[~25]: 'type=@apache_common:%[
   {"type":"ipv4", "name":"ip"},{"type":"whitespace"},
   {"type":"word", "name":"ident"},{"type":"whitespace"},
   {"type":"word", "name":"user"},{"type":"literal", "text":"
   ["},{"type":"char-to", "name":"date", "extradata":"]"},
   {"type":"literal", "text":"] \""},{"type":"word",
   "name":"method"},{"type":"whitespace"}, {"type":"char-to",
   "name":"request", "extradata":" "}, {"type":"literal", "text":"
   HTTP/"},{"type":"float", "name":"httpversion"},   
   {"type":"literal", "text":"\""}, {"type":"whitespace"},   
   {"type":"number", "name":"response"},{"type":"whitespace"},   
   {"type":"word", "name":"bytes"}]%'

   liblognorm: type line to add: '@apache_common:%[{"type":"ipv4",
   "name":"ip"},{"type":"whitespace"},{"type":"word",
   "name":"ident"},{"type":"whitespace"},{"type":"word",
   "name":"user"},{"type":"literal", "text":" ["},
   {"type":"char-to", "name":"date", "extradata":"]"},
   {"type":"literal", "text":"] \""},{"type":"word",
   "name":"method"},{"type":"whitespace"}, {"type":"char-to",
   "name":"request", "extradata":" "}, {"type":"literal", "text":"
   HTTP/"},{"type":"float", "name":"httpversion"},   
   {"type":"literal", "text":"\""}, {"type":"whitespace"},   
   {"type":"number", "name":"response"},{"type":"whitespace"},   
   {"type":"word", "name":"bytes"}]%'

   liblognorm: type name is '@apache_common'
   liblognorm: type line to add: '%[{"type":"ipv4", "name":"ip"},
   {"type":"whitespace"},{"type":"word", "name":"ident"},
   {"type":"whitespace"},{"type":"word", "name":"user"},
   {"type":"literal", "text":" ["},{"type":"char-to",
   "name":"date", "extradata":"]"},{"type":"literal", "text":"]
   \""},{"type":"word", "name":"method"}, {"type":"whitespace"},   
   {"type":"char-to", "name":"request", "extradata":" "},   
   {"type":"literal", "text":" HTTP/"}, {"type":"float",
   "name":"httpversion"},{"type":"literal", "text":"\""},   
   {"type":"whitespace"},{"type":"number", "name":"response"},   
   {"type":"whitespace"},{"type":"word", "name":"bytes"}]%'

   liblognorm: ln_pdagFindType, name '@apache_common', bAdd: 1, nTypes 0
   liblognorm: custom type '@apache_common' does not yet exist, adding...
   liblognorm: addSampToTree 0 of

Re: [rsyslog] liblognorm vs grok





I think it's a problem, several of the types require a space at the end,
and
I think they should all be modified to allow either a space or a
end-of-line.

ack. It's on my list for early next year.

better check if one exists, I also think David created one. This is
for the liblognorm project.
According to https://github.com/rsyslog/liblognorm/issues/207 the 
problem could be:

https://github.com/rsyslog/liblognorm/blob/master/src/parser.c#L2869
am I right? Should it accept SP and (\n) LF? If that's all I could PR...





* A or B (doc states it does)
* A or nothing (that was my real question)


I'm not sure if you can have a blank item on one branch or not. If not,
can
you have the branches both include a required item? (either the one
before
or the one after)

I *think* (but do not know for sure) this might work. Else I'll add
early next year as well. Conceptionally, it really is alternative with
a void branch.


Once I get an "alternative" working, I'll try to have an empty branch. If it
doesn't work, I'll open an issue :)

I think there also already one exists, from Radu.

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm vs grok



when troubleshooting things like this, create a rule file that is as 
minimal as you can get and parse with the -v option, it will show you 
what it's doing as it walks through the line.


I don't see how it parsed each message. Perhaps a debug option must be 
enabled?


   number of tree nodes: 20
   liblognorm: COMPONENT: @apache
   liblognorm: subDAG 0x7f97bae1a650 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'ipv4', name 'ip': 'UNKNOWN': called 0
   liblognorm: field type 'ipv4', name 'ip': 'UNKNOWN':
   liblognorm: subDAG 0x7f97bae1b050 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'whitespace', name '(null)': 'UNKNOWN': called 0
   liblognorm: field type 'whitespace', name '(null)': 'UNKNOWN':
   liblognorm: subDAG 0x7f97bae1b180 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'word', name 'ident': 'UNKNOWN': called 0
   liblognorm: field type 'word', name 'ident': 'UNKNOWN':
   liblognorm: subDAG 0x7f97bae1b3e0 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'whitespace', name '(null)': 'UNKNOWN': called 0
   liblognorm: field type 'whitespace', name '(null)': 'UNKNOWN':
   liblognorm: subDAG 0x7f97bae1b610 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'word', name 'user': 'UNKNOWN': called 0
   liblognorm: field type 'word', name 'user': 'UNKNOWN':
   liblognorm: subDAG 0x7f97bae1b780 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'literal', name '(null)': ' [': called 0
   liblognorm: field type 'literal', name '(null)': ' [':
   liblognorm: subDAG 0x7f97bae1b820 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'char-to', name 'date': 'UNKNOWN': called 0
   liblognorm: field type 'char-to', name 'date': 'UNKNOWN':
   liblognorm: subDAG 0x7f97bae1bc30 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'literal', name '(null)': '] "': called 0
   liblognorm: field type 'literal', name '(null)': '] "':
   liblognorm: subDAG 0x7f97bae1bdc0 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'word', name 'method': 'UNKNOWN': called 0
   liblognorm: field type 'word', name 'method': 'UNKNOWN':
   liblognorm: subDAG 0x7f97bae1c050 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'whitespace', name '(null)': 'UNKNOWN': called 0
   liblognorm: field type 'whitespace', name '(null)': 'UNKNOWN':
   liblognorm: subDAG 0x7f97bae1c3c0 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'char-to', name 'request': 'UNKNOWN': called 0
   liblognorm: field type 'char-to', name 'request': 'UNKNOWN':
   liblognorm: subDAG 0x7f97bae1c530 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'literal', name '(null)': ' HTTP/': called 0
   liblognorm: field type 'literal', name '(null)': ' HTTP/':
   liblognorm: subDAG 0x7f97bae1cbd0 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'float', name 'httpversion': 'UNKNOWN': called 0
   liblognorm: field type 'float', name 'httpversion': 'UNKNOWN':
   liblognorm: subDAG 0x7f97bae1cd50 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'literal', name '(null)': '"': called 0
   liblognorm: field type 'literal', name '(null)': '"':
   liblognorm: subDAG 0x7f97bae1cf90 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'whitespace', name '(null)': 'UNKNOWN': called 0
   liblognorm: field type 'whitespace', name '(null)': 'UNKNOWN':
   liblognorm: subDAG 0x7f97bae1d200 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'number', name 'response': 'UNKNOWN': called 0
   liblognorm: field type 'number', name 'response': 'UNKNOWN':
   liblognorm: subDAG 0x7f97bae1d350 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'whitespace', name '(null)': 'UNKNOWN': called 0
   liblognorm: field type 'whitespace', name '(null)': 'UNKNOWN':
   liblognorm: subDAG 0x7f97bae1d6e0 (children: 1 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: field type 'word', name 'bytes': 'UNKNOWN': called 0
   liblognorm: field type 'word', name 'bytes': 'UNKNOWN':
   liblognorm: subDAG [TERM] 0x7f97bae1da80 (children: 0 parsers, ref
   1) [called 0, backtracked 0]
   liblognorm: MAIN COMPONENT:
   liblognorm: subDAG 0x7f97bae190a0 (children: 0 parsers, ref 1)
   [called 0, backtracked 0]
   liblognorm: MAIN COMPONENT (alternative):
   liblognorm: 0x7f97bae190a0[ref 1]:
   To normalize: '127.0.0.1 - - [17/Mar/2016:18:06:58 +0100] "GET
   /redacted HTTP/1.1" 200 62957'
   liblognorm: 0: enter parser, dag node 0x7f97bae190a0, json
   0x7f97bae1ba20
   liblognorm: offs 0, strLen 102, isTerm 0
   liblognorm: 0 returns

Re: [rsyslog] liblognorm vs grok





almost, %@apache% makes no more sense than %word%, you need to give 
the match a name


so %log:@apache% would work, or if you want to move everything up a 
later (rather than having $!apache!ip) you could do %.:@apache%

That should work

How would that be using JSON syntax?
{"type":"@apache" name="."} ?


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm vs grok


Should something like this work?

{"type":"@apache"},
{"type":"alternative","parser":[
{},
{
{"type":"whitespace"},
...
}
]}


El 07/12/16 a las 11:08, Rainer Gerhards escribió:

2016-12-07 10:38 GMT+01:00 mosto...@gmail.com <mosto...@gmail.com>:



In this case, I seem to remember that number is defined as being
followed
by a space, so you can't use it if the number is followed by a newline.

I'll have to confirm that...but may I know why? Should I fill an issue
if
it's indeed that way?

I think it's a problem, several of the types require a space at the end,
and
I think they should all be modified to allow either a space or a
end-of-line.

ack. It's on my list for early next year.

May I create an issue somewhere?

better check if one exists, I also think David created one. This is
for the liblognorm project.


* A or B (doc states it does)
* A or nothing (that was my real question)

I'm not sure if you can have a blank item on one branch or not. If not,
can
you have the branches both include a required item? (either the one
before
or the one after)

I *think* (but do not know for sure) this might work. Else I'll add
early next year as well. Conceptionally, it really is alternative with
a void branch.

Once I get an "alternative" working, I'll try to have an empty branch. If it
doesn't work, I'll open an issue :)

I think there also already one exists, from Radu.

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm vs grok




In this case, I seem to remember that number is defined as being followed
by a space, so you can't use it if the number is followed by a newline.

I'll have to confirm that...but may I know why? Should I fill an issue if
it's indeed that way?


I think it's a problem, several of the types require a space at the end, and
I think they should all be modified to allow either a space or a
end-of-line.

ack. It's on my list for early next year.

May I create an issue somewhere?






* A or B (doc states it does)
* A or nothing (that was my real question)


I'm not sure if you can have a blank item on one branch or not. If not, can
you have the branches both include a required item? (either the one before
or the one after)

I *think* (but do not know for sure) this might work. Else I'll add
early next year as well. Conceptionally, it really is alternative with
a void branch.


Once I get an "alternative" working, I'll try to have an empty branch. 
If it doesn't work, I'll open an issue :)


Thank you all!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm vs grok



when troubleshooting things like this, create a rule file that is as 
minimal as you can get and parse with the -v option, it will show you 
what it's doing as it walks through the line.

Ok :)



In this case, I seem to remember that number is defined as being 
followed by a space, so you can't use it if the number is followed by 
a newline.
I'll have to confirm that...but may I know why? Should I fill an issue 
if it's indeed that way?


almost, %@apache% makes no more sense than %word%, you need to give 
the match a name


so %log:@apache% would work, or if you want to move everything up a 
later (rather than having $!apache!ip) you could do %.:@apache%

That should work

there is the alternative capability in the v2 language, or define 
multiple rules
Multiple rules is what I'm trying now. I have tested alternate and I'm 
not able to get it working.

Does alternative work for both...?

 * A or B (doc states it does)
 * A or nothing (that was my real question)

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm vs grok


I forgot:

With provided rule file...why I'm getting a bunch of this errors when 
using /usr/lib/lognorm/lognormalizer?


{ "originalmsg": "127.0.0.1 - - [17\/Mar\/2016:18:15:31 +0100] \"GET 
\/redacted\/page HTTP\/1.1\" 200 1234", "unparsed-data": "" }



El 05/12/16 a las 15:41, mosto...@gmail.com escribió:

Hi


Coming back to liblognorm, I have a few questions I'll love an expert 
reply.  0:D


*- Documentation [1] states how to define a type, but not how to use 
it. Are we properly using defined type "apache" in the configuration 
below?*


- Apache access log seem to have 2 formats: common and combined [2]
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif 
HTTP/1.0" 200 2326
127.0.0.1 - - [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif 
HTTP/1.0" 200 2326 "referrer" "useragent"

*How should we define our rulesets to have /optional/ fields?

- Our current workaround is to have a defined type and use it as part 
of a longer rule. Is that ok?*


*- How could we define logic to set a filed to "0" when content is 
"-"? (bytes field)*


type=@apache:%[
{"type":"ipv4", "name":"ip"},
{"type":"literal", "text":" "},
{"type":"word", "name":"ident"},
{"type":"literal", "text":" "},
{"type":"word", "name":"user"},
{"type":"literal", "text":" ["},
{"type":"char-to", "name":"date", "extradata":"]"},
{"type":"literal", "text":"] \""},
{"type":"word", "name":"method"},
{"type":"literal", "text":" "},
{"type":"char-to", "name":"request", "extradata":" "},
{"type":"literal", "text":" HTTP/"},
{"type":"float", "name":"httpversion"},
{"type":"literal", "text":"\" "},
{"type":"number", "name":"response"},
{"type":"literal", "text":" "},
{"type":"number", "name":"bytes"}
]%

rule=access:%[
{"type":"@apache"},
{"type":"literal", "text":"\""},
{"type":"char-to", "name":"referrer", "extradata":"\""},
{"type":"literal", "text":"\""},
{"type":"char-to", "name":"useragent", "extradata":"\""}
]%
rule=access:%@apache%

[1] http://www.liblognorm.com/files/manual/configuration.html
[2] https://httpd.apache.org/docs/2.4/logs.html#accesslog


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm vs grok


Hi


Coming back to liblognorm, I have a few questions I'll love an expert 
reply.  0:D


*- Documentation [1] states how to define a type, but not how to use it. 
Are we properly using defined type "apache" in the configuration below?*


- Apache access log seem to have 2 formats: common and combined [2]
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif 
HTTP/1.0" 200 2326
127.0.0.1 - - [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif 
HTTP/1.0" 200 2326 "referrer" "useragent"

*How should we define our rulesets to have /optional/ fields?

- Our current workaround is to have a defined type and use it as part of 
a longer rule. Is that ok?*


*- How could we define logic to set a filed to "0" when content is "-"? 
(bytes field)*


type=@apache:%[
{"type":"ipv4", "name":"ip"},
{"type":"literal", "text":" "},
{"type":"word", "name":"ident"},
{"type":"literal", "text":" "},
{"type":"word", "name":"user"},
{"type":"literal", "text":" ["},
{"type":"char-to", "name":"date", "extradata":"]"},
{"type":"literal", "text":"] \""},
{"type":"word", "name":"method"},
{"type":"literal", "text":" "},
{"type":"char-to", "name":"request", "extradata":" "},
{"type":"literal", "text":" HTTP/"},
{"type":"float", "name":"httpversion"},
{"type":"literal", "text":"\" "},
{"type":"number", "name":"response"},
{"type":"literal", "text":" "},
{"type":"number", "name":"bytes"}
]%

rule=access:%[
{"type":"@apache"},
{"type":"literal", "text":"\""},
{"type":"char-to", "name":"referrer", "extradata":"\""},
{"type":"literal", "text":"\""},
{"type":"char-to", "name":"useragent", "extradata":"\""}
]%
rule=access:%@apache%

[1] http://www.liblognorm.com/files/manual/configuration.html
[2] https://httpd.apache.org/docs/2.4/logs.html#accesslog
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm vs grok


Is that documentation stored on a github like rsyslog's?

http://www.liblognorm.com/files/manual/index.html


El 05/12/16 a las 11:15, David Lang escribió:

On Mon, 5 Dec 2016, mosto...@gmail.com wrote:


Hi.

Is there an online liblognorm tester to check the rules we are writing?

Otherwise, could you provide a testing guide 
(http://www.liblognorm.com/files/manual/installation.html#testing) to 
build lognormalizer to test?


the liblognorm package includes lognormalizer, but it doesn't put it 
in a place where it's picked up by the default path


/usr/lib/lognorm/lognormalizer




El 04/10/16 a las 19:27, mosto...@gmail.com escribió:

Hi Radu


After reading 
http://lists.adiscon.net/pipermail/rsyslog/2013-December/035122.html 
and considering several years have passed, I would like to get some 
feedback of your experience, to help me choose between raw 
forwarding messages+logstash or split before forwarding with 
mmnormalize.


If this decision should've been made today, what you would've choosen?

Of course, everybody is welcome to join the thread.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT 
POST if you DON'T LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm vs grok


Hi.

Is there an online liblognorm tester to check the rules we are writing?

Otherwise, could you provide a testing guide 
(http://www.liblognorm.com/files/manual/installation.html#testing) to 
build lognormalizer to test?



El 04/10/16 a las 19:27, mosto...@gmail.com escribió:

Hi Radu


After reading 
http://lists.adiscon.net/pipermail/rsyslog/2013-December/035122.html 
and considering several years have passed, I would like to get some 
feedback of your experience, to help me choose between raw forwarding 
messages+logstash or split before forwarding with mmnormalize.


If this decision should've been made today, what you would've choosen?

Of course, everybody is welcome to join the thread.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] REK stack




Rek Project makes us sound like dangerous anarchists, or possibly a dub-techno 
outfit.

LOL
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] REK stack


TREK (long journey) seems a more friendly name for me.

It also stands for "(The) Rsyslog ElasticSearch Kibana" project/stack

:P
(I can't stop posting on this list)


El 02/12/16 a las 11:48, Rainer Gerhards escribió:

Hi all,

I start a new thread as the other one has a million of different topics now ;-)

Just a short note: I think we should finally call this projekt "REK
stack" vs. ERK and other ideas. This seems to be consensus, is logical
(rsyslog-ES-Kibna, in right order) and as Brian pointed out there
already is prior art ;-).

Violent objections please here. I have updated the rsyslog github REK project:

https://github.com/rsyslog/rsyslog/projects/1

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

Then, implementing (although seems hard and you already said not 
intended) something like


   /if exists $!rulesetname then/

could improve if/else performance, cause using direct pointers.

Anyhow I'm moving to existent/working if/else solution for short-term


El 02/12/16 a las 11:48, Rainer Gerhards escribió:

2016-12-02 11:45 GMT+01:00 mosto...@gmail.com <mosto...@gmail.com>:

El 02/12/16 a las 10:50, David Lang escribió:

remember that values in logs may be under the control of an attacker (just
about every field can be hand-crafted by a person sending you a log
message). I know you are thinking of the simple case where you are only
reading files and processing them, but as soon as you support logs from any
other sources (including writing logs to /dev/log), things get much messier.

some way to test if a ruleset by that name exists would be needed, but
keep in mind that you may have other rulesets that exist besides the ones
you want to call.

Didn't think about that. For sure that's an exploit vector.

to some extend you could mitigate this by

call "prefix-" & $syslogtag

just FYI (if we permit an expression at all, we permit full expresibility).

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?


El 02/12/16 a las 10:50, David Lang escribió:
remember that values in logs may be under the control of an attacker 
(just about every field can be hand-crafted by a person sending you a 
log message). I know you are thinking of the simple case where you are 
only reading files and processing them, but as soon as you support 
logs from any other sources (including writing logs to /dev/log), 
things get much messier.


some way to test if a ruleset by that name exists would be needed, but 
keep in mind that you may have other rulesets that exist besides the 
ones you want to call.

Didn't think about that. For sure that's an exploit vector.


I don't see a switch being noticably better.

Agree

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?



(as an optimization, make the most common apps the earliest in the 
directory)
And that's where that /if exists call $!tag/ idea makes more sense, 
cause you don't need to compare, neither to arrange most used 
applications at top.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?


Hi


Combined threads reply follow.


On Thu, 1 Dec 2016, David Lang wrote
sigh, this is getting a wee bit frustrating, you keep saying "it hurts 
when I do X", we say "that doesn't work well, do Y" and you come back 
a day or so later saying "but it really huts when I do X"... (it 
doesn't help when we ask you to provide information and you instead 
spend hours trying other things)

I understand you better than you think. I'm the pupil you never wanted.
Said so, in my defense, I would say I am testing the /Y/ things too.

now that I have expressed my frustration, you are finding bugs, 
helping to fix them, and raising some good questions along the way. 
Just understand why once in a while our answers seem a bit curt. 
Don't worry. I understand it but I'm learning a lot along the way!. You 
are having A LOT of patience and being really kind and very instructive.


while I can see the use cases for "call $.var", what would you do if 
you call a ruleset that doesn't exist? you would first have to do 'if 
$.var == [array of legal values] then' to be safe.
Sure, something like if $!var exists then { call $!var } (one if 
statement vs many)



1. just a bunch of if statements

  performance cost of doing a bunch of if tests
  easy to include additional tests from a directory of files

That's my best option so far.


2/3. if then else if...

This saves up to 199 if's. Would this be noticiable?


4. switch statement

This seems interesting...


5. variable call statements

  what to do if called ruleset doesn't exist?

I'm screwed


6. function lookup tables
This is the Nth time you mention those...start to think I *really* need 
to start messing with them.


I think that with elsif, the need for switch (#4) is low, and the 
restrictions of it only doing simple equivalence tests (no startswith, 
contains, etc) really limit it's use

Totally agree.

call var (#5) seems easy to implement, but I really don't like opening 
up the problem of calling a non-existant ruleset. We could have it 
silently do nothing, but that gets really messy and I am already 
cringing at the troubleshooting exhanges we will have to help people 
figure out what is/isn't happeing.

Is there a *exists* statement? (eg: /if exists "object-name" then/)

function pointers are by far the most complicated, and since they 
include ruleset parsing after startup, they have the potential to be 
really ugly to implement. On the other hand, they are also by far the 
most powerful. If we could do things like limiting the functions so 
that they can't do any of the startup-type things[1] and only include 
statements that are normally executed for each log type, this would 
also give us a back-door way of providing the dynamic configuration 
that many people have been asking for.

"Limiting the functions" sounds too /patchy/ for me.


On Thu, 1 Dec 2016, David Lang wrote

if  then {
set common things
}
$includeconfig /etc/rsyslog.d/apps.d/*.conf
else {
unknown app
}

in each of the apps.d/*.conf files do
else if  then {
stuff
}

This is what I had in mind.

(as an optimization, make the most common apps the earliest in the 
directory)

Loved alphabetical order, but now I understand why this is better.


On Fri, 2 Dec 2016, Rainer Gerhards wrote


What exactly do you do with the variables you set inside the if body?
Are they always the same? Where does the data originate from?

I try to understand the scenario better, because I vaguely think I may
be able to find a much simpler solution which would require possible
minimal code changes. But I can't express myself clearer at the
moment, it's more a gut feeling.

It would be good if you could post some *concrete* example of three or
four of the if blocks, that would definitely aid understanding.

As David said, I would love having separated config files for each app to:

1. normalize
2. define this application pipeline (after normalization, invoke geoip,
   add some fields to JSON...)
3. index or drop messages at app criterion.

Main config would be some common routines (like index)+combined app.conf 
files.



On Fri, 2 Dec 2016, David Lang wrote


if the liblognorm ruleset ammend=: line could assign variable 
contents, not just constant strings, it would address 90% of the issues.
It would let me add/remove/edit fields. Other things, like geoip 
tagging, should still be made manually (I guess)



On Fri, 2 Dec 2016, Rainer Gerhards wrote

Nothing I can do immediately, but can you pls open an issue tracker
with some explanation (copy?) on liblognorm. I have planned time
early next year to work on enhancements and this would fit in.

Music to my ears


Thank you both a lot for you replies.
Not only learning a lot, but feeling in debt with you, reason why I'm 
contributing as much as I can in the most ways I'm able.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

Re: [rsyslog] Are we building an ERK stack?


El 01/12/16 a las 23:08, David Lang escribió:

On Thu, 1 Dec 2016, mosto...@gmail.com wrote:

I think that you are going to end up with some grief, if the message 
could not be insterted into ES for some reason, I think the odds are 
good that you will find that rawmsg can't be inserted either.

After sending the email I though the same...

I would keep the errorfile as a file and look at it periodially. I 
expect that when you first start things up, you will run into a number 
of errors, but once you work your way though them, the error rate will 
be low.


Set your monitoring system to monitor the size of the errorfile, and 
it it starts growing significantly, generate an alert.

Would love to have a more unattended/XXth century way, if anyone knows.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog-doc github include rst?


Ok. One thing less on my TODO. :D


El 01/12/16 a las 19:57, Rainer Gerhards escribió:

GitHub has the doc sources, I don't think it can generate a proper version.
The official doc sits at rsyslog.com/doc and there it should be IMHO.

I like GitHub, but I do not want to bet the project on its availability.

Rainer

Sent from phone, thus brief.

Am 01.12.2016 19:52 schrieb "mosto...@gmail.com" <mosto...@gmail.com>:


within github

I guessed sphinx would do it for web-fronted, but I was asking for github.



El 01/12/16 a las 19:50, Rainer Gerhards escribió:


The headers etc are actually generated by sphinx. Have a look at the
README, it details the steps. What you see on the website is the output of
a generation run.

Let me know if this helps or more detail is required.

Rainer

Sent from phone, thus brief.

Am 01.12.2016 19:12 schrieb "mosto...@gmail.com" <mosto...@gmail.com>:

Hi


@radu-gheorghe @rgerhards: Is there any way to include a rst document
into
another within github?

I have been trying /raw/ and other directives without success (seems due
to security concerns)

(Trying to include legal foot/license for every page)

Thanks.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

___

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog-doc github include rst?


> within github

I guessed sphinx would do it for web-fronted, but I was asking for github.



El 01/12/16 a las 19:50, Rainer Gerhards escribió:

The headers etc are actually generated by sphinx. Have a look at the
README, it details the steps. What you see on the website is the output of
a generation run.

Let me know if this helps or more detail is required.

Rainer

Sent from phone, thus brief.

Am 01.12.2016 19:12 schrieb "mosto...@gmail.com" <mosto...@gmail.com>:


Hi


@radu-gheorghe @rgerhards: Is there any way to include a rst document into
another within github?

I have been trying /raw/ and other directives without success (seems due
to security concerns)

(Trying to include legal foot/license for every page)

Thanks.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?


Hi Bob.

Today we finally found some time to have an eye on our 
rsyslog-normalizer-indexer which uses omelasticsearch


According to 
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omelasticsearch.html 
indexing parameter *errorfile* helps to store failed indexing attempts.


How do you handle those errors?
We are thinking on

 * setting errorfile=file
 * imfile ruleset=omelasticsearch
 * elastic template like: {index="errors" msg="rawmsg" }, and keep an
   eye on that

What do you think?


El 24/11/16 a las 12:52, Bob Gregory escribió:

https://io.made.com/blog/rek-it/

I wrote this up earlier.

On Wed, 23 Nov 2016 at 19:38 mosto...@gmail.com <mosto...@gmail.com> wrote:


Working, spamming mail list and writing on wiki at the same time. A
lovely afternoon...

Please, add your lines: https://github.com/rsyslog/rsyslog/wiki
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?




yup, thx. IMHO makes sense. Let's wait what David says, but I think it
would make sense to open an issue refering to this thread. ... I know,
I also must find time to actually work on some of them... ;-)

This is our current /core.conf/ draft:

   global(
MaxMessageSize="32k"
workDirectory="/data"
parser.escapeControlCharactersOnReceive="off"
   )

   template(name="index" type="string" string="$!data!index")
   template(name="type" type="string" string="$!data!type")
   template(name="json" type="string" string="%$!data%")

   module(load="imelasticsearch")
   ruleset(name="elastic"){
set $!data=$msg;
set $!data!relay=$myhostname;
set $!data!from=$hostname;
action(
action.reportSuspension="on"
action.resumeRetryCount="-1"
#queue.filename="omrelp.qi"
queue.maxdiskspace="1G"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
type="omelasticsearch"
server="server"
serverport="9200"
searchIndex="index"
dynSearchIndex="on"
searchType="type"
dynSearchType="on"
template="json"
)
   }

   module(load="imrelp")
   input(
port="20514"
type="imrelp"
name="imrelp"
   )

   # All files under rsyslog.d are automatically included, each will be
   like:
   # app.conf
   # ruleset(name="app") {
   #   #normalize/parse. ideally using inline rules
   #   #whatever
   #   set $!data!index="myindex--MM-DD";
   #   set $!data!type="this_app_types_are_known_by_this_app";
   #   call another_app_in_pipeline
   #   stop
   # }

   call $!data!app

This is the best approach we have found so far.
Although @davidlang suggested having just one normalization ruleset 
could be faster, this appears to be flexible, easy to maintain, etc.

Thoughts.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] rsyslog-doc github include rst?


Hi


@radu-gheorghe @rgerhards: Is there any way to include a rst document 
into another within github?


I have been trying /raw/ and other directives without success (seems due 
to security concerns)


(Trying to include legal foot/license for every page)

Thanks.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?



El 01/12/16 a las 19:04, Rainer Gerhards escribió:

2016-12-01 18:56 GMT+01:00 mosto...@gmail.com <mosto...@gmail.com>:

El 01/12/16 a las 18:37, Rainer Gerhards escribió:

2016-12-01 18:33 GMT+01:00 mosto...@gmail.com <mosto...@gmail.com>:

Hi

Is there any way to dynamically invoke a ruleset? eg: call $var
(I'm trying to avoid having +200 if statements...

not yet, but 90% sure evrything is in place to make implementation easy.

Can you elaborate on the use case?


relay syslog forwarding multiple files to central location
central syslog must mmnormalize depending on syslogtag before indexing into
elastic
other tasks like geoip must be done whenever a message has a ip field (for
example)

current approach is as follows:

  * core.conf contains input and ruleset for indexing
  * appX.conf files are copied to /etc/rsyslogd.d/, to be loaded at start
  * each app has a .conf file to define both, additional transformations
+ mmnormalize rules (https://github.com/rsyslog/rsyslog/issues/625)
  * when a message is received, it must be processed by 1-N apps, which
would be great if done dynamically, but I don't think that's possible.


core.conf

ruleset("name="elastic") {
 action(type="omelasticsearch"
 #once this message has been processed by all modules, index
 )
}

app1.conf

if $!app equals "app1" then {
 #normalize (davidlang says it's better to have 1 normalizer on
core.conf. I need to think about it)
 #add some custom fields
 #set $!index="myindexname--MM-DD"
 call geoip
 stop
}

app2.conf

if $!app equals "app2" then {
 #normalize using inline rulebase
 #set $!index="otherindexname";
 stop
}

app200.conf

if $!app equals "app20" then {
 #whatever
}

geoip.conf

ruleset(name="geoip") {
 #geo tag this message
}

unk.conf

if message_has_not_been_proccessed then {
#set $!index="unknown";
}

I hope I explained myself properly...

where would you use "call $var" if it were available?


core.conf

ruleset("name="elastic") {
action(type="omelasticsearch"
#index
)
   }
   call %syslogtag%  #Really, this is actually stored at $!app, but I 
think you got the idea...(eg: appX)


appX.conf

ruleset(name="appX") {
#whatever
#set $!index="myindexname--MM-DD"
   }

This way, core and app configurations will be -IMHO- much simpler.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?


El 01/12/16 a las 18:37, Rainer Gerhards escribió:

2016-12-01 18:33 GMT+01:00 mosto...@gmail.com <mosto...@gmail.com>:

Hi

Is there any way to dynamically invoke a ruleset? eg: call $var
(I'm trying to avoid having +200 if statements...

not yet, but 90% sure evrything is in place to make implementation easy.

Can you elaborate on the use case?


relay syslog forwarding multiple files to central location
central syslog must mmnormalize depending on syslogtag before indexing 
into elastic
other tasks like geoip must be done whenever a message has a ip field 
(for example)


current approach is as follows:

 * core.conf contains input and ruleset for indexing
 * appX.conf files are copied to /etc/rsyslogd.d/, to be loaded at start
 * each app has a .conf file to define both, additional transformations
   + mmnormalize rules (https://github.com/rsyslog/rsyslog/issues/625)
 * when a message is received, it must be processed by 1-N apps, which
   would be great if done dynamically, but I don't think that's possible.


core.conf

   ruleset("name="elastic") {
action(type="omelasticsearch"
#once this message has been processed by all modules, index
)
   }

app1.conf

   if $!app equals "app1" then {
#normalize (davidlang says it's better to have 1 normalizer on
   core.conf. I need to think about it)
#add some custom fields
#set $!index="myindexname--MM-DD"
call geoip
stop
   }

app2.conf

   if $!app equals "app2" then {
#normalize using inline rulebase
#set $!index="otherindexname";
stop
   }

app200.conf

   if $!app equals "app20" then {
#whatever
   }

geoip.conf

   ruleset(name="geoip") {
#geo tag this message
   }

unk.conf

   if message_has_not_been_proccessed then {
   #set $!index="unknown";
   }

I hope I explained myself properly...
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?


Hi

Is there any way to dynamically invoke a ruleset? eg: call $var
(I'm trying to avoid having +200 if statements...




El 25/11/16 a las 14:13, David Lang escribió:

On Fri, 25 Nov 2016, mosto...@gmail.com wrote:

I may be confused about which part is on the sender and which part 
is on the receiver.

sender: a bunch of imfiles forwarded using RELP

receiver: receives a JSON with msg=plain/original message, normalize 
and extract fields (that should be added to JSON)
each "application" should define his own rules, and sometimes even 
transform the JSON after that (seem's that the hard part)






  *# Is addMetadata="on" needed in order to use 
$!metadata!filename?*



the easiest thing is to try it :-)
I'm going to create an issue for double checking...can't handle this 
while editing documentation!


adding fields you may be able to do with the ammend= capabilities in 
the ruleset


if you don't want a field to be reported, give it the name '-' in 
the ruleset.


unfortunantly, you can't rename fields or copy fields in the ruleset.
So: each application having one .conf file copied to rsyslog.d/ with 
the required steps it's the only way? Perhaps something like:

*app1.conf*

  if $!group == "group" and $!app == "app1" then {
   # and here's an example on when to use inline rules
   # https://github.com/rsyslog/rsyslog/issues/625
   # an inline rule here will make it possible to have
   # just 1 config file per app, instead of 2
   action(type="mmnormalize" rulebase=:/rule-for-app1.fb")
   if $parsesuccess then {
   # do additional steps, transforms and whatever you want
   # call foo
   }
   call index
   stop
  }


it's better to have a single ruleset that does the parsing once, and 
then calls a 'perapp' ruleset that is something like


ruleset(name="perapp"){
  include /etc/rsyslog.d/apps/*
}

and in /etc/rsyslog.d/apps/ you have per app files that have the 
per-app if statement and any manipulation needed


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?


Thanks David. It helped (sadly it arrived 2 hours late :P)

I'm now dealing setting a variable with timestamp:::date-rfc5424 format.


El 01/12/16 a las 15:57, Dave Caplinger escribió:

Try:

   set $!data!foo = $programname;

As far as I know, rainerscript can't inject variables/properties into string 
literals directly; so if you really want to use string concatenation do this:

   set $!data!foo = "this_might_work_better_" & $programname;

If you want to get any more complex than that, you can use a template and exec 
it:

   template(name="s_my_programname" type="string" string="%$programname%")
   set $!data!foo = exec_template("s_my_hostname");

(but you could make the template much more complex, such as stringing multiple 
things together, adding delimiters, etc.)

Hope one of these helps,

--
Dave Caplinger
Director, Technical Product Management

On Dec 1, 2016, at 8:28 AM, mosto...@gmail.com<mailto:mosto...@gmail.com> wrote:

After meal, as usually happens, those quotes sparkled. Doesn't rsyslog
conf grammar allows that neither?

/(eg: set $!data!foo="this_doesnt_seem_to_work_$programname";)/



El 01/12/16 a las 14:57, mosto...@gmail.com<mailto:mosto...@gmail.com> escribió:

This worked, but I have lost 2 hours and still don't see where's the evil.

Works:

module(load="omrelp")
ruleset(name="relp") {
set $!data!group=field($programname,47,1);
set $!data!msg=$msg;
action(
action.reportSuspension="on"
action.resumeRetryCount="-1"
port="20514"
queue.maxdiskspace="5M"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
target="server"
template="RSYSLOG_DebugFormat"
type="omrelp"
)
}

Doesn't work

module(load="omrelp")
ruleset(name="relp") {
set $!data!foo="$programname"; <just added
this! (Line 17)
set $!data!group=field($programname,47,1);
set $!data!msg=$msg;
action(
action.reportSuspension="on"
action.resumeRetryCount="-1"
port="20514"
queue.maxdiskspace="5M"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
target="server"
template="RSYSLOG_DebugFormat"
type="omrelp"
)
}

Rsyslog complains with:

rsyslogd: error during parsing file /etc/rsyslog.conf, on or before
line 17: invalid character '"' in expression - is there an invalid
escape sequence somewhere? [v8.23.0 try http://www.rsyslog.com/e/2207 ]


El 01/12/16 a las 11:14, Rainer Gerhards escribió:
maybe the complete debug log would also help (not sure).
Rainer

2016-12-01 11:12 GMT+01:00 David Lang<da...@lang.hm<mailto:da...@lang.hm>>:
no, we need to see the contents of programname (the RSYSLOG_DebugFormat will
show this.

David Lang
k

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Confidentiality Notice: The content of this communication, along with any 
attachments, is covered by federal and state law governing electronic 
communications and may contain confidential and legally privileged information. 
If the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution, use or copying of the 
information contained herein is strictly prohibited. If you have received this 
communication in error, please immediately contact us by telephone at 
402.361.3000 or e-mail secur...@solutionary.com.

Copyright 2000-2016 NTT Security (US) Inc., a

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

After meal, as usually happens, those quotes sparkled. Doesn't rsyslog 
conf grammar allows that neither?


/(eg: set $!data!foo="this_doesnt_seem_to_work_$programname";)/



El 01/12/16 a las 14:57, mosto...@gmail.com escribió:


This worked, but I have lost 2 hours and still don't see where's the evil.

Works:

module(load="omrelp")
ruleset(name="relp") {
set $!data!group=field($programname,47,1);
set $!data!msg=$msg;
action(
action.reportSuspension="on"
action.resumeRetryCount="-1"
port="20514"
queue.maxdiskspace="5M"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
target="server"
template="RSYSLOG_DebugFormat"
type="omrelp"
)
}

Doesn't work

module(load="omrelp")
ruleset(name="relp") {
set $!data!foo="$programname"; <just added
this! (Line 17)
set $!data!group=field($programname,47,1);
set $!data!msg=$msg;
action(
action.reportSuspension="on"
action.resumeRetryCount="-1"
port="20514"
queue.maxdiskspace="5M"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
target="server"
template="RSYSLOG_DebugFormat"
type="omrelp"
)
}

Rsyslog complains with:

rsyslogd: error during parsing file /etc/rsyslog.conf, on or before 
line 17: invalid character '"' in expression - is there an invalid 
escape sequence somewhere? [v8.23.0 try http://www.rsyslog.com/e/2207 ]



El 01/12/16 a las 11:14, Rainer Gerhards escribió:

maybe the complete debug log would also help (not sure).
Rainer

2016-12-01 11:12 GMT+01:00 David Lang<da...@lang.hm>:

no, we need to see the contents of programname (the RSYSLOG_DebugFormat will
show this.

David Lang
k

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?


This worked, but I have lost 2 hours and still don't see where's the evil.

Works:

   module(load="omrelp")
   ruleset(name="relp") {
set $!data!group=field($programname,47,1);
set $!data!msg=$msg;
action(
action.reportSuspension="on"
action.resumeRetryCount="-1"
port="20514"
queue.maxdiskspace="5M"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
target="server"
template="RSYSLOG_DebugFormat"
type="omrelp"
)
   }

Doesn't work

   module(load="omrelp")
   ruleset(name="relp") {
set $!data!foo="$programname"; :

no, we need to see the contents of programname (the RSYSLOG_DebugFormat will
show this.

David Lang
k

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] about imfile




A message without TAG (malformed RFC 3164 message), no matter if it's read
from file or it arrives from socket, won't have a tag
Hence, setting it only for imfile won't fix it for socket modules.

I am not ready for this discussion again. In rsyslog, rfc3164 messages
always have a tag. See previous lengthy discussions ;-)

You misread my message, but your other response just solved the thread ;)
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] about imfile


now, that makes sense! :D

Thanks


El 01/12/16 a las 13:06, Rainer Gerhards escribió:

2016-12-01 12:55 GMT+01:00 Rainer Gerhards <rgerha...@hq.adiscon.com>:

2016-12-01 11:54 GMT+01:00 mosto...@gmail.com <mosto...@gmail.com>:

because a syslog message contains tag.

mind-blowing explanation :P

Well, as the property is already there, why would you like to have a
config parameter for something that by definition will never be
needed?

A, I think I just understand where we have the misunderstanding:

im(p)tcp by definition processes syslog messages
imfile by definition processes text file lines (which are NOT syslog messages)
so im(p)tcp always has a tag, and hence needs no config parama
where imfile by definition does not have a tag and thus needs one configured.

Does that help?
Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] about imfile


El 01/12/16 a las 12:55, Rainer Gerhards escribió:

2016-12-01 11:54 GMT+01:00 mosto...@gmail.com <mosto...@gmail.com>:

because a syslog message contains tag.

mind-blowing explanation :P

Well, as the property is already there, why would you like to have a
config parameter for something that by definition will never be
needed?
A RFC 3164 formatted message contains a tag, no matter if it's read from 
file or it arrives from socket.

Hence, there's no need to have a TAG property for any of them.

A message without TAG (malformed RFC 3164 message), no matter if it's 
read from file or it arrives from socket, won't have a tag

Hence, setting it only for imfile won't fix it for socket modules.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] about imfile




read modes other than 0 currently seem to have issues in inotify mode

   Any open issues? it's an based-on-experienced-warning message? legacy?



I am not aware of one, which does not necessarily mean none exists. So
you need to check the issue trackers :-(

The longer-term question is if we should grandfather readMode. The
performance difference seems not to be much, and a single approach is
much better to maintain.
I won't remove that until it has been confirmed or not if that's 
actually an issue.








no, because other input modules don't hard-code these values, they set
them based on the message they receive. It doesn't make sense to have them
apply to all modules.

I don't understand your reasoning here.
Why it makes sense to set tag when using imfile but not with imtcp?

because a syslog message contains tag.

mind-blowing explanation :P

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] about imfile

El 30/11/16 a las 22:51, David Lang escribió:

On Wed, 30 Nov 2016, mosto...@gmail.com wrote:

According to documentation:

State files are used to track which parts of the monitored file are
already processed.

Do state files keep just "last reading position" or as doc suggests
a file can be processed in multiple chunks(parts)?

I'd have to look at the format of the state file to be absolutly sure,
but I think it just keeps track of the last reasing position. I don't
think you can have multiple threads reading the same file, so if you
read a file in chunks, each time you read a chunk it advances the
position.

I asked 'cause documentation wasn't clear enough for me.

Note that when $WorkDirectory is not set or set to a non-writable
location, the state file **will not be generated**.

Am I wrong or state files are written to / in this scenario?

no, without a work directory set, they don't get written to /. As the
doc says, they just don't get written anywhere.
This is not what is happening on my tests. Setting WorkDirectory to
non-existing directory make it create imfile-state on /. Just opened an
issue.

Regarding pollinginterval: During each polling interval, all files
are processed in a round-robin fashion.

I'm confused. Does this mean files are readed, sleep for X seconds,
and readed again...
or rsyslog reads documents during X seconds looping in a round-robin
fashion?

the first.

Thanks

readtimeout: This can be used with *startmsg.regex* (but not *readMode*)

Why it can't be used with readmode? (Apart from obviously not
implemented)

just not implemented (I actually expected that it would be implemented
for readmodes)

read modes other than 0 currently seem to have issues in inotify mode

Any open issues? it's an based-on-experienced-warning message? legacy?

good question

Rainer?

imfile has tag, facility and severity properties...

Is there any way this properties being /inherited/ for ALL modules?
(hence documented on "/input-modules/")

no, because other input modules don't hard-code these values, they set
them based on the message they receive. It doesn't make sense to have
them apply to all modules.

I don't understand your reasoning here.
Why it makes sense to set tag when using imfile but not with imtcp?

@radu-gheorghe @rgerhards could you have a look at
https://github.com/mostolog/rsyslog-doc/blob/imfile/source/configuration/modules/imfile.rst

my comments

re: examples needed TODOs, are these items really needed? It seems to
me that the explinations are pretty clear, I could see examples adding
as much confusion as clarification.

Ok.

re: windows/inode, this documentation is about the unix version. the
windows version is slightly different (it has a GUI amoung other
things), and it isn't free.

it's not always clear why you have TODO there. In most cases, the text
following the TODO seems appropriate, could you change this to either
put the description of what needs to change on it's own line, or
otherwise indicate what needs to be changed?

TODO=>WIP :P

I would group all the EXPERT options in one section, with the big
warning at the top of them that if you don't understand them you
should not set them.

LGTM

I would also add a warning that they almost never need to be changed,
even on high load systems, so benchmarks should be run before and
after changing any of them because they sometimes have non-intuitive
performance impact.

I would not set escapelf as an expert option, but rather make a
grouping of options under the category "dealing with multi-line logs"
and put it there along with readmode, regex.startmsg and the related
timeouts.

Ok.

trimlineoverbytes should actually apply to all modes, why only to some?

I'm just a monkey typing...ask someone who knows!

is reopen on truncate really still experimental?

It was marked so...

I would put the depriciated items in their own section.

Done

Thank you a lot David.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?


I don't know if this is what you are asking for...

3421.997587883:main Q:Reg/w0  : SET !data!aapp =
3421.997596172:main Q:Reg/w0  :   function 'field' (id:9, params:3)
3421.997607766:main Q:Reg/w0  : var 'programname'
3421.997687716:main Q:Reg/w0  : 47
3421.997714715:main Q:Reg/w0  : 2
3421.997728731:main Q:Reg/w0  : END SET



El 30/11/16 a las 22:32, Rainer Gerhards escribió:

Can you show what the properties are for this message?

Sent from phone, thus brief.

Am 30.11.2016 20:18 schrieb "mosto...@gmail.com" <mosto...@gmail.com>:


Hi

I'm still not able to get it, and *perhaps reproduced an issue related to
properties not being accessible*...
Consider the following relay.conf:

global(
MaxMessageSize="32k"
workDirectory="/var/spool/rsyslog"
parser.escapeControlCharactersOnReceive="off"
)

template(
 name="json"
string="<%pri%>%timestamp:::date-rfc3339% %hostname%
logs/%$!data!group%/%$!data!aapp%: %$!data%"
 type="string"
)

module(load="omrelp")
ruleset(name="relp") {
 set $!data!aapp=field($programname,"/",2);
 set $!data!file="$!metadata!filename";
 set $!data!group=field($programname,"/",1);
 set $!data!msg=$msg;
 action(
action.reportSuspension="on"
action.resumeRetryCount="-1"
port="20514"
queue.maxdiskspace="5M"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
target="server"
template="json"
type="omrelp"
 )
}
ruleset(name="apps") {
 call relp
 stop
}

module(load="imfile")

input(type="imfile" file="/logs/apache/app1/app.log"
tag="group/app1" addMetadata="on" ruleset="apps"
PersistStateInterval="1")

input(type="imfile" file="/logs/apache/app2/app.log"
tag="group/app2" addMetadata="on" ruleset="apps"
PersistStateInterval="1")
ruleset(name="app_server1") {
 set $!data!containerApps="app1,app2";
 call relp
 stop
}

input(type="imfile" file="/logs/server1/app1.log"
tag="group1/server1" addMetadata="on" ruleset="app_server1"
startmsg.regex="^" readTimeout="5" PersistStateInterval="1")
input(type="imfile" file="/logs/server1/app2.log"
tag="group1/server1" addMetadata="on" ruleset="app_server1"
startmsg.regex="^" readTimeout="5" PersistStateInterval="1")

ruleset(name="app_server2") {
 set $!data!containerApps="app2,app3";
 call relp
 stop
}
input(type="imfile" file="/logs/server2/app2.log"
tag="group2/server2" addMetadata="on" ruleset="app_server2"
startmsg.regex="^" readTimeout="5" PersistStateInterval="1")
input(type="imfile" file="/logs/server2/app3.log"
tag="group2/server2" addMetadata="on" ruleset="app_server2"
startmsg.regex="^" readTimeout="5" PersistStateInterval="1")


With this configuration I'm getting messages like:

200 syslog 911 <133>2016-11-30T20:02:30.210405+01:00 my-rsyslog
logs/group/***FIELD NOT FOUND***: { "aapp": "***FIELD NOT FOUND***",
"file": "\/logs\/apache\/app1\/app.log", "group": "group", "msg":
"17 mar 2016 13:27:28,934  INFO REDACTED..." }


Why /aapp/ field isn't found?

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] about imfile

According to documentation:

State files are used to track which parts of the monitored file are
already processed.

Do state files keep just "last reading position" or as doc suggests
a file can be processed in multiple chunks(parts)?

Note that when $WorkDirectory is not set or set to a non-writable
location, the state file **will not be generated**.

Am I wrong or state files are written to / in this scenario?

Regarding pollinginterval: During each polling interval, all files are
processed in a round-robin fashion.

I'm confused. Does this mean files are readed, sleep for X seconds,
and readed again...
or rsyslog reads documents during X seconds looping in a round-robin
fashion?

readtimeout: This can be used with *startmsg.regex* (but not *readMode*)

Why it can't be used with readmode? (Apart from obviously not
implemented)

read modes other than 0 currently seem to have issues in inotify mode

Any open issues? it's an based-on-experienced-warning message? legacy?

imfile has tag, facility and severity properties...

Is there any way this properties being /inherited/ for ALL modules?
(hence documented on "/input-modules/")

@radu-gheorghe @rgerhards could you have a look at
https://github.com/mostolog/rsyslog-doc/blob/imfile/source/configuration/modules/imfile.rst
? Feedback appreciated.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?


Hi

I'm still not able to get it, and *perhaps reproduced an issue related 
to properties not being accessible*...

Consider the following relay.conf:

   global(
   MaxMessageSize="32k"
   workDirectory="/var/spool/rsyslog"
   parser.escapeControlCharactersOnReceive="off"
   )

   template(
name="json"
   string="<%pri%>%timestamp:::date-rfc3339% %hostname%
   logs/%$!data!group%/%$!data!aapp%: %$!data%"
type="string"
   )

   module(load="omrelp")
   ruleset(name="relp") {
set $!data!aapp=field($programname,"/",2);
set $!data!file="$!metadata!filename";
set $!data!group=field($programname,"/",1);
set $!data!msg=$msg;
action(
   action.reportSuspension="on"
   action.resumeRetryCount="-1"
   port="20514"
   queue.maxdiskspace="5M"
   queue.SaveOnShutdown="on"
   queue.type="LinkedList"
   target="server"
   template="json"
   type="omrelp"
)
   }
   ruleset(name="apps") {
call relp
stop
   }

   module(load="imfile")

   input(type="imfile" file="/logs/apache/app1/app.log"
   tag="group/app1" addMetadata="on" ruleset="apps"
   PersistStateInterval="1")

   input(type="imfile" file="/logs/apache/app2/app.log"
   tag="group/app2" addMetadata="on" ruleset="apps"
   PersistStateInterval="1")
   ruleset(name="app_server1") {
set $!data!containerApps="app1,app2";
call relp
stop
   }

   input(type="imfile" file="/logs/server1/app1.log"
   tag="group1/server1" addMetadata="on" ruleset="app_server1"
   startmsg.regex="^" readTimeout="5" PersistStateInterval="1")
   input(type="imfile" file="/logs/server1/app2.log"
   tag="group1/server1" addMetadata="on" ruleset="app_server1"
   startmsg.regex="^" readTimeout="5" PersistStateInterval="1")

   ruleset(name="app_server2") {
set $!data!containerApps="app2,app3";
call relp
stop
   }
   input(type="imfile" file="/logs/server2/app2.log"
   tag="group2/server2" addMetadata="on" ruleset="app_server2"
   startmsg.regex="^" readTimeout="5" PersistStateInterval="1")
   input(type="imfile" file="/logs/server2/app3.log"
   tag="group2/server2" addMetadata="on" ruleset="app_server2"
   startmsg.regex="^" readTimeout="5" PersistStateInterval="1")


With this configuration I'm getting messages like:

   200 syslog 911 <133>2016-11-30T20:02:30.210405+01:00 my-rsyslog
   logs/group/***FIELD NOT FOUND***: { "aapp": "***FIELD NOT FOUND***",
   "file": "\/logs\/apache\/app1\/app.log", "group": "group", "msg":
   "17 mar 2016 13:27:28,934  INFO REDACTED..." }


Why /aapp/ field isn't found?

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] auto-generating module documentation


> Is there some standard tool that can generate docs from C source?

doxyfile


Said that, after Rainer's reply, I have nothing to add :)


El 30/11/16 a las 10:27, Rainer Gerhards escribió:

Longer reply follows a bit later. I do not like this idea. It complicates
things, is a lot of work (at least a couple of days), and boils down to
that everyone but developers loses the ability to write module doc.

Rainer

Sent from phone, thus brief.

Am 30.11.2016 09:36 schrieb "David Lang" <da...@lang.hm>:


On Wed, 30 Nov 2016, mosto...@gmail.com wrote:

Probably a stupid idea...will it make sense that information being

populated from rsysloc-doc? (or viceversa, like javadoc)


Since this is the code that creates parameter and variable names in the
modules, I don't see how the -doc project could push it into the source

I'm thinking that rsyslog-doc would create these files from the rsyslog
source (or that the rsyslog source would output the data to be picked up by
the doc process)

Having everything on one place looks great to me
the question is how.

Part of this comes down to nuances in C that I don't know. If we add two
string pointers to the array, can we declare the array with constants and
have the compiler store the constants somewhere and create the appropriate
pointers to them

Is there some standard tool that can generate docs from C source? I think
I've seen people talk about sphinx (or something similar), but I've never
followed things very closely, and I don't have any idea if it can deal with
things in an array.


And then there is the problem of backwards compatibility. Is there a way
to make this change a module at a time? or do we have to change all of
rsyslog (and break any out-of-tree modules) in a single step?

David Lang




El 28/11/16 a las 17:50, David Lang escribió:


we have a few cases where the documentation doesn't match the module
parameters (wrong action parameters shown). While we can go through and fix
the ones that we find as we find them, I think we should look at finding a
way to make this more automated

Is there a reasonable way to expand the array that defines the v6
parameters, which currently defined as:

/* the following defines describe the parameter block for puling
  * config parameters. Note that the focus is on ease and saveness of
  * use, not performance. For example, we address parameters by name
  * instead of index, because the former is less error-prone. The (severe)
  * performance hit does not matter, as it is a one-time hit during config
  * load but never during actual processing. So there is really no reason
  * to care.
  */
struct cnfparamdescr { /* first the param description */
 const char *name;/**< not a es_str_t to ease definition in code
*/
 ecslCmdHdrlType type;
 unsigned flags;
};

to contain the default value and a description.

Then if we can add a module general description field, we should have
the ability to create allmost all of the module documenation pages
automatically (except for the legacy config stuff, and that could be either
added as another variable in the module, or maintained outside of the
module)

Thoughts?

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE

Re: [rsyslog] auto-generating module documentation

Probably a stupid idea...will it make sense that information being 
populated from rsysloc-doc? (or viceversa, like javadoc)


Having everything on one place looks great to me


El 28/11/16 a las 17:50, David Lang escribió:
we have a few cases where the documentation doesn't match the module 
parameters (wrong action parameters shown). While we can go through 
and fix the ones that we find as we find them, I think we should look 
at finding a way to make this more automated


Is there a reasonable way to expand the array that defines the v6 
parameters, which currently defined as:


/* the following defines describe the parameter block for puling
 * config parameters. Note that the focus is on ease and saveness of
 * use, not performance. For example, we address parameters by name
 * instead of index, because the former is less error-prone. The (severe)
 * performance hit does not matter, as it is a one-time hit during config
 * load but never during actual processing. So there is really no reason
 * to care.
 */
struct cnfparamdescr { /* first the param description */
const char *name;/**< not a es_str_t to ease definition in 
code */

ecslCmdHdrlType type;
unsigned flags;
};

to contain the default value and a description.

Then if we can add a module general description field, we should have 
the ability to create allmost all of the module documenation pages 
automatically (except for the legacy config stuff, and that could be 
either added as another variable in the module, or maintained outside 
of the module)


Thoughts?

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?



I may be confused about which part is on the sender and which part is 
on the receiver.

sender: a bunch of imfiles forwarded using RELP

receiver: receives a JSON with msg=plain/original message, normalize and 
extract fields (that should be added to JSON)
each "application" should define his own rules, and sometimes even 
transform the JSON after that (seem's that the hard part)







  *# Is addMetadata="on" needed in order to use $!metadata!filename?*



the easiest thing is to try it :-)
I'm going to create an issue for double checking...can't handle this 
while editing documentation!


adding fields you may be able to do with the ammend= capabilities in 
the ruleset


if you don't want a field to be reported, give it the name '-' in the 
ruleset.


unfortunantly, you can't rename fields or copy fields in the ruleset.
So: each application having one .conf file copied to rsyslog.d/ with the 
required steps it's the only way? Perhaps something like:

*app1.conf*

   if $!group == "group" and $!app == "app1" then {
# and here's an example on when to use inline rules
# https://github.com/rsyslog/rsyslog/issues/625
# an inline rule here will make it possible to have
# just 1 config file per app, instead of 2
action(type="mmnormalize" rulebase=:/rule-for-app1.fb")
if $parsesuccess then {
# do additional steps, transforms and whatever you want
# call foo
}
call index
stop
   }


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?



string="<%pri%>%timestamp:::date-rfc3339% %hostname% 
logs/$!data!group/$!data!app: %$!data%") to be fully correct

Done.

also watch out, the programname is limited to 32 characters, don't let 
your group and app names get too long.
Wasn't it possible to change that? IIRC we had some issues with 
hostnames/tags being too long and were able to handle longer.

Anyway, we'll try to stay within boundaries.

since you changed the programname to be logs/group/app this would be 
field 3

Are you sure?
*ruleset apps* is invoked for each input using TAG=group/app, and AFTER 
that ruleset relp uses template json, which prefix "logs/"





  *# Is addMetadata="on" needed in order to use $!metadata!filename?*

I think so.
As I'm double checking everything while updating docs, I would love to 
have a more confident statement on this. Rainer?


it's probably a good idea to put stop here to make it clear that you 
don't intend for there to be any other processing of the log message. 
With the input bound to a ruleset, I don't think it makes a 
difference, but better to be explicit.

Done

correct, although mmjsonparse defaults to needing @cee: in front of 
the json, so the line below needs to be changed to:


module(load="mmjsonparse" cookie="")
I forgot! Nice catch (...I'll have to check if cookie goes in module or 
action...)


yes, the script can either populate the rules file with includes, or 
just combine them into one file (probably faster at startup to have 
them combined, but it may not be measureable)

I'll combine them (if able)




  *# Once all operations have ended, it should be indexed**
  **# Is there any way apps not only define rules, but aditional
  transformations?**
  **# I guess having a .conf file with if+ruleset could work...*
no, the mmnormalize ruleset cannot apply any transformations. I would 
probably try to do that on the sending side if I could.

That's why I played with rulesets to make something like

   a.conf
  normalize
  add some fields
   b.conf
  normalize
   c.conf
  normalize
  remove some fields

That would make the combination script behave differently




  *# It is possible to use $!index here? Workaround?**

this is what dynsearchindex and dynparent are for.
so: dynSearchIndex="on" searchIndex="mytemplate" and template="$!index", 
right?





  **# How could EACH app specify his own index pattern?**
they can't directly, but the template can be "%$.manual%" and you use 
rainerscript commands to set $.manual to whatever you want it to me 
(another good use for a lookup table if it's complex enough ;-)

I didn't understand this...but having each app.conf file could also work.

The background idea is to combine this with "dynamic configuration 
reload" to be able to change "an application pipeline"


This is looking pretty good now.

Thanks. I think the same!

Seems next pending issue is to solve "each application can do different 
things, like adding hiw own index pattern or additional 
transforms/steps" in his config file.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?


What about...?

*remote.conf*(Please, notice there are commented questions)

   global(
   MaxMessageSize="32k"
)

   template(name="json" type="string"
   string="%timestamp:::date-rfc3339% %hostname%
   logs/$!data!group/$!data!app %$!data%")

   module(load="omrelp")
   ruleset(name="relp"){
   action(
   port="20514"
   *# It is possible to use $!server here? Workaround?*
   target="server"
   template="json"
   type="omrelp"
)
}

   ruleset(name="apps") {
set $!data!app=field($programname,"/",2);
   *# Is addMetadata="on" needed in order to use $!metadata!filename?*
set $!data!file="$!metadata!filename";
set $!data!group=field($programname,"/",1);
set $!data!msg=$msg;
   call relp
}

   input(type="imfile" file="/logs/apps/app1/app1.log"
   tag="mygroup/myapp1" addMetadata="on" ruleset="apps"
   startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
   readTimeout="5" PersistStateInterval="1")

...

   input(type="imfile" file="/logs/apps/anotherapp/file.log"
   tag="anothergroup/anotherapp" addMetadata="on" ruleset="apps"
   readTimeout="5" PersistStateInterval="1")


*rsyslog.conf*(Please, notice there are commented questions)

   global(
   MaxMessageSize="32k"
   parser.escapeControlCharactersOnReceive="off"
)

   *# Message is parsed as json on receive, to be able to use
   $!whatever field, right?*
   module(load="mmjsonparse")
   ruleset(name="json"){
   action(
   type="mmjsonparse"
)
}
   module(load="imrelp")
input(
   name="imrelp"
   port="20514"
   type="imrelp"
   ruleset="json"
)

set $.line = $!group $!app + " " + $!msg;
action(
   type="mmnormalize"
   variable="$.line"
   *# As I don't know the list of apps, **
   **# the only way to combine all rules is an script**
   **# isnt it?*
   rulebase=:/path/to/combined/rules.fb"
)

   *# IIUC, messages will be procesed by above rule**
   **# AFTER that, they will be processed by the following, right?*
if message contains "ip" field then {
# TODO lookup_table
}

   *# Once all operations have ended, it should be indexed**
   **# Is there any way apps not only define rules, but aditional
   transformations?**
   **# I guess having a .conf file with if+ruleset could work...*

   template(name="json" type="string" string="%$!%")
   module(load="omelasticsearch")
action(
   template="json"
   type="omelasticsearch"
   *# It is possible to use $!index here? Workaround?**
   **# How could EACH app specify his own index pattern?**
   **#   set $!index="$!app2_$$year-$$month-$$day"**
   **#   set $!index="$!app2_$$year-$$month-$$day-$$hour"*
   searchIndex="$!index"
)


The background idea is to combine this with "dynamic configuration 
reload" to be able to change "an application pipeline"

Thanks a lot for your help. I'll contribute as much as I can in exchange ;)


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

Agains't Rainer's advice, we are using adiscon repos and we're quite 
happy with them...



El 25/11/16 a las 09:59, Kosta Psimoulis escribió:

Thank you for much guys for your support, right now I have a much better
picture of what is going on. I am aware how to build from source but this
would be something difficult to maintain, I would probably need to create a
custom repo. I think I have enough information right now to reconsider and
evaluate a business decision whether to use rsyslog or look for another
solution. I have some queue files that I can recover and rebuild with the
information you have given but I am still not sure of how they got
corrupted and what happened to the ones that were in memory, was everything
saved on the hard drive or was there information that was lost.

Kind Regards,
Kosta

On Fri, Nov 25, 2016 at 3:42 AM, David Lang <da...@lang.hm> wrote:


On Fri, 25 Nov 2016, mosto...@gmail.com wrote:

TBH, it depends if you prefer building from source or instability

Kidding away, I know there are problems in older versions, they are


Isn't adiscon repo valid?


It looks like we don't have a repo for Jessie, at least it's not included
in the scripts/config.sh for rsyslog-pkg-debian

This probably also means we don't have a Travis test box running Jessie.

Packages for Wheezy will probably work (the only thing I can think of that
would be likely to break is gnutls related dependencies)

David Lang

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

After sleeping on it, I'm still thinking about "defining a separate 
pipeline for each application". To sum up, each application could do his 
own thing and return the message to the queue, in order to be processed 
by other modules...until it's done, and indexed into ES
Is there a way to put the already processed message into the input queue 
to be processed again by another *sibling* modules?


   input-> queue -> app
   <-
   -> geoip
   <-
   -> index

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?


Thanks!

It's your mmdblookup opensourced?


El 25/11/16 a las 03:46, chenlin rao escribió:

re-upload an english version. The content was a little old though.

2016-11-23 22:39 GMT+08:00 mosto...@gmail.com <mosto...@gmail.com>:


http://www.slideshare.net/chenryn/elk-stack-at-weibocom

I NEED the english version :P

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue




TBH, it depends if you prefer building from source or instability
Kidding away, I know there are problems in older versions, they are

Isn't adiscon repo valid?
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] making config changes to a running rsyslog




what are people's thoughts on these ideas?


Notice there can be multiple reload scenarios:
 - reload rsyslog config (new modules, inputs, rulesets, actions...)
 - add new inputs
 - modify a template
 - resize a queue

The simplest approach I can imagine is to signal HUP to reload: when 
signal is received, everything is stopped, reloaded and resumed. It may 
be faster than restart, cause modules are already loaded or objects 
(templates, inputs...) still in memory.


There's a lot of space for improvement: unload unneeded modules, restart 
only modified objects, rollover updates...but TBH I don't know if I 
would go to such API.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?


Once I use it ;)

Yet have to change overview to RST...


El 24/11/16 a las 17:36, Rainer Gerhards escribió:

2016-11-24 17:21 GMT+01:00 mosto...@gmail.com <mosto...@gmail.com>:

# I would consider adding a section here to look for parsing failures and
log them to someplace for later investigation, probably in raw format

That's mandatory. is there any easy way to catch norm failures?

Have a look here:

http://www.rsyslog.com/using-mongodb-with-rsyslog-and-loganalyzer/

$parsesuccess is your friend... and looking at it, it doesn't seem
documented. Would you like to document it or add an rsyslog-doc issue
tracker?

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?


El 24/11/16 a las 17:42, Rainer Gerhards escribió:

I added a project to rsyslog on github, where we can bind Issues to:
shouldn't that work for documentation? (as commented previously, I would 
love to have 1 repo!)



https://github.com/rsyslog/rsyslog/projects/1

I guess I must make the relationship, just let me know what you think
qualifies whenever you open something new.
Maybe you could match project to milestones, but I think projects are 
"wider".


BTW: With Bob's approval, I think a good application name can be TREK 
(trekkies could become angry :P)




Rainer

2016-11-24 13:27 GMT+01:00 Rainer Gerhards :

2016-11-24 12:52 GMT+01:00 Bob Gregory :

https://io.made.com/blog/rek-it/

I wrote this up earlier.

very good! Love to see the work coming in and participate in the effort!

Rainer

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?



this way you use the tag that you set on the input line to create the 
metadata fields you want with only one ruleset for all the app inputs.

cool!

The other thing you could do is to create a table lookup to map the 
programname (the value you set as the tag in the input) to the values 
you want. This would let you use arbitrary values for tags rather than 
having to have them follow a specific format.
Or even it could help me to link groups and apps, in order not to 
hardcode them.

I'll let that for version 2, if you don't mind :P


if $group == ["group1","group2"] then {
  set $.logline = $programname + " " + $!msg;
  action(name="parseapp" type="mmnormalize" variable="$.logline" 
rulebase=:/path/to/combined/rules.fb")

  call index
  stop
}

I'm not sure I understood this, so let me ask a few questions:
 - what you are proposing is mmnormalize to parse "APP MSG", isn't it?
 - I guess you are parsing "any group $programname", cause same 
$programname may share same lognorm format, isn't it?
 - how could I combine multiple rule files without hardcoded includes? 
I guess my only option is a script that merges...


I like the way you solved that with variable :D
Pending further testing/playing.

# I would consider adding a section here to look for parsing failures 
and log them to someplace for later investigation, probably in raw format

That's mandatory. is there any easy way to catch norm failures?

This way you can do per-app parser files and include them in your 
combined rulebase file if you want.


This still leaves you listing the groups in the main file. If this is 
enough of a problem, you could create a table that mapped all known 
programnames to 'known' and the test would be something like


if lookup_table("apps","$programname") == "known" then {

(with some other stuff earlier to load the table)
I'll have a look on lookup_tables later, although they seem to be great 
savers!



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?


or docker swarm mode :D


El 24/11/16 a las 16:22, David Lang escribió:

On Thu, 24 Nov 2016, mosto...@gmail.com wrote:

As we are concerned about high availability and load balancing, we 
plan to deploy multiple instances.


just a note that while rsyslog doesn't implement load balancing 
itself, it has features to support load balancing environments, so you 
pick the load balancer you want on the receiving end and have rsyslog 
disconnect every X messages to give the load balancer a chance to work.


I think this only works if you do IP based load balancing, rather than 
DNS based load balancing (especially as so many systems now run a 
caching DNS locally)


Personally, I use corosync (clusterlabs.org) but you can also use 
haproxy, lvs, or a commercial load balancer like f5


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?



not really, but we haven't had anyone experiment with thousands of 
them, so it's possible, but unlikely that there would be a measureable 
slowdown as rsyslog finds the right one to use.


The bigger overhead is in interpreting the template, that's where 
simplifying it to be $! or $!foo would be a big win (or writing a 
string module)


Memory went above 5GB for our first dirty try (several rulesets, several 
queues...). I'll change that soon.


there isn't a good writeup, but if you read on how to use the maxmind 
database, the perl example has you create an array where the first 
element is the decimal equivalent of the first IP address that matches 
the data.


This is exactly the structure that a sparse array lookup table is 
intended for. I beleive there is a function that will take an IPv4 
address and return a decimal number (if not, we need to add one). Use 
that function to create a number, lookup the number in the lookup 
table, and have it return the data.


The second paragraph is correct, however I haven't used them yet in 
rsyslog. I'll document them then.


Thanks a lot, David, for your kind help, experienced comments and wise 
advice.

You deserve another prize ;)
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?


Last hours have been quite busy!

After reading your comments, here's where we are. Please, advice and 
comment.


On some remote hosts, logs are readed from files. Each remote will have 
a configuration like:


   # remote.conf
global(
   MaxMessageSize="32k"
)
   template(name="json" type="string"
   string="%timestamp:::date-rfc3339% %hostname%
   logs/$!data!group/$!data!app $!data")

   module(load="omrelp")
   ruleset(name="relp"){
   action(
   port="20514"
   target="server"
   template="json"
   type="omrelp"
)
}

   ruleset(name="myapp1") {
set data!app="myapp1"
   $!data!file="$!metadata!filename"
   $!data!group="mygroup"
   $!data!msg="%msg"
   call relp
}
   input(type="imfile" file="/logs/apps/app1/app1.log"
   tag="mygroup/myapp1" addMetadata="on" ruleset="myapp1"
   startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
   readTimeout="5" PersistStateInterval="1")

   ruleset(name="myapp2") {
set data!app="myapp2"
   $!data!file="$!metadata!filename"
   $!data!group="mygroup"
   $!data!msg="%msg"
   call relp
}
   input(type="imfile" file="/logs/apps/app2/app2.log"
   tag="mygroup/myapp2" addMetadata="on" ruleset="myapp2"
   readTimeout="5" PersistStateInterval="1")



As depicted above, logs will be sent to server using RELP. We plan to 
make a config-generator-script, to add imfile+ruleset for each file.
If you know a better way for having multiple files, each having their 
own information, just let me know.


We also played a bit with mmpstrucdata, but seems json is easier (once 
you need it for elastic)


On the server side, we would like to accept and index any groups 
applications, but each app config should be on their own file, making it 
easier to change, understand...
Our current approach is based on copying multiple files to rsyslog.d 
directory, being all configurations loaded. This is what we got so far:


   # core.conf
global(
MaxMessageSize="32k"
parser.escapeControlCharactersOnReceive="off"
)
module(load="imrelp")
input(
port="20514"
type="imrelp"
name="imrelp"
)

template(name="json" type="list") {
constant(value="{")
property(name="$!msg")
constant(value="}")
}
module(load="omelasticsearch")
ruleset(name="index"){
action(
type="omelasticsearch"
template="json"
searchIndex="$!index"
)
}
module(load="mmnormalize")

# app1.conf
# Perhaps something like $!group == "mygroup" could work
if $syslogtag startswith "logs/group/" then {
rule=:%[
{"type":"ipv4", "name":"ip"},
{"type":"literal", "text:" "},
{"type":"literal", "text:"-"},
{"type":"rest", "name":"r"}
]%
set index="$!app_$$year-$$month-$$day"
action(type="mmnormalize" rule="$rule" version="2")
call index
stop
}
# app2.conf
if $!app == "myapp2" then {
rule=:%[
{"type":"ipv4", "name":"ip"},
{"type":"literal", "text:" "},
{"type":"literal", "text:"-"},
{"type":"rest", "name":"r"}
]%
action(type="mmnormalize" rule="$rule" version="2")
call index
stop
}


*Does it makes sense for you? Any improvements? Anything that can't be 
done?*

@bobthemighty: feedback!

Thanks in advance
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?


Doing the **same** here.

Currently I'm dealing with https://github.com/rsyslog/rsyslog/issues/625 
in order to have "one configuration file for each application", and 
copying them to rsyslog.d directory.


As we are concerned about high availability and load balancing, we plan 
to deploy multiple instances.
Still pending to decide if RELP->ES is done by the same rsyslog process 
or spplited in several stages.


Any discussion is much appreciated and highly valuable :)


El 24/11/16 a las 12:52, Bob Gregory escribió:

https://io.made.com/blog/rek-it/

I wrote this up earlier.

On Wed, 23 Nov 2016 at 19:38 mosto...@gmail.com <mosto...@gmail.com> wrote:


Working, spamming mail list and writing on wiki at the same time. A
lovely afternoon...

Please, add your lines: https://github.com/rsyslog/rsyslog/wiki
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Coding doc


I mean https://github.com/rsyslog/rsyslog-doc/pull/188


El 23/11/16 a las 21:25, Rainer Gerhards escribió:

Sent from phone, thus brief.

Am 23.11.2016 21:01 schrieb "mosto...@gmail.com" <mosto...@gmail.com>:

What did happen to the "new documentation format" effort?

It's well alive and I would object a change without a very good reason.
It's easy versioned and integrated well with distro package maintainer's
workflow.


I think wiki could be fair enough, but I don't know how it will behave

for old-releases.

If I'm not wrong, current system allow to have different documentation

pages according to each version, so perhaps it worths keep it.

On the other hand, markdown is more friendly thanwhatever is that. :P

Markdown is not simpler than RST, just has fewer features.

Rainer

El 23/11/16 a las 20:57, Rainer Gerhards escribió:

Hi all, especially histology,

I wonder if it would make sense to document coding concepts on the github
wiki - or is files better? Based on other discussions today, I think it
would make sense to write up a little bit more for potential contributors
(lowering the energy barrier).

What do you think?

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Coding doc


What did happen to the "new documentation format" effort?

I think wiki could be fair enough, but I don't know how it will behave 
for old-releases.
If I'm not wrong, current system allow to have different documentation 
pages according to each version, so perhaps it worths keep it.

On the other hand, markdown is more friendly thanwhatever is that. :P

El 23/11/16 a las 20:57, Rainer Gerhards escribió:

Hi all, especially histology,

I wonder if it would make sense to document coding concepts on the github
wiki - or is files better? Based on other discussions today, I think it
would make sense to write up a little bit more for potential contributors
(lowering the energy barrier).

What do you think?

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

Working, spamming mail list and writing on wiki at the same time. A 
lovely afternoon...


Please, add your lines: https://github.com/rsyslog/rsyslog/wiki
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project



do you have an example of what was sent over the wire and how it was 
mishandled? It may be that we need to tweak the default parsers.


https://github.com/rsyslog/rsyslog/issues/1191

no need to convert, just add the ability to use the slower, but more 
flexible mmnormalize.

You said it has bugs with rawmsg... :P


pm and mm don't work on the same data.

pm deals with the raw message that arrives on the wire

mm works with all the variables that exist at the time the mm module 
is called


mm modules can look at the data as it arrived over the wire, and they 
can also look at variables that have been created later.


I would like to see a pmmm module, or at least a pmnormalize module 
created, but I wouldn't want to try and replace the existing parsers 
with them.
IMHO it doesn't makes sense pm's parsing anything if I'll use mmnorm 
with rawmsg, right?

That's why I consider pm==mm regarding message handling.

90% of it is just getting good documentation of the over-the-wire 
protocol


the rest of it comes into play with things like the current omkafka 
problem where the module is telling rsyslog that the message delivery 
succeeded when it has really failed. That's where the deeper knowledge 
of the other system is needed.

where was the devil...? :P
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?



The problem is the fact that there are so many ways timestamp data can 
be scattered in a log message. take a look at the output of date 
--help and look at all the formatting options. I guarantee that some 
log somewhere will use every one of them.


IIRC, you had found a solution to this...
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] A couple of questions regarding rsyslog debug.




yup, the function is named DBGPRINTF() and some helpers. Being used
since ~1970 (the original dbgprintf() call stems back to the original
code ;)). Sometimes we use #ifdef DEBUG, but only if we actually need
to compute something beyond simple output generation (e.g. iterate
over a structure). This can't be done efficiently without #ifdef.

Perhaps replacing

   #ifdef foo
   whatever lines you may want
   #endif

with

   CODE_TO_HANDLE_WHATEVER macro, defined on file
   /code_to_handle_whatever_macro.c/

?

Maybe I'm wrong, but if it's between ifdef, it can be ommited at first 
level sight...



similar, a verbosity level, and partial support is available via the
RSYSLOG_DEBUG variable (you can turn on/off some debugging features).
But it's still in its infancy. Would really be good to have.

Ok!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?



you can combine variables to form a string that looks like a date in 
the output, but you can't take arbitrary date parts in a log message 
and parse them into a real timestamp field that would let you output 
it in different formats.

back on my pipeline proposal, wouldn't this solve the issue?
pipeline {
input()
processor() //extract %year%,%month%,%day%
processor() //merge "%year%:%month%:%day%" as date type 
property/field

output()
}

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] omriemann Re: Are we building an ERK stack?

As main promoter (ring the bell and run like hell), could you make some 
tests comparing filebeat vs imfile performance and footprint?



El 23/11/16 a las 19:32, Bob Gregory escribió:

I can easily enough knock together an omriemann - it's protobuf over TCP or
UDP.  TCP allows for message ack.

There are a couple of C clients that are useful as prior art, and I've
worked with a bunch of clients in python, haskell and golang.

On Wed, 23 Nov 2016 at 18:18 David Lang  wrote:


On Wed, 23 Nov 2016, Bob Gregory wrote:


For that, I'd like to see better support for GeoIP tagging, a Riemann
output plugin, some better guidance on "failed message queues", etc. etc.
etc.

With a bit of digging, I can't find where Riemann defines what the
over-the-wire
format is that you would need to deliver logs to it.

I see hints that it uses protobuf to serialize things, and has an
application-level ack mechanism similar to what we have in relp, but the
levels
of indirection are stacked high, and the API documenation only points you
at the
function defintions.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project


We both reached an agreement about exceptions and I'm happy with that :)

Actually my main concern is, each time I read a line like:

   CHKiRet(statsobj.Construct(>statsobj));

I have to start digging what CHKiRet is, or what the hell is:

   CODE_STD_STRING_REQUESTparseSelectorAct(1)

Anyway, it takes time to get used to someone else code and project...so 
don't take me too seriously.
And, btw, all this discussion and explanations are helping me a lot with 
this!

:D


El 23/11/16 a las 19:19, Rainer Gerhards escribió:

Your explanation is so simple that I'm wondering why source seems so
obscure to me...


The devil is in the details, and when you are working to make things very
fast, it gets messy.

The question to ask (honestly) is what looks obscure to you. From past
conversations I know that you tend toward C++ and exceptions, and I
have given reason why we want not use that.

If you have more points, we can work through them. Some may be legacy,
some may be needed, some may be ugly, but not important enough to
change (but maybe somebody steps in?). The code base is always
evolving, and it is doing lots of complex things. It's of course not
perfect, and we always refactored some ugly things out of it.

Having concrete pointers of what exactly looks obscure is usually the
first step in changing it (or providing reason why it is).

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project



why add the overhead of transporting the data between machines if you 
don't need to? Rsyslog is FAST, it routinely does things on one 
machine that other systems need many farms of machines to do.


adding more machines and network connectivity will add complexity and 
delays, reducing performance and reliability

Ok!



that depends how you configure it. All queues can be configured to 
operate in multiple modes

Ok. I forgot the main queue can be configured too!



there are three stages in decoding a message

1. the framing of a message (i.e. UDP a packet is a message, TCP 
newline indicates the end of a message)


2. the parsing of the message. The syslog format is a header followed 
by arbitrary text. This step is decoding the header


note: many log installations don't do anything more than this.

3. parsing the free-form text message, i.e. 'normalizing' the logs

I've made the suggestion in the past that we create a pmnormalize that 
uses liblognorm rules against the raw message and can populate the 
standard properties, but it hasn't been a high priority (the overhead 
of parsing the data with an existing parser and then dealing with it 
with mmnormalize just isn't very high)
I'm somehow with you on this. I didn't like my messages being "parsed" 
by default RFCs (hence, splitting json)
I understand rsyslog was born to handle logs (and their lack of 
standards), but I wouldn't forget it's also message-shipper, hence it 
could make sense to convert pm to mm and to define the pipeline as I 
exposed before.



The devil is in the details, and when you are working to make things 
very fast, it gets messy.


I deliberatly simplified a bunch of things to give you the architecture.

You need to first think about what is it that you are trying to 
understand when looking at the code.
Although I have progressed a bit since the beginning, it stills causes 
me some headaches.


If you want to deal with a logsource that can't comply with the very 
simple syslog protocol, you need a pm module. This is mostly for 
things like Cisco adding a field to the messae, AIX adding "message 
forwarded from", etc (we actually could use one that deals with 
syslog-ng inserting the severity in the header)
again, IMHO, using pm and mm doesn't make much sense. They could be, 
both, sequential actions.


to do the imhiredis module, the biggest issue is someone with a really 
good understanding of how redis works and the over-the-wire protocol

Bob!
I have played a bit with Redis, and doesn't seem very complex (rsyslog 
neither...and look!!!)


Thanks a lot for your kind help
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?




The ugly way to do this would be a series of

if $programname = "group/appX" then set $.owner = 
"\"group\":\"group\","\"unit\":\"unit\",\"app\":\"appX\",";
do having multiple templates affect performance? (what I really noticed 
it's they affect loading time!)



a far more elegant way to do this would be to do a table lookup on the 
programname and have it return the string.
I have readed about how lookup tables can be used for geoIP. Could you 
provide a link to doc where there's an example?


you can also simplify the template a bit. Instead of crafting the json 
in the template, create a variable that has what you want in/under it 
and output that variable. but compared to collapsing all the templates 
together, that's a minor change :-)

one variable for each file and one template which use it, isnt it?

I question the value of having a separate sending queue for each app. 
I think it's better to send them in one combined firehose and split 
them on the receiving side. It makes it less disruptive when you find 
you want to change the groupings of things and all those queues on the 
sender can eat up a lot of ram.

Probably this is because i came from redis.
Talking about elastic, probably ingest node would be the best option, 
while having index name as metadata.


I'll have an eye on that too.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?



My recommended architecture [1] is to have a local relay picking up 
logs from each network/datacenter, add useful metadata (fromhost-ip, 
what environment this is, which relay processed it, timestamp of when 
the log was processed on the relay, etc) and then forward the message 
to a central log system in json format [2].

This is actually what we are doing



On the central system, I should then have relativly clean data to deal 
with. I receive it and parse the json out.


I then created a template.

t = "$timestamp $hostname $syslogtag $!msg"

note the msg is not $msg (which would be json), but rather $!msg, 
which is the field inside the json that contains the original message.

I was using property(name="msg" position.from="3") to skip first JSON "{"
Another _hidden gem_




set $.m = exec_template("t")

which creates a variable that contains a line like:

Nov 23 06:19:38 bifrost dhcpd: DHCPREQUEST for 10.2.0.122 from 
00:90:f5:d6:7f:2a via eth2


I didn't played with exec_template, cause I didn't know if that was a 
"tricky approach", recommended or anything else.
Usually whatever_exec is run as command, hence having a serious impact 
on performance.

I'll have a look.




prefix=%timestamp:date% %hostname:word% dhcp:
rule=dhcp,foo: DHCPREQUEST for %ip:ipv4" from %mac:word% via 
%interface:word%

what's the foo part?
As someone with regex-way-of-thinking, I would've expected
rule=%prefix% DHCPREQUEST for %ip:ipv4" from %mac:word% via 
%interface:word%

but it isn't.

This then parses eveything apart, and creates a variable event.tags = 
["dhcp",:"foo"]

I'm lost!


if $programname = "dhcp" then { call dhcprules; stop }

As with _exec, someone told me if are evil (wasn't nginx)

...and that gives you a rocket speed...
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?


Hi all

In order to improve the first draft of ERK project, I would like to get 
some feedback from you.


What features are you missing/you think it may be improved in rsyslog?

Please, try to be as more clear/self-explanatory/simple as you can for 
better understanding.


 * logstash memory footprint is quite high compared to rsyslog,
   although both "doing the same".
 * rsyslog configuration can't be reloaded live
 * dynamic variables (calculated on each message processing) aren't
   supported on templates
 * combine multiple variables into one to build a "date" field isn't
   possible

Regards

PS: those with deep knowledge, please, start thinking how you'll solve 
them...


El 23/11/16 a las 12:52, Bob Gregory escribió:

There've been a few discussions over the last few days that are all
pointing in the same direction:

* Is it better to use Rsyslog's omelasticsearch rather than pushing to
logstash?
* Should we have a minimal log shipper component as distinct from rsyslog's
processing capabilities?
* Ought we to have an imhiredis module?

Really what we're talking about is replacing Logstash (and the various
beats) with rsyslog. I'm perfectly happy with that, Logstash is a
resource-expensive and fickle beast that spoils my otherwise pristine log
pipeline, but I do think the community ought to think about whether this is
the direction they want to take.

For my part, I'm quite happy to help build an imhiredis (and imkafka?)
module but only if I can actually dogfood it, which means replacing
Logstash in our own environment.

For that, I'd like to see better support for GeoIP tagging, a Riemann
output plugin, some better guidance on "failed message queues", etc. etc.
etc.

Are we jointly interested in building the REK stack and, if so, can we
start to work out the feature set we're missing, and the documentation we'd
need for this to work? I'm a little concerned that if we tackle the usecase
piece-meal, we'll end up with lots of disjointed parts that don't really
solve the problem: logstash is not an adequate logstash.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project



Does this help? 
Although I had all these clear, reading it helped me to better 
understand. Thank you.


A few questions appear:

 * You talk about threads...may we considered a distributed application
   architecture/other process reading that queue? That could be done
   with a redis/kafka reliable delivery, ie: imhiredis/omhiredis +
   http://redis.io/commands/rpoplpush
 * Is the main queue reliable/disk assisted? Of course UDP messages can
   be lost if the application crash, but shouldn't with TCP
 * By default, messages are parsed to match RFCs and then sent to _mm_.
   is there any way to disable parsers? IMHO pipelines could be
   simplified if parsers/mm/actions...are considered equals:
# conditionals can be used here
pipeline {
# and here
input() //multiple inputs can be used, order doesn't matter
processor(type="...") //multiple processors are handled as
   a sequence (a parser, a mm...)
processor(type="...") //this processor is run after the
   previous
output() //multiple outputs can be used, order doesn't matter
}


Your explanation is so simple that I'm wondering why source seems so 
obscure to me...



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project


LOL

What about "The Fellowship of the log" :P
Already writing.

El 23/11/16 a las 17:43, Rainer Gerhards escribió:

If the start already gets blocked by a process to find a code name, I
think we can give up... How about Elasticsearch-rsyslog-Kibana?

Rainer

2016-11-23 17:36 GMT+01:00 mosto...@gmail.com <mosto...@gmail.com>:

https://en.wikipedia.org/wiki/Erk

...I know you can do it better... :P


El 23/11/16 a las 17:34, Rainer Gerhards escribió:


ERK

2016-11-23 17:34 GMT+01:00 mosto...@gmail.com <mosto...@gmail.com>:

Let me start writing, cause it may be helpful as documentation too!


Please do.


I still need a codename! Sorry for that...but I NEED IT!

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project


https://en.wikipedia.org/wiki/Erk

...I know you can do it better... :P


El 23/11/16 a las 17:34, Rainer Gerhards escribió:

ERK

2016-11-23 17:34 GMT+01:00 mosto...@gmail.com <mosto...@gmail.com>:

Let me start writing, cause it may be helpful as documentation too!


Please do.


I still need a codename! Sorry for that...but I NEED IT!

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project






Let me start writing, cause it may be helpful as documentation too!


Please do.


I still need a codename! Sorry for that...but I NEED IT!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project




That's a permission issue: We need to be much more restrictive
(security) with who has permissions to the code than to the doc. Thus
we have two repos. I'd prefer a single one, too, but that's not
possible.

Understood...does this happened in real life or just in paper? :P
I mean: if there are reviewers, I wouldn't care.

sorry, I don't understand what you mean


When you said "who has permissions to the code than to the doc", I guess you're 
talking about git push.
Aren't those reviewed by you? Even more, perhaps it could help/ease to have 
documentation up to date.


I guess that's not for general consumption. There is some learning
curve to LaTex ;-)

markdown is the winner (but still, i want to learn latex!)






I am not connected to them. Given the fact that the paid big $ for
logstash, I don't think they would be overly enthusiastic... But I may
be wrong ;-)

I can spam them to know what they think...it seems they try to fill the gap
with Beats, but maybe they didn't.

I have no issue if you try...

On my TODO list ;)
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project

Totally agreed with your comments. That's why I want a collaborative 
doc, where I can randomly write and you'll wisely answer.


Let me start writing, cause it may be helpful as documentation too!

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project




That's a permission issue: We need to be much more restrictive
(security) with who has permissions to the code than to the doc. Thus
we have two repos. I'd prefer a single one, too, but that's not
possible.

Understood...does this happened in real life or just in paper? :P
I mean: if there are reviewers, I wouldn't care.



Is gdocs really that visible? Does anyone agree on it? I even think
some corp folks cannot access it (at least I've seen that when working
with consulting customers). If we do that move, we need bold support
from the community. I personally am skeptic. Besides, I'd prefer LaTex
;-)


I don't know latex yet (but I want to start someday...what about NOW?), 
but google docs is easy to setup for a bunch of people, permissiosn can 
be easily managed, and it will allow a fast-editing doc, as 
brainstroming for the project.

Once is solid, we can switch to github

Another option: https://www.sharelatex.com/




I am not connected to them. Given the fact that the paid big $ for
logstash, I don't think they would be overly enthusiastic... But I may
be wrong ;-)
I can spam them to know what they think...it seems they try to fill the 
gap with Beats, but maybe they didn't.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project




I don't think a fork make sense, why keep two versions? As a branch in
rsyslog-repo, that makes a lot of sense, but I'd move in smaller
refactoring steps and merge each one as early as possible. A total
rewrite from scratch is out of question, except if somone has a couple
of month to years time.

Agree, but that scratch couldn't be far... :P


Let's treat this as experiment:https://github.com/rsyslog/rsyslog/wiki
I think this wiki should only host development-related content, not
user doc. User doc should go to rsyslog-doc.


IMHO a single repo could be enough, if we just have a doc 
directory...otherwise, user documentation and design docs wont be on the 
same place/can't easily reference code snippets.


*Again (based on my experience, it sucks): something more 
collaborative/multiple editing, like gdocs than github?*


Have you/anyone considered involving/getting feedback from elastic people?
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project




Open question to the community: does it really make sense to create a
new repo at this stage?
I'll try to start as clean as possible, but I agree I can start writing 
anywhere




Name + Scope
I mean this project will NOT host rsyslog code, so what will it host?

Shouldn't we benefit from rsyslog code and lessons?

I think it can be faced two ways:
 - a new/fresh code, without backward compatibility
 - a rsyslog-ng branch, with maybe heavy refactoring, but keeping 
backwards compat


Perhaps anyone think's another way? Comments are more than welcome.


gut feeling: wouldn't it better to start by just documenting the
project goals on rsyslog wiki and after that decide of how to move
forward. Just an idea... I don't think my vote counts more than anyone
elses...

Agree. Google-docs /something without git perhaps it's better.
Anyway, just give me a link where to start writing :)

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project

github wiki seems to be the correct place. could you create a _draft_ 
project repo for eRk ?


considering rsyslog is a Rocket-fast SYStem for LOG processing 
http://www.rsyslog.com


perhaps we could use a development codename until a popular/commercial 
name is found


what about: logwalk? (I really don't care, but I feel this ERK idea is 
getting traction...)



El 23/11/16 a las 16:20, Rainer Gerhards escribió:

If we enable the GitHub wiki, that would be a better place for such doc
imho. It is always easy to copy something over once it is written... ;-)

Rainer

Sent from phone, thus brief.

Am 23.11.2016 16:03 schrieb "David Lang" <da...@lang.hm>:


On Wed, 23 Nov 2016, mosto...@gmail.com wrote:

Could any of you, aware of rsyslog internals, provide a design

diagram/architecture view of how the components are working?

Perhaps you could use something like: https://docs.google.com/presen
tation/d/1UJSO0t2J6mrIyBNSkCUvrR5Q4RWbLcOof2WgB2_qA3g/edit#
slide=id.g13bdc00843_0_3

I think, apart from documentation, it could help a lot understanding how
to improve rsyslog.
In fact, I'm missing a few complete examples (from file to file using
queues, mmnorm, json...) to understand the workflow. Is there anyone
anywhere?


I may try to modify your documentation shortly, but I think it's actually
much simpler than you are thinking or Rainer's documents make it seem

you have N input modules, each operating independently (each is at least
one thread).

When an input module receives a message, it adds it to the main queue [1]

There is one (sometimes more) worker thread that grabs a message from the
queue [2] and then steps through the config file.

As the worker is going through the config file, it can do the following:

   set variables
   apply filters
   invoke message modification module code (mm*) which set/modify variables
   format the message per a template and call an action (action() which
invoke om* modules)
   deliver to a different queue (which has it's own workers)
   abandon processing of this message (stop)


In addition to the threads that process logs, there is an 'admin' thread
that polices everything else, does garbage collection, etc.


Everything else is encapsulated into modules.

There are several module types:

im* input modules which get logs from ??? (including impstats which
gathers data from rsyslog internals)

pm* parser modules which may be used by input modules to understand the
log format [3]

mm* message modification modules, these can modify the message itself or
create/modify variables. Almost all of them just create/modify variables.

sm* string modules which are C implementations of templates (speed
optimization)

om* output modules, which take messages (or a batch of messages) and
deliver them to something

Each module provides code to specify what config options it supports,
which action() parameters it supports/requires, and the code to perform
work when called on each message.


so your file->file processing would be imfile reading a file and adding
messages to the main queue, and a worker process that reads the main queue
and writes the messages to a file using omfile

Does this help?

David Lang


[1] slight simplification, it could add it to a different queue if so
configured. It may add multiple messages to the queue at one time for
efficency.

[2] again, a slight simplification, it can grab more messages, see batch
procesing. But each message is then processed individually.

[3] parser modules are supposed to look at the message buffer provided to
it by the input module and populate the standard properties. If they don't
understand the format, they are supposed to report failure and not change
anything. There are a couple that 'cheat' and modify the message buffer to
fix known malformed messages and then claim they fail to let the standard
parsers then work on the now well-formed message. the pmaix* and the first
pmcisco* modules did this.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a

Re: [rsyslog] Are we building an ERK stack?


Of course it will help.

Let me take the risk: is there a rsyslog wiki where I could start 
documenting what ERK should look like? :P


markdown is mandatory.


El 23/11/16 a las 15:57, Rainer Gerhards escribió:

Would the capability to add metadata in imfile input help? That would be
easy to add. If not, what would give you the Mets data?

Rainer

Sent from phone, thus brief.

Am 23.11.2016 15:51 schrieb "mosto...@gmail.com" <mosto...@gmail.com>:


there are probably ways to simplify the configs, 5K lines of configs seems

excessive :-) how much of this is rulebase config vs rsyslog config?


Each app generates app-access.log, app-tomcat.log, app-application.log
files. imfile allow me to add filename as metadata, but nothing more.
As each application belongs to a workgroup, part of an organizational unit
and is running on some (multiple) hosts, at the end I have approximately
this for each app:

template(name="json_appX" type="list") {
 property(name="hostname")
 constant(value=" ")
 property(name="syslogtag")
 constant(value=" {")
constant(value="\"group\":\"group\","\"unit\":\"unit\",\"app\":\"appX\",")
 constant(value="\",\"file\":\"")
 property(name="$!metadata!filename")
 constant(value="\",\"msg\":\"")
 property(name="msg" format="jsonr")
 constant(value="\"}")
}
ruleset(name="json_appX") {
 action(
 template="json_appX"
 type="omrelp"
 target="server"
 port="20514"
 action.resumeRetryCount="-1"
 action.reportSuspension="on"
 queue.maxdiskspace="5M"
 queue.type="LinkedList"
 queue.filename="appX.qi"
 queue.SaveOnShutdown="on"
 )
}
input(type="imfile" file="/logs/appX/access.log" tag="group/appX"
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
input(type="imfile" file="/logs/appX/tomcat.log" tag="group/appX"
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
input(type="imfile" file="/logs/appX/application.log" tag="group/appX"
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")


which becomes 5K lines of config file.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?



there are probably ways to simplify the configs, 5K lines of configs 
seems excessive :-) how much of this is rulebase config vs rsyslog config?


Each app generates app-access.log, app-tomcat.log, app-application.log 
files. imfile allow me to add filename as metadata, but nothing more.
As each application belongs to a workgroup, part of an organizational 
unit and is running on some (multiple) hosts, at the end I have 
approximately this for each app:


template(name="json_appX" type="list") {
property(name="hostname")
constant(value=" ")
property(name="syslogtag")
constant(value=" {")
constant(value="\"group\":\"group\","\"unit\":\"unit\",\"app\":\"appX\",")
constant(value="\",\"file\":\"")
property(name="$!metadata!filename")
constant(value="\",\"msg\":\"")
property(name="msg" format="jsonr")
constant(value="\"}")
}
ruleset(name="json_appX") {
action(
template="json_appX"
type="omrelp"
target="server"
port="20514"
action.resumeRetryCount="-1"
action.reportSuspension="on"
queue.maxdiskspace="5M"
queue.type="LinkedList"
queue.filename="appX.qi"
queue.SaveOnShutdown="on"
)
}
input(type="imfile" file="/logs/appX/access.log" tag="group/appX" 
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
input(type="imfile" file="/logs/appX/tomcat.log" tag="group/appX" 
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")
input(type="imfile" file="/logs/appX/application.log" tag="group/appX" 
addMetadata="on" ruleset="json_appX" PersistStateInterval="1")



which becomes 5K lines of config file.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?



http://www.slideshare.net/chenryn/elk-stack-at-weibocom

I NEED the english version :P
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?


+1


Our current scenario (dockerized!):

   imfile_forwarder-->imrelp-->rsyslog-->redis-->logstash(grok+geoip)-->elastic

We are using redis as memory buffer and to split into multiple 
channels/lists (using dynakey ATM). We see kafka on the horizon.


We are also using several logstash containers to balance load, prevent 
single point of failure, etc.


What we're thinking after past days messages:

   imfile_forwarder-->imrelp-->rsyslog-->elastic

Having multiple rsyslog instances with simpler configs (instead of 5k 
lines with thousand of rulesets, templates and so), being able to geoip, 
reliable queues...


I wont dare to say it's time to review/refactor rsyslog, but 
maybe...https://www.youtube.com/watch?v=0O5h4enjrHw



El 23/11/16 a las 12:52, Bob Gregory escribió:

There've been a few discussions over the last few days that are all
pointing in the same direction:

* Is it better to use Rsyslog's omelasticsearch rather than pushing to
logstash?
* Should we have a minimal log shipper component as distinct from rsyslog's
processing capabilities?
* Ought we to have an imhiredis module?

Really what we're talking about is replacing Logstash (and the various
beats) with rsyslog. I'm perfectly happy with that, Logstash is a
resource-expensive and fickle beast that spoils my otherwise pristine log
pipeline, but I do think the community ought to think about whether this is
the direction they want to take.

For my part, I'm quite happy to help build an imhiredis (and imkafka?)
module but only if I can actually dogfood it, which means replacing
Logstash in our own environment.

For that, I'd like to see better support for GeoIP tagging, a Riemann
output plugin, some better guidance on "failed message queues", etc. etc.
etc.

Are we jointly interested in building the REK stack and, if so, can we
start to work out the feature set we're missing, and the documentation we'd
need for this to work? I'm a little concerned that if we tackle the usecase
piece-meal, we'll end up with lots of disjointed parts that don't really
solve the problem: logstash is not an adequate logstash.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project



./configure --help shows these options. I'm wondering if having a 
minimal build that created a different binary, used a different config 
and stripped everything down would be useful


Didn't notice I could use --disable-whatever on those features having 
[default=yes]. Anyway, imjournal is disabled by default.


Is there any option to disable "hardcoded parsers"? those which 
translate my message into bundled fields like priority, host, tag...

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] mmnormalize with mutiple input: conditionals?


Hi


Considering our imrelp module receives a lot of messages from different 
applications/hosts (with different TAGs), what would be the best way to 
split/forward each message to proper ruleset/parser?


Should I use /_if/_?

Does 
http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html 
have something to "process only specific messages" ?


Quoting https://github.com/rsyslog/rsyslog/issues/625#issuecomment-262286487

> If you can combine the programname with the message and include the 
programname in your rules (prefix works wonders here), liblognorm is 
extremely efficient in only using the relevant rules.


I understand it's just having TAG as part of rule, isnt it?


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] A couple of questions regarding rsyslog debug.


Hi


In the past I have to build a few applications with C/C++ and I found 
#ifdef directives everywhere to be ugly and confusing. For example, 
using ifdef DEBUG to enable/disable debug messages.


That's why I opt to use functions which behavior was defined depending 
on compile headers/source files. For example, define log_debug(...) as 
NOOP or sleep() depending on platform.


Have you considered doing something like this?


Also, some time ago I started playing with "component" debugging, which 
enabled debug only for specific components. For example, setting 
RSYSLOG_DEBUG_MODULES_C will enable debug just for modules_c file, but 
it will be very verbosed


Have you considered doing something like this?


Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project

Could any of you, aware of rsyslog internals, provide a design 
diagram/architecture view of how the components are working?


Perhaps you could use something like: 
https://docs.google.com/presentation/d/1UJSO0t2J6mrIyBNSkCUvrR5Q4RWbLcOof2WgB2_qA3g/edit#slide=id.g13bdc00843_0_3


I think, apart from documentation, it could help a lot understanding how 
to improve rsyslog.
In fact, I'm missing a few complete examples (from file to file using 
queues, mmnorm, json...) to understand the workflow. Is there anyone 
anywhere?


Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Would imhiredis make sense?



Logstash needs something like redis because it can't do any queueing 
itself. Rsyslog is built around queues, and has the ability to 
create multiple queues and piplines internally, you don't need to 
run multiple instances.

I want multiples instances in order to:

* Being able to process pipelines on different containers/hosts


much less needed on rsyslog due to the higher effiency. I've had 
rsyslog handling over a hundred thousand logs/sec on a single host.


This is our current scenario (each element deployed within a docker 
container):


   logs-->RELP-->rsyslog-->redis-->logstash_app_1/N...


This allow us to have multiple simpler configurations for logstash, 
splitting traffic between multiple workers/containers on different 
hosts, high availability, load balancing...





* Isolate pipelines to prevent problems on one affecting others


rulesets with queues on each ruleset solvs this for you.
One segfault while processing one ruleset/action (actually, it happened 
a lot with 8.22) crash the whole process.




All processing from that point on will take place in different 
threads working on different queues for each category.
Will I be able to "reload" rsyslog configuration to add/delete new 
rulesets/pipelines?


you can stop/start rsyslog, but there is not a way to change the 
config on the fly.

:(

However, if you really want to go this way, one thing you can do is to 
make use of the multicast mac feature in ethernet to distribute the 
same logs to multiple systems/containers and have each container throw 
away all logs except what it's configured to handle.


This lets you add/remove log processing at any time and even have 
multiple systems processing the same logs in different ways


https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david 


Network traffic x2
Actually, we are using a similar environment for other things, but I 
don't think that's the way to go.


KISS, start simple and only add complexity when you find it's actually 
needed. Have plans for how to scale out when you hit limits, but you 
usually find that you hit limits far later than expected. Yes, you may 
have to eventually do the same work, but by having a solid system now 
with less work, you can spend the time saved now to improve other things.
KISS is great, but we are looking to build a dynamic pipeline, and we 
found rsyslog is close to be the proper tool, with a couple of changes!



Somehow related with Rainer's new file reader proposal, I think a 
rsyslog code review/refactor will help with this.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Would imhiredis make sense?

2016-11-22 Thread mosto...@gmail.com



What sort of log volume are you talking about here? (logs/sec type of 
thing)

From 0 to thousand-thousands/sec

Logstash needs something like redis because it can't do any queueing 
itself. Rsyslog is built around queues, and has the ability to create 
multiple queues and piplines internally, you don't need to run 
multiple instances.

I want multiples instances in order to:

 * Being able to process pipelines on different containers/hosts
 * Isolate pipelines to prevent problems on one affecting others
 * (others)


What you would do is create a ruleset for each application (pipeline) 
and give that ruleset it's own queue.
I know it can be done, but not what I'm looking for. Moreover, I would 
love to be a "dynamic" configuration


As new logs arrive, you then sort them by application, and for each 
application (or application category), you call the appropriate ruleset.
And, if there are a lot of evt/sec, you may have a bottleneck. I'll 
probably have a rsyslog cluster based on docker swarm mode


All processing from that point on will take place in different threads 
working on different queues for each category.
Will I be able to "reload" rsyslog configuration to add/delete new 
rulesets/pipelines?



Give it a try, I'll bet that you find the result much simpler and faster.

I expecting your reply ;)

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-22 Thread mosto...@gmail.com


Quite interested on this thread


http://blog.gerhards.net/2016/11/would-creating-simple-linux-log-file.html 

IMHO: the "issue" we're having with rsyslog is understanding the code 
rather than performance. Why you'll invest time doing a "faster reader" 
if you already have a "fast-enough processor"? Has anyone complained 
about speed? What would be the benefit from such development? Will it 
increase reading speed by 2x? 4x? 10x?


I suggest you taking this approach:

 * Read file line by line (consider also it handles multiline)
 * Measure speed (lines/sec, MB/sec, chars/sec...
 * Read the same file with imfile with similar reading configuration
 * Measure speed (lines/sec, MB/sec, chars/sec...
 * Compare both times and think what you can do to make it faster :P


I have not found a good tool yet (I've written or seen written a 
couple over the years)
What are the issues with imfile? I don't know when rsyslog development 
started, but perhaps time for refactoring has come.


I suspect that a stripped down compile of rsyslog (no input modules 
other than imfile, especially no imjournal, etc) would end up being 
competitive to just about any special-purpose program.
I don't know if I understood properly: is there such a thing as 
--disable-imjournal? will this make rsyslog faster?
Is there any place where I can see "everything than can be disabled in 
order to make rsyslog even faster"?



IMHO, The biggest problem with using rsyslog to do this is the same 
problem we have with using rsyslog to create /dev/log in containers, 
the fact that the config is fixed at startup time.

That's somehow why I was asking about imhiredis.
I don't know yet how rsyslog works internally, but I'll love being able 
to reload/add/remove configuration/pipelines during execution.


Having a command socket that rsyslog listened to that would let you 
add/remove inputs (files or unix sockets), but didn't allow you to 
change anything else in the config would let you easily tell rsyslog 
to start watching a new container or file as needed, and then stop 
watching so that it doesn't prevent the container or directory from 
going away when the app/container is removed.
It's all this because rsyslogs polls for new files/dir changes or 
something like that?


TBH, I'll love having more expertise to better contribute.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Would imhiredis make sense?

2016-11-22 Thread mosto...@gmail.com


Hi


We've been playing with logstash, rsyslog and redis for a while in order 
to *index into elasticsearch a bunch of application logs*. Briefly: 
app1-file1.log, app1-file2.log...appN-fileX.log -> pipeline -> 
elasticsearch.


So far, we are using *redis queues and _each application_ processing was 
made by one logstash instance* (docker container). Of course, this works 
with 5-10 applications, but it doesn't when you plan to deploy 100 apps 
cause each logstash instance requires ~512MB of RAM.


We've been thinking about rsyslog since the beginning, because it takes 
fewer RAM, but just noticed it doesn't have a *redis input module (aka: 
imhiredis)*


We still plan to have independent instances (one rsyslog for each 
application), but we're wondering if you'll consider it makes sense to 
implement this module.


Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] lowercasing structured-data SD names


I'm confused now.

Did you make everything case-insensitive a long time ago, and then made 
it case-sensitive again (in order to solve "same name, different casing" 
json fields, among others)?


It's supposed to be case-sensitive now, but fields created by rsyslog 
are created lowercase?


Regards

El 16/11/16 a las 16:05, Rainer Gerhards escribió:

2016-11-16 15:38 GMT+01:00 mosto...@gmail.com <mosto...@gmail.com>:

A quick look up returned:
https://github.com/rsyslog/rsyslog/search?utf8=%E2%9C%93=tolower

This is relevant:

https://github.com/rsyslog/rsyslog/issues/481

Rainer


El 16/11/16 a las 11:48, mosto...@gmail.com escribió:


Rainer, what do you think?

A "removed-lowercase" branch could work for this, while helping us to
solve #1116



El 16/11/16 a las 11:11, David Lang escribió:

On Wed, 16 Nov 2016, mosto...@gmail.com wrote:


According to https://tools.ietf.org/html/rfc5424#section-6.3.2 and
https://tools.ietf.org/html/rfc5424#section-6.3.3 both are *case-sensitive*.

As stated in
http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmpstrucdata.html
: "Note that the case of the RFC5424 parameter names has been converted to
lower case."
*Why it's being lowercased? *


There was a time when rsyslog didn't allow you to access upper case items
(and per a discussion we are currently having, that may not have been fully
fixed yet)


I can confirm that having a look at:
https://github.com/rsyslog/rsyslog/blob/master/plugins/mmpstrucdata/mmpstrucdata.c#L219
Perhaps this is somehow related to
https://github.com/rsyslog/rsyslog/issues/1116
May I open an issue?


you can always open an issue :-)

you can also always open a PR

now, they may get closed with an explination instead of worked on, but
you can always open them :-P

That said, It does sound like in this case we should remove the
lowercasing from the mm module, but we need to double-check that we allow
access to the upper-case variables now.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] lowercasing structured-data SD names

A quick look up returned: 
https://github.com/rsyslog/rsyslog/search?utf8=%E2%9C%93=tolower



El 16/11/16 a las 11:48, mosto...@gmail.com escribió:

Rainer, what do you think?

A "removed-lowercase" branch could work for this, while helping us to 
solve #1116




El 16/11/16 a las 11:11, David Lang escribió:

On Wed, 16 Nov 2016, mosto...@gmail.com wrote:

According to https://tools.ietf.org/html/rfc5424#section-6.3.2 and 
https://tools.ietf.org/html/rfc5424#section-6.3.3 both are 
*case-sensitive*.


As stated in 
http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmpstrucdata.html 
: "Note that the case of the RFC5424 parameter names has been 
converted to lower case."

*Why it's being lowercased? *


There was a time when rsyslog didn't allow you to access upper case 
items (and per a discussion we are currently having, that may not 
have been fully fixed yet)


I can confirm that having a look at: 
https://github.com/rsyslog/rsyslog/blob/master/plugins/mmpstrucdata/mmpstrucdata.c#L219 

Perhaps this is somehow related to 
https://github.com/rsyslog/rsyslog/issues/1116

May I open an issue?


you can always open an issue :-)

you can also always open a PR

now, they may get closed with an explination instead of worked on, 
but you can always open them :-P


That said, It does sound like in this case we should remove the 
lowercasing from the mm module, but we need to double-check that we 
allow access to the upper-case variables now.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT 
POST if you DON'T LIKE THAT.




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] lowercasing structured-data SD names


Rainer, what do you think?

A "removed-lowercase" branch could work for this, while helping us to 
solve #1116




El 16/11/16 a las 11:11, David Lang escribió:

On Wed, 16 Nov 2016, mosto...@gmail.com wrote:

According to https://tools.ietf.org/html/rfc5424#section-6.3.2 and 
https://tools.ietf.org/html/rfc5424#section-6.3.3 both are 
*case-sensitive*.


As stated in 
http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmpstrucdata.html 
: "Note that the case of the RFC5424 parameter names has been 
converted to lower case."

*Why it's being lowercased? *


There was a time when rsyslog didn't allow you to access upper case 
items (and per a discussion we are currently having, that may not have 
been fully fixed yet)


I can confirm that having a look at: 
https://github.com/rsyslog/rsyslog/blob/master/plugins/mmpstrucdata/mmpstrucdata.c#L219 

Perhaps this is somehow related to 
https://github.com/rsyslog/rsyslog/issues/1116

May I open an issue?


you can always open an issue :-)

you can also always open a PR

now, they may get closed with an explination instead of worked on, but 
you can always open them :-P


That said, It does sound like in this case we should remove the 
lowercasing from the mm module, but we need to double-check that we 
allow access to the upper-case variables now.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] structured data parsing


We aren't, but discussion is always welcome

Instead of:

- Setting rfc5424-sd as default
- If jsonRoot is '', fields become top-level objects (or whatever 
specified otherwise)


I suggest:

- Setting top-level fields as default
- If jsonRoot is set, fields will be nested under that element.

I think this approach is simpler. Created issue (without solution 
approach) at https://github.com/rsyslog/rsyslog/issues/1262



El 16/11/16 a las 11:16, David Lang escribió:
I think you are saying the right thing, I'll restate things to try and 
make sure we are on the same page.


currently jsonroot is defaulted to null (but configurable), and the 
mmdata is always set to go under rfc5424-sd under jsonroot.


What should happen is that jsonroot should be set to rfc5424-sd as 
it's default (able to be overridden by the config) and then the mmdata 
is set under this directly


so that if jsonroot is set to '', the fields become top-level objects.

David Lang

On Wed, 16 Nov 2016, mosto...@gmail.com wrote:


Date: Wed, 16 Nov 2016 09:45:08 +0100
From: "mosto...@gmail.com" <mosto...@gmail.com>
Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
To: rsyslog@lists.adiscon.com
Subject: Re: [rsyslog] structured data parsing

To be more clear:

seems by default jsonroot = NULL, which IMHO should use "root as 
parent", instead of "rfc5424-sd".


Having "rfc5424-sd" as default it's mandatory for backwards 
compatibility?



El 16/11/16 a las 09:31, mosto...@gmail.com escribió:


Seems 
https://github.com/rsyslog/rsyslog/blob/master/plugins/mmpstrucdata/mmpstrucdata.c#L327 
it's the proper place.


Would changing this to:

/json_object_object_add//(jroot, //"msg"//, json);/

do the trick?

Any disagreement to PR?




El 16/11/16 a las 02:00, David Lang escribió:

On Tue, 15 Nov 2016, mosto...@gmail.com wrote:

According to 
http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmpstrucdata.html, 
structure data ID and parameters are stored under *rfc5424-sd* 
element in JSON.


It's there any way to store them as /firstchild/ elements (not 
using rfc5424-sd as parent, but !) ?


Otherwise, what's the best way to move those parameters /one level 
up/?


I don't think there is currently a good way to do this.

I think the right answer is to allow for the parent to be specified 
in the call to mmpstructdata similar to the way it can be set in 
mmnormalize.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT 
POST if you DON'T LIKE THAT.




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT 
POST if you DON'T LIKE THAT.





___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] structured data parsing


To be more clear:

seems by default jsonroot = NULL, which IMHO should use "root as 
parent", instead of "rfc5424-sd".


Having "rfc5424-sd" as default it's mandatory for backwards compatibility?


El 16/11/16 a las 09:31, mosto...@gmail.com escribió:


Seems 
https://github.com/rsyslog/rsyslog/blob/master/plugins/mmpstrucdata/mmpstrucdata.c#L327 
it's the proper place.


Would changing this to:

/json_object_object_add//(jroot, //"msg"//, json);/

do the trick?

Any disagreement to PR?




El 16/11/16 a las 02:00, David Lang escribió:

On Tue, 15 Nov 2016, mosto...@gmail.com wrote:

According to 
http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmpstrucdata.html, 
structure data ID and parameters are stored under *rfc5424-sd* 
element in JSON.


It's there any way to store them as /firstchild/ elements (not using 
rfc5424-sd as parent, but !) ?


Otherwise, what's the best way to move those parameters /one level up/?


I don't think there is currently a good way to do this.

I think the right answer is to allow for the parent to be specified 
in the call to mmpstructdata similar to the way it can be set in 
mmnormalize.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT 
POST if you DON'T LIKE THAT.




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] lowercasing structured-data SD names