Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

Then, implementing (although seems hard and you already said not 
intended) something like


   /if exists $!rulesetname then/

could improve if/else performance, cause using direct pointers.

Anyhow I'm moving to existent/working if/else solution for short-term


El 02/12/16 a las 11:48, Rainer Gerhards escribió:

2016-12-02 11:45 GMT+01:00 mosto...@gmail.com :

El 02/12/16 a las 10:50, David Lang escribió:

remember that values in logs may be under the control of an attacker (just
about every field can be hand-crafted by a person sending you a log
message). I know you are thinking of the simple case where you are only
reading files and processing them, but as soon as you support logs from any
other sources (including writing logs to /dev/log), things get much messier.

some way to test if a ruleset by that name exists would be needed, but
keep in mind that you may have other rulesets that exist besides the ones
you want to call.

Didn't think about that. For sure that's an exploit vector.

to some extend you could mitigate this by

call "prefix-" & $syslogtag

just FYI (if we permit an expression at all, we permit full expresibility).

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

2016-12-02 11:45 GMT+01:00 mosto...@gmail.com :
> El 02/12/16 a las 10:50, David Lang escribió:
>>
>> remember that values in logs may be under the control of an attacker (just
>> about every field can be hand-crafted by a person sending you a log
>> message). I know you are thinking of the simple case where you are only
>> reading files and processing them, but as soon as you support logs from any
>> other sources (including writing logs to /dev/log), things get much messier.
>>
>> some way to test if a ruleset by that name exists would be needed, but
>> keep in mind that you may have other rulesets that exist besides the ones
>> you want to call.
>
> Didn't think about that. For sure that's an exploit vector.

to some extend you could mitigate this by

call "prefix-" & $syslogtag

just FYI (if we permit an expression at all, we permit full expresibility).

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

El 02/12/16 a las 10:50, David Lang escribió:
remember that values in logs may be under the control of an attacker 
(just about every field can be hand-crafted by a person sending you a 
log message). I know you are thinking of the simple case where you are 
only reading files and processing them, but as soon as you support 
logs from any other sources (including writing logs to /dev/log), 
things get much messier.


some way to test if a ruleset by that name exists would be needed, but 
keep in mind that you may have other rulesets that exist besides the 
ones you want to call.

Didn't think about that. For sure that's an exploit vector.


I don't see a switch being noticably better.

Agree

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

important word missing:

switch is primarily a tool for compiler optimization IMHO and

**only**

makes
sense when scalar values are evaluated. This can then be replaced by
jump tables, reducing O(n) evaluation time to O(1). It doesn't gain
you anything if used on vector values (like strings).

Other than that, switch is just another way of wrinting "else if" chains.

Sorry for that,
Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

2016-12-02 10:44 GMT+01:00 mosto...@gmail.com :
>
>> (as an optimization, make the most common apps the earliest in the
>> directory)
>
> And that's where that /if exists call $!tag/ idea makes more sense, cause
> you don't need to compare, neither to arrange most used applications at top.

I, too, think this idea is useful. We just need to make sure that a
good error message is emitted when the ruleset cannot be found. It's
conceptually a function pointer in C.


The problem is that many people throw away rsyslog messages, it's even
default in most distros. That's also the reason why I change the error
message system so that at least on systemd systems the error messages
will go into the systemctl status display. I hope this helps resolving
all these unnecessary grief caused by ignored error messages.


Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Fri, 2 Dec 2016, mosto...@gmail.com wrote:

while I can see the use cases for "call $.var", what would you do if you 
call a ruleset that doesn't exist? you would first have to do 'if $.var == 
[array of legal values] then' to be safe.
Sure, something like if $!var exists then { call $!var } (one if statement vs 
many)


remember that values in logs may be under the control of an attacker (just about 
every field can be hand-crafted by a person sending you a log message). I know 
you are thinking of the simple case where you are only reading files and 
processing them, but as soon as you support logs from any other sources 
(including writing logs to /dev/log), things get much messier.


some way to test if a ruleset by that name exists would be needed, but keep in 
mind that you may have other rulesets that exist besides the ones you want to 
call.



1. just a bunch of if statements

  performance cost of doing a bunch of if tests
  easy to include additional tests from a directory of files

That's my best option so far.


2/3. if then else if...

This saves up to 199 if's. Would this be noticiable?


it depends on the volume of logs, but yes, it can be noticable, especially if 
you are in the situation of most people where a few log sources generate a large 
percentage of your logs.


note that you can use stop in a traditional if statement to avoid wasting time 
processing any other rules after that point



4. switch statement

This seems interesting...


but note it's limited to exact matches, there's a lot of times where 
'startswith' is really a good fit. In your case, the same app run by multiple 
teams, or multiple instances of the same app (and therefor the same log 
manipulation being needed) could be combined with the other options, but not 
with a switch.


In it's basic implementation, a switch is just a fancy way of typing if then 
else statements.


With what Rainer pointed out about

if then
else if
else if
else

I don't see a switch being noticably better.


5. variable call statements

  what to do if called ruleset doesn't exist?

I'm screwed


6. function lookup tables
This is the Nth time you mention those...start to think I *really* need to 
start messing with them.


they are an incredibly powerful tool to deal with distilling a lot of stuff down 
to a few results.


geoip lookups were one of the use cases I was thinking of when I created the 
spec for lookup tables, but another was categorization, either giving it a list 
of machine and what environment they are in (dev/QA/prod/DR/etc) so that alerts 
could be adjusted/redirected/silenced depending on what system generated the 
identical log message.


the problem of calling a non-existant ruleset. We could have it silently do 
nothing, but that gets really messy and I am already cringing at the 
troubleshooting exhanges we will have to help people figure out what 
is/isn't happeing.

Is there a *exists* statement? (eg: /if exists "object-name" then/)


not currently (there is an open issue for it)

and an exists statement would (at least initially) just be checking for a 
variable existing (as opposed to if $.var = '' that we have to do now)


checking if a ruleset exists is a very different thing,

function pointers are by far the most complicated, and since they include 
ruleset parsing after startup, they have the potential to be really ugly to 
implement. On the other hand, they are also by far the most powerful. If we 
could do things like limiting the functions so that they can't do any of 
the startup-type things[1] and only include statements that are normally 
executed for each log type, this would also give us a back-door way of 
providing the dynamic configuration that many people have been asking for.

"Limiting the functions" sounds too /patchy/ for me.


The problem is that there are a lot of things that only really make sense at 
startup. A lot of them have no way to undo them later.


loading modules is a good case in point. loading a module not only adds code, it 
adds syntax, there isn't any sane way to unload a module, so if a function 
loaded a module and it was redefined to not load the module, behavior would be 
different than if the module was never loaded in the first place.


another problem is that a lot of things get defined before the multiple threads 
are forked, and they then go on their separate ways. Trying to redefine these 
things across all threads in a consistant manner would be a nightmare.


and if you think about what you want to do differently for different logs, when 
would you need to change some global state? 99.9%+ of the time, you want to 
change variable assignements, do if statements, change where the output goes


adding all that complexity and risk to support the <0.1% use case would be 
enough to eliminate any possibility of it being implemented (even if Rainer 
thought it was a good idea, the work needed to implement it would delay it 
forever)


going for the limited 

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

(as an optimization, make the most common apps the earliest in the 
directory)
And that's where that /if exists call $!tag/ idea makes more sense, 
cause you don't need to compare, neither to arrange most used 
applications at top.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

Hi


Combined threads reply follow.


On Thu, 1 Dec 2016, David Lang wrote
sigh, this is getting a wee bit frustrating, you keep saying "it hurts 
when I do X", we say "that doesn't work well, do Y" and you come back 
a day or so later saying "but it really huts when I do X"... (it 
doesn't help when we ask you to provide information and you instead 
spend hours trying other things)

I understand you better than you think. I'm the pupil you never wanted.
Said so, in my defense, I would say I am testing the /Y/ things too.

now that I have expressed my frustration, you are finding bugs, 
helping to fix them, and raising some good questions along the way. 
Just understand why once in a while our answers seem a bit curt. 
Don't worry. I understand it but I'm learning a lot along the way!. You 
are having A LOT of patience and being really kind and very instructive.


while I can see the use cases for "call $.var", what would you do if 
you call a ruleset that doesn't exist? you would first have to do 'if 
$.var == [array of legal values] then' to be safe.
Sure, something like if $!var exists then { call $!var } (one if 
statement vs many)



1. just a bunch of if statements

  performance cost of doing a bunch of if tests
  easy to include additional tests from a directory of files

That's my best option so far.


2/3. if then else if...

This saves up to 199 if's. Would this be noticiable?


4. switch statement

This seems interesting...


5. variable call statements

  what to do if called ruleset doesn't exist?

I'm screwed


6. function lookup tables
This is the Nth time you mention those...start to think I *really* need 
to start messing with them.


I think that with elsif, the need for switch (#4) is low, and the 
restrictions of it only doing simple equivalence tests (no startswith, 
contains, etc) really limit it's use

Totally agree.

call var (#5) seems easy to implement, but I really don't like opening 
up the problem of calling a non-existant ruleset. We could have it 
silently do nothing, but that gets really messy and I am already 
cringing at the troubleshooting exhanges we will have to help people 
figure out what is/isn't happeing.

Is there a *exists* statement? (eg: /if exists "object-name" then/)

function pointers are by far the most complicated, and since they 
include ruleset parsing after startup, they have the potential to be 
really ugly to implement. On the other hand, they are also by far the 
most powerful. If we could do things like limiting the functions so 
that they can't do any of the startup-type things[1] and only include 
statements that are normally executed for each log type, this would 
also give us a back-door way of providing the dynamic configuration 
that many people have been asking for.

"Limiting the functions" sounds too /patchy/ for me.


On Thu, 1 Dec 2016, David Lang wrote

if  then {
set common things
}
$includeconfig /etc/rsyslog.d/apps.d/*.conf
else {
unknown app
}

in each of the apps.d/*.conf files do
else if  then {
stuff
}

This is what I had in mind.

(as an optimization, make the most common apps the earliest in the 
directory)

Loved alphabetical order, but now I understand why this is better.


On Fri, 2 Dec 2016, Rainer Gerhards wrote


What exactly do you do with the variables you set inside the if body?
Are they always the same? Where does the data originate from?

I try to understand the scenario better, because I vaguely think I may
be able to find a much simpler solution which would require possible
minimal code changes. But I can't express myself clearer at the
moment, it's more a gut feeling.

It would be good if you could post some *concrete* example of three or
four of the if blocks, that would definitely aid understanding.

As David said, I would love having separated config files for each app to:

1. normalize
2. define this application pipeline (after normalization, invoke geoip,
   add some fields to JSON...)
3. index or drop messages at app criterion.

Main config would be some common routines (like index)+combined app.conf 
files.



On Fri, 2 Dec 2016, David Lang wrote


if the liblognorm ruleset ammend=: line could assign variable 
contents, not just constant strings, it would address 90% of the issues.
It would let me add/remove/edit fields. Other things, like geoip 
tagging, should still be made manually (I guess)



On Fri, 2 Dec 2016, Rainer Gerhards wrote

Nothing I can do immediately, but can you pls open an issue tracker
with some explanation (copy?) on liblognorm. I have planned time
early next year to work on enhancements and this would fit in.

Music to my ears


Thank you both a lot for you replies.
Not only learning a lot, but feeling in debt with you, reason why I'm 
contributing as much as I can in the most ways I'm able.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

2016-12-02 9:11 GMT+01:00 David Lang :
> On Fri, 2 Dec 2016, Rainer Gerhards wrote:
>
>>> This could be because the source came in with names already (json, cef,
>>> name-value, etc), or because you need to take multiple fields in the log
>>> and
>>> combine them.
>>>
>>> if the liblognorm ruleset ammend=: line could assign variable contents,
>>> not
>>> just constant strings, it would address 90% of the issues.
>>
>>
>> can you give an example of what you think? I ask because liblognorm
>> does not know rsyslog variables (it cannot, as it is not a rsyslog
>> thingy).
>
>
> I'm just talking about referencing variables defined as a part of the rule
> that was just run.
>
> I've had cases where I parse two named items in a rule, but really will be
> wanting to treat them as a single item later, so being able to say a=$b+'
> '+$c would be very useful.
>
> or cases where the log arrives as json and I really want to combine fields
> in it, or rename them.
>
> I saw this a lot with windows logs, they like to have date and time as
> separate fields, or have names of fields that don't match up with the names
> used in other log sources, so a simple ammend=:a=$b would help a lot.
>
> There are cases where real logic/math is needed, and I think those are
> inappropriate for such things, but there's a lot that can be done with
> simple assignments that can contain variables.

Nothing I can do immediately, but can you pls open an issue tracker
with some explanation (copy?) on liblognorm. I have planned time
early next year to work on enhancements and this would fit in.

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Fri, 2 Dec 2016, Rainer Gerhards wrote:


This could be because the source came in with names already (json, cef,
name-value, etc), or because you need to take multiple fields in the log and
combine them.

if the liblognorm ruleset ammend=: line could assign variable contents, not
just constant strings, it would address 90% of the issues.


can you give an example of what you think? I ask because liblognorm
does not know rsyslog variables (it cannot, as it is not a rsyslog
thingy).


I'm just talking about referencing variables defined as a part of the rule that 
was just run.


I've had cases where I parse two named items in a rule, but really will be 
wanting to treat them as a single item later, so being able to say a=$b+' '+$c 
would be very useful.


or cases where the log arrives as json and I really want to combine fields in 
it, or rename them.


I saw this a lot with windows logs, they like to have date and time as separate 
fields, or have names of fields that don't match up with the names used in other 
log sources, so a simple ammend=:a=$b would help a lot.


There are cases where real logic/math is needed, and I think those are 
inappropriate for such things, but there's a lot that can be done with simple 
assignments that can contain variables.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

2016-12-02 8:44 GMT+01:00 David Lang :
> On Fri, 2 Dec 2016, Rainer Gerhards wrote:
>
>> 2016-12-02 8:27 GMT+01:00 David Lang :
>> These "possible sources" and "log type" and "multiple variables" are
>> exactly what I would like to clarify. Depending on the fine details,
>> there may be a simple solution, maybe not. At least something to think
>> about. I am thinking about someting along the lines of e.g. imfile
>> input with an extra metadata parameter, e.g.
>>
>> metadata=["!var=foo", "!bar=another text"]
>
>
> while that would be very useful, it's not this situation. In this case he is
> able to deal with this by setting tag to a structured value and then parse
> it later rather than setting multiple metadata values.
>
> In this case, we're working much later in the process, where it doesn't
> matter that the source was from an imfile. It's working with the data after
> it's been parsed by mmnormalize and then cleaning up the things where
> different logs still result in non-normalized data.

ok, this wasn't clear to me.

>
> This could be because the source came in with names already (json, cef,
> name-value, etc), or because you need to take multiple fields in the log and
> combine them.
>
> if the liblognorm ruleset ammend=: line could assign variable contents, not
> just constant strings, it would address 90% of the issues.

can you give an example of what you think? I ask because liblognorm
does not know rsyslog variables (it cannot, as it is not a rsyslog
thingy).

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Fri, 2 Dec 2016, Rainer Gerhards wrote:


2016-12-02 8:27 GMT+01:00 David Lang :

On Fri, 2 Dec 2016, Rainer Gerhards wrote:


What exactly do you do with the variables you set inside the if body?
Are they always the same? Where does the data originate from?

I try to understand the scenario better, because I vaguely think I may
be able to find a much simpler solution which would require possible
minimal code changes. But I can't express myself clearer at the
moment, it's more a gut feeling.

It would be good if you could post some *concrete* example of three or
four of the if blocks, that would definitely aid understanding.



he gave a couple of examples a while back.

An example would be populating one variable from several possible sources,
depending on the log type. In some cases it may require combining multiple
variables so it can't be solved in the parser


These "possible sources" and "log type" and "multiple variables" are
exactly what I would like to clarify. Depending on the fine details,
there may be a simple solution, maybe not. At least something to think
about. I am thinking about someting along the lines of e.g. imfile
input with an extra metadata parameter, e.g.

metadata=["!var=foo", "!bar=another text"]


while that would be very useful, it's not this situation. In this case he is 
able to deal with this by setting tag to a structured value and then parse it 
later rather than setting multiple metadata values.


In this case, we're working much later in the process, where it doesn't 
matter that the source was from an imfile. It's working with the data after 
it's been parsed by mmnormalize and then cleaning up the things where 
different logs still result in non-normalized data.


This could be because the source came in with names already (json, cef, 
name-value, etc), or because you need to take multiple fields in the log and 
combine them.


if the liblognorm ruleset ammend=: line could assign variable contents, not just 
constant strings, it would address 90% of the issues.


David Lang


But for me it is very hard to really think about it without a concrete
case. Looking at an abstract level, things seem easy, but the rest of
the conversation indicates to me that the abstraction may not
necessarily be correct, so I would like to see the dirty *real life,
real code" that shall be used.



along the lines of:

if type = 1 then
  set var = $foo
else if type = 2 then
  set var = $bar
else if type = 3 then
  set var = $baz + $bar

another thing that can be done in this sort of structure is to save
different logs in different places (type 1 & 2 go in dir A, type 3 goes in
dir B)

or writing the different logs using different templates.

If you look at the rules that I had, you see that I did a lot of that type
of thing. But I just did it as if statements


Again, it's all to abstract to keep me thinking further into that direction.

I want to avoid that I abstract something based on an already
abstracted idea which might not 100% match reality.

Hope this clarifies,
Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

2016-12-02 8:27 GMT+01:00 David Lang :
> On Fri, 2 Dec 2016, Rainer Gerhards wrote:
>
>> What exactly do you do with the variables you set inside the if body?
>> Are they always the same? Where does the data originate from?
>>
>> I try to understand the scenario better, because I vaguely think I may
>> be able to find a much simpler solution which would require possible
>> minimal code changes. But I can't express myself clearer at the
>> moment, it's more a gut feeling.
>>
>> It would be good if you could post some *concrete* example of three or
>> four of the if blocks, that would definitely aid understanding.
>
>
> he gave a couple of examples a while back.
>
> An example would be populating one variable from several possible sources,
> depending on the log type. In some cases it may require combining multiple
> variables so it can't be solved in the parser

These "possible sources" and "log type" and "multiple variables" are
exactly what I would like to clarify. Depending on the fine details,
there may be a simple solution, maybe not. At least something to think
about. I am thinking about someting along the lines of e.g. imfile
input with an extra metadata parameter, e.g.

metadata=["!var=foo", "!bar=another text"]

But for me it is very hard to really think about it without a concrete
case. Looking at an abstract level, things seem easy, but the rest of
the conversation indicates to me that the abstraction may not
necessarily be correct, so I would like to see the dirty *real life,
real code" that shall be used.

>
> along the lines of:
>
> if type = 1 then
>   set var = $foo
> else if type = 2 then
>   set var = $bar
> else if type = 3 then
>   set var = $baz + $bar
>
> another thing that can be done in this sort of structure is to save
> different logs in different places (type 1 & 2 go in dir A, type 3 goes in
> dir B)
>
> or writing the different logs using different templates.
>
> If you look at the rules that I had, you see that I did a lot of that type
> of thing. But I just did it as if statements

Again, it's all to abstract to keep me thinking further into that direction.

I want to avoid that I abstract something based on an already
abstracted idea which might not 100% match reality.

Hope this clarifies,
Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Fri, 2 Dec 2016, Rainer Gerhards wrote:


What exactly do you do with the variables you set inside the if body?
Are they always the same? Where does the data originate from?

I try to understand the scenario better, because I vaguely think I may
be able to find a much simpler solution which would require possible
minimal code changes. But I can't express myself clearer at the
moment, it's more a gut feeling.

It would be good if you could post some *concrete* example of three or
four of the if blocks, that would definitely aid understanding.


he gave a couple of examples a while back.

An example would be populating one variable from several possible sources, 
depending on the log type. In some cases it may require combining multiple 
variables so it can't be solved in the parser


along the lines of:

if type = 1 then
  set var = $foo
else if type = 2 then
  set var = $bar
else if type = 3 then
  set var = $baz + $bar

another thing that can be done in this sort of structure is to save different 
logs in different places (type 1 & 2 go in dir A, type 3 goes in dir B)


or writing the different logs using different templates.

If you look at the rules that I had, you see that I did a lot of that type of 
thing. But I just did it as if statements


David Lang

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

2016-12-01 23:30 GMT+01:00 David Lang :
> On Thu, 1 Dec 2016, David Lang wrote:
>
>>> Is there any way to dynamically invoke a ruleset? eg: call $var
>>> (I'm trying to avoid having +200 if statements...
>
>
> so to summarize, what I recommend that you do for for your use case is:
>
> 1. a single combined mmnormalize ruleset
>
> if  then {
> set common things
> }
> $includeconfig /etc/rsyslog.d/apps.d/*.conf
> else { unknown app }
>
>
> in each of the apps.d/*.conf files do
>
> else if  then {
>   stuff
> }
>
> (as an optimization, make the most common apps the earliest in the
> directory)
>
> This gets you your custom assignments per app, and the efficient parsing of
> the logs.

full ack, but a question to @mostolog

What exactly do you do with the variables you set inside the if body?
Are they always the same? Where does the data originate from?

I try to understand the scenario better, because I vaguely think I may
be able to find a much simpler solution which would require possible
minimal code changes. But I can't express myself clearer at the
moment, it's more a gut feeling.

It would be good if you could post some *concrete* example of three or
four of the if blocks, that would definitely aid understanding.

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Thu, 1 Dec 2016, David Lang wrote:


Is there any way to dynamically invoke a ruleset? eg: call $var
(I'm trying to avoid having +200 if statements...


so to summarize, what I recommend that you do for for your use case is:

1. a single combined mmnormalize ruleset

if  then {
set common things
}
$includeconfig /etc/rsyslog.d/apps.d/*.conf
else { unknown app }


in each of the apps.d/*.conf files do

else if  then {
  stuff
}

(as an optimization, make the most common apps the earliest in the directory)

This gets you your custom assignments per app, and the efficient parsing of the 
logs.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

No, braces (blocks) are just to form a single statement out of multiple. If
you add a single one (if), you do not need them.

Grammar: if stmt else stmt

Rainer

Sent from phone, thus brief.

Am 01.12.2016 23:22 schrieb "David Lang" :

> Ok, my mistake was thinking each else needed it's own {}, which results in
> a lot of closing } at the end of the sequence.
>
> David Lang
>
> On Thu, 1 Dec 2016, Rainer Gerhards wrote:
>
> Just on elseif... We have it, it's just a question of writing style. Insert
>> a space and you get:
>>
>> If expr
>> Else if expr
>> Else if expr
>> Else
>>
>> So there is no need for a special statement. Note that for the very same
>> reason, elseif does not exist in many programming languages. C, for
>> example, does not have it.
>>
>> Rainer
>>
>> Sent from phone, thus brief.
>>
>> Am 01.12.2016 23:05 schrieb "David Lang" :
>>
>> On Thu, 1 Dec 2016, mosto...@gmail.com wrote:
>>>
>>> Hi
>>>

 Is there any way to dynamically invoke a ruleset? eg: call $var
 (I'm trying to avoid having +200 if statements...


>>> sigh, this is getting a wee bit frustrating, you keep saying "it hurts
>>> when I do X", we say "that doesn't work well, do Y" and you come back a
>>> day
>>> or so later saying "but it really huts when I do X"... (it doesn't help
>>> when we ask you to provide information and you instead spend hours trying
>>> other things)
>>>
>>> some tools require you to split the logs up early and have completely
>>> seprate processing of each log type in order to scale (mostly because
>>> their
>>> speed in any one thread is so horrible), rsyslog is architected for very
>>> high performance when you keep everything together and only split it up
>>> in
>>> limited cases.
>>>
>>> now that I have expressed my frustration, you are finding bugs, helping
>>> to
>>> fix them, and raising some good questions along the way. Just understand
>>> why once in a while our answers seem a bit curt.
>>>
>>>
>>>
>>> while I can see the use cases for "call $.var", what would you do if you
>>> call a ruleset that doesn't exist? you would first have to do 'if $.var
>>> ==
>>> [array of legal values] then' to be safe.
>>>
>>> You don't like having lots of
>>>
>>> if  then {
>>>   stuff
>>>   stop
>>> }
>>>
>>> there are several approaches to doing this
>>>
>>> 1. just a bunch of if statements
>>>
>>>   performance cost of doing a bunch of if tests
>>>   easy to include additional tests from a directory of files
>>>
>>> 2. if then else
>>>
>>>   same number of tests
>>>   no need for stop
>>>   lots of trailing brackets
>>>   not include friendly
>>>
>>> if  then {
>>> } else {
>>> if  then {
>>> }
>>> }
>>>
>>> 3. add elsif
>>>
>>>   same number of tests
>>>   no need for stop
>>>   no odd trailing brackets
>>>   more include friendly thatn #2, but not as much as #1
>>>
>>> if  then {
>>> }
>>> elseif  then {
>>> }
>>>
>>> 4. switch statement
>>>
>>>   no faster than the above, but with potential for config optimization
>>>   cannot do more complex conditions
>>>   similar include impact as #3
>>>
>>> switch $.var {
>>>   case "value" {}
>>>   case "value" {}
>>> }
>>>
>>> 5. variable call statements
>>>
>>>   what to do if called ruleset doesn't exist?
>>>   include friendly if you don't have to have a guard test first
>>>
>>> call $.var
>>>
>>>
>>> 6. function lookup tables
>>>
>>>   solves the 'unknown thing to call' proclem
>>>   requires compiling config snippets at table load time
>>>   cannot do complex tests
>>>   table lookup could be extended to expand the sparse-array concept to
>>> string (solving the common 'startswith' type of test)
>>>
>>> exec table_lookup(functions,"$.var")
>>>
>>>
>>>
>>> now, looking at them in terms of complexity to implement
>>>
>>> #1 and #2 exist today
>>>
>>> #3 (elsif) seems like a fairly simple change to support
>>>
>>> #4 (switch) is a bit more complex
>>>
>>> #5 (call var) sounds like it's easy to implement
>>>
>>> #6 (function pointers) is significantly more work
>>>
>>>
>>> Personal opinion
>>>
>>> I'd like to see us add elsif (#3), it significantly cleans up long
>>> if-then-else cascades.
>>>
>>> I think that with elsif, the need for switch (#4) is low, and the
>>> restrictions of it only doing simple equivalence tests (no startswith,
>>> contains, etc) really limit it's use
>>>
>>> call var (#5) seems easy to implement, but I really don't like opening up
>>> the problem of calling a non-existant ruleset. We could have it silently
>>> do
>>> nothing, but that gets really messy and I am already cringing at the
>>> troubleshooting exhanges we will have to help people figure out what
>>> is/isn't happeing.
>>>
>>>
>>> function pointers are by far the most complicated, and since they include
>>> ruleset parsing after startup, they have the potential to be really ugly
>>> to
>>> implement. On the other hand, they are also by far the most powerful. If
>>> we
>>> could do things like limiting the functions so that they 

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

Ok, my mistake was thinking each else needed it's own {}, which results in a lot 
of closing } at the end of the sequence.


David Lang

On Thu, 1 Dec 2016, Rainer Gerhards wrote:


Just on elseif... We have it, it's just a question of writing style. Insert
a space and you get:

If expr
Else if expr
Else if expr
Else

So there is no need for a special statement. Note that for the very same
reason, elseif does not exist in many programming languages. C, for
example, does not have it.

Rainer

Sent from phone, thus brief.

Am 01.12.2016 23:05 schrieb "David Lang" :


On Thu, 1 Dec 2016, mosto...@gmail.com wrote:

Hi


Is there any way to dynamically invoke a ruleset? eg: call $var
(I'm trying to avoid having +200 if statements...



sigh, this is getting a wee bit frustrating, you keep saying "it hurts
when I do X", we say "that doesn't work well, do Y" and you come back a day
or so later saying "but it really huts when I do X"... (it doesn't help
when we ask you to provide information and you instead spend hours trying
other things)

some tools require you to split the logs up early and have completely
seprate processing of each log type in order to scale (mostly because their
speed in any one thread is so horrible), rsyslog is architected for very
high performance when you keep everything together and only split it up in
limited cases.

now that I have expressed my frustration, you are finding bugs, helping to
fix them, and raising some good questions along the way. Just understand
why once in a while our answers seem a bit curt.



while I can see the use cases for "call $.var", what would you do if you
call a ruleset that doesn't exist? you would first have to do 'if $.var ==
[array of legal values] then' to be safe.

You don't like having lots of

if  then {
  stuff
  stop
}

there are several approaches to doing this

1. just a bunch of if statements

  performance cost of doing a bunch of if tests
  easy to include additional tests from a directory of files

2. if then else

  same number of tests
  no need for stop
  lots of trailing brackets
  not include friendly

if  then {
} else {
if  then {
}
}

3. add elsif

  same number of tests
  no need for stop
  no odd trailing brackets
  more include friendly thatn #2, but not as much as #1

if  then {
}
elseif  then {
}

4. switch statement

  no faster than the above, but with potential for config optimization
  cannot do more complex conditions
  similar include impact as #3

switch $.var {
  case "value" {}
  case "value" {}
}

5. variable call statements

  what to do if called ruleset doesn't exist?
  include friendly if you don't have to have a guard test first

call $.var


6. function lookup tables

  solves the 'unknown thing to call' proclem
  requires compiling config snippets at table load time
  cannot do complex tests
  table lookup could be extended to expand the sparse-array concept to
string (solving the common 'startswith' type of test)

exec table_lookup(functions,"$.var")



now, looking at them in terms of complexity to implement

#1 and #2 exist today

#3 (elsif) seems like a fairly simple change to support

#4 (switch) is a bit more complex

#5 (call var) sounds like it's easy to implement

#6 (function pointers) is significantly more work


Personal opinion

I'd like to see us add elsif (#3), it significantly cleans up long
if-then-else cascades.

I think that with elsif, the need for switch (#4) is low, and the
restrictions of it only doing simple equivalence tests (no startswith,
contains, etc) really limit it's use

call var (#5) seems easy to implement, but I really don't like opening up
the problem of calling a non-existant ruleset. We could have it silently do
nothing, but that gets really messy and I am already cringing at the
troubleshooting exhanges we will have to help people figure out what
is/isn't happeing.


function pointers are by far the most complicated, and since they include
ruleset parsing after startup, they have the potential to be really ugly to
implement. On the other hand, they are also by far the most powerful. If we
could do things like limiting the functions so that they can't do any of
the startup-type things[1] and only include statements that are normally
executed for each log type, this would also give us a back-door way of
providing the dynamic configuration that many people have been asking for.

Internally, this could be implemented by creating rulesets and calling
them based on the result of the lookup.

[1] no changes to global() or main()
no loading modules.
probably no creating inputs
possibly allow template creation
just set and action() calls

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites 

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

Just on elseif... We have it, it's just a question of writing style. Insert
a space and you get:

If expr
Else if expr
Else if expr
Else

So there is no need for a special statement. Note that for the very same
reason, elseif does not exist in many programming languages. C, for
example, does not have it.

Rainer

Sent from phone, thus brief.

Am 01.12.2016 23:05 schrieb "David Lang" :

> On Thu, 1 Dec 2016, mosto...@gmail.com wrote:
>
> Hi
>>
>> Is there any way to dynamically invoke a ruleset? eg: call $var
>> (I'm trying to avoid having +200 if statements...
>>
>
> sigh, this is getting a wee bit frustrating, you keep saying "it hurts
> when I do X", we say "that doesn't work well, do Y" and you come back a day
> or so later saying "but it really huts when I do X"... (it doesn't help
> when we ask you to provide information and you instead spend hours trying
> other things)
>
> some tools require you to split the logs up early and have completely
> seprate processing of each log type in order to scale (mostly because their
> speed in any one thread is so horrible), rsyslog is architected for very
> high performance when you keep everything together and only split it up in
> limited cases.
>
> now that I have expressed my frustration, you are finding bugs, helping to
> fix them, and raising some good questions along the way. Just understand
> why once in a while our answers seem a bit curt.
>
>
>
> while I can see the use cases for "call $.var", what would you do if you
> call a ruleset that doesn't exist? you would first have to do 'if $.var ==
> [array of legal values] then' to be safe.
>
> You don't like having lots of
>
> if  then {
>   stuff
>   stop
> }
>
> there are several approaches to doing this
>
> 1. just a bunch of if statements
>
>   performance cost of doing a bunch of if tests
>   easy to include additional tests from a directory of files
>
> 2. if then else
>
>   same number of tests
>   no need for stop
>   lots of trailing brackets
>   not include friendly
>
> if  then {
> } else {
> if  then {
> }
> }
>
> 3. add elsif
>
>   same number of tests
>   no need for stop
>   no odd trailing brackets
>   more include friendly thatn #2, but not as much as #1
>
> if  then {
> }
> elseif  then {
> }
>
> 4. switch statement
>
>   no faster than the above, but with potential for config optimization
>   cannot do more complex conditions
>   similar include impact as #3
>
> switch $.var {
>   case "value" {}
>   case "value" {}
> }
>
> 5. variable call statements
>
>   what to do if called ruleset doesn't exist?
>   include friendly if you don't have to have a guard test first
>
> call $.var
>
>
> 6. function lookup tables
>
>   solves the 'unknown thing to call' proclem
>   requires compiling config snippets at table load time
>   cannot do complex tests
>   table lookup could be extended to expand the sparse-array concept to
> string (solving the common 'startswith' type of test)
>
> exec table_lookup(functions,"$.var")
>
>
>
> now, looking at them in terms of complexity to implement
>
> #1 and #2 exist today
>
> #3 (elsif) seems like a fairly simple change to support
>
> #4 (switch) is a bit more complex
>
> #5 (call var) sounds like it's easy to implement
>
> #6 (function pointers) is significantly more work
>
>
> Personal opinion
>
> I'd like to see us add elsif (#3), it significantly cleans up long
> if-then-else cascades.
>
> I think that with elsif, the need for switch (#4) is low, and the
> restrictions of it only doing simple equivalence tests (no startswith,
> contains, etc) really limit it's use
>
> call var (#5) seems easy to implement, but I really don't like opening up
> the problem of calling a non-existant ruleset. We could have it silently do
> nothing, but that gets really messy and I am already cringing at the
> troubleshooting exhanges we will have to help people figure out what
> is/isn't happeing.
>
>
> function pointers are by far the most complicated, and since they include
> ruleset parsing after startup, they have the potential to be really ugly to
> implement. On the other hand, they are also by far the most powerful. If we
> could do things like limiting the functions so that they can't do any of
> the startup-type things[1] and only include statements that are normally
> executed for each log type, this would also give us a back-door way of
> providing the dynamic configuration that many people have been asking for.
>
> Internally, this could be implemented by creating rulesets and calling
> them based on the result of the lookup.
>
> [1] no changes to global() or main()
> no loading modules.
> probably no creating inputs
> possibly allow template creation
> just set and action() calls
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE 

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Dave Caplinger]

Just to add to David's suggestion, here are some examples from one of my 
configs:

  template(name="s_relay_time" type="list") {
property(name="timegenerated" dateFormat="rfc5424")
  }
  template(name="s_relay_utime" type="list") {
property(name="timegenerated" dateFormat="unixtimestamp")
constant(value=".")
property(name="timegenerated" dateFormat="subseconds")
  }

Then use exec_template() to populate the variables/properties you like:

  set $!relay_time = exec_template("s_relay_time");
  set $!relay_utime = exec_template("s_relay_utime");

--
Dave Caplinger

> On Dec 1, 2016, at 3:22 PM, David Lang  wrote:
>
> On Thu, 1 Dec 2016, mosto...@gmail.com wrote:
>
>> Thanks David. It helped (sadly it arrived 2 hours late :P)
>>
>> I'm now dealing setting a variable with timestamp:::date-rfc5424 format.
>
> the only way to do that is with a template.
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
>
> Confidentiality Notice: The content of this communication, along with any 
> attachments, is covered by federal and state law governing electronic 
> communications and may contain confidential and legally privileged 
> information. If the reader of this message is not the intended recipient, you 
> are hereby notified that any dissemination, distribution, use or copying of 
> the information contained herein is strictly prohibited. If you have received 
> this communication in error, please immediately contact us by telephone at 
> 402.361.3000 or e-mail secur...@solutionary.com.
>
> Copyright 2000-2016 NTT Security (US) Inc., a wholly-owned subsidiary of NTT 
> Group. All rights reserved. ActiveGuard and Solutionary are registered 
> trademarks and NTT Security is a trademark of NTT Security (US) Inc. 
> Solutionary, the ActiveGuard logo icon, and the Solutionary logo icon are 
> registered service marks of NTT Security (US) Inc.
>


Confidentiality Notice: The content of this communication, along with any 
attachments, is covered by federal and state law governing electronic 
communications and may contain confidential and legally privileged information. 
If the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution, use or copying of the 
information contained herein is strictly prohibited. If you have received this 
communication in error, please immediately contact us by telephone at 
402.361.3000 or e-mail secur...@solutionary.com.

Copyright 2000-2016 NTT Security (US) Inc., a wholly-owned subsidiary of NTT 
Group. All rights reserved. ActiveGuard and Solutionary are registered 
trademarks and NTT Security is a trademark of NTT Security (US) Inc. 
Solutionary, the ActiveGuard logo icon, and the Solutionary logo icon are 
registered service marks of NTT Security (US) Inc.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Thu, 1 Dec 2016, mosto...@gmail.com wrote:


Thanks David. It helped (sadly it arrived 2 hours late :P)

I'm now dealing setting a variable with timestamp:::date-rfc5424 format.


the only way to do that is with a template.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

yup, thx. IMHO makes sense. Let's wait what David says, but I think it
would make sense to open an issue refering to this thread. ... I know,
I also must find time to actually work on some of them... ;-)

This is our current /core.conf/ draft:

   global(
MaxMessageSize="32k"
workDirectory="/data"
parser.escapeControlCharactersOnReceive="off"
   )

   template(name="index" type="string" string="$!data!index")
   template(name="type" type="string" string="$!data!type")
   template(name="json" type="string" string="%$!data%")

   module(load="imelasticsearch")
   ruleset(name="elastic"){
set $!data=$msg;
set $!data!relay=$myhostname;
set $!data!from=$hostname;
action(
action.reportSuspension="on"
action.resumeRetryCount="-1"
#queue.filename="omrelp.qi"
queue.maxdiskspace="1G"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
type="omelasticsearch"
server="server"
serverport="9200"
searchIndex="index"
dynSearchIndex="on"
searchType="type"
dynSearchType="on"
template="json"
)
   }

   module(load="imrelp")
   input(
port="20514"
type="imrelp"
name="imrelp"
   )

   # All files under rsyslog.d are automatically included, each will be
   like:
   # app.conf
   # ruleset(name="app") {
   #   #normalize/parse. ideally using inline rules
   #   #whatever
   #   set $!data!index="myindex--MM-DD";
   #   set $!data!type="this_app_types_are_known_by_this_app";
   #   call another_app_in_pipeline
   #   stop
   # }

   call $!data!app

This is the best approach we have found so far.
Although @davidlang suggested having just one normalization ruleset 
could be faster, this appears to be flexible, easy to maintain, etc.

Thoughts.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

2016-12-01 19:08 GMT+01:00 mosto...@gmail.com :
>
> El 01/12/16 a las 19:04, Rainer Gerhards escribió:
>
>> 2016-12-01 18:56 GMT+01:00 mosto...@gmail.com :
>>>
>>> El 01/12/16 a las 18:37, Rainer Gerhards escribió:

 2016-12-01 18:33 GMT+01:00 mosto...@gmail.com :
>
> Hi
>
> Is there any way to dynamically invoke a ruleset? eg: call $var
> (I'm trying to avoid having +200 if statements...

 not yet, but 90% sure evrything is in place to make implementation easy.

 Can you elaborate on the use case?
>>>
>>>
>>> relay syslog forwarding multiple files to central location
>>> central syslog must mmnormalize depending on syslogtag before indexing
>>> into
>>> elastic
>>> other tasks like geoip must be done whenever a message has a ip field
>>> (for
>>> example)
>>>
>>> current approach is as follows:
>>>
>>>   * core.conf contains input and ruleset for indexing
>>>   * appX.conf files are copied to /etc/rsyslogd.d/, to be loaded at start
>>>   * each app has a .conf file to define both, additional transformations
>>> + mmnormalize rules (https://github.com/rsyslog/rsyslog/issues/625)
>>>   * when a message is received, it must be processed by 1-N apps, which
>>> would be great if done dynamically, but I don't think that's
>>> possible.
>>>
>>>
>>> core.conf
>>>
>>> ruleset("name="elastic") {
>>>  action(type="omelasticsearch"
>>>  #once this message has been processed by all modules, index
>>>  )
>>> }
>>>
>>> app1.conf
>>>
>>> if $!app equals "app1" then {
>>>  #normalize (davidlang says it's better to have 1 normalizer on
>>> core.conf. I need to think about it)
>>>  #add some custom fields
>>>  #set $!index="myindexname--MM-DD"
>>>  call geoip
>>>  stop
>>> }
>>>
>>> app2.conf
>>>
>>> if $!app equals "app2" then {
>>>  #normalize using inline rulebase
>>>  #set $!index="otherindexname";
>>>  stop
>>> }
>>>
>>> app200.conf
>>>
>>> if $!app equals "app20" then {
>>>  #whatever
>>> }
>>>
>>> geoip.conf
>>>
>>> ruleset(name="geoip") {
>>>  #geo tag this message
>>> }
>>>
>>> unk.conf
>>>
>>> if message_has_not_been_proccessed then {
>>> #set $!index="unknown";
>>> }
>>>
>>> I hope I explained myself properly...
>>
>> where would you use "call $var" if it were available?
>
>
> core.conf
>
> ruleset("name="elastic") {
> action(type="omelasticsearch"
> #index
> )
>}
>call %syslogtag%  #Really, this is actually stored at $!app, but I think
> you got the idea...(eg: appX)

yup, thx. IMHO makes sense. Let's wait what David says, but I think it
would make sense to open an issue refering to this thread. ... I know,
I also must find time to actually work on some of them... ;-)

Rainer
>
> appX.conf
>
> ruleset(name="appX") {
> #whatever
> #set $!index="myindexname--MM-DD"
>}
>
> This way, core and app configurations will be -IMHO- much simpler.
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

El 01/12/16 a las 19:04, Rainer Gerhards escribió:

2016-12-01 18:56 GMT+01:00 mosto...@gmail.com :

El 01/12/16 a las 18:37, Rainer Gerhards escribió:

2016-12-01 18:33 GMT+01:00 mosto...@gmail.com :

Hi

Is there any way to dynamically invoke a ruleset? eg: call $var
(I'm trying to avoid having +200 if statements...

not yet, but 90% sure evrything is in place to make implementation easy.

Can you elaborate on the use case?


relay syslog forwarding multiple files to central location
central syslog must mmnormalize depending on syslogtag before indexing into
elastic
other tasks like geoip must be done whenever a message has a ip field (for
example)

current approach is as follows:

  * core.conf contains input and ruleset for indexing
  * appX.conf files are copied to /etc/rsyslogd.d/, to be loaded at start
  * each app has a .conf file to define both, additional transformations
+ mmnormalize rules (https://github.com/rsyslog/rsyslog/issues/625)
  * when a message is received, it must be processed by 1-N apps, which
would be great if done dynamically, but I don't think that's possible.


core.conf

ruleset("name="elastic") {
 action(type="omelasticsearch"
 #once this message has been processed by all modules, index
 )
}

app1.conf

if $!app equals "app1" then {
 #normalize (davidlang says it's better to have 1 normalizer on
core.conf. I need to think about it)
 #add some custom fields
 #set $!index="myindexname--MM-DD"
 call geoip
 stop
}

app2.conf

if $!app equals "app2" then {
 #normalize using inline rulebase
 #set $!index="otherindexname";
 stop
}

app200.conf

if $!app equals "app20" then {
 #whatever
}

geoip.conf

ruleset(name="geoip") {
 #geo tag this message
}

unk.conf

if message_has_not_been_proccessed then {
#set $!index="unknown";
}

I hope I explained myself properly...

where would you use "call $var" if it were available?


core.conf

ruleset("name="elastic") {
action(type="omelasticsearch"
#index
)
   }
   call %syslogtag%  #Really, this is actually stored at $!app, but I 
think you got the idea...(eg: appX)


appX.conf

ruleset(name="appX") {
#whatever
#set $!index="myindexname--MM-DD"
   }

This way, core and app configurations will be -IMHO- much simpler.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

2016-12-01 18:56 GMT+01:00 mosto...@gmail.com :
> El 01/12/16 a las 18:37, Rainer Gerhards escribió:
>>
>> 2016-12-01 18:33 GMT+01:00 mosto...@gmail.com :
>>>
>>> Hi
>>>
>>> Is there any way to dynamically invoke a ruleset? eg: call $var
>>> (I'm trying to avoid having +200 if statements...
>>
>> not yet, but 90% sure evrything is in place to make implementation easy.
>>
>> Can you elaborate on the use case?
>
>
> relay syslog forwarding multiple files to central location
> central syslog must mmnormalize depending on syslogtag before indexing into
> elastic
> other tasks like geoip must be done whenever a message has a ip field (for
> example)
>
> current approach is as follows:
>
>  * core.conf contains input and ruleset for indexing
>  * appX.conf files are copied to /etc/rsyslogd.d/, to be loaded at start
>  * each app has a .conf file to define both, additional transformations
>+ mmnormalize rules (https://github.com/rsyslog/rsyslog/issues/625)
>  * when a message is received, it must be processed by 1-N apps, which
>would be great if done dynamically, but I don't think that's possible.
>
>
> core.conf
>
>ruleset("name="elastic") {
> action(type="omelasticsearch"
> #once this message has been processed by all modules, index
> )
>}
>
> app1.conf
>
>if $!app equals "app1" then {
> #normalize (davidlang says it's better to have 1 normalizer on
>core.conf. I need to think about it)
> #add some custom fields
> #set $!index="myindexname--MM-DD"
> call geoip
> stop
>}
>
> app2.conf
>
>if $!app equals "app2" then {
> #normalize using inline rulebase
> #set $!index="otherindexname";
> stop
>}
>
> app200.conf
>
>if $!app equals "app20" then {
> #whatever
>}
>
> geoip.conf
>
>ruleset(name="geoip") {
> #geo tag this message
>}
>
> unk.conf
>
>if message_has_not_been_proccessed then {
>#set $!index="unknown";
>}
>
> I hope I explained myself properly...

where would you use "call $var" if it were available?

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

El 01/12/16 a las 18:37, Rainer Gerhards escribió:

2016-12-01 18:33 GMT+01:00 mosto...@gmail.com :

Hi

Is there any way to dynamically invoke a ruleset? eg: call $var
(I'm trying to avoid having +200 if statements...

not yet, but 90% sure evrything is in place to make implementation easy.

Can you elaborate on the use case?


relay syslog forwarding multiple files to central location
central syslog must mmnormalize depending on syslogtag before indexing 
into elastic
other tasks like geoip must be done whenever a message has a ip field 
(for example)


current approach is as follows:

 * core.conf contains input and ruleset for indexing
 * appX.conf files are copied to /etc/rsyslogd.d/, to be loaded at start
 * each app has a .conf file to define both, additional transformations
   + mmnormalize rules (https://github.com/rsyslog/rsyslog/issues/625)
 * when a message is received, it must be processed by 1-N apps, which
   would be great if done dynamically, but I don't think that's possible.


core.conf

   ruleset("name="elastic") {
action(type="omelasticsearch"
#once this message has been processed by all modules, index
)
   }

app1.conf

   if $!app equals "app1" then {
#normalize (davidlang says it's better to have 1 normalizer on
   core.conf. I need to think about it)
#add some custom fields
#set $!index="myindexname--MM-DD"
call geoip
stop
   }

app2.conf

   if $!app equals "app2" then {
#normalize using inline rulebase
#set $!index="otherindexname";
stop
   }

app200.conf

   if $!app equals "app20" then {
#whatever
   }

geoip.conf

   ruleset(name="geoip") {
#geo tag this message
   }

unk.conf

   if message_has_not_been_proccessed then {
   #set $!index="unknown";
   }

I hope I explained myself properly...
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

2016-12-01 18:33 GMT+01:00 mosto...@gmail.com :
> Hi
>
> Is there any way to dynamically invoke a ruleset? eg: call $var
> (I'm trying to avoid having +200 if statements...

not yet, but 90% sure evrything is in place to make implementation easy.

Can you elaborate on the use case?

Rainer
>
>
>
>
> El 25/11/16 a las 14:13, David Lang escribió:
>
>> On Fri, 25 Nov 2016, mosto...@gmail.com wrote:
>>
 I may be confused about which part is on the sender and which part is on
 the receiver.
>>>
>>> sender: a bunch of imfiles forwarded using RELP
>>>
>>> receiver: receives a JSON with msg=plain/original message, normalize and
>>> extract fields (that should be added to JSON)
>>> each "application" should define his own rules, and sometimes even
>>> transform the JSON after that (seem's that the hard part)
>>>

>>
>>>   *# Is addMetadata="on" needed in order to use $!metadata!filename?*
>>
>>
 the easiest thing is to try it :-)
>>>
>>> I'm going to create an issue for double checking...can't handle this
>>> while editing documentation!
>>>
 adding fields you may be able to do with the ammend= capabilities in the
 ruleset

 if you don't want a field to be reported, give it the name '-' in the
 ruleset.

 unfortunantly, you can't rename fields or copy fields in the ruleset.
>>>
>>> So: each application having one .conf file copied to rsyslog.d/ with the
>>> required steps it's the only way? Perhaps something like:
>>> *app1.conf*
>>>
>>>   if $!group == "group" and $!app == "app1" then {
>>># and here's an example on when to use inline rules
>>># https://github.com/rsyslog/rsyslog/issues/625
>>># an inline rule here will make it possible to have
>>># just 1 config file per app, instead of 2
>>>action(type="mmnormalize" rulebase=:/rule-for-app1.fb")
>>>if $parsesuccess then {
>>># do additional steps, transforms and whatever you want
>>># call foo
>>>}
>>>call index
>>>stop
>>>   }
>>
>>
>> it's better to have a single ruleset that does the parsing once, and then
>> calls a 'perapp' ruleset that is something like
>>
>> ruleset(name="perapp"){
>>   include /etc/rsyslog.d/apps/*
>> }
>>
>> and in /etc/rsyslog.d/apps/ you have per app files that have the per-app
>> if statement and any manipulation needed
>>
>> David Lang
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

Hi

Is there any way to dynamically invoke a ruleset? eg: call $var
(I'm trying to avoid having +200 if statements...




El 25/11/16 a las 14:13, David Lang escribió:

On Fri, 25 Nov 2016, mosto...@gmail.com wrote:

I may be confused about which part is on the sender and which part 
is on the receiver.

sender: a bunch of imfiles forwarded using RELP

receiver: receives a JSON with msg=plain/original message, normalize 
and extract fields (that should be added to JSON)
each "application" should define his own rules, and sometimes even 
transform the JSON after that (seem's that the hard part)






  *# Is addMetadata="on" needed in order to use 
$!metadata!filename?*



the easiest thing is to try it :-)
I'm going to create an issue for double checking...can't handle this 
while editing documentation!


adding fields you may be able to do with the ammend= capabilities in 
the ruleset


if you don't want a field to be reported, give it the name '-' in 
the ruleset.


unfortunantly, you can't rename fields or copy fields in the ruleset.
So: each application having one .conf file copied to rsyslog.d/ with 
the required steps it's the only way? Perhaps something like:

*app1.conf*

  if $!group == "group" and $!app == "app1" then {
   # and here's an example on when to use inline rules
   # https://github.com/rsyslog/rsyslog/issues/625
   # an inline rule here will make it possible to have
   # just 1 config file per app, instead of 2
   action(type="mmnormalize" rulebase=:/rule-for-app1.fb")
   if $parsesuccess then {
   # do additional steps, transforms and whatever you want
   # call foo
   }
   call index
   stop
  }


it's better to have a single ruleset that does the parsing once, and 
then calls a 'perapp' ruleset that is something like


ruleset(name="perapp"){
  include /etc/rsyslog.d/apps/*
}

and in /etc/rsyslog.d/apps/ you have per app files that have the 
per-app if statement and any manipulation needed


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

Thanks David. It helped (sadly it arrived 2 hours late :P)

I'm now dealing setting a variable with timestamp:::date-rfc5424 format.


El 01/12/16 a las 15:57, Dave Caplinger escribió:

Try:

   set $!data!foo = $programname;

As far as I know, rainerscript can't inject variables/properties into string 
literals directly; so if you really want to use string concatenation do this:

   set $!data!foo = "this_might_work_better_" & $programname;

If you want to get any more complex than that, you can use a template and exec 
it:

   template(name="s_my_programname" type="string" string="%$programname%")
   set $!data!foo = exec_template("s_my_hostname");

(but you could make the template much more complex, such as stringing multiple 
things together, adding delimiters, etc.)

Hope one of these helps,

--
Dave Caplinger
Director, Technical Product Management

On Dec 1, 2016, at 8:28 AM, mosto...@gmail.com wrote:

After meal, as usually happens, those quotes sparkled. Doesn't rsyslog
conf grammar allows that neither?

/(eg: set $!data!foo="this_doesnt_seem_to_work_$programname";)/



El 01/12/16 a las 14:57, mosto...@gmail.com escribió:

This worked, but I have lost 2 hours and still don't see where's the evil.

Works:

module(load="omrelp")
ruleset(name="relp") {
set $!data!group=field($programname,47,1);
set $!data!msg=$msg;
action(
action.reportSuspension="on"
action.resumeRetryCount="-1"
port="20514"
queue.maxdiskspace="5M"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
target="server"
template="RSYSLOG_DebugFormat"
type="omrelp"
)
}

Doesn't work

module(load="omrelp")
ruleset(name="relp") {
set $!data!foo="$programname"; >:
no, we need to see the contents of programname (the RSYSLOG_DebugFormat will
show this.

David Lang
k

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Confidentiality Notice: The content of this communication, along with any 
attachments, is covered by federal and state law governing electronic 
communications and may contain confidential and legally privileged information. 
If the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution, use or copying of the 
information contained herein is strictly prohibited. If you have received this 
communication in error, please immediately contact us by telephone at 
402.361.3000 or e-mail secur...@solutionary.com.

Copyright 2000-2016 NTT Security (US) Inc., a wholly-owned subsidiary of NTT 
Group. All rights reserved. ActiveGuard and Solutionary are registered 
trademarks and NTT Security is a trademark of NTT Security (US) Inc. 
Solutionary, the ActiveGuard logo icon, and the Solutionary logo icon are 
registered service marks of NTT Security (US) Inc.




Confidentiality Notice: The 

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

You can save time by ensuring that the config file is error-free
before you try out anything. As long as there is at least one error,
you never know what that error affects (well, you can know, but then
you need to be deep into the architecture). It's the same thing as
with compiler error messages...

Also, it would have been helpful if you had posted the requested info,
then we could have helped. But without that debug info, we really
don't know what goes on (I still don't know about what was the cause
of your original issue).

Rainer
Rainer

2016-12-01 14:57 GMT+01:00 mosto...@gmail.com :
> This worked, but I have lost 2 hours and still don't see where's the evil.
>
> Works:
>
>module(load="omrelp")
>ruleset(name="relp") {
> set $!data!group=field($programname,47,1);
> set $!data!msg=$msg;
> action(
> action.reportSuspension="on"
> action.resumeRetryCount="-1"
> port="20514"
> queue.maxdiskspace="5M"
> queue.SaveOnShutdown="on"
> queue.type="LinkedList"
> target="server"
> template="RSYSLOG_DebugFormat"
> type="omrelp"
> )
>}
>
> Doesn't work
>
>module(load="omrelp")
>ruleset(name="relp") {
> set $!data!foo="$programname"; this! (Line 17)
> set $!data!group=field($programname,47,1);
> set $!data!msg=$msg;
> action(
> action.reportSuspension="on"
> action.resumeRetryCount="-1"
> port="20514"
> queue.maxdiskspace="5M"
> queue.SaveOnShutdown="on"
> queue.type="LinkedList"
> target="server"
> template="RSYSLOG_DebugFormat"
> type="omrelp"
> )
>}
>
> Rsyslog complains with:
>
> rsyslogd: error during parsing file /etc/rsyslog.conf, on or before line 17:
> invalid character '"' in expression - is there an invalid escape sequence
> somewhere? [v8.23.0 try http://www.rsyslog.com/e/2207 ]
>
>
> El 01/12/16 a las 11:14, Rainer Gerhards escribió:
>
>> maybe the complete debug log would also help (not sure).
>> Rainer
>>
>> 2016-12-01 11:12 GMT+01:00 David Lang :
>>>
>>> no, we need to see the contents of programname (the RSYSLOG_DebugFormat
>>> will
>>> show this.
>>>
>>> David Lang
>>> k
>>>
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
>>
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Dave Caplinger]

Try:

  set $!data!foo = $programname;

As far as I know, rainerscript can't inject variables/properties into string 
literals directly; so if you really want to use string concatenation do this:

  set $!data!foo = "this_might_work_better_" & $programname;

If you want to get any more complex than that, you can use a template and exec 
it:

  template(name="s_my_programname" type="string" string="%$programname%")
  set $!data!foo = exec_template("s_my_hostname");

(but you could make the template much more complex, such as stringing multiple 
things together, adding delimiters, etc.)

Hope one of these helps,

--
Dave Caplinger
Director, Technical Product Management

On Dec 1, 2016, at 8:28 AM, mosto...@gmail.com wrote:

After meal, as usually happens, those quotes sparkled. Doesn't rsyslog
conf grammar allows that neither?

/(eg: set $!data!foo="this_doesnt_seem_to_work_$programname";)/



El 01/12/16 a las 14:57, mosto...@gmail.com escribió:

This worked, but I have lost 2 hours and still don't see where's the evil.

Works:

   module(load="omrelp")
   ruleset(name="relp") {
   set $!data!group=field($programname,47,1);
   set $!data!msg=$msg;
   action(
   action.reportSuspension="on"
   action.resumeRetryCount="-1"
   port="20514"
   queue.maxdiskspace="5M"
   queue.SaveOnShutdown="on"
   queue.type="LinkedList"
   target="server"
   template="RSYSLOG_DebugFormat"
   type="omrelp"
   )
   }

Doesn't work

   module(load="omrelp")
   ruleset(name="relp") {
   set $!data!foo="$programname"; >:
no, we need to see the contents of programname (the RSYSLOG_DebugFormat will
show this.

David Lang
k

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Confidentiality Notice: The content of this communication, along with any 
attachments, is covered by federal and state law governing electronic 
communications and may contain confidential and legally privileged information. 
If the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution, use or copying of the 
information contained herein is strictly prohibited. If you have received this 
communication in error, please immediately contact us by telephone at 
402.361.3000 or e-mail secur...@solutionary.com.

Copyright 2000-2016 NTT Security (US) Inc., a wholly-owned subsidiary of NTT 
Group. All rights reserved. ActiveGuard and Solutionary are registered 
trademarks and NTT Security is a trademark of NTT Security (US) Inc. 
Solutionary, the ActiveGuard logo icon, and the Solutionary logo icon are 
registered service marks of NTT Security (US) Inc.




Confidentiality Notice: The content of this communication, along with any 
attachments, is covered by federal and state law governing electronic 
communications and may contain confidential and legally privileged information. 
If the reader of this 

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

After meal, as usually happens, those quotes sparkled. Doesn't rsyslog 
conf grammar allows that neither?


/(eg: set $!data!foo="this_doesnt_seem_to_work_$programname";)/



El 01/12/16 a las 14:57, mosto...@gmail.com escribió:


This worked, but I have lost 2 hours and still don't see where's the evil.

Works:

module(load="omrelp")
ruleset(name="relp") {
set $!data!group=field($programname,47,1);
set $!data!msg=$msg;
action(
action.reportSuspension="on"
action.resumeRetryCount="-1"
port="20514"
queue.maxdiskspace="5M"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
target="server"
template="RSYSLOG_DebugFormat"
type="omrelp"
)
}

Doesn't work

module(load="omrelp")
ruleset(name="relp") {
set $!data!foo="$programname"; :

no, we need to see the contents of programname (the RSYSLOG_DebugFormat will
show this.

David Lang
k

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

This worked, but I have lost 2 hours and still don't see where's the evil.

Works:

   module(load="omrelp")
   ruleset(name="relp") {
set $!data!group=field($programname,47,1);
set $!data!msg=$msg;
action(
action.reportSuspension="on"
action.resumeRetryCount="-1"
port="20514"
queue.maxdiskspace="5M"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
target="server"
template="RSYSLOG_DebugFormat"
type="omrelp"
)
   }

Doesn't work

   module(load="omrelp")
   ruleset(name="relp") {
set $!data!foo="$programname"; :

no, we need to see the contents of programname (the RSYSLOG_DebugFormat will
show this.

David Lang
k

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

maybe the complete debug log would also help (not sure).
Rainer

2016-12-01 11:12 GMT+01:00 David Lang :
> no, we need to see the contents of programname (the RSYSLOG_DebugFormat will
> show this.
>
> David Lang
> k
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

no, we need to see the contents of programname (the RSYSLOG_DebugFormat will 
show this.


David Lang
k
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

I don't know if this is what you are asking for...

3421.997587883:main Q:Reg/w0  : SET !data!aapp =
3421.997596172:main Q:Reg/w0  :   function 'field' (id:9, params:3)
3421.997607766:main Q:Reg/w0  : var 'programname'
3421.997687716:main Q:Reg/w0  : 47
3421.997714715:main Q:Reg/w0  : 2
3421.997728731:main Q:Reg/w0  : END SET



El 30/11/16 a las 22:32, Rainer Gerhards escribió:

Can you show what the properties are for this message?

Sent from phone, thus brief.

Am 30.11.2016 20:18 schrieb "mosto...@gmail.com" :


Hi

I'm still not able to get it, and *perhaps reproduced an issue related to
properties not being accessible*...
Consider the following relay.conf:

global(
MaxMessageSize="32k"
workDirectory="/var/spool/rsyslog"
parser.escapeControlCharactersOnReceive="off"
)

template(
 name="json"
string="<%pri%>%timestamp:::date-rfc3339% %hostname%
logs/%$!data!group%/%$!data!aapp%: %$!data%"
 type="string"
)

module(load="omrelp")
ruleset(name="relp") {
 set $!data!aapp=field($programname,"/",2);
 set $!data!file="$!metadata!filename";
 set $!data!group=field($programname,"/",1);
 set $!data!msg=$msg;
 action(
action.reportSuspension="on"
action.resumeRetryCount="-1"
port="20514"
queue.maxdiskspace="5M"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
target="server"
template="json"
type="omrelp"
 )
}
ruleset(name="apps") {
 call relp
 stop
}

module(load="imfile")

input(type="imfile" file="/logs/apache/app1/app.log"
tag="group/app1" addMetadata="on" ruleset="apps"
PersistStateInterval="1")

input(type="imfile" file="/logs/apache/app2/app.log"
tag="group/app2" addMetadata="on" ruleset="apps"
PersistStateInterval="1")
ruleset(name="app_server1") {
 set $!data!containerApps="app1,app2";
 call relp
 stop
}

input(type="imfile" file="/logs/server1/app1.log"
tag="group1/server1" addMetadata="on" ruleset="app_server1"
startmsg.regex="^" readTimeout="5" PersistStateInterval="1")
input(type="imfile" file="/logs/server1/app2.log"
tag="group1/server1" addMetadata="on" ruleset="app_server1"
startmsg.regex="^" readTimeout="5" PersistStateInterval="1")

ruleset(name="app_server2") {
 set $!data!containerApps="app2,app3";
 call relp
 stop
}
input(type="imfile" file="/logs/server2/app2.log"
tag="group2/server2" addMetadata="on" ruleset="app_server2"
startmsg.regex="^" readTimeout="5" PersistStateInterval="1")
input(type="imfile" file="/logs/server2/app3.log"
tag="group2/server2" addMetadata="on" ruleset="app_server2"
startmsg.regex="^" readTimeout="5" PersistStateInterval="1")


With this configuration I'm getting messages like:

200 syslog 911 <133>2016-11-30T20:02:30.210405+01:00 my-rsyslog
logs/group/***FIELD NOT FOUND***: { "aapp": "***FIELD NOT FOUND***",
"file": "\/logs\/apache\/app1\/app.log", "group": "group", "msg":
"17 mar 2016 13:27:28,934  INFO REDACTED..." }


Why /aapp/ field isn't found?

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

Can you show what the properties are for this message?

Sent from phone, thus brief.

Am 30.11.2016 20:18 schrieb "mosto...@gmail.com" :

> Hi
>
> I'm still not able to get it, and *perhaps reproduced an issue related to
> properties not being accessible*...
> Consider the following relay.conf:
>
>global(
>MaxMessageSize="32k"
>workDirectory="/var/spool/rsyslog"
>parser.escapeControlCharactersOnReceive="off"
>)
>
>template(
> name="json"
>string="<%pri%>%timestamp:::date-rfc3339% %hostname%
>logs/%$!data!group%/%$!data!aapp%: %$!data%"
> type="string"
>)
>
>module(load="omrelp")
>ruleset(name="relp") {
> set $!data!aapp=field($programname,"/",2);
> set $!data!file="$!metadata!filename";
> set $!data!group=field($programname,"/",1);
> set $!data!msg=$msg;
> action(
>action.reportSuspension="on"
>action.resumeRetryCount="-1"
>port="20514"
>queue.maxdiskspace="5M"
>queue.SaveOnShutdown="on"
>queue.type="LinkedList"
>target="server"
>template="json"
>type="omrelp"
> )
>}
>ruleset(name="apps") {
> call relp
> stop
>}
>
>module(load="imfile")
>
>input(type="imfile" file="/logs/apache/app1/app.log"
>tag="group/app1" addMetadata="on" ruleset="apps"
>PersistStateInterval="1")
>
>input(type="imfile" file="/logs/apache/app2/app.log"
>tag="group/app2" addMetadata="on" ruleset="apps"
>PersistStateInterval="1")
>ruleset(name="app_server1") {
> set $!data!containerApps="app1,app2";
> call relp
> stop
>}
>
>input(type="imfile" file="/logs/server1/app1.log"
>tag="group1/server1" addMetadata="on" ruleset="app_server1"
>startmsg.regex="^" readTimeout="5" PersistStateInterval="1")
>input(type="imfile" file="/logs/server1/app2.log"
>tag="group1/server1" addMetadata="on" ruleset="app_server1"
>startmsg.regex="^" readTimeout="5" PersistStateInterval="1")
>
>ruleset(name="app_server2") {
> set $!data!containerApps="app2,app3";
> call relp
> stop
>}
>input(type="imfile" file="/logs/server2/app2.log"
>tag="group2/server2" addMetadata="on" ruleset="app_server2"
>startmsg.regex="^" readTimeout="5" PersistStateInterval="1")
>input(type="imfile" file="/logs/server2/app3.log"
>tag="group2/server2" addMetadata="on" ruleset="app_server2"
>startmsg.regex="^" readTimeout="5" PersistStateInterval="1")
>
>
> With this configuration I'm getting messages like:
>
>200 syslog 911 <133>2016-11-30T20:02:30.210405+01:00 my-rsyslog
>logs/group/***FIELD NOT FOUND***: { "aapp": "***FIELD NOT FOUND***",
>"file": "\/logs\/apache\/app1\/app.log", "group": "group", "msg":
>"17 mar 2016 13:27:28,934  INFO REDACTED..." }
>
>
> Why /aapp/ field isn't found?
>
> Regards
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Wed, 30 Nov 2016, mosto...@gmail.com wrote:

I'm still not able to get it, and *perhaps reproduced an issue related to 
properties not being accessible*...

Consider the following relay.conf:

  global(
  MaxMessageSize="32k"
  workDirectory="/var/spool/rsyslog"
  parser.escapeControlCharactersOnReceive="off"
  )

  template(
   name="json"
  string="<%pri%>%timestamp:::date-rfc3339% %hostname%
  logs/%$!data!group%/%$!data!aapp%: %$!data%"
   type="string"
  )


first off, this template is still not forming a valid syslog message.

syslog messages can be one of two formats

rfc-3164

<44>Jan 12 11:22:33 host program[pid]: message
(with [pid] being optional)

or rfc-5424

<44>1 2003-08-24T05:14:15.03-07:00 host.f.d.n program pid - message

what you are producing is neither because you are using the wrong time format, 
change date-rfc3339 to date-rfc3164.




  module(load="omrelp")
  ruleset(name="relp") {
   set $!data!aapp=field($programname,"/",2);
   set $!data!file="$!metadata!filename";
   set $!data!group=field($programname,"/",1);
   set $!data!msg=$msg;
   action(
  action.reportSuspension="on"
  action.resumeRetryCount="-1"
  port="20514"
  queue.maxdiskspace="5M"
  queue.SaveOnShutdown="on"
  queue.type="LinkedList"
  target="server"
  template="json"
  type="omrelp"
   )
  }
  ruleset(name="apps") {
   call relp
   stop
  }

  module(load="imfile")

  input(type="imfile" file="/logs/apache/app1/app.log"
  tag="group/app1" addMetadata="on" ruleset="apps"
  PersistStateInterval="1")

  input(type="imfile" file="/logs/apache/app2/app.log"
  tag="group/app2" addMetadata="on" ruleset="apps"
  PersistStateInterval="1")
  ruleset(name="app_server1") {
   set $!data!containerApps="app1,app2";
   call relp
   stop
  }

  input(type="imfile" file="/logs/server1/app1.log"
  tag="group1/server1" addMetadata="on" ruleset="app_server1"
  startmsg.regex="^" readTimeout="5" PersistStateInterval="1")
  input(type="imfile" file="/logs/server1/app2.log"
  tag="group1/server1" addMetadata="on" ruleset="app_server1"
  startmsg.regex="^" readTimeout="5" PersistStateInterval="1")

  ruleset(name="app_server2") {
   set $!data!containerApps="app2,app3";
   call relp
   stop
  }
  input(type="imfile" file="/logs/server2/app2.log"
  tag="group2/server2" addMetadata="on" ruleset="app_server2"
  startmsg.regex="^" readTimeout="5" PersistStateInterval="1")
  input(type="imfile" file="/logs/server2/app3.log"
  tag="group2/server2" addMetadata="on" ruleset="app_server2"
  startmsg.regex="^" readTimeout="5" PersistStateInterval="1")


With this configuration I'm getting messages like:

  200 syslog 911 <133>2016-11-30T20:02:30.210405+01:00 my-rsyslog
  logs/group/***FIELD NOT FOUND***: { "aapp": "***FIELD NOT FOUND***",
  "file": "\/logs\/apache\/app1\/app.log", "group": "group", "msg":
  "17 mar 2016 13:27:28,934  INFO REDACTED..." }


Why /aapp/ field isn't found?


any time you don't get fields like you expect, write a log in the format 
RSYSLOG_DebugFormat and look at what is actually set


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

Hi

I'm still not able to get it, and *perhaps reproduced an issue related 
to properties not being accessible*...

Consider the following relay.conf:

   global(
   MaxMessageSize="32k"
   workDirectory="/var/spool/rsyslog"
   parser.escapeControlCharactersOnReceive="off"
   )

   template(
name="json"
   string="<%pri%>%timestamp:::date-rfc3339% %hostname%
   logs/%$!data!group%/%$!data!aapp%: %$!data%"
type="string"
   )

   module(load="omrelp")
   ruleset(name="relp") {
set $!data!aapp=field($programname,"/",2);
set $!data!file="$!metadata!filename";
set $!data!group=field($programname,"/",1);
set $!data!msg=$msg;
action(
   action.reportSuspension="on"
   action.resumeRetryCount="-1"
   port="20514"
   queue.maxdiskspace="5M"
   queue.SaveOnShutdown="on"
   queue.type="LinkedList"
   target="server"
   template="json"
   type="omrelp"
)
   }
   ruleset(name="apps") {
call relp
stop
   }

   module(load="imfile")

   input(type="imfile" file="/logs/apache/app1/app.log"
   tag="group/app1" addMetadata="on" ruleset="apps"
   PersistStateInterval="1")

   input(type="imfile" file="/logs/apache/app2/app.log"
   tag="group/app2" addMetadata="on" ruleset="apps"
   PersistStateInterval="1")
   ruleset(name="app_server1") {
set $!data!containerApps="app1,app2";
call relp
stop
   }

   input(type="imfile" file="/logs/server1/app1.log"
   tag="group1/server1" addMetadata="on" ruleset="app_server1"
   startmsg.regex="^" readTimeout="5" PersistStateInterval="1")
   input(type="imfile" file="/logs/server1/app2.log"
   tag="group1/server1" addMetadata="on" ruleset="app_server1"
   startmsg.regex="^" readTimeout="5" PersistStateInterval="1")

   ruleset(name="app_server2") {
set $!data!containerApps="app2,app3";
call relp
stop
   }
   input(type="imfile" file="/logs/server2/app2.log"
   tag="group2/server2" addMetadata="on" ruleset="app_server2"
   startmsg.regex="^" readTimeout="5" PersistStateInterval="1")
   input(type="imfile" file="/logs/server2/app3.log"
   tag="group2/server2" addMetadata="on" ruleset="app_server2"
   startmsg.regex="^" readTimeout="5" PersistStateInterval="1")


With this configuration I'm getting messages like:

   200 syslog 911 <133>2016-11-30T20:02:30.210405+01:00 my-rsyslog
   logs/group/***FIELD NOT FOUND***: { "aapp": "***FIELD NOT FOUND***",
   "file": "\/logs\/apache\/app1\/app.log", "group": "group", "msg":
   "17 mar 2016 13:27:28,934  INFO REDACTED..." }


Why /aapp/ field isn't found?

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

I may be confused about which part is on the sender and which part is 
on the receiver.

sender: a bunch of imfiles forwarded using RELP

receiver: receives a JSON with msg=plain/original message, normalize and 
extract fields (that should be added to JSON)
each "application" should define his own rules, and sometimes even 
transform the JSON after that (seem's that the hard part)







  *# Is addMetadata="on" needed in order to use $!metadata!filename?*



the easiest thing is to try it :-)
I'm going to create an issue for double checking...can't handle this 
while editing documentation!


adding fields you may be able to do with the ammend= capabilities in 
the ruleset


if you don't want a field to be reported, give it the name '-' in the 
ruleset.


unfortunantly, you can't rename fields or copy fields in the ruleset.
So: each application having one .conf file copied to rsyslog.d/ with the 
required steps it's the only way? Perhaps something like:

*app1.conf*

   if $!group == "group" and $!app == "app1" then {
# and here's an example on when to use inline rules
# https://github.com/rsyslog/rsyslog/issues/625
# an inline rule here will make it possible to have
# just 1 config file per app, instead of 2
action(type="mmnormalize" rulebase=:/rule-for-app1.fb")
if $parsesuccess then {
# do additional steps, transforms and whatever you want
# call foo
}
call index
stop
   }


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Fri, 25 Nov 2016, mosto...@gmail.com wrote:

string="<%pri%>%timestamp:::date-rfc3339% %hostname% 
logs/$!data!group/$!data!app: %$!data%") to be fully correct

Done.

also watch out, the programname is limited to 32 characters, don't let your 
group and app names get too long.
Wasn't it possible to change that? IIRC we had some issues with 
hostnames/tags being too long and were able to handle longer.

Anyway, we'll try to stay within boundaries.

since you changed the programname to be logs/group/app this would be field 
3

Are you sure?
*ruleset apps* is invoked for each input using TAG=group/app, and AFTER that 
ruleset relp uses template json, which prefix "logs/"


I may be confused about which part is on the sender and which part is on the 
receiver.





  *# Is addMetadata="on" needed in order to use $!metadata!filename?*

I think so.
As I'm double checking everything while updating docs, I would love to have a 
more confident statement on this. Rainer?


the easiest thing is to try it :-)

correct, although mmjsonparse defaults to needing @cee: in front of the 
json, so the line below needs to be changed to:


module(load="mmjsonparse" cookie="")
I forgot! Nice catch (...I'll have to check if cookie goes in module or 
action...)


it should be action, sorry


  *# Once all operations have ended, it should be indexed**
  **# Is there any way apps not only define rules, but aditional
  transformations?**
  **# I guess having a .conf file with if+ruleset could work...*
no, the mmnormalize ruleset cannot apply any transformations. I would 
probably try to do that on the sending side if I could.

That's why I played with rulesets to make something like

  a.conf
 normalize
 add some fields
  b.conf
 normalize
  c.conf
 normalize
 remove some fields

That would make the combination script behave differently


adding fields you may be able to do with the ammend= capabilities in the ruleset

if you don't want a field to be reported, give it the name '-' in the ruleset.

unfortunantly, you can't rename fields or copy fields in the ruleset.




  *# It is possible to use $!index here? Workaround?**

this is what dynsearchindex and dynparent are for.
so: dynSearchIndex="on" searchIndex="mytemplate" and template="$!index", 
right?


yep.




  **# How could EACH app specify his own index pattern?**
they can't directly, but the template can be "%$.manual%" and you use 
rainerscript commands to set $.manual to whatever you want it to me 
(another good use for a lookup table if it's complex enough ;-)

I didn't understand this...but having each app.conf file could also work.


you can do

set ...
set ...
action()

but not

action(set)

set is a statement, action is a statement. Action takes parameters, but not 
statements inside the ()


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

string="<%pri%>%timestamp:::date-rfc3339% %hostname% 
logs/$!data!group/$!data!app: %$!data%") to be fully correct

Done.

also watch out, the programname is limited to 32 characters, don't let 
your group and app names get too long.
Wasn't it possible to change that? IIRC we had some issues with 
hostnames/tags being too long and were able to handle longer.

Anyway, we'll try to stay within boundaries.

since you changed the programname to be logs/group/app this would be 
field 3

Are you sure?
*ruleset apps* is invoked for each input using TAG=group/app, and AFTER 
that ruleset relp uses template json, which prefix "logs/"





  *# Is addMetadata="on" needed in order to use $!metadata!filename?*

I think so.
As I'm double checking everything while updating docs, I would love to 
have a more confident statement on this. Rainer?


it's probably a good idea to put stop here to make it clear that you 
don't intend for there to be any other processing of the log message. 
With the input bound to a ruleset, I don't think it makes a 
difference, but better to be explicit.

Done

correct, although mmjsonparse defaults to needing @cee: in front of 
the json, so the line below needs to be changed to:


module(load="mmjsonparse" cookie="")
I forgot! Nice catch (...I'll have to check if cookie goes in module or 
action...)


yes, the script can either populate the rules file with includes, or 
just combine them into one file (probably faster at startup to have 
them combined, but it may not be measureable)

I'll combine them (if able)




  *# Once all operations have ended, it should be indexed**
  **# Is there any way apps not only define rules, but aditional
  transformations?**
  **# I guess having a .conf file with if+ruleset could work...*
no, the mmnormalize ruleset cannot apply any transformations. I would 
probably try to do that on the sending side if I could.

That's why I played with rulesets to make something like

   a.conf
  normalize
  add some fields
   b.conf
  normalize
   c.conf
  normalize
  remove some fields

That would make the combination script behave differently




  *# It is possible to use $!index here? Workaround?**

this is what dynsearchindex and dynparent are for.
so: dynSearchIndex="on" searchIndex="mytemplate" and template="$!index", 
right?





  **# How could EACH app specify his own index pattern?**
they can't directly, but the template can be "%$.manual%" and you use 
rainerscript commands to set $.manual to whatever you want it to me 
(another good use for a lookup table if it's complex enough ;-)

I didn't understand this...but having each app.conf file could also work.

The background idea is to combine this with "dynamic configuration 
reload" to be able to change "an application pipeline"


This is looking pretty good now.

Thanks. I think the same!

Seems next pending issue is to solve "each application can do different 
things, like adding hiw own index pattern or additional 
transforms/steps" in his config file.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Fri, 25 Nov 2016, mosto...@gmail.com wrote:


Date: Fri, 25 Nov 2016 11:22:08 +0100
From: "mosto...@gmail.com" <mosto...@gmail.com>
Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
To: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] mmnormalize with mutiple input: conditionals?

What about...?

*remote.conf*(Please, notice there are commented questions)

  global(
  MaxMessageSize="32k"
   )

  template(name="json" type="string"
  string="%timestamp:::date-rfc3339% %hostname%
  logs/$!data!group/$!data!app %$!data%")


change this to:

string="<%pri%>%timestamp:::date-rfc3339% %hostname% logs/$!data!group/$!data!app: 
%$!data%")

to be fully correct

also watch out, the programname is limited to 32 characters, don't let your 
group and app names get too long.



  module(load="omrelp")
  ruleset(name="relp"){
  action(
  port="20514"
  *# It is possible to use $!server here? Workaround?*
  target="server"
  template="json"
  type="omrelp"
   )
   }

  ruleset(name="apps") {
   set $!data!app=field($programname,"/",2);


since you changed the programname to be logs/group/app this would be field 3


  *# Is addMetadata="on" needed in order to use $!metadata!filename?*


I think so.


   set $!data!file="$!metadata!filename";
   set $!data!group=field($programname,"/",1);


as per above, field 2


   set $!data!msg=$msg;
  call relp


it's probably a good idea to put stop here to make it clear that you don't 
intend for there to be any other processing of the log message. With the input 
bound to a ruleset, I don't think it makes a difference, but better to be 
explicit.



   }

  input(type="imfile" file="/logs/apps/app1/app1.log"
  tag="mygroup/myapp1" addMetadata="on" ruleset="apps"
  startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
  readTimeout="5" PersistStateInterval="1")

   ...

  input(type="imfile" file="/logs/apps/anotherapp/file.log"
  tag="anothergroup/anotherapp" addMetadata="on" ruleset="apps"
  readTimeout="5" PersistStateInterval="1")


*rsyslog.conf*(Please, notice there are commented questions)

  global(
  MaxMessageSize="32k"
  parser.escapeControlCharactersOnReceive="off"
   )

  *# Message is parsed as json on receive, to be able to use
  $!whatever field, right?*


correct, although mmjsonparse defaults to needing @cee: in front of the json, so 
the line below needs to be changed to:


module(load="mmjsonparse" cookie="")


  module(load="mmjsonparse")
  ruleset(name="json"){
  action(
  type="mmjsonparse"
   )
   }
  module(load="imrelp")
   input(
  name="imrelp"
  port="20514"
  type="imrelp"
  ruleset="json"
   )

   set $.line = $!group $!app + " " + $!msg;
   action(
  type="mmnormalize"
  variable="$.line"
  *# As I don't know the list of apps, **
  **# the only way to combine all rules is an script**
  **# isnt it?*


yes, the script can either populate the rules file with includes, or just 
combine them into one file (probably faster at startup to have them combined, 
but it may not be measureable)



  rulebase=:/path/to/combined/rules.fb"
   )

  *# IIUC, messages will be procesed by above rule**
  **# AFTER that, they will be processed by the following, right?*


yes


   if message contains "ip" field then {
   # TODO lookup_table
   }

  *# Once all operations have ended, it should be indexed**
  **# Is there any way apps not only define rules, but aditional
  transformations?**
  **# I guess having a .conf file with if+ruleset could work...*


no, the mmnormalize ruleset cannot apply any transformations. I would probably 
try to do that on the sending side if I could.



  template(name="json" type="string" string="%$!%")
  module(load="omelasticsearch")
   action(
  template="json"
  type="omelasticsearch"
  *# It is possible to use $!index here? Workaround?**


this is what dynsearchindex and dynparent are for.


  **# How could EACH app specify his own index pattern?**


they can't directly, but the template can be "%$.manual%" and you use 
rainerscript commands to set $.manual to whatever you want it to me (another 
good use for a lookup table if it's complex enough ;-)



  **#   set $!index="$!app2_$$year-$$month-$$day"**
  **#   set $!index="$!app2_$$year-$$month-$$day-$$hour"*


set statements cant' be inside an action() s

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

What about...?

*remote.conf*(Please, notice there are commented questions)

   global(
   MaxMessageSize="32k"
)

   template(name="json" type="string"
   string="%timestamp:::date-rfc3339% %hostname%
   logs/$!data!group/$!data!app %$!data%")

   module(load="omrelp")
   ruleset(name="relp"){
   action(
   port="20514"
   *# It is possible to use $!server here? Workaround?*
   target="server"
   template="json"
   type="omrelp"
)
}

   ruleset(name="apps") {
set $!data!app=field($programname,"/",2);
   *# Is addMetadata="on" needed in order to use $!metadata!filename?*
set $!data!file="$!metadata!filename";
set $!data!group=field($programname,"/",1);
set $!data!msg=$msg;
   call relp
}

   input(type="imfile" file="/logs/apps/app1/app1.log"
   tag="mygroup/myapp1" addMetadata="on" ruleset="apps"
   startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
   readTimeout="5" PersistStateInterval="1")

...

   input(type="imfile" file="/logs/apps/anotherapp/file.log"
   tag="anothergroup/anotherapp" addMetadata="on" ruleset="apps"
   readTimeout="5" PersistStateInterval="1")


*rsyslog.conf*(Please, notice there are commented questions)

   global(
   MaxMessageSize="32k"
   parser.escapeControlCharactersOnReceive="off"
)

   *# Message is parsed as json on receive, to be able to use
   $!whatever field, right?*
   module(load="mmjsonparse")
   ruleset(name="json"){
   action(
   type="mmjsonparse"
)
}
   module(load="imrelp")
input(
   name="imrelp"
   port="20514"
   type="imrelp"
   ruleset="json"
)

set $.line = $!group $!app + " " + $!msg;
action(
   type="mmnormalize"
   variable="$.line"
   *# As I don't know the list of apps, **
   **# the only way to combine all rules is an script**
   **# isnt it?*
   rulebase=:/path/to/combined/rules.fb"
)

   *# IIUC, messages will be procesed by above rule**
   **# AFTER that, they will be processed by the following, right?*
if message contains "ip" field then {
# TODO lookup_table
}

   *# Once all operations have ended, it should be indexed**
   **# Is there any way apps not only define rules, but aditional
   transformations?**
   **# I guess having a .conf file with if+ruleset could work...*

   template(name="json" type="string" string="%$!%")
   module(load="omelasticsearch")
action(
   template="json"
   type="omelasticsearch"
   *# It is possible to use $!index here? Workaround?**
   **# How could EACH app specify his own index pattern?**
   **#   set $!index="$!app2_$$year-$$month-$$day"**
   **#   set $!index="$!app2_$$year-$$month-$$day-$$hour"*
   searchIndex="$!index"
)


The background idea is to combine this with "dynamic configuration 
reload" to be able to change "an application pipeline"

Thanks a lot for your help. I'll contribute as much as I can in exchange ;)


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Fri, 25 Nov 2016, David Lang wrote:


On Fri, 25 Nov 2016, mosto...@gmail.com wrote:

After sleeping on it, I'm still thinking about "defining a separate 
pipeline for each application". To sum up, each application could do his 
own thing and return the message to the queue, in order to be processed by 
other modules...until it's done, and indexed into ES
Is there a way to put the already processed message into the input queue to 
be processed again by another *sibling* modules?


  input-> queue -> app
  <-
  -> geoip
  <-
  -> index


that's exactly what a message modification module (mm*) does, it changes the 
messages in the queue so that things after them in the config see the 
modified version.


you can also simulate it by having separate rulesets all call one common 
ruleset, but that's really not merging the work back into one flow, it's just 
executing the same code in many different flows.


Also, rsyslog really isn't structured to support per-application pipelines the 
way you are thinking of them. While you could simulate them, they are a bad fit 
for how rsyslog actually works, and as a result, your performance and resource 
usage will be substantially sub-optimal.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Fri, 25 Nov 2016, mosto...@gmail.com wrote:

After sleeping on it, I'm still thinking about "defining a separate pipeline 
for each application". To sum up, each application could do his own thing and 
return the message to the queue, in order to be processed by other 
modules...until it's done, and indexed into ES
Is there a way to put the already processed message into the input queue to 
be processed again by another *sibling* modules?


  input-> queue -> app
  <-
  -> geoip
  <-
  -> index


that's exactly what a message modification module (mm*) does, it changes the 
messages in the queue so that things after them in the config see the modified 
version.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

After sleeping on it, I'm still thinking about "defining a separate 
pipeline for each application". To sum up, each application could do his 
own thing and return the message to the queue, in order to be processed 
by other modules...until it's done, and indexed into ES
Is there a way to put the already processed message into the input queue 
to be processed again by another *sibling* modules?


   input-> queue -> app
   <-
   -> geoip
   <-
   -> index

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

2016-11-24 18:18 GMT+01:00 David Lang :
> On Thu, 24 Nov 2016, Rainer Gerhards wrote:
>
>> 2016-11-24 17:21 GMT+01:00 mosto...@gmail.com :

 # I would consider adding a section here to look for parsing failures
 and
 log them to someplace for later investigation, probably in raw format
>>>
>>>
>>> That's mandatory. is there any easy way to catch norm failures?
>>
>>
>> Have a look here:
>>
>> http://www.rsyslog.com/using-mongodb-with-rsyslog-and-loganalyzer/
>>
>> $parsesuccess is your friend... and looking at it, it doesn't seem
>> documented. Would you like to document it or add an rsyslog-doc issue
>> tracker?
>
>
> It was documented in the past, but when I tried to use it with mmnormalize,

Unfortunately the change of doc to RST lost quite some content (that's
why I insist so much this doesn't happen again when restructuring).
Should be documented anyhow.

> it didn't work, I had to fall back to looking for unparsed-data existing.

That's definitely a bug. I have co-incidently looked at the code
yesterday, the status is set, so it *should* work. Bug Tracker?

Rainer
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Thu, 24 Nov 2016, Rainer Gerhards wrote:


2016-11-24 17:21 GMT+01:00 mosto...@gmail.com :

# I would consider adding a section here to look for parsing failures and
log them to someplace for later investigation, probably in raw format


That's mandatory. is there any easy way to catch norm failures?


Have a look here:

http://www.rsyslog.com/using-mongodb-with-rsyslog-and-loganalyzer/

$parsesuccess is your friend... and looking at it, it doesn't seem
documented. Would you like to document it or add an rsyslog-doc issue
tracker?


It was documented in the past, but when I tried to use it with mmnormalize, it 
didn't work, I had to fall back to looking for unparsed-data existing.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Thu, 24 Nov 2016, mosto...@gmail.com wrote:

this way you use the tag that you set on the input line to create the 
metadata fields you want with only one ruleset for all the app inputs.

cool!

The other thing you could do is to create a table lookup to map the 
programname (the value you set as the tag in the input) to the values you 
want. This would let you use arbitrary values for tags rather than having 
to have them follow a specific format.
Or even it could help me to link groups and apps, in order not to hardcode 
them.

I'll let that for version 2, if you don't mind :P


if $group == ["group1","group2"] then {
  set $.logline = $programname + " " + $!msg;
  action(name="parseapp" type="mmnormalize" variable="$.logline" 
rulebase=:/path/to/combined/rules.fb")

  call index
  stop
}

I'm not sure I understood this, so let me ask a few questions:
- what you are proposing is mmnormalize to parse "APP MSG", isn't it?


yes, instead of having 500 if clauses, one per app (or one per app/group), this 
would let you just have one, which would be significantly faster.


- I guess you are parsing "any group $programname", cause same $programname 
may share same lognorm format, isn't it?


I'm saying that the one rule file would have rules to handle all message types.

so if you have a log for group1/app1 that contains

this is a messae.

the string sent to mmnormalize would be

group1/app1 this is a message.

so you would have rules like:

rule=tag:group1/app1 %var:word% is a message.

This lets you have completely unambiguous parser rules, no possibility of 
confusing two similar messages that need different rules,


if all app1 messages are the same, you could simplify the rules with

rule=tag:%-:char-to:/%/app1 %var:word% is a message.

or take other shortcuts.

but this is just an administrative option, the performance difference between 
having 500 rules starting with groupX and one rule starting with %-:char-to:/% 
is very small (adding 1100 rules before the first one that matches was a 20% 
difference in speed in Rainer's paper)


- how could I combine multiple rule files without hardcoded includes? I 
guess my only option is a script that merges...


well, if things are generated programatically, a script that merges is a good 
idea. If the include function in liblognorm can't handle a directory or 
wildcard, we should look at getting that added.



I like the way you solved that with variable :D
Pending further testing/playing.

# I would consider adding a section here to look for parsing failures and 
log them to someplace for later investigation, probably in raw format

That's mandatory. is there any easy way to catch norm failures?


look for the variable unparsed-data, if it exists the parse failed.

There was a property documented to contain the success/failure of a parser, but 
that seems to have gone away. It would be handy to have for all mm modules.


This way you can do per-app parser files and include them in your combined 
rulebase file if you want.


This still leaves you listing the groups in the main file. If this is 
enough of a problem, you could create a table that mapped all known 
programnames to 'known' and the test would be something like


if lookup_table("apps","$programname") == "known" then {

(with some other stuff earlier to load the table)
I'll have a look on lookup_tables later, although they seem to be great 
savers!


one of the really neat things about lookup_tables is that you can change them on 
the fly (although, since procesing doesn't stop when you go to load a table, 
some logs will be processed with the old table)


you can have something like

if $msg == " reload lookup table X" then table_load("X","/path/to/file")

as I've said elsewhere, one of the things I had in mind when specifying this was 
to handle the maxmind geoip database, so it is designed to handle very large 
tables that will take a long time (at least as far as rsyslog is concerned) to 
load, and need to be updated wihtout interrupting log processing.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

Once I use it ;)

Yet have to change overview to RST...


El 24/11/16 a las 17:36, Rainer Gerhards escribió:

2016-11-24 17:21 GMT+01:00 mosto...@gmail.com :

# I would consider adding a section here to look for parsing failures and
log them to someplace for later investigation, probably in raw format

That's mandatory. is there any easy way to catch norm failures?

Have a look here:

http://www.rsyslog.com/using-mongodb-with-rsyslog-and-loganalyzer/

$parsesuccess is your friend... and looking at it, it doesn't seem
documented. Would you like to document it or add an rsyslog-doc issue
tracker?

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

2016-11-24 17:21 GMT+01:00 mosto...@gmail.com :
>> # I would consider adding a section here to look for parsing failures and
>> log them to someplace for later investigation, probably in raw format
>
> That's mandatory. is there any easy way to catch norm failures?

Have a look here:

http://www.rsyslog.com/using-mongodb-with-rsyslog-and-loganalyzer/

$parsesuccess is your friend... and looking at it, it doesn't seem
documented. Would you like to document it or add an rsyslog-doc issue
tracker?

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

this way you use the tag that you set on the input line to create the 
metadata fields you want with only one ruleset for all the app inputs.

cool!

The other thing you could do is to create a table lookup to map the 
programname (the value you set as the tag in the input) to the values 
you want. This would let you use arbitrary values for tags rather than 
having to have them follow a specific format.
Or even it could help me to link groups and apps, in order not to 
hardcode them.

I'll let that for version 2, if you don't mind :P


if $group == ["group1","group2"] then {
  set $.logline = $programname + " " + $!msg;
  action(name="parseapp" type="mmnormalize" variable="$.logline" 
rulebase=:/path/to/combined/rules.fb")

  call index
  stop
}

I'm not sure I understood this, so let me ask a few questions:
 - what you are proposing is mmnormalize to parse "APP MSG", isn't it?
 - I guess you are parsing "any group $programname", cause same 
$programname may share same lognorm format, isn't it?
 - how could I combine multiple rule files without hardcoded includes? 
I guess my only option is a script that merges...


I like the way you solved that with variable :D
Pending further testing/playing.

# I would consider adding a section here to look for parsing failures 
and log them to someplace for later investigation, probably in raw format

That's mandatory. is there any easy way to catch norm failures?

This way you can do per-app parser files and include them in your 
combined rulebase file if you want.


This still leaves you listing the groups in the main file. If this is 
enough of a problem, you could create a table that mapped all known 
programnames to 'known' and the test would be something like


if lookup_table("apps","$programname") == "known" then {

(with some other stuff earlier to load the table)
I'll have a look on lookup_tables later, although they seem to be great 
savers!



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Thu, 24 Nov 2016, mosto...@gmail.com wrote:


Last hours have been quite busy!

After reading your comments, here's where we are. Please, advice and comment.

On some remote hosts, logs are readed from files. Each remote will have a 
configuration like:


  # remote.conf
   global(
  MaxMessageSize="32k"
   )
  template(name="json" type="string"
  string="%timestamp:::date-rfc3339% %hostname%
  logs/$!data!group/$!data!app $!data")

  module(load="omrelp")
  ruleset(name="relp"){
  action(
  port="20514"
  target="server"
  template="json"
  type="omrelp"
   )
   }

  ruleset(name="myapp1") {
   set data!app="myapp1"
  $!data!file="$!metadata!filename"
  $!data!group="mygroup"
  $!data!msg="%msg"
  call relp
   }
  input(type="imfile" file="/logs/apps/app1/app1.log"
  tag="mygroup/myapp1" addMetadata="on" ruleset="myapp1"
  startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
  readTimeout="5" PersistStateInterval="1")

  ruleset(name="myapp2") {
   set data!app="myapp2"
  $!data!file="$!metadata!filename"
  $!data!group="mygroup"
  $!data!msg="%msg"
  call relp
   }
  input(type="imfile" file="/logs/apps/app2/app2.log"
  tag="mygroup/myapp2" addMetadata="on" ruleset="myapp2"
  readTimeout="5" PersistStateInterval="1")


I would actually look at doing something like:

input(type="imfile" file="/logs/apps/app1/app1.log"
  tag="mygroup/myapp1" addMetadata="on" ruleset="myapp"
  startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
  readTimeout="5" PersistStateInterval="1")

input(type="imfile" file="/logs/apps/app2/app2.log"
  tag="mygroup/myapp2" addMetadata="on" ruleset="myapp"
  readTimeout="5" PersistStateInterval="1")

ruleset(name="myapp") {
  set data!app=field($programname,"/",2);
  set $!data!file="$!metadata!filename"';
  set $!data!group=field($progranmame,"/",1);;
  set $!data!msg=$msg;
  call relp
}

this way you use the tag that you set on the input line to create the metadata 
fields you want with only one ruleset for all the app inputs.


The other thing you could do is to create a table lookup to map the programname 
(the value you set as the tag in the input) to the values you want. This would 
let you use arbitrary values for tags rather than having to have them follow a 
specific format.


As depicted above, logs will be sent to server using RELP. We plan to make a 
config-generator-script, to add imfile+ruleset for each file.
If you know a better way for having multiple files, each having their own 
information, just let me know.


We also played a bit with mmpstrucdata, but seems json is easier (once you 
need it for elastic)


yeah, rfc5424 structured data was going after a good idea (having structured 
data rather than everything being free-form), but it never took off. there isn't 
a good or standard api for apps to use to create structured data (they would 
have to form the entire syslog line as a string) and json took off as a big 
thing just a year or so later.


All in all it's best to ignore the structured data fields and just use json.

On the server side, we would like to accept and index any groups 
applications, but each app config should be on their own file, making it 
easier to change, understand...
Our current approach is based on copying multiple files to rsyslog.d 
directory, being all configurations loaded. This is what we got so far:


  # core.conf
   global(
   MaxMessageSize="32k"
   parser.escapeControlCharactersOnReceive="off"
   )
   module(load="imrelp")
   input(
   port="20514"
   type="imrelp"
   name="imrelp"
   )

   template(name="json" type="list") {
   constant(value="{")
   property(name="$!msg")
   constant(value="}")
   }
   module(load="omelasticsearch")
   ruleset(name="index"){
   action(
   type="omelasticsearch"
   template="json"
   searchIndex="$!index"
   )
   }
   module(load="mmnormalize")

   # app1.conf
   # Perhaps something like $!group == "mygroup" could work
   if $syslogtag startswith "logs/group/" then {
   rule=:%[
   {"type":"ipv4", "name":"ip"},
   {"type":"literal", "text:" "},
   {"type":"literal", "text:"-"},
   {"type":"rest", "name":"r"}
   ]%
   set index="$!app_$$year-$$month-$$day"
   action(type="mmnormalize" rule="$rule" version="2")
   call index
   stop
   }
   # app2.conf
   if $!app == "myapp2" then {
   rule=:%[
   {"type":"ipv4", "name":"ip"},
   {"type":"literal", "text:" "},
   {"type":"literal", "text:"-"},
   {"type":"rest", "name":"r"}
   ]%
   action(type="mmnormalize" rule="$rule" version="2")
   call index
   stop
   }


*Does it makes sense for you? Any improvements? 

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

Last hours have been quite busy!

After reading your comments, here's where we are. Please, advice and 
comment.


On some remote hosts, logs are readed from files. Each remote will have 
a configuration like:


   # remote.conf
global(
   MaxMessageSize="32k"
)
   template(name="json" type="string"
   string="%timestamp:::date-rfc3339% %hostname%
   logs/$!data!group/$!data!app $!data")

   module(load="omrelp")
   ruleset(name="relp"){
   action(
   port="20514"
   target="server"
   template="json"
   type="omrelp"
)
}

   ruleset(name="myapp1") {
set data!app="myapp1"
   $!data!file="$!metadata!filename"
   $!data!group="mygroup"
   $!data!msg="%msg"
   call relp
}
   input(type="imfile" file="/logs/apps/app1/app1.log"
   tag="mygroup/myapp1" addMetadata="on" ruleset="myapp1"
   startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
   readTimeout="5" PersistStateInterval="1")

   ruleset(name="myapp2") {
set data!app="myapp2"
   $!data!file="$!metadata!filename"
   $!data!group="mygroup"
   $!data!msg="%msg"
   call relp
}
   input(type="imfile" file="/logs/apps/app2/app2.log"
   tag="mygroup/myapp2" addMetadata="on" ruleset="myapp2"
   readTimeout="5" PersistStateInterval="1")



As depicted above, logs will be sent to server using RELP. We plan to 
make a config-generator-script, to add imfile+ruleset for each file.
If you know a better way for having multiple files, each having their 
own information, just let me know.


We also played a bit with mmpstrucdata, but seems json is easier (once 
you need it for elastic)


On the server side, we would like to accept and index any groups 
applications, but each app config should be on their own file, making it 
easier to change, understand...
Our current approach is based on copying multiple files to rsyslog.d 
directory, being all configurations loaded. This is what we got so far:


   # core.conf
global(
MaxMessageSize="32k"
parser.escapeControlCharactersOnReceive="off"
)
module(load="imrelp")
input(
port="20514"
type="imrelp"
name="imrelp"
)

template(name="json" type="list") {
constant(value="{")
property(name="$!msg")
constant(value="}")
}
module(load="omelasticsearch")
ruleset(name="index"){
action(
type="omelasticsearch"
template="json"
searchIndex="$!index"
)
}
module(load="mmnormalize")

# app1.conf
# Perhaps something like $!group == "mygroup" could work
if $syslogtag startswith "logs/group/" then {
rule=:%[
{"type":"ipv4", "name":"ip"},
{"type":"literal", "text:" "},
{"type":"literal", "text:"-"},
{"type":"rest", "name":"r"}
]%
set index="$!app_$$year-$$month-$$day"
action(type="mmnormalize" rule="$rule" version="2")
call index
stop
}
# app2.conf
if $!app == "myapp2" then {
rule=:%[
{"type":"ipv4", "name":"ip"},
{"type":"literal", "text:" "},
{"type":"literal", "text:"-"},
{"type":"rest", "name":"r"}
]%
action(type="mmnormalize" rule="$rule" version="2")
call index
stop
}


*Does it makes sense for you? Any improvements? Anything that can't be 
done?*

@bobthemighty: feedback!

Thanks in advance
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Wed, 23 Nov 2016, mosto...@gmail.com wrote:

My recommended architecture [1] is to have a local relay picking up logs 
from each network/datacenter, add useful metadata (fromhost-ip, what 
environment this is, which relay processed it, timestamp of when the log 
was processed on the relay, etc) and then forward the message to a central 
log system in json format [2].

This is actually what we are doing


good, this makes things much easier

On the central system, I should then have relativly clean data to deal 
with. I receive it and parse the json out.


I then created a template.

t = "$timestamp $hostname $syslogtag $!msg"

note the msg is not $msg (which would be json), but rather $!msg, which is 
the field inside the json that contains the original message.

I was using property(name="msg" position.from="3") to skip first JSON "{"
Another _hidden gem_


if your message is in json, you should parse it in json, not try to skip the 
first characters.



set $.m = exec_template("t")

which creates a variable that contains a line like:

Nov 23 06:19:38 bifrost dhcpd: DHCPREQUEST for 10.2.0.122 from 
00:90:f5:d6:7f:2a via eth2


I didn't played with exec_template, cause I didn't know if that was a "tricky 
approach", recommended or anything else.
Usually whatever_exec is run as command, hence having a serious impact on 
performance.

I'll have a look.


in rsyslog terms, it's not great, but as I've said elsewhere, you can do a lot 
of 'not great' things and still be fast :-)


using templates is 'slow' because the template format is interpreted when you 
use it. You can create a string module (sm*) that does this directly in C code, 
and that makes templates extremely fast. When we first introduces them for the 
standard templates, it caused a 10%+ improvement in the overall speed of 
rsyslog.



prefix=%timestamp:date% %hostname:word% dhcp:
rule=dhcp,foo: DHCPREQUEST for %ip:ipv4" from %mac:word% via 
%interface:word%

what's the foo part?
As someone with regex-way-of-thinking, I would've expected
   rule=%prefix% DHCPREQUEST for %ip:ipv4" from %mac:word% via 
%interface:word%

but it isn't.

This then parses eveything apart, and creates a variable event.tags = 
["dhcp",:"foo"]

I'm lost!


a rule in liblognorm is:

rule=:

most of the example rules have no tags, so you see

rule=: 

when you have one or more tags on a rule, the variable $!event.tags that 
contains an array of the tags that are on that rule. You can have multiple tags 
on one rule


so you could say that for a given log message, you want it to be processed as 
multiple categories (login and ssh for example)





if $programname = "dhcp" then { call dhcprules; stop }

As with _exec, someone told me if are evil (wasn't nginx)

...and that gives you a rocket speed...


prior to rsyslog v8, if statements were significantly slower than other 
conditionals. With the config optimizer in v8, all tests are equally fast.


tests still cost, so if you can replace a bank of

if  then /var/log/fooX

statements with a dynafile statement that used a variable that you have defined 
as part of the path, you will see a huge improvement in speed.


Remember, premature optimization is the root of all evil :-)

Rsyslog is very fast and has been getting faster, start off with your config 
optimized for ease of management and measure the resulting speed. If it's not 
fast enough for your environment, then look at making changes. You will be 
surprised at what you can do without having to worry about perfromance.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

My recommended architecture [1] is to have a local relay picking up 
logs from each network/datacenter, add useful metadata (fromhost-ip, 
what environment this is, which relay processed it, timestamp of when 
the log was processed on the relay, etc) and then forward the message 
to a central log system in json format [2].

This is actually what we are doing



On the central system, I should then have relativly clean data to deal 
with. I receive it and parse the json out.


I then created a template.

t = "$timestamp $hostname $syslogtag $!msg"

note the msg is not $msg (which would be json), but rather $!msg, 
which is the field inside the json that contains the original message.

I was using property(name="msg" position.from="3") to skip first JSON "{"
Another _hidden gem_




set $.m = exec_template("t")

which creates a variable that contains a line like:

Nov 23 06:19:38 bifrost dhcpd: DHCPREQUEST for 10.2.0.122 from 
00:90:f5:d6:7f:2a via eth2


I didn't played with exec_template, cause I didn't know if that was a 
"tricky approach", recommended or anything else.
Usually whatever_exec is run as command, hence having a serious impact 
on performance.

I'll have a look.




prefix=%timestamp:date% %hostname:word% dhcp:
rule=dhcp,foo: DHCPREQUEST for %ip:ipv4" from %mac:word% via 
%interface:word%

what's the foo part?
As someone with regex-way-of-thinking, I would've expected
rule=%prefix% DHCPREQUEST for %ip:ipv4" from %mac:word% via 
%interface:word%

but it isn't.

This then parses eveything apart, and creates a variable event.tags = 
["dhcp",:"foo"]

I'm lost!


if $programname = "dhcp" then { call dhcprules; stop }

As with _exec, someone told me if are evil (wasn't nginx)

...and that gives you a rocket speed...
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Wed, 23 Nov 2016, mosto...@gmail.com wrote:



Considering our imrelp module receives a lot of messages from different 
applications/hosts (with different TAGs), what would be the best way to 
split/forward each message to proper ruleset/parser?


Should I use /_if/_?

Does 
http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html 
have something to "process only specific messages" ?


Quoting https://github.com/rsyslog/rsyslog/issues/625#issuecomment-262286487

If you can combine the programname with the message and include the 
programname in your rules (prefix works wonders here), liblognorm is 
extremely efficient in only using the relevant rules.


I understand it's just having TAG as part of rule, isnt it?


a call to mmnormalize has three options for what to process

1. rawlog (what arrived on the wire)

2. $msg

3. a variable you define.

unfortunantly, parsing on rawlog is full of problems, some things send data in 
rfc3184 format, some send it in rfc5424 format, but the data is otherwise 
identical, and then there are the malformed messages


My recommended architecture [1] is to have a local relay picking up logs from 
each network/datacenter, add useful metadata (fromhost-ip, what environment this 
is, which relay processed it, timestamp of when the log was processed on the 
relay, etc) and then forward the message to a central log system in json format 
[2].


On the central system, I should then have relativly clean data to deal with. I 
receive it and parse the json out.


I then created a template.

t = "$timestamp $hostname $syslogtag $!msg"

note the msg is not $msg (which would be json), but rather $!msg, which is the 
field inside the json that contains the original message.


I then do

set $.m = exec_template("t")

which creates a variable that contains a line like:

Nov 23 06:19:38 bifrost dhcpd: DHCPREQUEST for 10.2.0.122 from 
00:90:f5:d6:7f:2a via eth2

I then have a ruleset that looks something like:

prefix=%timestamp:date% %hostname:word% dhcp:
rule=dhcp,foo: DHCPREQUEST for %ip:ipv4" from %mac:word% via %interface:word%

This then parses eveything apart, and creates a variable event.tags = 
["dhcp",:"foo"]


I then do one of two things.

I can do something like:

if $programname = "dhcp" then { call dhcprules; stop }

which sends the logs to the dhcp rules and then stops processing the message. 
This works well for cases where the programname tells you exactly what you want 
to do with the log.


or I can do a foreach loop on event.tags and have a series of if statemetns 
based on what is in the event.tags array.


This works well for cases where you have a lot of different sources of logs you 
want to treat the same way. A good example of this is if you want to do 
something with all failed logins, and don't care if they are console logins, ssh 
logins, http logins, VPN logins (or even appication logins)


Does this help?

David Lang

[1] 
https://www.usenix.org/publications/login/august-2013-volume-38-number-4/enterprise-logging

[2] This is slightly simplified as it leaves out details of replicating the data 
to a backup datacenter, etc



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

more or less a duplicate answers for those that later find it via search engines

2016-11-23 12:54 GMT+01:00 mosto...@gmail.com :
> Hi
>
>
> Considering our imrelp module receives a lot of messages from different
> applications/hosts (with different TAGs), what would be the best way to
> split/forward each message to proper ruleset/parser?
>
> Should I use /_if/_?

Depends on what you want to do. If processing is actually different:
yes. If you just want different rules for lognorm, this usually
doesn't improve performance.

>
> Does
> http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html
> have something to "process only specific messages" ?

You can filter before calling into it (like "if" above)

>
> Quoting https://github.com/rsyslog/rsyslog/issues/625#issuecomment-262286487
>
>> If you can combine the programname with the message and include the
>> programname in your rules (prefix works wonders here), liblognorm is
>> extremely efficient in only using the relevant rules.
>
> I understand it's just having TAG as part of rule, isnt it?

No, the full message is interpreted. See here

https://www.researchgate.net/publication/310545144_Efficient_Normalization_of_IT_Log_Messages_under_Realtime_Conditions

(sorry it's long, but you are asking very specific questions, and the
specific answer can't be brief, there is a lot of theory involved to
*really* understand and appreciate the system - alternatively, you
simply need to try it out, but it looks like you want to know "the
real thing").

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.