Re: [Simple-evcorr-users] SEC CPU utilization

[Dusan Sovic]

I using similar approach as David mention. I processing syslog messages from 
network devices (various Vendors like Arista, Cisco, Juniper etc.).

We have about 7k patterns defined for match. As SEC cannot handle such regexp 
volume, I using syslog-ng PatternDB for pattern matching, classification and 
tagging.

Than all matched logs are written in normalized format into file what monitored 
by SEC and pre-defined correlation schema is applied. This help me to reduce 
flow into SEC to 30 msg/s while incoming syslog message flow is about ~20k 
msg/s (from about 20k unique network devices). My SEC instance is eating ~ 20 – 
30 % of CPU (with about 500 active event correlation operations with ~ 4k 
active contexts in avg).


Od: David Lang 
Odoslané: štvrtok 26. marca 2020 1:24
Komu: Richard Ostrochovský 
Kópia: simple-evcorr-users@lists.sourceforge.net 

Predmet: Re: [Simple-evcorr-users] SEC CPU utilization

Much of the time you can trivially split your rules and then run multiple copies
of SEC, each processing a subnet of the rules on the logs that will match them.

Rsyslog has a very efficient parsing capability (mmnormalize), you can use it to
just classify the logs, or you can go a step further and have it extract the
relevant fields and pass them to SEC in ways that make it cheaper to parse and
process.

where you do have rules that correlate across different types of logs, you may
be able to just combine those logs into one ruleset, or you can have parallel
instances output messages to say that their part of the correlation has tested
true and then have another instance that processes these partial correlation
messages and decide if the combined correlation is matched.

David Lang


___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] How to introduce new match variable

[Dusan Sovic]

Hi Risto,

Thank you for your explanation. All works well for me now.
I using SEC v 2.7.12 therefore I see that compilation error with lcall and :> 
operator.

Thank you,
Dusan

Od: Risto Vaarandi 
Odoslané: streda 19. februára 2020 14:52
Komu: Dusan Sovic 
Kópia: simple-evcorr-users@lists.sourceforge.net 

Predmet: Re: [Simple-evcorr-users] How to introduce new match variable

hi Dusan,

you can find my comments below:

>
> I try to add new variable using “context” and :> operator also using “lcall” 
> action but no luck.
> Any idea how to achieve this?
>
> This is what I have produced so far:
>
> Config file: dusko.sec
> 
> rem=Rule 1
> type=Single
> ptype=RegExp
> pattern=^(?\S+) (?\S+)$
> varmap=MY_EVENT
> continue=TakeNext
> desc=Parsing Event
> action=write - R1: Parsing event: $+{EVENT} $+{SEVERITY}
>
> rem=Rule 2
> type=Single
> ptype=Cached
> pattern=MY_EVENT
> context=MY_EVENT :> ( sub { return $_[0]->{"NEW"} = "new_entry"; } )
> desc=Introducing new variable
> action=lcall %o MY_EVENT -> ( sub { $_[0]->{"NEW"} = "value" } ); \
> write - R2: NEW = $+{NEW}
>

Rule #2 is not having an expected effect, since SEC rule matching involves 
several steps in the following order:
1) pattern is matched against an incoming event
2) if pattern matched the event, collect match variable values for 
substitutions (e.g., substitutions in 'context' field of the rule)
3) evaluate the context expression of the rule (provided with 'context' field)

If any new match variables are created during step 3, they are not used during 
substitutions within the current rule, since the set of match variables and 
their values were fixed during previous step. However, the match variable would 
be visible in the following rules. In order to make the variable visible 
immediately in the current rule, you can enclose the context expression in 
square brackets [ ], which means that context expression has to be evaluated 
*before* the pattern match (in other words, step 3 would be taken before step 1 
now). For example:

rem=Rule 2
type=Single
ptype=Cached
pattern=MY_EVENT
context=[ MY_EVENT :> ( sub { return $_[0]->{"NEW"} = "new_entry"; } ) ]
desc=Introducing new variable
action=write - R2: NEW = $+{NEW}

The use of [ ] operator involves one caveat -- since match variables (e.g., $1 
or $2) are produced by pattern match, they will not have any values yet when 
context expression is evaluated, and are therefore not substituted. However, 
this is not a problem for the above rule, since the context expression in this 
rule contains no references to match variables (such as $1 or $+{NEW}).

>
> Also if I want to replace “->” with “:>” for lcall action:
> action=lcall %o MY_EVENT :> ( sub { $_[0]->{"NEW"} = "value" } ); \
> write - R2: NEW = $+{NEW}
>
> I got compilation error:
> Rule in ./dusko.sec at line 10: Eval '{"NEW"} = "value" } )' didn't return a 
> code reference: syntax error at (eval 9) line 1, near "} ="
> Unmatched right curly bracket at (eval 9) line 1, at end of line
> Rule in ./dusko.sec at line 10: Invalid action list ' lcall %o MY_EVENT :> ( 
> sub { $_[0]->{"NEW"} = "value" } ); write - R2: NEW = $+{NEW} '

This is because the :> operator for 'lcall' action was introduced in sec-2.8.0, 
and is not supported by previous versions (such as sec-2.7.X). When I tried 
your rule with sec-2.8.2, everything worked fine, but testing it with 
sec-2.7.12 produced the same error message. Therefore I suspect that you have 
an earlier version than 2.8.0, and would recommend to upgrade to 2.8.2 (the 
latest version). But with the above workaround, you would not need 'lcall %o 
MY_EVENT :> ( sub { $_[0]->{"NEW"} = "value" } )' action anyway.

Hope this helps,
risto

>
> Thanks for any help,
> Dusan
>
> ___
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net<mailto:Simple-evcorr-users@lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] How to introduce new match variable

[Dusan Sovic]

Hi SEC users,

I want to create / introduce new match variable in my rules.
I search forum posts and found this:

"Once you have cached match results, they become visible across all rules
and you can modify them. In order to do this, you have to use the :>
context expression operator for getting a reference to the set of cached
match variables. Once you have the reference, you can not only modify
individual variables, but you can also delete existing match variables, and
even introduce new variables (for example, $_[0]->{"newvariable"} = 1 would
set the variable $+{newvariable} to 1)."

I try to add new variable using “context” and :> operator also using “lcall” 
action but no luck.
Any idea how to achieve this?

This is what I have produced so far:

Config file: dusko.sec

rem=Rule 1
type=Single
ptype=RegExp
pattern=^(?\S+) (?\S+)$
varmap=MY_EVENT
continue=TakeNext
desc=Parsing Event
action=write - R1: Parsing event: $+{EVENT} $+{SEVERITY}

rem=Rule 2
type=Single
ptype=Cached
pattern=MY_EVENT
context=MY_EVENT :> ( sub { return $_[0]->{"NEW"} = "new_entry"; } )
desc=Introducing new variable
action=lcall %o MY_EVENT -> ( sub { $_[0]->{"NEW"} = "value" } ); \
write - R2: NEW = $+{NEW}

Star sec
---
sec -input=- -conf=./dusko.sec -intevents -intcontexts --debug=6

Put this input event:
---
Event1 Normal

Result into:

R1: Parsing event: Event1 Normal
R2: NEW =

Also if I want to replace “->” with “:>” for lcall action:
action=lcall %o MY_EVENT :> ( sub { $_[0]->{"NEW"} = "value" } ); \
write - R2: NEW = $+{NEW}

I got compilation error:
Rule in ./dusko.sec at line 10: Eval '{"NEW"} = "value" } )' didn't return a 
code reference: syntax error at (eval 9) line 1, near "} ="
Unmatched right curly bracket at (eval 9) line 1, at end of line
Rule in ./dusko.sec at line 10: Invalid action list ' lcall %o MY_EVENT :> ( 
sub { $_[0]->{"NEW"} = "value" } ); write - R2: NEW = $+{NEW} '

Thanks for any help,
Dusan

___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] SingleWithThreshold reference current input line

[Dusan Sovic]

Hi Risto,

Thank you.
EventGroup rule working well in my use case.

Regards,
Dusan

Od: Risto Vaarandi 
Odoslané: štvrtok, 8. novembra 2018 21:15
Komu: dusan.so...@hotmail.sk
Kópia: simple-evcorr-users@lists.sourceforge.net
Predmet: Re: [Simple-evcorr-users] SingleWithThreshold reference current input 
line

hi Dusan,

the problem lies in the fact that when SingleWithThreshold rule starts a 
counting operation, match variables in the 'action' field receive their values 
from the first event which triggered that operation (that is done for staying 
consistent with substitution of variables in other fields, where values from 
first event have to be used). In order to solve this issue, the best solution 
is to employ EventGroup rule instead of SingleWithThreshold, since EventGroup 
is a more general counting rule that supports a number of useful extensions.

One such extension is support for the 'count' field which allows for executing 
action(s) on each matching event. Unlike 'action' field, match variables in 
'count' field are set from *each* matching event. For example, consider the 
following rule:

type=EventGroup
ptype=RegExp
pattern=.
desc=count any event
count=assign %lastline $0
action=write - %lastline
thresh=3
window=60

After each matching event, action list variable %lastline is set to the current 
event, and when the third matching event is observed in 60 second time window, 
this event is written to standard output. Since unlike match variables in 
'action' field, action list variables like %lastline are always substituted  at 
action list execution, %lastline will hold the value of last matching line.

For employing this technique for your ruleset, EventGroup rule could be used in 
the following fashion:

rem=Parse My Event
type=Single
ptype=RegExp
pattern=^\S+ (?\S+)
varmap=MY_EVENT
continue=TakeNext
desc=Parse Event
action=none

rem=Rule1
type=EventGroup
ptype=Cached
pattern=MY_EVENT
desc=Rule1 $+{EVENT}
count=assign %lastline $0
action=write - %lastline
window=60
thresh=2

When submitting three example events to this ruleset, the following output 
should be displayed:

Assigning '2018-11-11T00:00:01+00:00 Event1' to variable '%lastline'
Assigning '2018-11-11T00:00:02+00:00 Event1' to variable '%lastline'
Writing event '2018-11-11T00:00:02+00:00 Event1' to file '-'
2018-11-11T00:00:02+00:00 Event1 <--- second event that was written to 
standard output
Assigning '2018-11-11T00:00:03+00:00 Event1' to variable '%lastline'

Hope this helps,
risto


Kontakt Dusan Sovic (mailto:dusan.so...@hotmail.sk>>) 
kirjutas kuupäeval N, 8. november 2018 kell 16:11:
Hello SEC Users,

I using SingleWithSuppress rule to process timestamped input events. I want to 
take action after 2nd event occurrence within 60 seconds.
Problem what I have is that after second event match, action is taken and event 
($0) is written to the output but it use timestamp of first received event 
(that one what started correlation operation).
On the output I would like to see the *timestamp* of the second event or more 
general whole input message of second event as is.

Let me demonstrate this on example:

Config File: ccr.sec

rem=Parse My Event
type=Single
ptype=RegExp
pattern=^\S+ (?\S+)
varmap=MY_EVENT
continue=TakeNext
desc=Parse Event
action=none

rem=Rule1
type=SingleWithThreshold
ptype=Cached
pattern=MY_EVENT
desc=Rule1 $+{EVENT}
action=write - $0
window=60
thresh=2

Run sec: sec -conf=./ccr.sec -input=-

Input following line:
2018-11-11T00:00:01+00:00 Event1
2018-11-11T00:00:02+00:00 Event1
2018-11-11T00:00:03+00:00 Event1

Output action:
Writing event '2018-11-11T00:00:01+00:00 Event1' to file '-'

What I want to achieve / see:
Writing event '2018-11-11T00:00:02+00:00 Event1' to file '-'

Thanks,
Dusan



___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net<mailto:Simple-evcorr-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] SingleWithThreshold reference current input line

[Dusan Sovic]

Hello SEC Users,

I using SingleWithSuppress rule to process timestamped input events. I want to 
take action after 2nd event occurrence within 60 seconds.
Problem what I have is that after second event match, action is taken and event 
($0) is written to the output but it use timestamp of first received event 
(that one what started correlation operation).
On the output I would like to see the *timestamp* of the second event or more 
general whole input message of second event as is.

Let me demonstrate this on example:

Config File: ccr.sec

rem=Parse My Event
type=Single
ptype=RegExp
pattern=^\S+ (?\S+)
varmap=MY_EVENT
continue=TakeNext
desc=Parse Event
action=none

rem=Rule1
type=SingleWithThreshold
ptype=Cached
pattern=MY_EVENT
desc=Rule1 $+{EVENT}
action=write - $0
window=60
thresh=2

Run sec: sec -conf=./ccr.sec -input=-

Input following line:
2018-11-11T00:00:01+00:00 Event1
2018-11-11T00:00:02+00:00 Event1
2018-11-11T00:00:03+00:00 Event1

Output action:
Writing event '2018-11-11T00:00:01+00:00 Event1' to file '-'

What I want to achieve / see:
Writing event '2018-11-11T00:00:02+00:00 Event1' to file '-'

Thanks,
Dusan



___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] Suppress rule and continue filed support

[Dusan Sovic]

Hi Risto,

Thank you for documentation update. Look good.

Dusan

Od: Risto Vaarandi 
Odoslané: nedeľa, 14. októbra 2018 13:22
Komu: dusan.so...@hotmail.sk
Kópia: simple-evcorr-users@lists.sourceforge.net
Predmet: Re: [Simple-evcorr-users] Suppress rule and continue filed support




I think it will be beneficial to highlight this fact in Rule Types -> Suppress 
paragraph that other “techniques” have to be used to achieve suppression across 
multiple configuration files.

that's a good idea -- I will modify the documentation accordingly.
risto


I like idea to add this as separate FAQ entry.




Documentation updates have been committed to github, and the changes are now 
visible in online documentation:
http://simple-evcorr.github.io/man.html

In addition, FAQ has been updated with a new entry:
http://simple-evcorr.github.io/FAQ.html#25

kind regards,
risto



Thank you,

Dusan


Od: Risto Vaarandi mailto:risto.vaara...@gmail.com>>
Odoslané: piatok, 12. októbra 2018 11:25
Komu: dusan.so...@hotmail.sk
Kópia: 
simple-evcorr-users@lists.sourceforge.net
Predmet: Re: [Simple-evcorr-users] Suppress rule and continue filed support


Hi Risto,

Thank you very much for your suggestions and explanation.
Must agree with your arguments that introducing "continue" field option support 
into "Suppress" rule will introduce some confusions.
I have never thought that this type of suppression logic can be achieved by 
"Jump" rule. Thanks for bringing it up.

hi Dusan,

it is my bad that I forgot to mention this opportunity in the man page, but 
I'll incorporate it in the documentation for the next version. Also, this topic 
would actually make a good FAQ entry.


Thanks for this great piece of software and I really appreciate your support 
and help.

I am happy that you like sec and have found it useful :-)

kind regards,
risto


Dusan


Od: Risto Vaarandi mailto:risto.vaara...@gmail.com>>
Odoslané: štvrtok, 11. októbra 2018 22:47
Komu: dusan.so...@hotmail.sk
Kópia: 
simple-evcorr-users@lists.sourceforge.net
Predmet: Re: [Simple-evcorr-users] Suppress rule and continue filed support



Hello SEC Users,



hi Dusan,


Base on SEC documentation *Suppress* rules doesn’t support “continue” field 
like other rules.

My understanding is that if suppress rule match event the search for matching 
rules ends in the *current* configuration file.

That's correct, the Suppress rule works indeed within the scope of the current 
rule file only.




Let’s consider this simple example with two config files:



Config file: 01.sec



type=Suppress

ptype=RegExp

pattern=foo

desc=$0



Config file: 02.sec



type=Single

ptype=RegExp

pattern=foo

continue=EndMatch

desc=$0

action=write - foo matched



If you launch sec:

sec -conf=./*.sec -input=-



And put “foo” in the input:

*  In first configuration file suppression rule match "foo" and switch to next 
configuration file.

*  In second configuration file Single rule match "foo" and action is taken.



I think it can be beneficial if "Suppress" rule will support *continue* filed 
so I can tell that I don't want to continue to find match in *all* next 
configuration files.

That approach has one downside -- it would introduce confusion among users, 
since some values for "continue" like "TakeNext" and "GoTo" imply that the 
matching process will not stop but rather proceed from another rule. In sec 
code, it is of course possible to either ignore such values or report an error, 
but from the design perspective it isn't the best solution. I would rather 
favor the creation of a separate rule type (e.g., Stop instead of Suppress) -- 
but if you consider already existing rule types, the Jump rule is actually 
meeting your needs.

The whole purpose of this rule is to actually control the rule processing flow, 
and you can utilize it for continuing processing in another rule file. When you 
omit the "cfset" field from Jump, the Jump rule can also be used for 
controlling the rule processing order within a single rule file, provided that 
the "continue" field has been set to "GoTo". Also, without "cfset" field and 
"continue" omitted (or set to "DontCont"), Jump is identical to Suppress. Both 
of those special cases have been described in the documentation for the Jump 
rule.

However, if you omit "cfset" and set "continue" to "EndMatch", the Jump rule 
will do exactly what you want. For example, the following rule will end 
processing for all events which contain the substring "test":

type=Jump
ptype=substr
pattern=test
continue=endmatch

I acknowledge that the documentation should also explicitly describe this 
particular case, in order to make it obvious to the reader.




I know that I can achieve this by replacing "Suppress" rule 

Re: [Simple-evcorr-users] Suppress rule and continue filed support

[Dusan Sovic]

Hi Risto,



I think it will be beneficial to highlight this fact in Rule Types -> Suppress 
paragraph that other “techniques” have to be used to achieve suppression across 
multiple configuration files.

I like idea to add this as separate FAQ entry.



Thank you,

Dusan


Od: Risto Vaarandi 
Odoslané: piatok, 12. októbra 2018 11:25
Komu: dusan.so...@hotmail.sk
Kópia: simple-evcorr-users@lists.sourceforge.net
Predmet: Re: [Simple-evcorr-users] Suppress rule and continue filed support


Hi Risto,

Thank you very much for your suggestions and explanation.
Must agree with your arguments that introducing "continue" field option support 
into "Suppress" rule will introduce some confusions.
I have never thought that this type of suppression logic can be achieved by 
"Jump" rule. Thanks for bringing it up.

hi Dusan,

it is my bad that I forgot to mention this opportunity in the man page, but 
I'll incorporate it in the documentation for the next version. Also, this topic 
would actually make a good FAQ entry.


Thanks for this great piece of software and I really appreciate your support 
and help.

I am happy that you like sec and have found it useful :-)

kind regards,
risto


Dusan


Od: Risto Vaarandi mailto:risto.vaara...@gmail.com>>
Odoslané: štvrtok, 11. októbra 2018 22:47
Komu: dusan.so...@hotmail.sk
Kópia: 
simple-evcorr-users@lists.sourceforge.net
Predmet: Re: [Simple-evcorr-users] Suppress rule and continue filed support



Hello SEC Users,



hi Dusan,


Base on SEC documentation *Suppress* rules doesn’t support “continue” field 
like other rules.

My understanding is that if suppress rule match event the search for matching 
rules ends in the *current* configuration file.

That's correct, the Suppress rule works indeed within the scope of the current 
rule file only.




Let’s consider this simple example with two config files:



Config file: 01.sec



type=Suppress

ptype=RegExp

pattern=foo

desc=$0



Config file: 02.sec



type=Single

ptype=RegExp

pattern=foo

continue=EndMatch

desc=$0

action=write - foo matched



If you launch sec:

sec -conf=./*.sec -input=-



And put “foo” in the input:

*  In first configuration file suppression rule match "foo" and switch to next 
configuration file.

*  In second configuration file Single rule match "foo" and action is taken.



I think it can be beneficial if "Suppress" rule will support *continue* filed 
so I can tell that I don't want to continue to find match in *all* next 
configuration files.

That approach has one downside -- it would introduce confusion among users, 
since some values for "continue" like "TakeNext" and "GoTo" imply that the 
matching process will not stop but rather proceed from another rule. In sec 
code, it is of course possible to either ignore such values or report an error, 
but from the design perspective it isn't the best solution. I would rather 
favor the creation of a separate rule type (e.g., Stop instead of Suppress) -- 
but if you consider already existing rule types, the Jump rule is actually 
meeting your needs.

The whole purpose of this rule is to actually control the rule processing flow, 
and you can utilize it for continuing processing in another rule file. When you 
omit the "cfset" field from Jump, the Jump rule can also be used for 
controlling the rule processing order within a single rule file, provided that 
the "continue" field has been set to "GoTo". Also, without "cfset" field and 
"continue" omitted (or set to "DontCont"), Jump is identical to Suppress. Both 
of those special cases have been described in the documentation for the Jump 
rule.

However, if you omit "cfset" and set "continue" to "EndMatch", the Jump rule 
will do exactly what you want. For example, the following rule will end 
processing for all events which contain the substring "test":

type=Jump
ptype=substr
pattern=test
continue=endmatch

I acknowledge that the documentation should also explicitly describe this 
particular case, in order to make it obvious to the reader.




I know that I can achieve this by replacing "Suppress" rule in 01.sec 
configuration file by "Single" rule and do not take any action and define 
continue=EndMatch.

One of the drawbacks of this approach is the need to set the "action" field to 
"none", while it would be more convenient not to define that field at all. 
However, with the Jump rule you would not have this issue. Also, since Jump is 
specifically designed for exercising control over rule processing (and Suppress 
rule is actually a special case of Jump),  the use of Jump would be quite 
appropriate here.

hope that helps,
risto




Thanks,
Dusan

___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net

Re: [Simple-evcorr-users] Suppress rule and continue filed support

[Dusan Sovic]

Hi Risto,

Thank you very much for your suggestions and explanation.
Must agree with your arguments that introducing "continue" field option support 
into "Suppress" rule will introduce some confusions.
I have never thought that this type of suppression logic can be achieved by 
"Jump" rule. Thanks for bringing it up.

Thanks for this great piece of software and I really appreciate your support 
and help.

Dusan


Od: Risto Vaarandi 
Odoslané: štvrtok, 11. októbra 2018 22:47
Komu: dusan.so...@hotmail.sk
Kópia: simple-evcorr-users@lists.sourceforge.net
Predmet: Re: [Simple-evcorr-users] Suppress rule and continue filed support



Hello SEC Users,



hi Dusan,


Base on SEC documentation *Suppress* rules doesn’t support “continue” field 
like other rules.

My understanding is that if suppress rule match event the search for matching 
rules ends in the *current* configuration file.

That's correct, the Suppress rule works indeed within the scope of the current 
rule file only.




Let’s consider this simple example with two config files:



Config file: 01.sec



type=Suppress

ptype=RegExp

pattern=foo

desc=$0



Config file: 02.sec



type=Single

ptype=RegExp

pattern=foo

continue=EndMatch

desc=$0

action=write - foo matched



If you launch sec:

sec -conf=./*.sec -input=-



And put “foo” in the input:

*  In first configuration file suppression rule match "foo" and switch to next 
configuration file.

*  In second configuration file Single rule match "foo" and action is taken.



I think it can be beneficial if "Suppress" rule will support *continue* filed 
so I can tell that I don't want to continue to find match in *all* next 
configuration files.

That approach has one downside -- it would introduce confusion among users, 
since some values for "continue" like "TakeNext" and "GoTo" imply that the 
matching process will not stop but rather proceed from another rule. In sec 
code, it is of course possible to either ignore such values or report an error, 
but from the design perspective it isn't the best solution. I would rather 
favor the creation of a separate rule type (e.g., Stop instead of Suppress) -- 
but if you consider already existing rule types, the Jump rule is actually 
meeting your needs.

The whole purpose of this rule is to actually control the rule processing flow, 
and you can utilize it for continuing processing in another rule file. When you 
omit the "cfset" field from Jump, the Jump rule can also be used for 
controlling the rule processing order within a single rule file, provided that 
the "continue" field has been set to "GoTo". Also, without "cfset" field and 
"continue" omitted (or set to "DontCont"), Jump is identical to Suppress. Both 
of those special cases have been described in the documentation for the Jump 
rule.

However, if you omit "cfset" and set "continue" to "EndMatch", the Jump rule 
will do exactly what you want. For example, the following rule will end 
processing for all events which contain the substring "test":

type=Jump
ptype=substr
pattern=test
continue=endmatch

I acknowledge that the documentation should also explicitly describe this 
particular case, in order to make it obvious to the reader.




I know that I can achieve this by replacing "Suppress" rule in 01.sec 
configuration file by "Single" rule and do not take any action and define 
continue=EndMatch.

One of the drawbacks of this approach is the need to set the "action" field to 
"none", while it would be more convenient not to define that field at all. 
However, with the Jump rule you would not have this issue. Also, since Jump is 
specifically designed for exercising control over rule processing (and Suppress 
rule is actually a special case of Jump),  the use of Jump would be quite 
appropriate here.

hope that helps,
risto




Thanks,
Dusan

___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] Suppress rule and continue filed support

[Dusan Sovic]

Hello SEC Users,



Base on SEC documentation *Suppress* rules doesn’t support “continue” field 
like other rules.

My understanding is that if suppress rule match event the search for matching 
rules ends in the *current* configuration file.



Let’s consider this simple example with two config files:



Config file: 01.sec



type=Suppress

ptype=RegExp

pattern=foo

desc=$0



Config file: 02.sec



type=Single

ptype=RegExp

pattern=foo

continue=EndMatch

desc=$0

action=write - foo matched



If you launch sec:

sec -conf=./*.sec -input=-



And put “foo” in the input:

*  In first configuration file suppression rule match "foo" and switch to next 
configuration file.

*  In second configuration file Single rule match "foo" and action is taken.



I think it can be beneficial if "Suppress" rule will support *continue* filed 
so I can tell that I don't want to continue to find match in *all* next 
configuration files.



I know that I can achieve this by replacing "Suppress" rule in 01.sec 
configuration file by "Single" rule and do not take any action and define 
continue=EndMatch.



Thanks,
Dusan

___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] Regexp matching against context names

[Dusan Sovic]

Hi Risto,


This helps me a lot!

Thank you very much for your help.


Dusan


Od: Risto Vaarandi <risto.vaara...@gmail.com>
Odoslané: 20. apríla 2017 15:52:56
Komu: Dusan Sovic
Kópia: simple-evcorr-users@lists.sourceforge.net
Predmet: Re: [Simple-evcorr-users] Regexp matching against context names

hi Dusan,

the post you are referring to originates from 2012 when the most
recent sec version was 2.6.2. This version didn't indeed have support
for looping in action lists. While 2.7.X versions do not have a
specific action for regular expression based filtering of context
names, it can be done in a straightforward way with 'lcall' action.
For example:

lcall %names -> ( sub { grep(/^MY/, keys %main::context_list) } )

The above action takes advantage of the fact that context names act as
keys in the %main::context_list hash, and uses a simple grep(/regex/,
...) filter for extracting all relevant names. I would argue that
using 'lcall' instead of a 'findcont' action allows for more generic
filtering, since you can define more complex conditions inside a Perl
code snippet that just one regular expression.

Also, here is an example how to use the above action for actual deletion:

type=single
ptype=regexp
pattern=create (\S+)
desc=set up context $1
action=create $1

type=single
ptype=substr
pattern=delete
desc=find context names
action=lcall %names -> ( sub { grep(/^MY/, keys %main::context_list) } ); \
   fill BUFFER %names; getsize %n BUFFER; \
   while %n ( shift BUFFER %context; delete %context; getsize %n BUFFER)


If you start up sec with this rule and feed in events

create AAA
create BBB
create CCC
create MYCONT1
create MYCONT2

then five contexts AAA, BBB, CCC, MYCONT1 and MYCONT2 are created by
the first rule. If you feed a line to sec that contains the string
"delete", the second rule uses regular expression ^MY to find out
contexts with prefix "MY" and deletes them. In the case of above five
contexts, MYCONT1 and MYCONT2 get deleted.

In order to accomplish deletion in the while-loop, context names are
pushed into the event store of the context called BUFFER, and the body
of the loop gets executed until all context names have been consumed
for BUFFER. During each iteration, the first name from the event store
of BUFFER is taken away with the 'shift' action, deleting the context
with that name. After that, the number of remaining names is
established with the 'getsize' action, and if getsize returns 0, the
loop terminates.

Hope this helps,
risto



2017-04-20 12:51 GMT+03:00 Dusan Sovic <dusan.so...@hotmail.sk>:
> Dear mailing list users,
>
> I would like to know if there is a possibility or plan to support regular 
> expression matching against context names.
> In my case sometimes it can be useful to have option to delete all contexts 
> matching given regexp pattern like 'delete device01.com.*'.
> Contexts what I creating contains device FQDN and additional information so 
> for one device I can have one or more contexts created 1:N.
>
> Example:
> Context Name: 
> device01.com|H3C-LAGG-LAGG_INACTIVE_PARTNER|Ten-GigabitEthernet1/0/7-BAGG57
> Context Name: 
> device01.com|H3C-LAGG-LAGG_INACTIVE_PARTNER|Ten-GigabitEthernet1/0/8-BAGG58
>
> In the past this topic was discussed in this post: 
> https://sourceforge.net/p/simple-evcorr/mailman/message/30011331/ and Risto 
> proposed to create "action 'findcont %o regexp', that would allow for 
> creating repeated actions over data in variable %o."
>
> Thanks and Best Regards,
> Dusan
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] Test IF correlation operation exist then take action

[Dusan Sovic]

Hi Risto,


This is exactly what I have looking for!

Thank you,
Dusan


Od: Risto Vaarandi <risto.vaara...@gmail.com>
Odoslané: 15. januára 2017 20:54
Komu: Dusan Sovic
Kópia: simple-evcorr-users@lists.sourceforge.net
Predmet: Re: [Simple-evcorr-users] Test IF correlation operation exist then 
take action

hi Dusan,

the use of 'getwpos' is probably the best way to accomplish this task.
As an alternative, one could check sec internal data structures, but
it is more complex and makes the rules less readable. Since 'getwpos'
assigns the beginning of the event correlation window (as seconds
since epoch) to a variable, and does not alter the variable if the
operation does not exist, the presence of operations could be checked
in the following way:

type=SingleWithSuppress
ptype=RegExp
pattern=event: (\d+)
desc=suppress event $1
action=write - %s
window=30

type=Single
ptype=RegExp
pattern=check: (\d+)
desc=check if operation for event $1 exists
action=assign %exists 0; getwpos %exists -1 suppress event $1; \
   if %exists ( write - Operation for event $1 exists ) \
 else ( write - Operation for event $1 does not exist )

In this very simple example, the second rule verifies if a
SingleWithSuppress operation is currently running for a given event
number. If the operation does not exist, %exists variable will keep
its initial value 0, and thus the 'if-else' action will print the
right status string to standard output.

Hope this helps,
risto

2017-01-15 18:32 GMT+02:00 Dusan Sovic <dusan.so...@hotmail.sk>:
> Dear mailing list users,
>
> In one of my rule I need to conditionally take action if given correlation 
> operation exist. From SEC man page, I can see that under rule *action* I can 
> use actions 'reset', 'getwpos' and 'setwpos' to work with correlation 
> operation(s).
> I learn how to use 'reset' action and it's working well for me.
> In one of my rule I need to call two actions if given operation exist and do 
> nothing if operation doesn't exist.
>
> The "pseudo code" may looks like:
>
> action= if( exist([] []) ) { write - some_output_$0; reset 
> [] []; }
>
> One theoretical solution what I see is to use 'getwpos' action and use return 
> value % for test in next *if* statement, but maybe there is a "smarter" 
> way how to test for the presence / existence of correlation operation.
>
> Thank you,
> Dusan
>
> --
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
> ___
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
--
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] Test IF correlation operation exist then take action

[Dusan Sovic]

Dear mailing list users,

In one of my rule I need to conditionally take action if given correlation 
operation exist. From SEC man page, I can see that under rule *action* I can 
use actions ‘reset’, ‘getwpos’ and ‘setwpos’ to work with correlation 
operation(s). 
I learn how to use ‘reset’ action and it’s working well for me.
In one of my rule I need to call two actions if given operation exist and do 
nothing if operation doesn’t exist.

The "pseudo code" may looks like:

action= if( exist([] []) ) { write - some_output_$0; reset 
[] []; }

One theoretical solution what I see is to use 'getwpos' action and use return 
value % for test in next *if* statement, but maybe there is a "smarter" 
way how to test for the presence / existence of correlation operation.

Thank you,
Dusan

--
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] Fw: Content of pattern match cache after synthetic event injection

[Dusan Sovic]

Hi Risto,


Thank you very much for your answer!

The use of  the [ ] operator is the key and make my complex rule-set to work in 
way I want.


Thanks & Best Regards,

Dusan


Od: Risto Vaarandi <risto.vaara...@gmail.com>
Odoslané: 31. decembra 2016 10:51
Komu: Dusan Sovic
Kópia: simple-evcorr-users@lists.sourceforge.net
Predmet: Re: [Simple-evcorr-users] Content of pattern match cache after 
synthetic event injection

hi Dusan,

you have asked an excellent question. Behavior you are seeing is
actually something expected, since pattern match caching is done after
a successful RegExp pattern match, but *before* the 'context' field of
the rule definition is evaluated. It is also discussed in the
documentation of the varmap statement, and in the man page section
INPUT PROCESSING AND TIMING there is also a relevant rule example. In
the case of the example that you provided, the following will happen
after the synthetic event has been injected:

1) the 'pattern' field of the first rule will match the synthetic
event, since the regular expression (?\S+)
(?Problem|Resolution) produces a match
2) the 'varmap' statement creates a pattern match cache entry MY_EVENT
3) the 'context' field of the rule evaluates false, and therefore the
action "write - R1: Parsing my event" is not executed
4) since the rule 'continue' field is set to TakeNext, the synthetic
event is passed to following rules for further matching, and therefore
another pattern match cache entry SYNTHETIC_EVENT will be created

This behavior of sec is not a bug but intentional, since it allows to
cache successful pattern matches at the earliest possible opportunity.
If you have several rules with identical patterns but different
context expressions, you can cache the match result in the first rule
and reuse it in all following rules.

Fortunately, there is also a fix for your ruleset and it involves the
use of the [ ] operator -- if the context expression is enclosed in
square brackets [ ], then it is evaluated *before* the pattern match.
Therefore, if the context expression evaluates false, the 'pattern'
field of the rule will not be tried, and consequently the 'varmap'
statement will not be triggered. If we modify the first two rules of
your example accordingly:


rem=Rule 1
type=Single
ptype=RegExp
pattern=(?\S+) (?Problem|Resolution)
varmap=MY_EVENT
context=[ !_INTERNAL_EVENT ]
continue=TakeNext
desc=Parse My Event
action=write - R1: Parsing my event

rem=Rule 2
type=Single
ptype=RegExp
pattern=SYNTHETIC (?\S+) (?Problem|Resolution)
varmap=SYNTHETIC_EVENT
context=[ _INTERNAL_EVENT ]
continue=TakeNext
desc=Parse Synthetic Event
action=write - R2: Parsing synthetic event


then the ruleset starts to work in the way we want:

R1: Parsing my event
R4: Injecting synthetic event
R2: Parsing synthetic event
R5: SYNTHETIC Event1 Resolution

I hope my answer was helpful, and thanks for posting such an
interesting question to the mailing list!

kind regards,
risto

2016-12-30 18:20 GMT+02:00 Dusan Sovic <dusan.so...@hotmail.sk>:
> Hello,
> In my SEC rules I using pattern match cache. I would like to know is the 
> pattern match cache content after injection of synthetics event. Is there any 
> possibility to clear record from pattern match cache on demand?
>
> Consider the following SEC rule config (t.sec) :
> 
> rem=Rule 1
> type=Single
> ptype=RegExp
> pattern=(?\S+) (?Problem|Resolution)
> varmap=MY_EVENT
> context=!_INTERNAL_EVENT
> continue=TakeNext
> desc=Parse My Event
> action=write - R1: Parsing my event
>
> rem=Rule 2
> type=Single
> ptype=RegExp
> pattern=SYNTHETIC (?\S+) (?Problem|Resolution)
> varmap=SYNTHETIC_EVENT
> context=_INTERNAL_EVENT
> continue=TakeNext
> desc=Parse Synthetic Event
> action=write - R2: Parsing synthetic event
>
> rem=Rule 3
> type=Single
> ptype=Cached
> pattern=MY_EVENT
> context=MY_EVENT :> ( sub { return $_[0]->{"TYPE"} eq "Problem"; } )
> desc=Problem_$+{EVENT}
> action=write - R3: Problem: $+{EVENT}
>
> rem=Rule 4
> type=Single
> ptype=Cached
> pattern=MY_EVENT
> context=MY_EVENT :> ( sub { return $_[0]->{"TYPE"} eq "Resolution"; } )
> desc=Resolution_$+{EVENT}
> action=event 0 SYNTHETIC $0; write - R4: Injecting synthetic event
>
> rem=Rule 5
> type=Single
> ptype=Cached
> pattern=SYNTHETIC_EVENT
> context=SYNTHETIC_EVENT :> ( sub { return $_[0]->{"TYPE"} eq "Resolution"; } )
> desc=Resolution_$+{EVENT}
> action=write - R5: $0
>
> Run the sec instance:
> sec -input=- -conf=./t.sec -intevents -intcontexts
>
>
> and put this input event:
> Event1 Resolution
>
> SEC will match:
> *  Rule 1
> *  Rule 4 -> inject synthetic event
> *  Rule 2
> *  R

[Simple-evcorr-users] Content of pattern match cache after synthetic event injection

[Dusan Sovic]

Hello,
In my SEC rules I using pattern match cache. I would like to know is the 
pattern match cache content after injection of synthetics event. Is there any 
possibility to clear record from pattern match cache on demand?

Consider the following SEC rule config (t.sec) :

rem=Rule 1
type=Single
ptype=RegExp
pattern=(?\S+) (?Problem|Resolution)
varmap=MY_EVENT
context=!_INTERNAL_EVENT
continue=TakeNext
desc=Parse My Event
action=write - R1: Parsing my event

rem=Rule 2
type=Single
ptype=RegExp
pattern=SYNTHETIC (?\S+) (?Problem|Resolution)
varmap=SYNTHETIC_EVENT
context=_INTERNAL_EVENT
continue=TakeNext
desc=Parse Synthetic Event
action=write - R2: Parsing synthetic event

rem=Rule 3
type=Single
ptype=Cached
pattern=MY_EVENT
context=MY_EVENT :> ( sub { return $_[0]->{"TYPE"} eq "Problem"; } )
desc=Problem_$+{EVENT}
action=write - R3: Problem: $+{EVENT} 

rem=Rule 4
type=Single
ptype=Cached
pattern=MY_EVENT
context=MY_EVENT :> ( sub { return $_[0]->{"TYPE"} eq "Resolution"; } )
desc=Resolution_$+{EVENT}
action=event 0 SYNTHETIC $0; write - R4: Injecting synthetic event

rem=Rule 5
type=Single
ptype=Cached
pattern=SYNTHETIC_EVENT
context=SYNTHETIC_EVENT :> ( sub { return $_[0]->{"TYPE"} eq "Resolution"; } )
desc=Resolution_$+{EVENT}
action=write - R5: $0

Run the sec instance:
sec -input=- -conf=./t.sec -intevents -intcontexts


and put this input event:
Event1 Resolution

SEC will match:
*  Rule 1
*  Rule 4 -> inject synthetic event
*  Rule 2
*  Rule 4
*  Rule 2
*  Rule 4
*  etc.

I would expect that after synthetic event injection (2nd rule), sec will match 
5th rule. 
As from doc: "Note that before processing each new input line, previous content 
of the pattern match cache is cleared."

Instead of, it will match 2-4-2-4..etc rules.
This means, that after first synthetics event injection (4th rule) and match by 
2nd rule, pattern match cache must contains two match records: "MY_EVENT" and 
"SYNTHETIC_EVENT".
Therefore rule 5 never match.

One solution what I see is to add additional context test in rules  3-4-5 for 
presence of "_INTERNAL_EVENT" context.
Question is, if is possible to "somehow" clear the content or given record from 
pattern cache after synthetics event injection to allow rule5 match?

Thanks,
Dusan

--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users