hi Dusan,
you have asked an excellent question. Behavior you are seeing is
actually something expected, since pattern match caching is done after
a successful RegExp pattern match, but *before* the 'context' field of
the rule definition is evaluated. It is also discussed in the
documentation of the varmap statement, and in the man page section
INPUT PROCESSING AND TIMING there is also a relevant rule example. In
the case of the example that you provided, the following will happen
after the synthetic event has been injected:
1) the 'pattern' field of the first rule will match the synthetic
event, since the regular expression (?<EVENT>\S+)
(?<TYPE>Problem|Resolution) produces a match
2) the 'varmap' statement creates a pattern match cache entry MY_EVENT
3) the 'context' field of the rule evaluates false, and therefore the
action "write - R1: Parsing my event" is not executed
4) since the rule 'continue' field is set to TakeNext, the synthetic
event is passed to following rules for further matching, and therefore
another pattern match cache entry SYNTHETIC_EVENT will be created
This behavior of sec is not a bug but intentional, since it allows to
cache successful pattern matches at the earliest possible opportunity.
If you have several rules with identical patterns but different
context expressions, you can cache the match result in the first rule
and reuse it in all following rules.
Fortunately, there is also a fix for your ruleset and it involves the
use of the [ ] operator -- if the context expression is enclosed in
square brackets [ ], then it is evaluated *before* the pattern match.
Therefore, if the context expression evaluates false, the 'pattern'
field of the rule will not be tried, and consequently the 'varmap'
statement will not be triggered. If we modify the first two rules of
your example accordingly:
rem=Rule 1
type=Single
ptype=RegExp
pattern=(?<EVENT>\S+) (?<TYPE>Problem|Resolution)
varmap=MY_EVENT
context=[ !_INTERNAL_EVENT ]
continue=TakeNext
desc=Parse My Event
action=write - R1: Parsing my event
rem=Rule 2
type=Single
ptype=RegExp
pattern=SYNTHETIC (?<EVENT>\S+) (?<TYPE>Problem|Resolution)
varmap=SYNTHETIC_EVENT
context=[ _INTERNAL_EVENT ]
continue=TakeNext
desc=Parse Synthetic Event
action=write - R2: Parsing synthetic event
then the ruleset starts to work in the way we want:
R1: Parsing my event
R4: Injecting synthetic event
R2: Parsing synthetic event
R5: SYNTHETIC Event1 Resolution
I hope my answer was helpful, and thanks for posting such an
interesting question to the mailing list!
kind regards,
risto
2016-12-30 18:20 GMT+02:00 Dusan Sovic <dusan.so...@hotmail.sk>:
> Hello,
> In my SEC rules I using pattern match cache. I would like to know is the
> pattern match cache content after injection of synthetics event. Is there any
> possibility to clear record from pattern match cache on demand?
>
> Consider the following SEC rule config (t.sec) :
> ----------------------------------------------------
> rem=Rule 1
> type=Single
> ptype=RegExp
> pattern=(?<EVENT>\S+) (?<TYPE>Problem|Resolution)
> varmap=MY_EVENT
> context=!_INTERNAL_EVENT
> continue=TakeNext
> desc=Parse My Event
> action=write - R1: Parsing my event
>
> rem=Rule 2
> type=Single
> ptype=RegExp
> pattern=SYNTHETIC (?<EVENT>\S+) (?<TYPE>Problem|Resolution)
> varmap=SYNTHETIC_EVENT
> context=_INTERNAL_EVENT
> continue=TakeNext
> desc=Parse Synthetic Event
> action=write - R2: Parsing synthetic event
>
> rem=Rule 3
> type=Single
> ptype=Cached
> pattern=MY_EVENT
> context=MY_EVENT :> ( sub { return $_[0]->{"TYPE"} eq "Problem"; } )
> desc=Problem_$+{EVENT}
> action=write - R3: Problem: $+{EVENT}
>
> rem=Rule 4
> type=Single
> ptype=Cached
> pattern=MY_EVENT
> context=MY_EVENT :> ( sub { return $_[0]->{"TYPE"} eq "Resolution"; } )
> desc=Resolution_$+{EVENT}
> action=event 0 SYNTHETIC $0; write - R4: Injecting synthetic event
>
> rem=Rule 5
> type=Single
> ptype=Cached
> pattern=SYNTHETIC_EVENT
> context=SYNTHETIC_EVENT :> ( sub { return $_[0]->{"TYPE"} eq "Resolution"; } )
> desc=Resolution_$+{EVENT}
> action=write - R5: $0
>
> Run the sec instance:
> sec -input=- -conf=./t.sec -intevents -intcontexts
>
>
> and put this input event:
> Event1 Resolution
>
> SEC will match:
> * Rule 1
> * Rule 4 -> inject synthetic event
> * Rule 2
> * Rule 4
> * Rule 2
> * Rule 4
> * etc.
>
> I would expect that after synthetic event injection (2nd rule), sec will
> match 5th rule.
> As from doc: "Note that before processing each new input line, previous
> content of the pattern match cache is cleared."
>
> Instead of, it will match 2-4-2-4..etc rules.
> This means, that after first synthetics event injection (4th rule) and match
> by 2nd rule, pattern match cache must contains two match records: "MY_EVENT"
> and "SYNTHETIC_EVENT".
> Therefore rule 5 never match.
>
> One solution what I see is to add additional context test in rules 3-4-5 for
> presence of "_INTERNAL_EVENT" context.
> Question is, if is possible to "somehow" clear the content or given record
> from pattern cache after synthetics event injection to allow rule5 match?
>
> Thanks,
> Dusan
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
