Re: [Simple-evcorr-users] Way to corelate 2 rules with treshold
hi Przemysław,
I have provided my answers inline:
Then i tried this to incorporate it into Your proposal as a separate rule
> (inserted it right after the first rule).
> # create context for changed messadeid preserving login
> type=single
> ptype=regexp
> pattern=postfix\/smtp.*\]: (?\S+):.*relay=127.0.0.1.* queued as
> (?\S+)\)
> context=MSGID_$+{MSGID}
> desc=set up a context for message ID $+{MSGID2}
> action=copy MSGID_$+{MSGID} %user;create MSGID_$+{MSGID2} 3600; fill
> MSGID_$+{MSGID2} %{user}
>
> This seems to copy user data from old msgid to new msgid and make even
> multiple filter hops work ok.
>
> My question are:
> Is this optimal performance wise?
>
yes, it is probably the best way for addressing the problem. Contexts are
internally implemented as Perl hashes, and above three actions (copy,
create and fill) simply create a new hash entry and set it to a value from
an already existing entry. Since operations with hashes (such as looking up
an entry) work extremely fast in Perl, the actions don't have any
performance issues.
> Should i modify the action line to be like the one bellow, to get rid of
> now obsolote oryginal context, or will this create a race condition?
>
> action=copy MSGID_$+{MSGID} %user;create MSGID_$+{MSGID2} 3600; fill
> MSGID_$+{MSGID2} %{user};delete MSGID_$+{MSGID}
>
If the old message ID is unnecessary and is no longer needed, I would
recommend to delete the context. Although individual context consumes very
little memory, contexts are checked periodically (by default each second)
by SEC for lifetime expiration, and inspecting thousands of unnecessary
contexts will involve extra cpu time. Therefore, deleting contexts that are
no longer needed helps to keep things optimal, even if you wouldn't notice
any performance difference :)
Also, there are no race conditions involved in the action list you have
defined. All actions in the list (copy, create, fill and delete) are
independent from each other, and the contexts they manipulate don't need to
exist throughout the execution of the entire action list. As for 'delete'
action, the context that is deleted does not necessarily need to exist, and
if it is not there, 'delete' is no-op (you would see a debug-level message
in the log about the missing context, though).
kind regards
risto
>
> Regards
> Przemyslaw Orzechowski
>
> W dniu 12.06.2018 o 21:12, Risto Vaarandi pisze:
>
> hi Przemysław,
>
> for addressing this problem, you could divide it into several independent
> subtasks:
> 1) memorize the mapping from msgid to username, so that each mapping would
> exist for a certain amount of time
> 2) when a bounce event appears in the log, generate a synthetic event for
> the bounce, so that msgid->username mapping is used for including the user
> name in the synthetic event
> 3) count and threshold synthetic events, and generate an alarm if more
> than 10 events have been seen in 1 hour for the given user name
>
> To illustrate how this could work, I've put together a small rulebase of
> three rules, with each rule corresponding to one of the subtasks above:
>
> # establish msgid->username mapping
>
> type=single
> ptype=regexp
> pattern=postfix.*\]: (?\S+): .* sasl_username=(?\S+)
> desc=set up a context for message ID $+{MSGID}
> action=create MSGID_$+{MSGID} 3600; fill MSGID_$+{MSGID} $+{LOGIN}
>
> # generate a synthetic bounce event which contains the user name
> # (username is obtained through msgid->username mapping)
>
> type=single
> ptype=regexp
> pattern=postfix\/smtp.*\]: (?\S+):.*dsn=5.*, status=bounced
> context=MSGID_$+{MSGID}
> desc=create a synthetic event for a bounce event with a username
> action=copy MSGID_$+{MSGID} %user; event BOUNCE_FOR_USER_%{user}_MSGID_
> $+{MSGID}
>
> # count and threshold synthetic events generated above
>
> type=singlewiththreshold
> ptype=regexp
> pattern=BOUNCE_FOR_USER_(?\S+)_MSGID_(\S+)
> desc=Too many bounces for user $+{LOGIN} during 1 hour
> action=write - %s
> thresh=10
> window=3600
>
> For recording the mappings, the first rule simply uses a context for each
> msgid, where the message ID is used as a context name, and user name is
> written into context's event store with the 'fill' action. Each context
> exists for 1 hour, but you can increase the mapping lifetime by using a
> larger value instead of 3600 seconds for the 'create' action.
>
> The second rule converts the original bounce event into a synthetic event
> which contains the user name. Note that the user name is taken from the
> context for the given msgid with the 'copy' action (if the context does not
> exist, the rule would not match the original bounce event due to 'context'
> field, and the synthetic event is not created).
>
> Finally, the third rule implements event counting which is done on user
> name basis. If you would like to do it per each unique (username,msgid)
> tuple, you can include the msgid into the 'desc' field (the regular
> expression would also need to be modified for setting the
Re: [Simple-evcorr-users] Way to corelate 2 rules with treshold
Hi
And thank You for fast and helpful answer.
I needed to slightly modify the proposed rules bellow my fully working
version (for now the action is left to write)
An "dsn=5.x.x" (x-any numeric digit) always denotes a permanent error,
so the text bounced is always there i removed ", status=bounced" that
actually made the regexp to fail for some reason
# etablish msgid->username mapping
type=single
ptype=regexp
pattern=postfix.*\]: (?\S+): .* sasl_username=(?\S+)
desc=set up a context for message ID $+{MSGID}
action=create MSGID_$+{MSGID} 3600; fill MSGID_$+{MSGID} $+{LOGIN}
# generate a synthetic bounce event which contains the user name
# (username is obtained through msgid->username mapping)
type=single
ptype=regexp
pattern=postfix\/smtp.*\]: (?\S+):.*dsn=5.*
context=MSGID_$+{MSGID}
desc=create a synthetic event for a bounce event with a username
action=copy MSGID_$+{MSGID} %user; event
BOUNCE_FOR_USER_%{user}_MSGID_$+{MSGID}
# count and threshold synthetic events generated above
type=singlewiththreshold
ptype=regexp
pattern=BOUNCE_FOR_USER_(?\S+)_MSGID_(\S+)
desc=Too many bounces for user $+{LOGIN} during 1 hour
action=write - %s
thresh=10
window=3600
I tried to add one more rule for a situation when the message is
redirected to a filter in postfix (in that case MSGID changes (can in
theory repeat multiple times so multiple changes of MSGID can occur -
will produce a log line similar to the one bellow each time).
Jun 5 20:00:12 server1 postfix/smtp[3454]: 4E8F9804CFE:
to=, relay=127.0.0.1[127.0.0.1]:10042, delay=2.2,
delays=1.6/0/0/0.67, dsn=2.0.0, status=sent (250 2.0.0 Ok, id=03546-04,
from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 0F7BB804D31)
as a test first I added this simple rule (MSGID is same as with Your
first rule for user mappaing) MSGID2 is the new MSGID that the message
will have from now on.
type=single
ptype=regexp
pattern=postfix\/smtp.*\]: (?\S+):.*relay=127.0.0.1.* queued as
(?\S+)\)
desc=test3
action=write - RELAY_$+{MSGID}_do_$+{MSGID2}_
this rule fires correctly when message passes thru postfix filter
Then i tried this to incorporate it into Your proposal as a separate
rule (inserted it right after the first rule).
# create context for changed messadeid preserving login
type=single
ptype=regexp
pattern=postfix\/smtp.*\]: (?\S+):.*relay=127.0.0.1.* queued as
(?\S+)\)
context=MSGID_$+{MSGID}
desc=set up a context for message ID $+{MSGID2}
action=copy MSGID_$+{MSGID} %user;create MSGID_$+{MSGID2} 3600; fill
MSGID_$+{MSGID2} %{user}
This seems to copy user data from old msgid to new msgid and make even
multiple filter hops work ok.
My question are:
Is this optimal performance wise?
Should i modify the action line to be like the one bellow, to get rid of
now obsolote oryginal context, or will this create a race condition?
action=copy MSGID_$+{MSGID} %user;create MSGID_$+{MSGID2} 3600; fill
MSGID_$+{MSGID2} %{user};delete MSGID_$+{MSGID}
Regards
Przemyslaw Orzechowski
W dniu 12.06.2018 o 21:12, Risto Vaarandi pisze:
hi Przemysław,
for addressing this problem, you could divide it into several
independent subtasks:
1) memorize the mapping from msgid to username, so that each mapping
would exist for a certain amount of time
2) when a bounce event appears in the log, generate a synthetic event
for the bounce, so that msgid->username mapping is used for including
the user name in the synthetic event
3) count and threshold synthetic events, and generate an alarm if more
than 10 events have been seen in 1 hour for the given user name
To illustrate how this could work, I've put together a small rulebase
of three rules, with each rule corresponding to one of the subtasks above:
# establish msgid->username mapping
type=single
ptype=regexp
pattern=postfix.*\]: (?\S+): .* sasl_username=(?\S+)
desc=set up a context for message ID $+{MSGID}
action=create MSGID_$+{MSGID} 3600; fill MSGID_$+{MSGID} $+{LOGIN}
# generate a synthetic bounce event which contains the user name
# (username is obtained through msgid->username mapping)
type=single
ptype=regexp
pattern=postfix\/smtp.*\]: (?\S+):.*dsn=5.*, status=bounced
context=MSGID_$+{MSGID}
desc=create a synthetic event for a bounce event with a username
action=copy MSGID_$+{MSGID} %user; event
BOUNCE_FOR_USER_%{user}_MSGID_$+{MSGID}
# count and threshold synthetic events generated above
type=singlewiththreshold
ptype=regexp
pattern=BOUNCE_FOR_USER_(?\S+)_MSGID_(\S+)
desc=Too many bounces for user $+{LOGIN} during 1 hour
action=write - %s
thresh=10
window=3600
For recording the mappings, the first rule simply uses a context for
each msgid, where the message ID is used as a context name, and user
name is written into context's event store with the 'fill' action.
Each context exists for 1 hour, but you can increase the mapping
lifetime by using a larger value instead of 3600 seconds for the
'create' action.
The second rule converts the original bounce event into a
Re: [Simple-evcorr-users] Way to corelate 2 rules with treshold
hi Przemysław,
for addressing this problem, you could divide it into several independent
subtasks:
1) memorize the mapping from msgid to username, so that each mapping would
exist for a certain amount of time
2) when a bounce event appears in the log, generate a synthetic event for
the bounce, so that msgid->username mapping is used for including the user
name in the synthetic event
3) count and threshold synthetic events, and generate an alarm if more than
10 events have been seen in 1 hour for the given user name
To illustrate how this could work, I've put together a small rulebase of
three rules, with each rule corresponding to one of the subtasks above:
# establish msgid->username mapping
type=single
ptype=regexp
pattern=postfix.*\]: (?\S+): .* sasl_username=(?\S+)
desc=set up a context for message ID $+{MSGID}
action=create MSGID_$+{MSGID} 3600; fill MSGID_$+{MSGID} $+{LOGIN}
# generate a synthetic bounce event which contains the user name
# (username is obtained through msgid->username mapping)
type=single
ptype=regexp
pattern=postfix\/smtp.*\]: (?\S+):.*dsn=5.*, status=bounced
context=MSGID_$+{MSGID}
desc=create a synthetic event for a bounce event with a username
action=copy MSGID_$+{MSGID} %user; event
BOUNCE_FOR_USER_%{user}_MSGID_$+{MSGID}
# count and threshold synthetic events generated above
type=singlewiththreshold
ptype=regexp
pattern=BOUNCE_FOR_USER_(?\S+)_MSGID_(\S+)
desc=Too many bounces for user $+{LOGIN} during 1 hour
action=write - %s
thresh=10
window=3600
For recording the mappings, the first rule simply uses a context for each
msgid, where the message ID is used as a context name, and user name is
written into context's event store with the 'fill' action. Each context
exists for 1 hour, but you can increase the mapping lifetime by using a
larger value instead of 3600 seconds for the 'create' action.
The second rule converts the original bounce event into a synthetic event
which contains the user name. Note that the user name is taken from the
context for the given msgid with the 'copy' action (if the context does not
exist, the rule would not match the original bounce event due to 'context'
field, and the synthetic event is not created).
Finally, the third rule implements event counting which is done on user
name basis. If you would like to do it per each unique (username,msgid)
tuple, you can include the msgid into the 'desc' field (the regular
expression would also need to be modified for setting the $+{MSGID}
variable).
Hope this helps,
risto
2018-06-12 17:55 GMT+03:00 Przemysław Orzechowski <
przemek.orzechow...@makolab.com>:
> Hi
>
>
> I have successfully created 2 rules that mach lines i want to act on but
> I'm struggling in finding a way to correlate them.
>
> First description what i want to get
>
> Generate an event for a threshold ie 10 events in 1 hour the problem is i
> don't quite get how to count them correctly
>
> First is a line that has username and mailid for example:
>
> Jun 3 06:46:13 server1 postfix/smtpd[3268]: 626C5802295:
> client=unknown[8.8.8.8], sasl_method=LOGIN, sasl_username=login@some.domai
> n.com
>
> Im maching it with pattern
>
> pattern=postfix.*\]: (?\S+): .* sasl_username=(?\S+)
>
> the threshold should be for unique LOGINS and should count events mached
> by the second rule ie lines like these:
>
>
> Jun 3 06:46:14 server1 postfix/smtp[23808]: 626C5802295: to=<
> somb...@somewhere.com>, relay=somewhere.com[9.9.9.9]:25, delay=1,
> delays=1/0.03/21/20, dsn=5.0.0, status=bounced (host somewhere.com[9.9.9.9]
> said: 550-The mail server could not deliver mail to somb...@somewhere.com.
> 550-The account or domain may not exist, they may be blacklisted, or
> missing 550 the proper dns entries. (in reply to RCPT TO command))
>
> Im maching this line with pattern
>
> pattern=postfix\/smtp.*\]: (?\S+):.*dsn=5.*
>
> the MSGID is identical in both lines
>
> Im trying to count the bounces and execute an action if there are too many
> of them for single sasl_username in a time period
>
> Btw MSGID can repeat themselvs in longer preiods of time but should be
> unique within the treshold period.
>
> I would be very thankfull for any help
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
