Hi
And thank You for fast and helpful answer.
I needed to slightly modify the proposed rules bellow my fully working
version (for now the action is left to write)
An "dsn=5.x.x" (x-any numeric digit) always denotes a permanent error,
so the text bounced is always there i removed ", status=bounced" that
actually made the regexp to fail for some reason
# etablish msgid->username mapping
type=single
ptype=regexp
pattern=postfix.*\]: (?<MSGID>\S+): .* sasl_username=(?<LOGIN>\S+)
desc=set up a context for message ID $+{MSGID}
action=create MSGID_$+{MSGID} 3600; fill MSGID_$+{MSGID} $+{LOGIN}
# generate a synthetic bounce event which contains the user name
# (username is obtained through msgid->username mapping)
type=single
ptype=regexp
pattern=postfix\/smtp.*\]: (?<MSGID>\S+):.*dsn=5.*
context=MSGID_$+{MSGID}
desc=create a synthetic event for a bounce event with a username
action=copy MSGID_$+{MSGID} %user; event
BOUNCE_FOR_USER_%{user}_MSGID_$+{MSGID}
# count and threshold synthetic events generated above
type=singlewiththreshold
ptype=regexp
pattern=BOUNCE_FOR_USER_(?<LOGIN>\S+)_MSGID_(\S+)
desc=Too many bounces for user $+{LOGIN} during 1 hour
action=write - %s
thresh=10
window=3600
I tried to add one more rule for a situation when the message is
redirected to a filter in postfix (in that case MSGID changes (can in
theory repeat multiple times so multiple changes of MSGID can occur -
will produce a log line similar to the one bellow each time).
Jun 5 20:00:12 server1 postfix/smtp[3454]: 4E8F9804CFE:
to=<somb...@somewhere.com>, relay=127.0.0.1[127.0.0.1]:10042, delay=2.2,
delays=1.6/0/0/0.67, dsn=2.0.0, status=sent (250 2.0.0 Ok, id=03546-04,
from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 0F7BB804D31)
as a test first I added this simple rule (MSGID is same as with Your
first rule for user mappaing) MSGID2 is the new MSGID that the message
will have from now on.
type=single
ptype=regexp
pattern=postfix\/smtp.*\]: (?<MSGID>\S+):.*relay=127.0.0.1.* queued as
(?<MSGID2>\S+)\)
desc=test3
action=write - RELAY_$+{MSGID}_do_$+{MSGID2}_
this rule fires correctly when message passes thru postfix filter
Then i tried this to incorporate it into Your proposal as a separate
rule (inserted it right after the first rule).
# create context for changed messadeid preserving login
type=single
ptype=regexp
pattern=postfix\/smtp.*\]: (?<MSGID>\S+):.*relay=127.0.0.1.* queued as
(?<MSGID2>\S+)\)
context=MSGID_$+{MSGID}
desc=set up a context for message ID $+{MSGID2}
action=copy MSGID_$+{MSGID} %user;create MSGID_$+{MSGID2} 3600; fill
MSGID_$+{MSGID2} %{user}
This seems to copy user data from old msgid to new msgid and make even
multiple filter hops work ok.
My question are:
Is this optimal performance wise?
Should i modify the action line to be like the one bellow, to get rid of
now obsolote oryginal context, or will this create a race condition?
action=copy MSGID_$+{MSGID} %user;create MSGID_$+{MSGID2} 3600; fill
MSGID_$+{MSGID2} %{user};delete MSGID_$+{MSGID}
Regards
Przemyslaw Orzechowski
W dniu 12.06.2018 o 21:12, Risto Vaarandi pisze:
hi Przemysław,
for addressing this problem, you could divide it into several
independent subtasks:
1) memorize the mapping from msgid to username, so that each mapping
would exist for a certain amount of time
2) when a bounce event appears in the log, generate a synthetic event
for the bounce, so that msgid->username mapping is used for including
the user name in the synthetic event
3) count and threshold synthetic events, and generate an alarm if more
than 10 events have been seen in 1 hour for the given user name
To illustrate how this could work, I've put together a small rulebase
of three rules, with each rule corresponding to one of the subtasks above:
# establish msgid->username mapping
type=single
ptype=regexp
pattern=postfix.*\]: (?<MSGID>\S+): .* sasl_username=(?<LOGIN>\S+)
desc=set up a context for message ID $+{MSGID}
action=create MSGID_$+{MSGID} 3600; fill MSGID_$+{MSGID} $+{LOGIN}
# generate a synthetic bounce event which contains the user name
# (username is obtained through msgid->username mapping)
type=single
ptype=regexp
pattern=postfix\/smtp.*\]: (?<MSGID>\S+):.*dsn=5.*, status=bounced
context=MSGID_$+{MSGID}
desc=create a synthetic event for a bounce event with a username
action=copy MSGID_$+{MSGID} %user; event
BOUNCE_FOR_USER_%{user}_MSGID_$+{MSGID}
# count and threshold synthetic events generated above
type=singlewiththreshold
ptype=regexp
pattern=BOUNCE_FOR_USER_(?<LOGIN>\S+)_MSGID_(\S+)
desc=Too many bounces for user $+{LOGIN} during 1 hour
action=write - %s
thresh=10
window=3600
For recording the mappings, the first rule simply uses a context for
each msgid, where the message ID is used as a context name, and user
name is written into context's event store with the 'fill' action.
Each context exists for 1 hour, but you can increase the mapping
lifetime by using a larger value instead of 3600 seconds for the
'create' action.
The second rule converts the original bounce event into a synthetic
event which contains the user name. Note that the user name is taken
from the context for the given msgid with the 'copy' action (if the
context does not exist, the rule would not match the original bounce
event due to 'context' field, and the synthetic event is not created).
Finally, the third rule implements event counting which is done on
user name basis. If you would like to do it per each unique
(username,msgid) tuple, you can include the msgid into the 'desc'
field (the regular expression would also need to be modified for
setting the $+{MSGID} variable).
Hope this helps,
risto
2018-06-12 17:55 GMT+03:00 Przemysław Orzechowski
<przemek.orzechow...@makolab.com
<mailto:przemek.orzechow...@makolab.com>>:
Hi
I have successfully created 2 rules that mach lines i want to act
on but I'm struggling in finding a way to correlate them.
First description what i want to get
Generate an event for a threshold ie 10 events in 1 hour the
problem is i don't quite get how to count them correctly
First is a line that has username and mailid for example:
Jun 3 06:46:13 server1 postfix/smtpd[3268]: 626C5802295:
client=unknown[8.8.8.8], sasl_method=LOGIN,
sasl_username=lo...@some.domain.com <mailto:lo...@some.domain.com>
Im maching it with pattern
pattern=postfix.*\]: (?<MSGID>\S+): .* sasl_username=(?<LOGIN>\S+)
the threshold should be for unique LOGINS and should count events
mached by the second rule ie lines like these:
Jun 3 06:46:14 server1 postfix/smtp[23808]: 626C5802295:
to=<somb...@somewhere.com <mailto:somb...@somewhere.com>>,
relay=somewhere.com <http://somewhere.com>[9.9.9.9]:25, delay=1,
delays=1/0.03/21/20, dsn=5.0.0, status=bounced (host somewhere.com
<http://somewhere.com>[9.9.9.9] said: 550-The mail server could
not deliver mail to somb...@somewhere.com
<mailto:somb...@somewhere.com>. 550-The account or domain may not
exist, they may be blacklisted, or missing 550 the proper dns
entries. (in reply to RCPT TO command))
Im maching this line with pattern
pattern=postfix\/smtp.*\]: (?<MSGID>\S+):.*dsn=5.*
the MSGID is identical in both lines
Im trying to count the bounces and execute an action if there are
too many of them for single sasl_username in a time period
Btw MSGID can repeat themselvs in longer preiods of time but
should be unique within the treshold period.
I would be very thankfull for any help
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
<mailto:Simple-evcorr-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
<https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
