Re: [Simple-evcorr-users] Way to corelate 2 rules with treshold
hi Przemysław, I have provided my answers inline: Then i tried this to incorporate it into Your proposal as a separate rule > (inserted it right after the first rule). > # create context for changed messadeid preserving login > type=single > ptype=regexp > pattern=postfix\/smtp.*\]: (?\S+):.*relay=127.0.0.1.* queued as > (?\S+)\) > context=MSGID_$+{MSGID} > desc=set up a context for message ID $+{MSGID2} > action=copy MSGID_$+{MSGID} %user;create MSGID_$+{MSGID2} 3600; fill > MSGID_$+{MSGID2} %{user} > > This seems to copy user data from old msgid to new msgid and make even > multiple filter hops work ok. > > My question are: > Is this optimal performance wise? > yes, it is probably the best way for addressing the problem. Contexts are internally implemented as Perl hashes, and above three actions (copy, create and fill) simply create a new hash entry and set it to a value from an already existing entry. Since operations with hashes (such as looking up an entry) work extremely fast in Perl, the actions don't have any performance issues. > Should i modify the action line to be like the one bellow, to get rid of > now obsolote oryginal context, or will this create a race condition? > > action=copy MSGID_$+{MSGID} %user;create MSGID_$+{MSGID2} 3600; fill > MSGID_$+{MSGID2} %{user};delete MSGID_$+{MSGID} > If the old message ID is unnecessary and is no longer needed, I would recommend to delete the context. Although individual context consumes very little memory, contexts are checked periodically (by default each second) by SEC for lifetime expiration, and inspecting thousands of unnecessary contexts will involve extra cpu time. Therefore, deleting contexts that are no longer needed helps to keep things optimal, even if you wouldn't notice any performance difference :) Also, there are no race conditions involved in the action list you have defined. All actions in the list (copy, create, fill and delete) are independent from each other, and the contexts they manipulate don't need to exist throughout the execution of the entire action list. As for 'delete' action, the context that is deleted does not necessarily need to exist, and if it is not there, 'delete' is no-op (you would see a debug-level message in the log about the missing context, though). kind regards risto > > Regards > Przemyslaw Orzechowski > > W dniu 12.06.2018 o 21:12, Risto Vaarandi pisze: > > hi Przemysław, > > for addressing this problem, you could divide it into several independent > subtasks: > 1) memorize the mapping from msgid to username, so that each mapping would > exist for a certain amount of time > 2) when a bounce event appears in the log, generate a synthetic event for > the bounce, so that msgid->username mapping is used for including the user > name in the synthetic event > 3) count and threshold synthetic events, and generate an alarm if more > than 10 events have been seen in 1 hour for the given user name > > To illustrate how this could work, I've put together a small rulebase of > three rules, with each rule corresponding to one of the subtasks above: > > # establish msgid->username mapping > > type=single > ptype=regexp > pattern=postfix.*\]: (?\S+): .* sasl_username=(?\S+) > desc=set up a context for message ID $+{MSGID} > action=create MSGID_$+{MSGID} 3600; fill MSGID_$+{MSGID} $+{LOGIN} > > # generate a synthetic bounce event which contains the user name > # (username is obtained through msgid->username mapping) > > type=single > ptype=regexp > pattern=postfix\/smtp.*\]: (?\S+):.*dsn=5.*, status=bounced > context=MSGID_$+{MSGID} > desc=create a synthetic event for a bounce event with a username > action=copy MSGID_$+{MSGID} %user; event BOUNCE_FOR_USER_%{user}_MSGID_ > $+{MSGID} > > # count and threshold synthetic events generated above > > type=singlewiththreshold > ptype=regexp > pattern=BOUNCE_FOR_USER_(?\S+)_MSGID_(\S+) > desc=Too many bounces for user $+{LOGIN} during 1 hour > action=write - %s > thresh=10 > window=3600 > > For recording the mappings, the first rule simply uses a context for each > msgid, where the message ID is used as a context name, and user name is > written into context's event store with the 'fill' action. Each context > exists for 1 hour, but you can increase the mapping lifetime by using a > larger value instead of 3600 seconds for the 'create' action. > > The second rule converts the original bounce event into a synthetic event > which contains the user name. Note that the user name is taken from the > context for the given msgid with the 'copy' action (if the context does not > exist, the rule would not match the original bounce event due to 'context' > field, and the synthetic event is not created). > > Finally, the third rule implements event counting which is done on user > name basis. If you would like to do it per each unique (username,msgid) > tuple, you can include the msgid into the 'desc' field (the regular > expression would also need to be modified for setting the
Re: [Simple-evcorr-users] Way to corelate 2 rules with treshold
Hi And thank You for fast and helpful answer. I needed to slightly modify the proposed rules bellow my fully working version (for now the action is left to write) An "dsn=5.x.x" (x-any numeric digit) always denotes a permanent error, so the text bounced is always there i removed ", status=bounced" that actually made the regexp to fail for some reason # etablish msgid->username mapping type=single ptype=regexp pattern=postfix.*\]: (?\S+): .* sasl_username=(?\S+) desc=set up a context for message ID $+{MSGID} action=create MSGID_$+{MSGID} 3600; fill MSGID_$+{MSGID} $+{LOGIN} # generate a synthetic bounce event which contains the user name # (username is obtained through msgid->username mapping) type=single ptype=regexp pattern=postfix\/smtp.*\]: (?\S+):.*dsn=5.* context=MSGID_$+{MSGID} desc=create a synthetic event for a bounce event with a username action=copy MSGID_$+{MSGID} %user; event BOUNCE_FOR_USER_%{user}_MSGID_$+{MSGID} # count and threshold synthetic events generated above type=singlewiththreshold ptype=regexp pattern=BOUNCE_FOR_USER_(?\S+)_MSGID_(\S+) desc=Too many bounces for user $+{LOGIN} during 1 hour action=write - %s thresh=10 window=3600 I tried to add one more rule for a situation when the message is redirected to a filter in postfix (in that case MSGID changes (can in theory repeat multiple times so multiple changes of MSGID can occur - will produce a log line similar to the one bellow each time). Jun 5 20:00:12 server1 postfix/smtp[3454]: 4E8F9804CFE: to=, relay=127.0.0.1[127.0.0.1]:10042, delay=2.2, delays=1.6/0/0/0.67, dsn=2.0.0, status=sent (250 2.0.0 Ok, id=03546-04, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 0F7BB804D31) as a test first I added this simple rule (MSGID is same as with Your first rule for user mappaing) MSGID2 is the new MSGID that the message will have from now on. type=single ptype=regexp pattern=postfix\/smtp.*\]: (?\S+):.*relay=127.0.0.1.* queued as (?\S+)\) desc=test3 action=write - RELAY_$+{MSGID}_do_$+{MSGID2}_ this rule fires correctly when message passes thru postfix filter Then i tried this to incorporate it into Your proposal as a separate rule (inserted it right after the first rule). # create context for changed messadeid preserving login type=single ptype=regexp pattern=postfix\/smtp.*\]: (?\S+):.*relay=127.0.0.1.* queued as (?\S+)\) context=MSGID_$+{MSGID} desc=set up a context for message ID $+{MSGID2} action=copy MSGID_$+{MSGID} %user;create MSGID_$+{MSGID2} 3600; fill MSGID_$+{MSGID2} %{user} This seems to copy user data from old msgid to new msgid and make even multiple filter hops work ok. My question are: Is this optimal performance wise? Should i modify the action line to be like the one bellow, to get rid of now obsolote oryginal context, or will this create a race condition? action=copy MSGID_$+{MSGID} %user;create MSGID_$+{MSGID2} 3600; fill MSGID_$+{MSGID2} %{user};delete MSGID_$+{MSGID} Regards Przemyslaw Orzechowski W dniu 12.06.2018 o 21:12, Risto Vaarandi pisze: hi Przemysław, for addressing this problem, you could divide it into several independent subtasks: 1) memorize the mapping from msgid to username, so that each mapping would exist for a certain amount of time 2) when a bounce event appears in the log, generate a synthetic event for the bounce, so that msgid->username mapping is used for including the user name in the synthetic event 3) count and threshold synthetic events, and generate an alarm if more than 10 events have been seen in 1 hour for the given user name To illustrate how this could work, I've put together a small rulebase of three rules, with each rule corresponding to one of the subtasks above: # establish msgid->username mapping type=single ptype=regexp pattern=postfix.*\]: (?\S+): .* sasl_username=(?\S+) desc=set up a context for message ID $+{MSGID} action=create MSGID_$+{MSGID} 3600; fill MSGID_$+{MSGID} $+{LOGIN} # generate a synthetic bounce event which contains the user name # (username is obtained through msgid->username mapping) type=single ptype=regexp pattern=postfix\/smtp.*\]: (?\S+):.*dsn=5.*, status=bounced context=MSGID_$+{MSGID} desc=create a synthetic event for a bounce event with a username action=copy MSGID_$+{MSGID} %user; event BOUNCE_FOR_USER_%{user}_MSGID_$+{MSGID} # count and threshold synthetic events generated above type=singlewiththreshold ptype=regexp pattern=BOUNCE_FOR_USER_(?\S+)_MSGID_(\S+) desc=Too many bounces for user $+{LOGIN} during 1 hour action=write - %s thresh=10 window=3600 For recording the mappings, the first rule simply uses a context for each msgid, where the message ID is used as a context name, and user name is written into context's event store with the 'fill' action. Each context exists for 1 hour, but you can increase the mapping lifetime by using a larger value instead of 3600 seconds for the 'create' action. The second rule converts the original bounce event into a
Re: [Simple-evcorr-users] Way to corelate 2 rules with treshold
hi Przemysław, for addressing this problem, you could divide it into several independent subtasks: 1) memorize the mapping from msgid to username, so that each mapping would exist for a certain amount of time 2) when a bounce event appears in the log, generate a synthetic event for the bounce, so that msgid->username mapping is used for including the user name in the synthetic event 3) count and threshold synthetic events, and generate an alarm if more than 10 events have been seen in 1 hour for the given user name To illustrate how this could work, I've put together a small rulebase of three rules, with each rule corresponding to one of the subtasks above: # establish msgid->username mapping type=single ptype=regexp pattern=postfix.*\]: (?\S+): .* sasl_username=(?\S+) desc=set up a context for message ID $+{MSGID} action=create MSGID_$+{MSGID} 3600; fill MSGID_$+{MSGID} $+{LOGIN} # generate a synthetic bounce event which contains the user name # (username is obtained through msgid->username mapping) type=single ptype=regexp pattern=postfix\/smtp.*\]: (?\S+):.*dsn=5.*, status=bounced context=MSGID_$+{MSGID} desc=create a synthetic event for a bounce event with a username action=copy MSGID_$+{MSGID} %user; event BOUNCE_FOR_USER_%{user}_MSGID_$+{MSGID} # count and threshold synthetic events generated above type=singlewiththreshold ptype=regexp pattern=BOUNCE_FOR_USER_(?\S+)_MSGID_(\S+) desc=Too many bounces for user $+{LOGIN} during 1 hour action=write - %s thresh=10 window=3600 For recording the mappings, the first rule simply uses a context for each msgid, where the message ID is used as a context name, and user name is written into context's event store with the 'fill' action. Each context exists for 1 hour, but you can increase the mapping lifetime by using a larger value instead of 3600 seconds for the 'create' action. The second rule converts the original bounce event into a synthetic event which contains the user name. Note that the user name is taken from the context for the given msgid with the 'copy' action (if the context does not exist, the rule would not match the original bounce event due to 'context' field, and the synthetic event is not created). Finally, the third rule implements event counting which is done on user name basis. If you would like to do it per each unique (username,msgid) tuple, you can include the msgid into the 'desc' field (the regular expression would also need to be modified for setting the $+{MSGID} variable). Hope this helps, risto 2018-06-12 17:55 GMT+03:00 Przemysław Orzechowski < przemek.orzechow...@makolab.com>: > Hi > > > I have successfully created 2 rules that mach lines i want to act on but > I'm struggling in finding a way to correlate them. > > First description what i want to get > > Generate an event for a threshold ie 10 events in 1 hour the problem is i > don't quite get how to count them correctly > > First is a line that has username and mailid for example: > > Jun 3 06:46:13 server1 postfix/smtpd[3268]: 626C5802295: > client=unknown[8.8.8.8], sasl_method=LOGIN, sasl_username=login@some.domai > n.com > > Im maching it with pattern > > pattern=postfix.*\]: (?\S+): .* sasl_username=(?\S+) > > the threshold should be for unique LOGINS and should count events mached > by the second rule ie lines like these: > > > Jun 3 06:46:14 server1 postfix/smtp[23808]: 626C5802295: to=< > somb...@somewhere.com>, relay=somewhere.com[9.9.9.9]:25, delay=1, > delays=1/0.03/21/20, dsn=5.0.0, status=bounced (host somewhere.com[9.9.9.9] > said: 550-The mail server could not deliver mail to somb...@somewhere.com. > 550-The account or domain may not exist, they may be blacklisted, or > missing 550 the proper dns entries. (in reply to RCPT TO command)) > > Im maching this line with pattern > > pattern=postfix\/smtp.*\]: (?\S+):.*dsn=5.* > > the MSGID is identical in both lines > > Im trying to count the bounces and execute an action if there are too many > of them for single sasl_username in a time period > > Btw MSGID can repeat themselvs in longer preiods of time but should be > unique within the treshold period. > > I would be very thankfull for any help > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Simple-evcorr-users mailing list > Simple-evcorr-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users > -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users