>I'd be shocked if more than 10 people out of the couple thousand remaining ASSP users (??) has an CorrectASSPcfg functions in a regex.
May be some perl experts, who wants to extend the regex features in assp. But IMHO this is not required and I never saw a related question. Even I don't do such things in a regex. And never the less: 'AllowCodeInRegex' is disabled per default for security reasons! https://perldoc.perl.org/perlretut#A-bit-of-magic:-executing-Perl-code-in-a-regular-expression The goal of the lib/CorrectASSPcfg.pm is to set configuration parameters, to implement custom code and to provide some callbacks ( I already spoke about) for assp.pl. Perl experts will know more ways how to implement custom code in to assp.- without modifying the original perl script. >but wasn't aware of the *FAIL syntax. https://perldoc.perl.org/perlretut#Backtracking-control-verbs It's a good idea to have https://perldoc.perl.org/perlretut in mind (or at hands) if you want to do perl regular expressions! >This is TERRIFIC. Terrific, terrific, terrific. Possibly yes. BUT SURELY it can be dangerous! You have to enable '$AllowCodeInRegex' , which applies to all custom regular expressions. If someone (a hacker) is smart enough to let assp fill logs, headers or bodys with bad content - it can be possible that assp executes malicious code (for example like log4j ). Not only at the time a mail is processed, this can be happen at any later time (analyze, blockreport, notify, ham/spam report, rebuildspamdb, ...) I'm sure this is not possible if 'AllowCodeInRegex' is disabled! Thomas Von: "K Post" <nntp.p...@gmail.com> An: "ASSP development mailing list" <assp-test@lists.sourceforge.net> Datum: 16.07.2022 13:32 Betreff: Re: [Assp-test] blockStrictDKIMRe -- also thoughts on DMARC rejects This is TERRIFIC. Terrific, terrific, terrific. I've done a lot of regex work in my days mostly in php, vb, and linux batch scripts, but wasn't aware of the *FAIL syntax. I was thinking a return of 1 or 0, not no return. That makes much more sense. And thanks for the continued examples. I'd be shocked if more than 10 people out of the couple thousand remaining ASSP users (??) has an CorrectASSPcfg functions in a regex. Three small charities that used to use my charity's ASSP installation for email filtering have gone direct to Office365 and removed ASSP from the equation in the last 2 years. I'm still in touch with their staff. While the like 365's features, they can't stand the spam filtering. You've spoiled them with never having to sort through a junk folder!! On Fri, Jul 15, 2022 at 6:37 AM Thomas Eckardt <thomas.ecka...@thockar.com > wrote: Ken - learn perl regular expressions! >Why is the *FAIL bit in your example of - the (*FAIL) or (*F) statement makes a regex fail, even a match was found. Because the return value (e.g. setting $_) from a code execution in a perl regex does not modify the 'match found/no match found' flag of the regex. But the $_ can be used in a conditional regex to tell the regex engine what to do (in which case). There are multiple ways to do it. And possibly there are better ways - but this one I found nice. (\@.+\.docusign\.net|next domain|next domain|...|...)(?(?{&CorrectASSPcfg::myWantedDKIMCheck($fh,$+)})|(*FAIL)) explanation: ( the matching strings/domains, match captured in $+) # if failed, the next parts of the regex are ignored and the regex fails - if matched 'match found' is set by the regex engine (?( # start of a conditional (yes|no) regex (?(cond)yes|no) ?{ # start of the code to be executed ?{code} &CorrectASSPcfg::myWantedDKIMCheck # call this sub ($fh,$+) # provide the filehandle and the last match result to the sub }) # end of the code and the condition - the return value of the sub is the conditional result | # the 'yes' part (before the pipe [empty]) - if the code returned 1, nothing is to do, regex keeps 'match found' (*FAIL) # the 'no' part (after the pipe) - if the sub returned 0 or undef make the regex fail (no match found) ) # end of the conditional regex > seems to return if there's no DKIM (return unless $this->{isDKIM};) wouldn't that not match the regex, so the 60 score wouldn't be applied? right! returns undef in case there is no DKIM-signature found - which makes the regex fail (*FAIL) -> no score If you need it the opposit way change the return value of the sub - or the regex ....})(*FAIL)|) <- here the 'yes' part sets 'no match found' and the 'no' part leaves the 'match found' flag How ever, these all are examples on how you can get such or similar problems solved. It's on you to define rules, to find the required $Con{$fh}->{.....} flags in assp, to change the regex as needed and to write the code. The sub used in CorrectASSPcfg should be solid rock against crashes (in doubt: eval {} is a good friend)! Keep in mind: the header checks and body checks are done in a fixed order (assp_check_order.txt) - for example, it is useless to require a Con-flag in a headerRe, if this flag was not already processed (set) before by assp! be careful if you modify such flags - know what you do! >Not just that, but it helps me to understand ways that CorrectASSPcfg can be used. Reading the SSL GUI section, makes this really clear! CallBack to Verify .... Call to Configure ..... .... .... To make it complete - the following subs in CorrectASSPcfg.pm will be called if they are available: from assp.pl: set - set hidden variables (prevents the requirement to modify assp.pl for hidden variable changes/settings) custom_mlog - do something with or because of log lines custom_reply - customize ASSP SMTP replies (literals, error explanations, .....) modMyHeader - check/modify X-ASSP-.... headers before they are added to the mail header translateReply - translate MTA SMTP reply codes (some MTA's provides bad reply codes) customAnalyze - called before a mail is parsed and processed by the analyzer from ASSP_AFC.pm: AFC_Executable_Detection - custome code to detect executable code in attachments, called if no code was found The following config parameters can point to subs in CorrectASSPcfg.pm: ASSP_ARCSelectCode ASSP_RSSSelectCode debugCode Thomas Von: "K Post" <nntp.p...@gmail.com> An: "ASSP development mailing list" < assp-test@lists.sourceforge.net> Datum: 14.07.2022 17:07 Betreff: Re: [Assp-test] blockStrictDKIMRe -- also thoughts on DMARC rejects Outstanding. Heavy coding to implement my original blockStrictDKIMRe concept alone makes it not worth it. That, added to the can of worms that I'd be opening up by planting the seed to have other exceptions, is enough to put the nail in the coffin for the idea. Though if the strict regex for SPF failures for certain domains hasn't caused the kind of flood of request you're worried about, if at some point you think of an >easy< way to code the option, then maybe it should be reconsidered -- but again, only if it can be done without heavy coding on your part!! I do very much appreciate the discussion and thoughtful explanation. The relatively simple solution of putting code into CorrectASSPcfg is genius. Not just that, but it helps me to understand ways that CorrectASSPcfg can be used. I previously didn't know that we could get function results right in a regex from CorrectASSPcfg. Using a function offers so much flexibility and promise! I hope it doesn't sound pathetic, but that's exciting! A couple more questions if (or when) you have the time and energy for this: Why is the *FAIL bit in your example of ~<<<(\@.+\.docusign\.net|next domain|next domain|...|...)(?(?{&CorrectASSPcfg::myWantedDKIMCheck($fh,$+)})|(*FAIL))>>>~=>60 I'm concerned about only matching (docusign.\net|otherdomains)(.*FAIL) I'm assuming you intended to have a period before the * Won't that match any header like: from: whate...@docusign.net subject: failure to complete submission The function example: sub CorrectASSPcfg::myWantedDKIMCheck { my ($fh,$match) = @_; my $this = ($fh && exists($main::Con{$fh})) ? $main::Con{$fh} :''; return unless $this; return unless $this->{isDKIM}; return 1 if $this->{dkimresult} eq 'pass'; return 1 if $this->{dkimverified} eq 'verified-OK'; my $re = qr/ domain1\.org | \.domain2\.org | user[^@]+?\@.+?\.domain3\.org /xis; return ($match !~ /$re/); } seems to return if there's no DKIM (return unless $this->{isDKIM};) wouldn't that not match the regex, so the 60 score wouldn't be applied? Part of my goal is to require DKIM signature for certain domain names, not only requiring valid DKIM. Thanks again Ken On Sat, Jul 9, 2022 at 4:53 AM Thomas Eckardt <thomas.ecka...@thockar.com> wrote: I'm sorry but the example \@.+\.docusign\.net(?{&CorrectASSPcfg::myWantedDKIMCheck($fh)})=>60 should be better ~<<<(\@.+\.docusign\.net|next domain|next domain|...|...)(?(?{&CorrectASSPcfg::myWantedDKIMCheck($fh,$+)})|(*FAIL))>>>~=>60 The first one does not fail if CorrectASSPcfg::myWantedDKIMCheck returns 0, The second provides $fh and the matched string to the sub CorrectASSPcfg::myWantedDKIMCheck. short example for CorrectASSPcfg::myWantedDKIMCheck sub CorrectASSPcfg::myWantedDKIMCheck { my ($fh,$match) = @_; my $this = ($fh && exists($main::Con{$fh})) ? $main::Con{$fh} :''; return unless $this; return unless $this->{isDKIM}; return 1 if $this->{dkimresult} eq 'pass'; return 1 if $this->{dkimverified} eq 'verified-OK'; my $re = qr/ domain1\.org | \.domain2\.org | user[^@]+?\@.+?\.domain3\.org /xis; return ($match !~ /$re/); } Thomas Von: "Thomas Eckardt" <thomas.ecka...@thockar.com> An: "ASSP development mailing list" < assp-test@lists.sourceforge.net> Datum: 08.07.2022 16:53 Betreff: Re: [Assp-test] blockStrictDKIMRe -- also thoughts on DMARC rejects If such a feature would be implemented, it will result in havy coding. >I want to outright block any message from @*.docusign.net that isn't signed or that has an invalid signature. I don't care if it's from a whitelisted email address, from an IP that's in the SPF record, and with a message body that is 100% great. You want not only to make the test domain based strict, you want to ignore flags like 'whitelisted' - that's ok - but if I would start to allow any flag exceptions, other users may want to have other or more flag exception. - noprocessing - whitelisted - spamlover - domain based scoring values - SMIME/PGP signed ..... Yes, a great feature - but who would need it? The best way would be to create a level 1 plugin for this purpose. There you can check the dkim result, flags, ip's ... what ever you want - and based on your logic, you can block or pass the mail. But knowing (and thinking like) assp, will open other ways (solution workarounds) - for example. we assume the DKIM check is set to scoring - and the scoring value is 20 points below the penalty limit. If a DKIM signature is invalid - assp scores. If the domain has ever sent a mail with a valid DKIM signature before (a DKIMCache entry is found), assp scores for DKIM if a mail without a DKIM signature from this domain is received. Now, if there was not added any other score (the mail is 100% ok, except DKIM) the mail will pass because the penalty limit is not reached. But you want to block the mail if the sender matches @*.docusign.net sender??? ... matches???... - assp has weighted regular expressions - like: bombSenderRe - where you can add or remove scoring points if you set there \@.+\.docusign\.net=>20 all mails from those domains will get a penalty of 20 points, which is harmless if there is everything else ok with the mail if dkim fails, the penalty limit will be reached and the mail will be blocked this can be finetuned using :>NWLI You are also able to implement code in to the regex (for example to check for the DKIM result). This is much less complicated than writing a plugin. \@.+\.docusign\.net(?{&CorrectASSPcfg::myWantedDKIMCheck($fh)})=>60 "score with 60 if the sender matches and the sub CorrectASSPcfg::myWantedDKIMCheck returned 1" Both examples should only show, that there are more ways to get wanted results in assp. If someone solved a similar problem using another way, it would be nice to hear, how this was done. Thomas Von: "K Post" <nntp.p...@gmail.com> An: "ASSP development mailing list" < assp-test@lists.sourceforge.net> Datum: 07.07.2022 15:56 Betreff: Re: [Assp-test] blockStrictDKIMRe -- also thoughts on DMARC rejects All of your points are clear, and the explanation is greatly appreciated. I now understand why it may be unwise to generally honor reject DMARC policy if we've overridden spf/dkim policy once we start manipulating results with ASSP. That makes sense. I still feel like a blockStrictDKIMRe type of new feature, where a failed OR missing dkim signature where the message matches the regex would be strictly blocked (just like we can do with blockstrictSPFRe for spf failures) would be helpful. For example (hopefully this is more illustrative of the desire), I want to outright block any message from @*.docusign.net that isn't signed or that has an invalid signature. I don't care if it's from a whitelisted email address, from an IP that's in the SPF record, and with a message body that is 100% great. If there's no DKIM signature or an invalid one for a message that matches the regex, reject the message (just like their DMARC policy says to do). Is there another way with current ASSP features to accomplish this only if a message matches this proposed regex? Ken On Fri, Jun 17, 2022 at 4:35 AM Thomas Eckardt <thomas.ecka...@thockar.com > wrote: >Would you please consider adding a feature to do the same for a failed DKIM signature? NO! Contrary to SPF, a DKIM signature has only two options : OK and FAIL - Based on the signature it self or based on a trusted forwarders authentication result (ARC). A DKIM signature has to be valid every time for any of the above reasons. > I score failed spf and score failed dkim, so DoDMARC is only scoring even though p=reject. What else makes sense? If SPF is scored and DKIM is scored and DMARC is score - AND the resulting score does'nt block the mail at the pealtybox, your settings are wrong! >If DMARC says p=reject, why shouldn't assp outright honor that, regardless of if we have spf / dkim failures set to only score? SPF has too many options to change/override the original result in assp (more or less strict, overwrite, skip ....), some these options also exists for DKIM. If we ignore/change/override .... sender policies for SPF and DKIM, it is not wise to honor the reject DMARC policy strictly. Thomas Von: "K Post" <nntp.p...@gmail.com> An: "ASSP development mailing list" < assp-test@lists.sourceforge.net> Datum: 16.06.2022 19:28 Betreff: [Assp-test] blockStrictDKIMRe -- also thoughts on DMARC rejects The ability to block failed SPF, instead of just scoring them, for delect regex matches has been a terrific feature of ASSP for a long time. (Block SPF Processing Regex* (blockstrictSPFRe) ) Would you please consider adding a feature to do the same for a failed DKIM signature? Outright blocking of a matching message that fails DKIM, regardless of the domain's DMARC settings. -- maybe that's not necessary if DoDMARC will honor =reject, see more below. Reasoning: I already score failed DKIM signatures, but I can't set that score too high because so many organizations still send messages through 3rd parties with invalid DKIM signatures. It really is incredible how many I see. But for frequently abused sender addresses (docusign for example), who are often spoofed but send otherwise unspammy content, I want to outright block if the DKIM signature fails. blockStrictSPFRe usually works because these bad DKIM sigs are on mails that also violate SPF rules, still though it would be helpful if I could also just say "if a specific regex is matched on an email with an invalid DKIM, reject the message" RELATED: DMARC p=reject should always reject if failed Docusign.net has a dmarc rule of p=reject. I want to honor that. The last scam that came in from them failed SPF and failed DKIM validation, but the message was from a whitelisted address.. DoDMARC says that the blocking will be the "most less aggressive" (least aggressive) and the published DMARC record. I score failed spf and score failed dkim, so DoDMARC is only scoring even though p=reject. Enable DMARC Check (DoDMARC) If enabled and ValidateSPF and DoDKIM are enabled and the sending domain has published a DMARC-record/policy, assp will act on the mail according to the senders DMARC-policy using the results of the SPF and DKIM check and validating the SPF/DKIM address/domain Identifier Alignment rules (RFC7489 section 3). It is safe to leave this feature ON, it will not produce false positives! The blocking mode (block, monitor, score, testmode) is adapted from the most less aggressive setting of ValidateSPF and DoDKIM - and the published DMARC record ([p][sp]=[reject][quarantine]). Scoring is done using dmarcValencePB. If DMARC says p=reject, why shouldn't assp outright honor that, regardless of if we have spf / dkim failures set to only score? Thanks Ken _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ******************************************************* _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ******************************************************* _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ******************************************************* _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ******************************************************* _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! *******************************************************
_______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
