Re: [Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam?
>What do you think about doing the DKIMNPAddress check even if hmm passes the threshold? DKIMNPAddress check is done in preDKIMCheck, which is a header check. HMM is a body check. >I think everyone would benefit from this. I don't know anyone. Thomas Von:"K Post" An: "ASSP development mailing list" Datum: 19.04.2018 23:21 Betreff: Re: [Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam? I am seeing this with a bunch of other addresses now. DKIM verifies. HMM fails, dkim not processed when accepting mail. What do you think about doing the DKIMNPAddress check even if hmm passes the threshold? I think everyone would benefit from this. On Tue, Apr 17, 2018 at 11:20 AM, K Post wrote: Okay, "regardless" was a bit much, I agree with the reasons you have listed for DKIM skipping and figured that was the case for all. What I'm seeing with 18103 is mail that scores with hmm/bayes only NOT being noprocessed due to a DKIMNPAddress hit. We know it's a good signature because analyze says it's a match, but for whatever reason ASSP isn't no processing it after a hmm or possibily other scoring threthold match. Unfortunately, I haven't seen an example since I made logging more verbose, or I'd have provide more info. Can you confirm that DKIMNPAddress is supposed to run and results be honored if a message's score already exceeds the reject threshold AND none of the exceptions you previously listed are true? I'll take a look at the graph to see what I can learn. You definitely peaked my interest when you mentioned the disclaimers file previously. About 70% of our staff uses signatures, generally unique to them. We're small enough that I guess I could start compiling a list for the disclaimers file. I assume that grabbing the signatures right from mail text files in the corpus would be okay? That would pick up the html markup they tend to use. Yes? At least 50% (guessing) of our legitimate inbound mail has signatures and disclaimers on them too. I can't see manually maintaining a disclaimers file with all of them in it. Which is better, having some inbound signatures in the file or NONE? I don't want to create a bias against signatures that aren't in the disclaimer file. On Tue, Apr 17, 2018 at 5:17 AM, Thomas Eckardt < thomas.ecka...@thockar.com> wrote: >If we put the address on the DKIMNPAddress list, shouldn't it honor that regardless of anything else? REGARDLESS ??? No . this makes no sense. The pre-DKIM check is skipped for (18103): invalidSenderDomain (no valid TLD) whitelisted acceptAllMail bounce mails outgoing mails RWL high trust contentOnly noprocessing (except noprocessing by size) noDKIMAddresses noDKIMIP --- >don't know if a change from 0.001 to 0.005 would be significant or make sense, Just use the 'Bayes/HMM confidence' graph. This requires 'enableGraphStats' to be enabled. >Then there's a ton of spam messages also advertising (fake) handbags and they're often using the SAME domain and sometimes identical from address as the legit mails. ASSP has alot of features to indentify the correctness of the origin of an email. And there is (IMHO) a big difference between maliciouse spam and normal (not dangerous) mails you don't want to get. The later are often problematic. Most times it is better to let them pass, than to block important mails. --- The next version will have an improvement for HMM and Bayesian. An real problem may become disclaimers and privat and corporate signatues. They are always added to outgoing mails, but also to spam reports. They can be found in most of the answers to our mails. And for example, in my case, they may be added by spammers to there spam mail. Nobody can say, how the occurrence of such a disclaimer will affect the HMM and Bayesian results. It may possible, that these results differs from day to day, or block always good mails, or.let spam pass. The only way to prevent such "wild" results is to remove the disclaimers, before the rebuildspamdb task builds the spamdb and HMMdb. I use this code for a month now and I'm really happy with the result. Thomas Von:"K Post" An: "ASSP development mailing list" < assp-test@lists.sourceforge.net> Datum:16.04.2018 17:43 Betreff:Re: [Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam? I'm always correcting HMM/Bayes by reviewing the block report on a daily basis and reporting. It's a horrible task that I dread, but it is worth it. I have the score set to 50, and 50 as the threshold for rejection. We've tried low
Re: [Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam?
I am seeing this with a bunch of other addresses now. DKIM verifies. HMM fails, dkim not processed when accepting mail. What do you think about doing the DKIMNPAddress check even if hmm passes the threshold? I think everyone would benefit from this. On Tue, Apr 17, 2018 at 11:20 AM, K Post wrote: > Okay, "regardless" was a bit much, I agree with the reasons you have > listed for DKIM skipping and figured that was the case for all. What I'm > seeing with 18103 is mail that scores with hmm/bayes only NOT being > noprocessed due to a DKIMNPAddress hit. We know it's a good signature > because analyze says it's a match, but for whatever reason ASSP isn't no > processing it after a hmm or possibily other scoring threthold match. > Unfortunately, I haven't seen an example since I made logging more verbose, > or I'd have provide more info. Can you confirm that DKIMNPAddress is > supposed to run and results be honored if a message's score already exceeds > the reject threshold AND none of the exceptions you previously listed are > true? > > I'll take a look at the graph to see what I can learn. > > You definitely peaked my interest when you mentioned the disclaimers file > previously. > > About 70% of our staff uses signatures, generally unique to them. We're > small enough that I guess I could start compiling a list for the > disclaimers file. I assume that grabbing the signatures right from mail > text files in the corpus would be okay? That would pick up the html markup > they tend to use. Yes? > > At least 50% (guessing) of our legitimate *inbound* mail has signatures > and disclaimers on them too. I can't see manually maintaining a > disclaimers file with all of them in it. Which is better, having some > inbound signatures in the file or NONE? I don't want to create a bias > against signatures that aren't in the disclaimer file. > > On Tue, Apr 17, 2018 at 5:17 AM, Thomas Eckardt < > thomas.ecka...@thockar.com> wrote: > >> >If we put the address on the DKIMNPAddress list, shouldn't it honor >> that regardless of anything else? >> >> REGARDLESS ??? >> >> No . this makes no sense. >> >> The pre-DKIM check is skipped for (18103): >> >> invalidSenderDomain (no valid TLD) >> whitelisted >> acceptAllMail >> bounce mails >> outgoing mails >> RWL high trust >> contentOnly >> noprocessing (except noprocessing by size) >> noDKIMAddresses >> noDKIMIP >> >> --- >> >don't know if a change from 0.001 to 0.005 would be significant or make >> sense, >> >> Just use the 'Bayes/HMM confidence' graph. This requires >> 'enableGraphStats' to be enabled. >> >> >Then there's a ton of spam messages also advertising (fake) handbags >> and they're often using the SAME domain and sometimes identical from >> address as the legit mails. >> >> ASSP has alot of features to indentify the correctness of the origin of >> an email. And there is (IMHO) a big difference between maliciouse spam and >> normal (not dangerous) mails you don't want to get. The later are often >> problematic. Most times it is better to let them pass, than to block >> important mails. >> >> --- >> >> The next version will have an improvement for HMM and Bayesian. >> An real problem may become disclaimers and privat and corporate >> signatues. They are always added to outgoing mails, but also to spam >> reports. They can be found in most of the answers to our mails. And for >> example, in my case, they may be added by spammers to there spam mail. >> Nobody can say, how the occurrence of such a disclaimer will affect the HMM >> and Bayesian results. It may possible, that these results differs from day >> to day, or block always good mails, or.let spam pass. >> >> The only way to prevent such "wild" results is to remove the disclaimers, >> before the rebuildspamdb task builds the spamdb and HMMdb. I use this code >> for a month now and I'm really happy with the result. >> >> Thomas >> >> >> >> >> >> Von:"K Post" >> An:"ASSP development mailing list" > et> >> Datum:16.04.2018 17:43 >> Betreff:Re: [Assp-test] Analyze shows DKIMNPAddress match as >> expected, but some messages still processed as spam? >> -- >> >> >> >> I'm always correcting HMM/Bayes by reviewing the block report on a daily &
Re: [Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam?
Okay, "regardless" was a bit much, I agree with the reasons you have listed for DKIM skipping and figured that was the case for all. What I'm seeing with 18103 is mail that scores with hmm/bayes only NOT being noprocessed due to a DKIMNPAddress hit. We know it's a good signature because analyze says it's a match, but for whatever reason ASSP isn't no processing it after a hmm or possibily other scoring threthold match. Unfortunately, I haven't seen an example since I made logging more verbose, or I'd have provide more info. Can you confirm that DKIMNPAddress is supposed to run and results be honored if a message's score already exceeds the reject threshold AND none of the exceptions you previously listed are true? I'll take a look at the graph to see what I can learn. You definitely peaked my interest when you mentioned the disclaimers file previously. About 70% of our staff uses signatures, generally unique to them. We're small enough that I guess I could start compiling a list for the disclaimers file. I assume that grabbing the signatures right from mail text files in the corpus would be okay? That would pick up the html markup they tend to use. Yes? At least 50% (guessing) of our legitimate *inbound* mail has signatures and disclaimers on them too. I can't see manually maintaining a disclaimers file with all of them in it. Which is better, having some inbound signatures in the file or NONE? I don't want to create a bias against signatures that aren't in the disclaimer file. On Tue, Apr 17, 2018 at 5:17 AM, Thomas Eckardt wrote: > >If we put the address on the DKIMNPAddress list, shouldn't it honor that > regardless of anything else? > > REGARDLESS ??? > > No . this makes no sense. > > The pre-DKIM check is skipped for (18103): > > invalidSenderDomain (no valid TLD) > whitelisted > acceptAllMail > bounce mails > outgoing mails > RWL high trust > contentOnly > noprocessing (except noprocessing by size) > noDKIMAddresses > noDKIMIP > > --- > >don't know if a change from 0.001 to 0.005 would be significant or make > sense, > > Just use the 'Bayes/HMM confidence' graph. This requires > 'enableGraphStats' to be enabled. > > >Then there's a ton of spam messages also advertising (fake) handbags and > they're often using the SAME domain and sometimes identical from address as > the legit mails. > > ASSP has alot of features to indentify the correctness of the origin of an > email. And there is (IMHO) a big difference between maliciouse spam and > normal (not dangerous) mails you don't want to get. The later are often > problematic. Most times it is better to let them pass, than to block > important mails. > > --- > > The next version will have an improvement for HMM and Bayesian. > An real problem may become disclaimers and privat and corporate signatues. > They are always added to outgoing mails, but also to spam reports. They can > be found in most of the answers to our mails. And for example, in my case, > they may be added by spammers to there spam mail. Nobody can say, how the > occurrence of such a disclaimer will affect the HMM and Bayesian results. > It may possible, that these results differs from day to day, or block > always good mails, or.let spam pass. > > The only way to prevent such "wild" results is to remove the disclaimers, > before the rebuildspamdb task builds the spamdb and HMMdb. I use this code > for a month now and I'm really happy with the result. > > Thomas > > > > > > Von:"K Post" > An:"ASSP development mailing list" net> > Datum:16.04.2018 17:43 > Betreff:Re: [Assp-test] Analyze shows DKIMNPAddress match as > expected, but some messages still processed as spam? > -- > > > > I'm always correcting HMM/Bayes by reviewing the block report on a daily > basis and reporting. It's a horrible task that I dread, but it is worth > it. I have the score set to 50, and 50 as the threshold for rejection. > We've tried lower, but too much spam is only tagged *solely* due to a HMM > hit and slips through. > > I never changed baysConf from the original 0.001 with baysprobablity to > 0.6, but I do have a 1.000 corpus norm. It's certainly a mature > installation (15+ years). The issue I'm having is legit messages, say a > message advertising handbags from a reputable seller that one of our staff > buys from (over their lunch hour of course!!). Then there's a ton of spam > messages also advertising (fake) handbags and they're often using the SAME > do
Re: [Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam?
>If we put the address on the DKIMNPAddress list, shouldn't it honor that regardless of anything else? REGARDLESS ??? No . this makes no sense. The pre-DKIM check is skipped for (18103): invalidSenderDomain (no valid TLD) whitelisted acceptAllMail bounce mails outgoing mails RWL high trust contentOnly noprocessing (except noprocessing by size) noDKIMAddresses noDKIMIP --- >don't know if a change from 0.001 to 0.005 would be significant or make sense, Just use the 'Bayes/HMM confidence' graph. This requires 'enableGraphStats' to be enabled. >Then there's a ton of spam messages also advertising (fake) handbags and they're often using the SAME domain and sometimes identical from address as the legit mails. ASSP has alot of features to indentify the correctness of the origin of an email. And there is (IMHO) a big difference between maliciouse spam and normal (not dangerous) mails you don't want to get. The later are often problematic. Most times it is better to let them pass, than to block important mails. --- The next version will have an improvement for HMM and Bayesian. An real problem may become disclaimers and privat and corporate signatues. They are always added to outgoing mails, but also to spam reports. They can be found in most of the answers to our mails. And for example, in my case, they may be added by spammers to there spam mail. Nobody can say, how the occurrence of such a disclaimer will affect the HMM and Bayesian results. It may possible, that these results differs from day to day, or block always good mails, or.let spam pass. The only way to prevent such "wild" results is to remove the disclaimers, before the rebuildspamdb task builds the spamdb and HMMdb. I use this code for a month now and I'm really happy with the result. Thomas Von:"K Post" An: "ASSP development mailing list" Datum: 16.04.2018 17:43 Betreff: Re: [Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam? I'm always correcting HMM/Bayes by reviewing the block report on a daily basis and reporting. It's a horrible task that I dread, but it is worth it. I have the score set to 50, and 50 as the threshold for rejection. We've tried lower, but too much spam is only tagged solely due to a HMM hit and slips through. I never changed baysConf from the original 0.001 with baysprobablity to 0.6, but I do have a 1.000 corpus norm. It's certainly a mature installation (15+ years). The issue I'm having is legit messages, say a message advertising handbags from a reputable seller that one of our staff buys from (over their lunch hour of course!!). Then there's a ton of spam messages also advertising (fake) handbags and they're often using the SAME domain and sometimes identical from address as the legit mails. HMM/Bayes is rightfully biased against all handbag email. You're saying that increasing baysConf will help the legit ones get through but still will block the spam ones (with almost identical content)? I've read through the gui for baysConf, but the problem is that which I understand HMM and Bayes from a concept standpoint, the calculations aren't something I understand, so I don't dare change the 0.0001 threshold without real guidance from you. I understand 1 is the max, but don't know if increases linearly or exponentially change levels, don't know if a change from 0.001 to 0.005 would be significant or make sense, etc. My hopes with the dkim np was to let hmm spammy mail through if it's a dkim match. Ignore all other results, if the DKIM is good, just let it through was my thinking. I know handbag seller X sends ad mail that's DKIM signed, but I don't know when IP they'll come from (or the IP is mailing service that I don't want to blanket allow). This has been quite successful with a whole lot of mail. I've become spoiled, now I want it to work for all mail when there's a DKIM match to the NP list. You raise another a good point about the 2 kind of DKIM checks, thanks for the reminder. Does it make any sense to always have ASSP do the second one and if it validates and matches dkimNPaddress or dkimWLadderss, process solely based on that match? For example, HMM might hit before the full body validation of DKIM, but so what? If we put the address on the DKIMNPAddress list, shouldn't it honor that regardless of anything else? NO processing, as I'm interpreting it, should mean, well, NONE, so if other hits have happened, they're ignored because we said don't process. I will temporarily change the logging level for a bit and see if I can figure out why dkim isn't being done for these messages, but I'm guessing th
Re: [Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam?
I'm always correcting HMM/Bayes by reviewing the block report on a daily basis and reporting. It's a horrible task that I dread, but it is worth it. I have the score set to 50, and 50 as the threshold for rejection. We've tried lower, but too much spam is only tagged *solely* due to a HMM hit and slips through. I never changed baysConf from the original 0.001 with baysprobablity to 0.6, but I do have a 1.000 corpus norm. It's certainly a mature installation (15+ years). The issue I'm having is legit messages, say a message advertising handbags from a reputable seller that one of our staff buys from (over their lunch hour of course!!). Then there's a ton of spam messages also advertising (fake) handbags and they're often using the SAME domain and sometimes identical from address as the legit mails. HMM/Bayes is rightfully biased against all handbag email. You're saying that increasing baysConf will help the legit ones get through but still will block the spam ones (with almost identical content)? I've read through the gui for baysConf, but the problem is that which I understand HMM and Bayes from a concept standpoint, the calculations aren't something I understand, so I don't dare change the 0.0001 threshold without real guidance from you. I understand 1 is the max, but don't know if increases linearly or exponentially change levels, don't know if a change from 0.001 to 0.005 would be significant or make sense, etc. My hopes with the dkim np was to let hmm spammy mail through if it's a dkim match. Ignore all other results, if the DKIM is good, just let it through was my thinking. I know handbag seller X sends ad mail that's DKIM signed, but I don't know when IP they'll come from (or the IP is mailing service that I don't want to blanket allow). This has been quite successful with a whole lot of mail. I've become spoiled, now I want it to work for all mail when there's a DKIM match to the NP list. You raise another a good point about the 2 kind of DKIM checks, thanks for the reminder. Does it make any sense to *always* have ASSP do the second one and if it validates and matches dkimNPaddress or dkimWLadderss, process solely based on that match? For example, HMM might hit before the full body validation of DKIM, but so what? If we put the address on the DKIMNPAddress list, shouldn't it honor that regardless of anything else? NO processing, as I'm interpreting it, should mean, well, NONE, so if other hits have happened, they're ignored because we said don't process. I will temporarily change the logging level for a bit and see if I can figure out why dkim isn't being done for these messages, but I'm guessing that it's by design. As always, thanks Ken On Mon, Apr 16, 2018 at 2:35 AM, Thomas Eckardt wrote: > >I'm still seeing scenarios where analyze shows a DKIM NP match, but the > message is still going to spam based on score from HMM. > > The best solution is to correct the corpus, to get better HMM results. You > may also decrease the scoring points for HMM and/or Bayesian. If the corpus > is corrected and the corpusnorm is ~ 1.0 , 'baysConf' will increase > detection correctness. > > > If assp receives a mail it acts as a statemachine. If and how a check is > done, depends on the previouse reached states. > > Using the analyzer, assp acts procedural. Every check is done without any > state dependency. This is done, to be able to show every feature match. > The analyzer uses the current configuration, hashes, lists and databases. > So it may be normal to get different results compared to the real mail > processing loggings, if a mail is analyzed. > > >DKIM NP match: > > The analyzer checks DKIM without any dependency and shows all results. > > But, if a mail is received, the DKIM check depends on several previouse > states > DKIM NP is a resulting state of the DKIM check. So - if any of the > previouse (DKIM depdency) states prevents the DKIM check, there will be no > DKIM > (DKIM NP) result. The mail will be processed the same way, as it was not > DKIM signed. > Every state, that depends on DKIM NP will not be reached. > > You should also remember, that assp use two DKIM checks. The full DKIM > check, which requires the full mail to be received - the results of this > check affects only the Plugin Level 2 (full mail) checks. > And the DKIM-Pre-Check - which is done after the MIME header is received > and if 'DKIMCacheInterval' is enabled. The results (states) of this check > affects most of the header checks and all body and full mail checks. > > If you expect but miss a match for 'DKIMWLAddresses' or 'DKIMNPAddresses' > , you may increase the logging level (ValidateSen
Re: [Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam?
>I'm still seeing scenarios where analyze shows a DKIM NP match, but the message is still going to spam based on score from HMM. The best solution is to correct the corpus, to get better HMM results. You may also decrease the scoring points for HMM and/or Bayesian. If the corpus is corrected and the corpusnorm is ~ 1.0 , 'baysConf' will increase detection correctness. If assp receives a mail it acts as a statemachine. If and how a check is done, depends on the previouse reached states. Using the analyzer, assp acts procedural. Every check is done without any state dependency. This is done, to be able to show every feature match. The analyzer uses the current configuration, hashes, lists and databases. So it may be normal to get different results compared to the real mail processing loggings, if a mail is analyzed. >DKIM NP match: The analyzer checks DKIM without any dependency and shows all results. But, if a mail is received, the DKIM check depends on several previouse states DKIM NP is a resulting state of the DKIM check. So - if any of the previouse (DKIM depdency) states prevents the DKIM check, there will be no DKIM (DKIM NP) result. The mail will be processed the same way, as it was not DKIM signed. Every state, that depends on DKIM NP will not be reached. You should also remember, that assp use two DKIM checks. The full DKIM check, which requires the full mail to be received - the results of this check affects only the Plugin Level 2 (full mail) checks. And the DKIM-Pre-Check - which is done after the MIME header is received and if 'DKIMCacheInterval' is enabled. The results (states) of this check affects most of the header checks and all body and full mail checks. If you expect but miss a match for 'DKIMWLAddresses' or 'DKIMNPAddresses' , you may increase the logging level (ValidateSenderLog, SessionLog, ipmatchLogging, slmatchLogging). Thomas Von:"K Post" An: "ASSP development mailing list" Datum: 15.04.2018 23:21 Betreff: Re: [Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam? I'm still seeing scenarios where analyze shows a DKIM NP match, but the message is still going to spam based on score from HMM. On Mon, Apr 9, 2018 at 12:19 PM, K Post wrote: cheers. On Mon, Apr 9, 2018 at 3:36 AM, Thomas Eckardt wrote: If assp has modified the original mail header (changed foreign X-ASSP- or removed cc,bcc or ...) the DKIM check is skipped and also DKIMNPAddress and DKIMWLAddress. The next version will try to check, if removed or changed headers are protected by a DKIM signature and do the check, if this is not the case. Thomas DISCLAIMER: *** This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! *** -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: *** This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! *** -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam?
I'm still seeing scenarios where analyze shows a DKIM NP match, but the message is still going to spam based on score from HMM. On Mon, Apr 9, 2018 at 12:19 PM, K Post wrote: > cheers. > > On Mon, Apr 9, 2018 at 3:36 AM, Thomas Eckardt > wrote: > >> If assp has modified the original mail header (changed foreign X-ASSP- or >> removed cc,bcc or ...) the DKIM check is skipped and also DKIMNPAddress and >> DKIMWLAddress. >> >> The next version will try to check, if removed or changed headers are >> protected by a DKIM signature and do the check, if this is not the case. >> >> Thomas >> >> >> >> DISCLAIMER: >> *** >> This email and any files transmitted with it may be confidential, legally >> privileged and protected in law and are intended solely for the use of the >> individual to whom it is addressed. >> This email was multiple times scanned for viruses. There should be no >> known virus in this email! >> *** >> >> >> >> -- >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> ___ >> Assp-test mailing list >> Assp-test@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/assp-test >> >> > -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam?
cheers. On Mon, Apr 9, 2018 at 3:36 AM, Thomas Eckardt wrote: > If assp has modified the original mail header (changed foreign X-ASSP- or > removed cc,bcc or ...) the DKIM check is skipped and also DKIMNPAddress and > DKIMWLAddress. > > The next version will try to check, if removed or changed headers are > protected by a DKIM signature and do the check, if this is not the case. > > Thomas > > > > DISCLAIMER: > *** > This email and any files transmitted with it may be confidential, legally > privileged and protected in law and are intended solely for the use of the > individual to whom it is addressed. > This email was multiple times scanned for viruses. There should be no > known virus in this email! > *** > > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Assp-test mailing list > Assp-test@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/assp-test > > -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam?
If assp has modified the original mail header (changed foreign X-ASSP- or removed cc,bcc or ...) the DKIM check is skipped and also DKIMNPAddress and DKIMWLAddress. The next version will try to check, if removed or changed headers are protected by a DKIM signature and do the check, if this is not the case. Thomas DISCLAIMER: *** This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! *** -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam?
Good point Thomas. Here's an example of the log and the analyze output. If you have a second, could you take a peek? This is a from a message that shows a NP DKIM match in analyze but is still rejected as spam. It's an ad for flowers and reads a whole lot like the myriad of spam messages that we actually do want to reject (so I'm not surprised by the HMM hit). This message is from a legitimate source though, and they always DKIM sign, so I just wanted to put it to okmail using DKIMNpAddresses. I can't figure out why ASSP wouldn't be doing DKIMNPAddress for this one. FYI, the other examples I saw of this were all when I had .domainname.com (leading dot) in DKIMNPAddress, but in this example the identity is the root domain and I have @domain.com in the list. Apr-07-18 11:01:36 Connected: session:116EC328 140.X.Y.Z:26515 > A.B.C.10:25 > A.B.C.11:25 Apr-07-18 11:01:37 140.X.Y.Z info: injected STARTTLS request to A.B.C.11 Apr-07-18 11:01:38 08637-54105 140.X.Y.Z < lakjfoaiejglainek5a...@bounce.domain.com> to: myu...@ourcharity.org *DKIM-Signature found* Apr-07-18 11:01:38 Info: enhanced Originated IP detection ignored IP's: 140.X.Y.Z Apr-07-18 11:01:38 08637-54105 140.X.Y.Z < lakjfoaiejglainek5a...@bounce.domain.com> to: myu...@ourcharity.org Received-RWL: from (list.dnswl.org->127.0.15.0,trust=0-[none] (category=Email Marketing Providers);) - high trust is 0-[none] - client-ip=140.X.Y.Z Apr-07-18 11:01:41 08637-54105 140.X.Y.Z < lakjfoaiejglainek5a...@bounce.domain.com> to: myu...@ourcharity.org *HMM Check [scoring] *- Prob: 0.7 - Confidence: 0.00894 => confident.spam - answer/query relation: 23% of 30 Apr-07-18 11:01:41 08637-54105 140.X.Y.Z < lakjfoaiejglainek5a...@bounce.domain.com> to: myu...@ourcharity.org Message-Score: *added 50 for HMM Probability: 0.7, total score for this message is now 50* Apr-07-18 11:01:41 08637-54105 140.X.Y.Z < lakjfoaiejglainek5a...@bounce.domain.com> to: myu...@ourcharity.org deleting spamming safelisted tuplet: (142.0.81.0,bounce.domain.com) age: 4s Apr-07-18 11:01:41 08637-54105 [MessageLimit] 140.X.Y.Z < lakjfoaiejglainek5a...@bounce.domain.com> to: myu...@ourcharity.org *[spam found] (MessageScore 50, limit 50) [*Flowers Today] -> messages/spam/Flowers-Today--2128465.txt; Apr-07-18 11:01:41 08637-54105 140.X.Y.Z < lakjfoaiejglainek5a...@bounce.domain.com> to: myu...@ourcharity.org [SMTP Error] 554 5.7.1 Error: Rejected email - unsolicited [08637-54105 116EC328] Apr-07-18 11:01:41 08637-54105 140.X.Y.Z < lakjfoaiejglainek5a...@bounce.domain.com> to: myu...@ourcharity.org finished message - received DATA size: 51.49 kByte - sent DATA size: 0 Byte Apr-07-18 11:01:41 08637-54105 140.X.Y.Z < lakjfoaiejglainek5a...@bounce.domain.com> to: myu...@ourcharity.org disconnected: session:116EC328 140.X.Y.Z - processing time 5 seconds associated analyze Feature Matching: • DKIM-check returned OK verified-OK for identity 'em...@domain.com' • DKIM-identity match (@domain.com) in DKIMNPAddresses -> noprocessing • SPF-check returned OK for 140.X.Y.Z -> lakjfoaiejglainek5a...@bounce.domain.com, smtp.some-listserv.net • Received-SPF: pass (bounce.domain.com: Sender is authorized to use ' lakjfoaiejglainek5a...@bounce.domain.com' in 'mfrom' identity (mechanism 'include:senderdomain.com' matched)) receiver=assp.ourcharity.org; identity=mailfrom; envelope-from="lakjfoaiejglainek5a...@bounce.domain.com"; helo=smtp.some-listserv.net; client-ip=140.X.Y.Z • URIBL check: 'OK' • Known Good HELO: 'smtp.some-listserv.net' • Valid Format of HELO: 'smtp.some-listserv.net' • IP in Helo check: 'OK' • AUTH would be disabled • RBLCacheCheck returned OK for 140.X.Y.Z: inserted as ok at 2018-04-08 10:31:00 • domain domain.com (in From) has a valid MX record: domain-com.mail.protection.outlook.com • domainMX domain-com.mail.protection.outlook.com has a valid A record: 216.32.x.y • domain bounce.domain.com (in Mail From: , Errors-To , List-Unsubscribe) has a valid MX record: bounce.some-listserv.net • domainMX bounce.some-listserv.net has a valid A record: 145.x.y.z• • 140.X.Y.Z PTR record via DNS: status=PTR OK - smtp.some-listserv.net • 140.X.Y.Z is in RWLCache: status=not listed • 140.X.Y.Z SenderBase: status=not classified, data=[CN=US, ORG=Some Listerv] Feature Matching Log: Apr-08-18 11:51:38 Info: analyze detected: IP: '140.X.Y.Z' , HELO: ' smtp.some-listserv.net' , assp-Host: 'assp.ourcharity.org' Apr-08-18 11:51:38 Info: enhanced Originated IP detection ignored IP's: 140.X.Y.Z Apr-08-18 11:51:39 Info: found DKIM signature identity 'em...@domain.com' Apr-08-18 11:51:39 em...@domain.com em...@domain.com,myu...@ourcharity.org matches @domain.com in DKIMNPAddresses Apr-08-18 11:51:39 [scoring] DKIM signature verified-OK - pass - identity is: em...@domain.com - sender policy is: accept - author policy is: accept - state changed to: noprocessing Apr-08-18 11:51:42 Info: analyzing attachments in incoming email On Sun, Apr 8, 2018 at 3:21 AM, Thomas Eckardt wrote: > >Analyze shows:
Re: [Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam?
>Analyze shows: analyzer shows every feature match - but at runtime the DKIM check may be skipped for several reasons. The maillog.txt for the mail should show what happens. Thomas Von:"K Post" An: "ASSP development mailing list" Datum: 07.04.2018 20:14 Betreff:[Assp-test] Analyze shows DKIMNPAddress match as expected, but some messages still processed as spam? I have several listings in DKIMNPAddresses like: .domain.org (with the leading dot) to allow a DKIM identity @.domain.org to be tagged as no processing, but not just @domain.org However, I've seen several examples where the mail is still flagged as spam (due hitting a limit, often no MX, no A and somewhat spammy content) even though the DKIM signature verifies. Analyze shows: • DKIM-check returned OK verified-OK for identity '@reply.domain.org' • DKIM-identity match (.domain.org) in DKIMNPAddresses -> noprocessing Shouldn't this no processing flag just let the mail through? Maybe I'm not understanding? Overall DKIMNPAddresses is working beautifully and is a wonderful addition. Thanks. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: *** This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! *** -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
