OK - I think I got it. Build 20037 will fix the problem.
Thomas Von: "K Post" <nntp.p...@gmail.com> An: "ASSP development mailing list" <assp-test@lists.sourceforge.net> Datum: 06.02.2020 02:31 Betreff: Re: [Assp-test] Missing MX, A, and FROM for specific sender. Unicode problem? Thanks Thomas, Here's a sample output from the maillog that I found. This is before I whitelisted the DKIM sig. Sometimes they sent through SparkPost, sometimes Amazon AWS. Missing FROM, even though it's listed Malformed reply to because it's quoted printable Here's one through SparkPost Mail log: Jan-14-20 11:11:44 34543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org malformed address: found in - Reply-To:=?utf-8?q?newsletter=40surveymonkey=2Ecom?= Jan-14-20 11:11:44 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org Message-Score: added 50 (nofromValencePB) for From-missing, total score for this message is now 50 Jan-14-20 11:11:44 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org DKIM-Signature found Jan-14-20 11:11:48 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org info: found DKIM signature identity '@t.outbound.surveymonkey.com' Jan-14-20 11:11:48 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org [scoring] DKIM signature verified-OK - header-passed - identity is: @ t.outbound.surveymonkey.com - sender policy is: neutral - author policy is: neutral Jan-14-20 11:11:52 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org whiteHost Regex: whiteSenderBaseRE 'surveymonkey.com' Jan-14-20 11:11:52 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org Message-Score: added -35 for White Host 'surveymonkey.com', total score for this message is now 15 Jan-14-20 11:11:52 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org [scoring] SenderBase -- White Host 'surveymonkey.com' Jan-14-20 11:11:52 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org Message-Score: added 10 for DNSBL: neutral, 52.40.63.21 listed in dnsbl-3.uceprotect.net, total score for this message is now 25 Jan-14-20 11:11:52 msg33543-22560 [DNSBL] 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org [scoring] DNSBL: neutral, 52.40.63.21 listed in (dnsbl-3.uceprotect.net <-127.0.0.2) Jan-14-20 11:11:52 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org checking MX/A for t.outbound.surveymonkey.com Jan-14-20 11:11:53 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org t.outbound.surveymonkey.com - MX 'surveymonkey.mx.e.sparkpost.com' - got IP (52.25.164.16) Jan-14-20 11:11:53 msg33543-22560 [MissingMX] 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org [[scoring]] MX missing: =?utf-8?q?newsletter=40surveymonkey=2ecom?= (Reply-To) Jan-14-20 11:11:53 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org Message-Score: added 8 (mxValencePB) for MX missing: =?utf-8?q?newsletter=40surveymonkey=2ecom?= (Reply-To), total score for this message is now 33 Jan-14-20 11:11:53 msg33543-22560 [MissingMXA] 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org [[scoring]] A record missing for MX: =?utf-8?q?newsletter=40surveymonkey=2ecom?= (Reply-To) Jan-14-20 11:11:53 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org deleting spamming safelisted tuplet: (52.40.63.0, t.outbound.surveymonkey.com) age: 10s Jan-14-20 11:11:53 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org Message-Score: added 15 (mxaValencePB) for A record missing for MX: =?utf-8?q?newsletter=40surveymonkey=2ecom?= (Reply-To), total score for this message is now 48 Jan-14-20 11:11:53 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org MX found: t.outbound.surveymonkey.com (Mail From: , From) -> surveymonkey.mx.e.sparkpost.com Jan-14-20 11:11:53 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org A record found for MX: t.outbound.surveymonkey.com (Mail From: , From) -> 52.25.164.16 Jan-14-20 11:11:53 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org HMM-Check has given less than 6 results - using monitoring mode only Jan-14-20 11:11:53 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org Bayesian Check [scoring] - Prob: 0.00000 - Confidence: 0.00000 => doubtful.ham - answer/query relation: 57% of 28 Jan-14-20 11:11:53 msg33543-22560 [MessageLimit][lowlimit] 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org [spam found] and possibly passing because messagescore(48) low [Your first survey response] -> messages/discarded/Your_first_survey_response--3831526.txt Jan-14-20 11:11:53 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org spam found and passing [Your first survey response] -> messages/discarded/Your_first_survey_response--3831526.txt Jan-14-20 11:11:53 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org [Plugin] calling plugin ASSP_AFC Jan-14-20 11:12:00 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org info: PB-IP-Score for '52.40.63.0' is 49, added 49 in this session Jan-14-20 11:12:00 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org finished message - received DATA size: 37.68 kByte - sent DATA size: 39.68 kByte Jan-14-20 11:12:00 msg33543-22560 52.40.63.21 < survey-nore...@t.outbound.surveymonkey.com> to: ouru...@ourcharity.org disconnected: session:4D4A64D8 52.40.63.21 - processing time 18 seconds And the headers from this one: X-ASSP-Message-Score: 50 (From-missing) X-ASSP-IP-Score: 50 (From-missing) X-ASSP-DKIMidentity: @t.outbound.surveymonkey.com X-Original-Authentication-Results: OurCharity.org; dkim=pass; spf=pass X-ASSP-Re-whiteSenderBaseRE: surveymonkey.com X-ASSP-Message-Score: -35 (White Host 'surveymonkey.com') X-ASSP-IP-Score: -34 (White Host 'surveymonkey.com') X-ASSP-Message-Score: 10 (DNSBL: neutral, 52.40.63.21 listed in dnsbl-3.uceprotect.net) X-ASSP-IP-Score: 10 (DNSBL: neutral, 52.40.63.21 listed in dnsbl-3.uceprotect.net) X-ASSP-DNSBL: neutral, 52.40.63.21 listed in (dnsbl-3.uceprotect.net <-127.0.0.2) X-ASSP-Message-Score: 8 (MX missing: =?utf-8?q?newsletter=40surveymonkey=2ecom?= (Reply-To)) X-ASSP-IP-Score: 8 (MX missing: =?utf-8?q?newsletter=40surveymonkey=2ecom?= (Reply-To)) X-ASSP-Message-Score: 15 (A record missing for MX: =?utf-8?q?newsletter=40surveymonkey=2ecom?= (Reply-To)) X-ASSP-IP-Score: 15 (A record missing for MX: =?utf-8?q?newsletter=40surveymonkey=2ecom?= (Reply-To)) X-ASSP-Detected-URI: surveymonkey.com(6), t.outbound.surveymonkey.com(1) X-ASSP-Tag: MessageLimit X-ASSP-Spam-Reason: MessageScore passed low limit X-ASSP-Message-Totalscore: 48 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;d= t.outbound.surveymonkey.com; s=scph; t=1579032308;i=@ t.outbound.surveymonkey.com; Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8" Authentication-Results: aa.mta2vsmtp.cc.prd.sparkpost smtp.user=<hidden>; auth=pass (PLAIN) Received: from [64.191.16.134] ([64.191.16.134:47402] helo=n9emlsvc110mgp1.n9.jungle.tech-event_subscriber_process) by aa.mta2vsmtp.cc.prd.sparkpost (envelope-from < survey-nore...@t.outbound.surveymonkey.com>) (ecelerity 4.3.1.69416 r(Core:4.3.1.4)) with ESMTPSA (cipher=AES-256-GCM) id 17/C3-02945-4FE1E1E5; Tue, 14 Jan 2020 15:05:08 +0000 Message-ID: <17.c3.02945.4fe1e...@aa.mta2vsmtp.cc.prd.sparkpost> MIME-Version: 1.0 From: SurveyMonkey <surveymon...@t.outbound.surveymonkey.com> To: ouru...@ourcharity.org Subject: =?utf-8?q?Your_first_survey_response!?= Date: Tue, 14 Jan 2020 15:05:08 +0000 Reply-To: =?utf-8?q?newsletter=40surveymonkey=2Ecom?= On Wed, Feb 5, 2020 at 2:53 AM Thomas Eckardt <thomas.ecka...@thockar.com> wrote: >FROM isn't being detected >From where you got this information? The real reason is only shown in the maillog.txt. I'm just trying to fix the issue for MX/A. Thomas Von: "K Post" <nntp.p...@gmail.com> An: "ASSP development mailing list" < assp-test@lists.sourceforge.net> Datum: 03.02.2020 21:53 Betreff: Re: [Assp-test] Missing MX, A, and FROM for specific sender. Unicode problem? I was able to get a hold of SurveyMonkey. I gave them examples of their quoted printable reply-to and to my surprise, they replied, and quickly! Here's the reply: My team let me know that we use UTF-8 encoding for our headers and that this can be fixed within the setup for your Exchange server. They also confirmed that we've used UTF-8 to encode our headers for awhile now so this isn't a new formatting on our end. I'm not sure what to reply with. Thomas, are you saying that Reply-To: =?utf-8?q?no-reply=40surveymonkey=2Ecom?= is not UTF-8 encoded correctly? I don't understand what is invalid with what Surveymonkey is doing here. I feel like I have the ear of the team there who can fix this, but I need to tell them clearly what they are doing wrong. You wrote previously "This is no unicode (or better ASCII in UTF-8 - which is the same). This is a quoted printable encoded email address, which is (and should not) interpreted as such one." but what specifically isn't allowed? It's a quoted printable email address which IS interpreted by ASSP as an address, but it should not be interpreted by ASSP as an email address? It seems to me that ASSP isn't interpreting this as an email address, but it should be. They're pointing the finger at you, you're saying it's them. I believe YOU are correct, but I don't know what to tell them next... Thanks for the help. On Mon, Feb 3, 2020 at 12:53 PM K Post <nntp.p...@gmail.com> wrote: Also, looking at my first post on this thread, FROM isn't being detected according to the message but the FROM line is in the header without funky formatting. Can you tell from the header I included why FROM is considered to be missing? The last one scored poorly because of a missing from, missing MX, and missing a record, but it actually had all of those things : X-ASSP-Message-Score: 50 (From-missing) X-ASSP-IP-Score: 50 (From-missing) X-ASSP-Message-Score: 8 (MX missing: =?utf-8?q?no-reply=40surveymonkey=2ecom?= (Reply-To)) X-ASSP-IP-Score: 8 (MX missing: =?utf-8?q?no-reply=40surveymonkey=2ecom?= (Reply-To)) X-ASSP-Message-Score: 15 (A record missing for MX: =?utf-8?q?no-reply=40surveymonkey=2ecom?= (Reply-To)) X-ASSP-IP-Score: 15 (A record missing for MX: =?utf-8?q?no-reply=40surveymonkey=2ecom?= (Reply-To)) interesting lines in the header: From: SurveyMonkey <surveymon...@t.outbound.surveymonkey.com> Subject: =?utf-8?q?New_login_alert?= Reply-To: =?utf-8?q?no-reply=40surveymonkey=2Ecom?= On Mon, Feb 3, 2020 at 12:44 PM K Post <nntp.p...@gmail.com> wrote: Okay. Is there an alternative so I can set it as we find these messages to specifically ignore these sender errors while still checking the overwhelming majority of messages that properly format their headers? Do I need to no-processing the entire message based on the IP or something? Is there a better way? Thanks On Mon, Feb 3, 2020 at 1:40 AM Thomas Eckardt <thomas.ecka...@thockar.com> wrote: >I believe =?utf-8?q?no-reply=40surveymonkey=2ecom?= is unicode for < no-re...@surveymonkey.com> or is that bad unicode? This is no unicode (or better ASCII in UTF-8 - which is the same). This is a quoted printable encoded email address, which is (and should not) interpreted as such one. ASSP does not allow (and removes any such EHLO-answer offer) 8-bit MIME headers. ... Also note that messages in this format require the use of the SMTPUTF8 extension [RFC6531] to be transferred via SMTP. ... Thomas Von: "K Post" <nntp.p...@gmail.com> An: "ASSP development mailing list" < assp-test@lists.sourceforge.net> Datum: 02.02.2020 18:49 Betreff: Re: [Assp-test] Missing MX, A, and FROM for specific sender. Unicode problem? You're correct in that RFC1342, a proposed standard (from 1992!!) does say: ...an encoded-word MUST NOT appear in any portion of an "address". However, RFC6532 https://tools.ietf.org/html/rfc6532, also just a proposed standard but from 2012, 20 years newer than 1342, and one that seems to have a lot of senders providers relying on it says: This document specifies an enhancement to the Internet Message Format and to MIME that allows use of Unicode in mail addresses and most header field content. So which proposed standard do you adhere to? It seems like if there's a more lenient one, or more feature full one, that's much newer and that people are using, that we should at least give that some real consideration. If there's enough senders putting addresses in unicode format, and ASSP obviously already knows how to decode them, is there any downside to having ASSP allow unicode in addresses and decode it? I've not seem spammers doing this, and even if they did try to obscure addresses in unicode, ASSP will still do its thing and check the discovered addresses the same way it would if they had not. I really don't know why senders are doing this, but they are, and it's mail we need to get through. The big one for us is SurveyMonkey, something that our staff relies heavily on, but there are others too. What do you think Thomas? If I'm being logical, is there any hope of getting this changed/enhanced? Thanks On Sat, Feb 1, 2020 at 8:09 AM Thomas Eckardt <thomas.ecka...@thockar.com> wrote: an email header field may contain encoded unicode - in commends how ever, if an email header field is used - it has to contain a valid email address - unicode is not allowed to be used in email addresses valid examples: reply-to: "any encoded unicode" < valid@email.address> reply-to: < valid@email.address> invalid example: reply-to: "any encoded unicode" Thomas Von: "K Post" <nntp.p...@gmail.com> An: "ASSP development mailing list" < assp-test@lists.sourceforge.net> Datum: 31.01.2020 17:30 Betreff: Re: [Assp-test] Missing MX, A, and FROM for specific sender. Unicode problem? I knew that unicode it was common in the subject, but not from/reply-to. Apparently it's legal in for all headers. https://tools.ietf.org/html/rfc1342 And according to the ever questionable wikipedia, https://en.wikipedia.org/wiki/Unicode_and_email : RFC 2047 provides support for encoding non-ASCII values such as real names and subject lines in email header[5] RFC 6532 allows the use of UTF-8 in a mail header section [7] My gut says that FROM/Reply-To (I don't know about the envelope itself) would need to be checked to see if they're unicode and converted prior to having the email address extracted from those lines and run through checks like MX and A, etc. What do we all think? I don't know if ASSP is already handling unicode in FROM and Reply-To and something's wrong with the formatting in my sample header above, or if ASSP doesn't accept UTF-8 encoded FROM/Reply-To. If it's the later, do you think we should ask Thomas to look into it? On Fri, Jan 31, 2020 at 10:51 AM Robert K Coffman Jr. -Info From Data Corp. <bcoff...@infofromdata.com> wrote: Ken, I can confirm I am seeing this also. I haven't had any complaints (I vaguely recollect way way back in ASSP time I might have had an issue with Survey Monkey) so I have taken no action on it. - Bob On 1/31/2020 10:26 AM, K Post wrote: > Interesting idea Doug. Do any of your users happen to get any > SurveyMonkey notifications? These are sent to the owners of surveys. > I'm curious if you're seeing the same malformed info in the headers. > Thanks > ken > > > On Thu, Jan 30, 2020 at 12:56 PM Doug Lytle <supp...@drdos.info > <mailto:supp...@drdos.info>> wrote: > > This is not a necessarily resolution, but possibly a workaround for you. > > In a past life, I've had some mail servers that just caused more > issues then they were worth, so I ended up identifying their mail > server(S) range of IP Addresses and placed those in an alias on the > firewall and did a NAT directly to the mail server instead of ASSP > If they were destined for port 25. > > Doug > > > _______________________________________________ > Assp-test mailing list > Assp-test@lists.sourceforge.net <mailto: Assp-test@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > _______________________________________________ > Assp-test mailing list > Assp-test@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/assp-test > _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ******************************************************* _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ******************************************************* _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ******************************************************* _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! *******************************************************
_______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
