Hi Matt, I'm not sure that the issue is attachments. There is nothing wrong with attachments using Unix/Apple linefeeds. But the RFCs for SMTP (and similar protocols) all require that each header line must end with CRLF and each header line ends with CRLFCRLF.
Anything else is not a legitimate SMTP mailer - but rather a poor attempt in faking one. We agree that in the headers CR only or LF only should be "treated" by Declude as if they were CRLF (and detected as header violations, nevertheless). I also have no problem if Declude were to "fix" those linefees to proper CRLF if it so desires. Best Regards Andy Schmidt Phone: +1 201 934-3414 x20 (Business) Fax: +1 201 934-9206 -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Monday, October 23, 2006 12:27 AM To: declude.junkmail@declude.com Subject: Re: [Declude.JunkMail] RE: On RFC Violation - Declude allows attachments and Virus to pass through untouched and unscanned Unfortunately, treating everything with only CR's or only LF's as a virus/vulnerability is not perfectly accurate. I definitely see legitimate E-mail coming this way. I believe the best solution is to just simply treat either CR, LF, or CRLF as line breaks when parsing the message in Declude for virus scanning, header modification, etc. Linux uses LF-only by default for line breaks in text files, and this is why this is common in both spamware and in homegrown E-mail scripts (which may well be legitimate). In fact, if someone attaches a text file created in Linux to an E-mail message, it should come through with a LF-only pattern in the text attachment, and that certainly isn't a virus. CR-only is used by at least old Macs (OSX is now essentially BSD so it's LF-only now), and I haven't seen anyone use this format for constructing E-mail, and I have my doubts about whether it is supported by E-mail clients if it was used in the MIME headers, but of course attachments could come as CR-only and be fully legitimate, though not necessarily displayable as anything but a single line in non-old Mac E-mail clients. Detecting and correcting for this definitely seems easy to do. In VBScript, this is how I approach it with Linux format (LF-only): ' Correct for non-CRLF linebreaks. regEx.Global = True regEx.IgnoreCase = False regEx.Pattern = "[^\r]\n" ' If regEx.Test(strMessage) Then boolTestBadCrLf = True strMessage = Replace(strMessage, vbLf, vbCrLf) strMessage = Replace(strMessage, vbCr & vbCrLf, vbCrLf) End If There are fancier ways to do the replacement as well, but this seemed appropriate for VBScript, and it only kicks in when the pattern is found, and that is rare. This also only affects what is in memory and not the file. It has been stated in the past that Declude tried resolving this much earlier when it was pointed out that the headers were being written to the end of the body. I don't see why this would be the case if a simple check and replace was done immediately after reading in the file's contents, and before all tests and before writing the headers. I do wish Declude would respond to this because it does concern me now that there is a parsing error that leads to huge vulnerability in Declude, and while there is currently a method of blocking such messages in Declude with a vulnerability switch, I know that this is not a universally accurate method, and I fear that it could tag things such as Linux style text attachments. Matt Andy Schmidt wrote: > Hi, > > Well, the necessary logic seems absolutely simple: > > A) parse token, up to EITHER CR or LF > > B) if "CR" found, see if followed by LF, skip over CF/LF and start new > line > > C) if only "CR" found, and not followed by LF, skip over CR and start > new line. Set SMTP Header violation flag. > > D) if "LF" found, skip over LF and start new line. > Set SMTP Header violation flag. > > The key is to detect an "apparent" end of headers - no matter if it's > a proper CRLFCRLF sequence, or if it's only a CRCR or LFLF or CRLFLF > combination so that spam can be recognized as such, virus can't be > hidden in none-standard line feeds and Declude doesn't attach headers > behind the body of the message. > > I'm certain that we are looking at one cause why obvious spam is > currently slipping past Declude in the current releases. > > Best Regards > Andy Schmidt > > Phone: +1 201 934-3414 x20 (Business) > Fax: +1 201 934-9206 > > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of > Matt > Sent: Friday, October 20, 2006 05:35 PM > To: declude.junkmail@declude.com > Subject: Re: SPAM-WARN: Re: [Possible Spam][Declude.JunkMail] On RFC > Violation - Declude allows attachments and Virus to pass through untouched > and unscanned > > I think that the real issue is that if Declude just simply understood > CR patterns, the messages could be properly parsed, and headers could > be properly inserted. > > There is a similar and longstanding issue with Declude failing to > decode base64 encoding in very long lines (i.e. +1000 or so characters > long). This also leads to failing to scan the attachments. One virus > has been spreading for the last year and a half with this symptom, and > the only reason why my Declude virus stops it is because it always > sends as an EXE, but this causes backscatter. If this was changed to > do something like a ZIP file, which most of us don't block, it could > result in viruses passing straight through Declude. > > While Declude does have a new option to treat CR patterns as a > vulnerability, this isn't something that I can use since there is > unfortunately a steady, yet small, stream of legitimate messages that > have this flaw. The flaw itself doesn't represent a vulnerability, in > fact this wouldn't be a vulnerability at all if Declude understood how > to parse the messages. I can't see how fixing this flaw, or the > base64 long line flaw would take them any more than a couple of hours. > I have a plug-in that I wrote for Declude that is fully capable of > understanding CR patterns as well as long base64 code without issue. > > Matt > > > > Michael Thomas - Mathbox wrote: > > David, > > In my opinion, which others may not share, Declude should detect all > RFC/MIME violations and flag them in some manner. There exist quite a > few > that are common to spam messages, but not flagged by Declude. > However, that > is a totally different subject than the point of my test. The RFC > violation > was simply a symptom. Some admins might not choose to delete or block > messages with that construction. There exist well-known web sites > that > generate response email messages where one or two lines out 50-100 > are > missing one of the Cr/Lf pair characters. > > The point is that for many (maybe most) people on this list, Declude > is the > lock and keys for securing email passing through their mail servers. > Declude > trusts that the email message will be well-formed. Because of that > mis-placed trust, Declude did not reliably detect any attachment, > regardless > of type, and therefore did not invoke the scanners. Note that this > test was > performed on Declude version 3.1.1. Maybe, the new gateway product is > not > quite so trusting. I have no idea. We are an ISP and would not pay > the fees > associated with ISP use of the gateway product. > > In any event, this is a well-known issue and has been for some time. > I > reported this issue to Declude and the list some time ago regarding > spam not > being scanned because of this issue. At that time, I was so focused > on the > spam problem, I did not think about the attachment/virus side of the > issue, > which should have been obvious to me. > > Michael Thomas > Mathbox > 978-683-6718 > 1-877-MATHBOX (Toll Free) > > > > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On > Behalf Of David Sullivan > Sent: Friday, October 20, 2006 3:51 PM > To: declude.junkmail@declude.com > Subject: SPAM-WARN: Re: [Possible Spam][Declude.JunkMail] On > > RFC Violation - Declude allows attachments and Virus to pass > > through untouched and unscanned > > Hello Michael, > > Thanks for the great research. > > Wouldn't this be the purpose of > Vulnerability detection in Declude? "Declude detects mal-formed > messages that can allow viruses to be hidden from email > server virus > scanners." > > We treat all vulnerabilities as viruses, send the notice and 86 the > message. > > -David > > > > Thursday, October 19, 2006, 10:52:25 PM, you wrote: > > MTM> Hi All, > > MTM> Well, when responding on declude.junkmail@declude.com to > Will about RFC > MTM> violations, I said I would test this and I did. > MTM> -------------------- > MTM> While writing this message, I happened to think about > attachments. It would > MTM> appear to me, that there is an implied possibility for > attachments and > MTM> therefore viruses to pass through undetected. All that > should be required is > MTM> that the lines that make up the entire email, including > > the attachment > MTM> section, be terminated with line feeds instead of > carriage return/line feed > MTM> pairs. Under such condition, Declude would see only one > > line and not find > MTM> the relevant sections. I will test this possibility. > MTM> -------------------- > > MTM> Tested: Declude v3.1.1 for IMail > > MTM> As it happens, my suspicions were accurate. I wrote a > script that could be > MTM> modified to remove either the carriage-returns or the > line-feeds from a > MTM> message file. I then created a message in Outlook > Express, added an > MTM> executable file (uptime.exe) as an attachment and saved > > it in my Draft > MTM> folder. I then dragged that message to the same location > as the script and > MTM> renamed it to match the file name in the script > (Rfc.eml) I ran the script, > MTM> which stripped the carriage-returns and produced > Rfc2.eml. I renamed > MTM> Rfc2.eml to RfcNoCr.eml. In the script, I then changed > vbCr to vbLf and ran > MTM> it again, which stripped the line-feeds and produced > Rfc2.eml. I renamed > MTM> Rfc2.eml to RfcNoLf.eml. > > MTM> Now, to get IIS SMTP to actually process the file, you > must edit each file > MTM> and remove the single Cr or Lf and press the Enter Key, > > producing a CrLf > MTM> pair after the To field and the From field. I also added > the string "No Cr" > MTM> to the end of the subject of RfcNoCr.eml and added No > Lf > to the subject of > MTM> RfcNoLf.eml. So for example change: > MTM> -------------------- > MTM> From: "Michael Thomas - Mathbox" <[EMAIL PROTECTED]> > <mailto:[EMAIL PROTECTED]> [Cr]To: > MTM> "[EMAIL PROTECTED]" <mailto:[EMAIL PROTECTED]> > [Cr]Subject: Test Attachment > Pass-Through on RFC > MTM> Violation[Cr]<line continues> > MTM> -------------------- > MTM> Change To > MTM> -------------------- > MTM> From: "Michael Thomas - Mathbox" <[EMAIL PROTECTED]> > <mailto:[EMAIL PROTECTED]> > MTM> To: "[EMAIL PROTECTED]" <mailto:[EMAIL PROTECTED]> > MTM> Subject: Test Attachment Pass-Through on RFC Violation > No Cr[Cr]<line > continues>> > MTM> -------------------- > > MTM> Now it so happens, a long time ago, I wrote a couple of > > tests to detect > MTM> these RFC violations, so first I had to disable them in > > my GLOBAL.CFG, which > MTM> I did by commenting them out. Note that I also BAN the > .EXE extension and I > MTM> left that enabled. > > MTM> Now copy and paste the two files into the pickup > directory of your favorite > MTM> IIS SMTP pickup directory. Viola, you just passed an > executable through > MTM> Declude and through your mail server. That executable > could very well have > MTM> been a virus. > > MTM> Note that Declude detected RfcNoLf.eml as [Outlook 'CR' > > Vulnerability]. Ok > MTM> good. > > MTM> But Declude let RfcNoCr.eml pass straight through > without calling the virus > MTM> scanners, because Declude did NOT see an attachment. > Also, because Declude > MTM> did not see an attachment, Declude did not ban the .EXE > > extension. > > MTM> Here are the log entries from RfcNoLf.eml > MTM> -------------------- > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Scanning > Time: 218ms > MTM> [kernel=31 user=187] > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Virus > scanner 1 reports exit > MTM> code of 0 > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Virus > detected. Not continuing > MTM> with remaining scanners. > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd 0: > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Starting > EXT check . > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd > MTM> C:\IMAIL\spool\proc\work\D1b2101b7000083ba.vir\*.* > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd 0 > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Deleted > MTM> C:\IMAIL\spool\proc\work\D1b2101b7000083ba.vir\0. > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd report.txt > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Deleted > MTM> > C:\IMAIL\spool\proc\work\D1b2101b7000083ba.vir\report.txt. > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd > han=13e9c0 b=False > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd File(s) > are INFECTED [[Outlook > MTM> 'CR' Vulnerability]: 0] > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd High > code=23. > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd AV > returned 23 > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Scanned: > CONTAINS A VIRUS > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd From: > [EMAIL PROTECTED] To: > MTM> [EMAIL PROTECTED] [incoming from XX.XXX.XXX.X] > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Subject: > Test Attachment > MTM> Pass-Through on RFC Violation No Lf > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Skipping > non-AV E-mail > MTM> BANnotify.eml > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd > MTM> C:\IMAIL\Declude\postmaster.eml > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd Starting > E-mail file > MTM> C:\IMAIL\Declude\postmaster.eml > MTM> 10/19/2006 20:41:23.471 q1b2101b7000083ba.smd > C:\IMAIL\IMail1.exe -h > MTM> "mathbox.com" -t "[EMAIL PROTECTED]" > <mailto:[EMAIL PROTECTED]> -u > "[EMAIL PROTECTED]" <mailto:[EMAIL PROTECTED]> -s > MTM> "Mathbox Email Virus Scanning detected and quarantined > a > virus" -f > MTM> "C:\IMAIL\spool\proc\work\D1b2101b7000083ba.sm0" > MTM> 10/19/2006 20:41:23.487 q1b2101b7000083ba.smd TempName > = > MTM> C:\IMAIL\Declude\postmaster.eml > MTM> -------------------- > > MTM> Here are the log entries from RfcNoCr.eml > MTM> -------------------- > MTM> 10/19/2006 20:41:10.690 q1b2101da000083bb.smd Setting > Scan File 1 to > MTM> C:\Progra~1\FSI\F-Prot\FPcmd.exe /TYPE /SILENT /SERVER > /NOMEM /ARCHIVE > MTM> /NOBOOT /DUMB /REPORT=report.txt. > MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd CFG: > Setting report parse 1 to > MTM> Infection. > MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Setting > Scan File 2 to > MTM> C:\imail\declude\runclamscan.exe log=3 > MTM> C:\clamav-devel\bin\clamscan.exe > MTM> --quiet --no-summary --tempdir=c:\tmp\ > MTM> --database=C:\clamav-devel\share\clamav\ --max-ratio=0 > --mbox -l report.txt. > MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd CFG: > Setting report parse 2 to > MTM> FOUND. > MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Setting > virus directory to: > MTM> C:\IMAIL\spool\virus > MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Incoming > E-mail scanning > MTM> turned ON > MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Outgoing > E-mail scanning > MTM> turned ON > MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Setting > AVAFTERJM to ON. > MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Setting > MAXATONCE to 20. > MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Setting > scanner timeout to 120 > MTM> seconds > MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Setting > AUTOFORGE to OFF. > MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Scanner 0 > > Virus Codes: 3 6 8 9 > MTM> 10 . OK Codes: > MTM> 10/19/2006 20:41:10.721 q1b2101da000083bb.smd Scanner 1 > > Virus Codes: 1 . OK > MTM> Codes: > MTM> 10/19/2006 20:41:10.908 q1b2101da000083bb.smd Skip > Extensions: GIF TXT MPG > MTM> PNG > MTM> 10/19/2006 20:41:10.955 q1b2101da000083bb.smd 48 Ban > Extensions: ADE ADP ASD > MTM> ASP BAS BAT BIN CAB CHM CMD COM CPL CRT DLL EXE HLP HTA > > HTO INF INS ISP JS > MTM> JSC JSE KSH LNK MDB MDE MSI OCX PCD PIF REG SCF SCR SCT > > SHB SHS SYS VB VBE > MTM> VBS VBX VSMACROS VXD WSC WSF WSH > MTM> 10/19/2006 20:41:11.002 q1b2101da000083bb.smd Virus Pro > > Registered > MTM> 10/19/2006 20:41:11.018 q1b2101da000083bb.smd Starting > locality check > MTM> (sender=mathbox.com; nr=1 ca=off). nHas=1. > MTM> 10/19/2006 20:41:11.018 q1b2101da000083bb.smd > [EMAIL PROTECTED] [0-0] is > MTM> local domain1 viaFM > MTM> 10/19/2006 20:41:11.018 q1b2101da000083bb.smd Ending > locality check > MTM> (cached), sender=local. > MTM> 10/19/2006 20:41:11.018 q1b2101da000083bb.smd Local host > = mathbox.com > MTM> 10/19/2006 20:41:11.018 q1b2101da000083bb.smd > MTM> [EMAIL PROTECTED] Offset=5 > MTM> Flags=1 > MTM> 10/19/2006 20:41:11.033 q1b2101da000083bb.smd Msgid: > MTM> 10/19/2006 20:41:11.049 q1b2101da000083bb.smd Subject: > Test Attachment > MTM> Pass-Through on RFC Violation No Cr > MTM> -------------------- > > > MTM> Here is the script to strip Cr or Lf, just change the > vbCr below to vbLf. > MTM> Just save it as: > MTM> Rfc.vbs > MTM> -------------------- > MTM> Dim InFile > MTM> Dim OutFile > MTM> Dim Fso, File > MTM> Dim AllText > > MTM> InFile = "Rfc.eml" > MTM> OutFile = "Rfc2.eml" > > MTM> Set Fso = CreateObject("Scripting.FileSystemObject") > MTM> If Fso.FileExists( InFile ) = True Then > MTM> Set File = Fso.OpenTextFile( InFile, 1, False, 0 > ) > MTM> AllText = File.ReadAll > MTM> File.Close > MTM> Set File = Nothing > MTM> AllText = Replace( AllText, vbCr, "" ) > MTM> Set File = Fso.OpenTextFile( OutFile, 2, True, 0 > ) > MTM> File.Write AllText > MTM> File.Close > MTM> Set File = Nothing > MTM> End If > MTM> Set Fso = Nothing > MTM> -------------------- > > > MTM> Finally, if you want to test for these RFC violations, see > MTM> http://www.mathbox.com/NoCrTest/NoCrTest.zip > > > MTM> Michael Thomas > MTM> Mathbox > MTM> 978-683-6718 > MTM> 1-877-MATHBOX (Toll Free) > > > > > > MTM> --- > MTM> This E-mail came from the Declude.JunkMail mailing list. To > MTM> unsubscribe, just send an E-mail to > [EMAIL PROTECTED], and > MTM> type "unsubscribe Declude.JunkMail". The archives can > be found > MTM> at http://www.mail-archive.com. > > > > > -- > Best regards, > David mailto:[EMAIL PROTECTED] > > > > > --- > This E-mail came from the Declude.JunkMail mailing list. To > unsubscribe, just send an E-mail to [EMAIL PROTECTED], and > type "unsubscribe Declude.JunkMail". The archives can be > found > at http://www.mail-archive.com. > > > > > > > > > > --- > This E-mail came from the Declude.JunkMail mailing list. To > unsubscribe, just send an E-mail to [EMAIL PROTECTED], and > type "unsubscribe Declude.JunkMail". The archives can be found > at http://www.mail-archive.com. > > > > > > > --- > This E-mail came from the Declude.JunkMail mailing list. To > unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type > "unsubscribe Declude.JunkMail". The archives can be found at > http://www.mail-archive.com. > > > > > --- > This E-mail came from the Declude.JunkMail mailing list. To > unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type > "unsubscribe Declude.JunkMail". The archives can be found at > http://www.mail-archive.com. > > > > --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
