Ahem, like I said, the attachment.
Andrew ;)
-----Original Message-----
From: Colbeck, Andrew
Sent: Monday, February 23, 2004 12:58 PM
To: '[EMAIL PROTECTED]'
Subject: RE: [Declude.JunkMail] Feature-itis
Matt, it's not that I particularly want the log files to be neatly ordered
instead of interleaved. Although it would be nice, I'm used to identifying
the Q... number first and then filtering out the one(s) I want for
examination.
My bugbear is that individual lines are garbled, the start of new log lines
appearing in the middle of other log lines. See the accompanying snippet
for a sample. I have not added or removed any CR/LF from that sample.
Back in the stone age, I implemented a very similar logging system that was
also multi-process, and locking was feared to be too much of a drain. For
readability, I wanted the log to not be interleaved, and I also wanted to
avoid generating unique names for the logs, so instead of printing the lines
instantly, I appended to a string, complete with CR/LF, and at the end of
the task, I then implemented a lock, spin, write, unlock routine.
Andrew 8(
-----Original Message-----
From: Matt [mailto:[EMAIL PROTECTED]]
Sent: Monday, February 23, 2004 11:49 AM
To: [EMAIL PROTECTED]
Subject: Re: [Declude.JunkMail] Feature-itis
Andrew,
I just wanted to chime in and say that I of course would love to see
non-text base64 stuff thrown out before scanning, and allow us to target
only unencoded text strings. The idea of scanning only the decoded text
would also be a big processor saver and the primary method, so maybe you
would make the decoded choice the BODY filter and have a BODYSOURCE
filter for the encoded version. It's important that a filter have the
ability to scan MIME attachment descriptors though, so a method to
provide for that would be necessary as well. Maybe a BODYSOURCE check
would match what a BODY filter does today even with the source of the
attachments.
Regarding the logging, I'm not sure about the garbling, but streamed log
files like Declude's, IMail's and all sorts of HTTP stuff is generally
done on demand instead of grouped together. I believe there are
definite advantages beyond speed for doing this. It does though lack a
unique identifier as a single field which might be nice for log parsing,
and maybe that's what's needed here. Something like the spool file name
with a "-1" appended to it which increments and appears on each line
would do the trick, am I right? That would certainly make things easier
to parse.
Matt
Colbeck, Andrew wrote:
Far be it for me to halt progress...
Scott, I can't wait to put in the new TESTSFAILED logic. I've wanted
exactly this to keep certain multi-answer ip4r tests in check, and Matt is
off to a great start in combining tests...
I also find that CMDSPACE is very handy and has low false positives.
Seamless decoding of BASE64 Subjects (and quoted printables?) is also a
good
thing(tm).
SPF testing and the time-based DOW and HOUR features could be very handy.
But for my two cents, I have other priorities:
Priority #1 (by far)
====================
I want cleaner logs. This has been discussed in the list before, and I'm
pretty sure that Pete and Sandy agreed that they'd seen the behaviour
elsewhere, i.e. that multiple processes of writing to the same log file are
garbling the text file, and that per se, the garbling wasn't strictly
declude's doing.
I find that I need to run at loglevel HIGH to get the reporting I need on
text filtering, which means bigger log files and presumably more time spent
by each instance of declude while it or Windows races to the end of the
file
to append the text.
Without good logging, I'm very much put off my log analysis. Filtering the
logs when I get a false positive during my mailserver's "morning rush" is a
major pain due to all the overlapping loglines.
I can think of a couple of techniques, and I'm a lousy programmer. I don't
think you'll need my help there...
The simplest thing might be to give us a variable in the global.cfg to turn
on file locking, so that we can control whether the performance hit is
important in our environment. I realize that would likely add a lot of
lines of code to your source, but it could also be trivial to implement
inside a function.
Sending to a syslog server might also be easy to implement, but the only
experience I have with using the logs in a resulting syslog server is with
Kiwi, and there, I was using the text log it creates rather than any kind
of
interface to syslog (I don't know if that's the norm, nor what the IMail
users with syslog do with their logging.)
Ideally, the logs would be sent directly by declude.exe to an ODBC DSN and
the particular SQL database of our choosing, but I know that's really a
stretch.
Priority #2
===========
I get pecked to death by ducks on the small-weight false positives I get on
short text matches that are matching the encoded body of BASE64
attachments.
I know that you've mentioned several times before that going beyond the
current functionality would require a big leap in going to full MIME
decoding, but I hope that my aim is lower: I want to skip matching the
BASE64 encoding.
Sure, it would also be great to skip decoding MIME attachments that aren't
text or HTML (I get false positives on the binary contents of decoded .zip
files, too), but that would probably be Priority #3.
I know that at least one person on the list relies on declude to match text
inside the BASE64 attachments to catch viruses, but perhaps matching that
could be toggled with a flag, or make it a new test, e.g. instead of
specifying
BODY x CONTAINS abcdefghij
that this would be appropriate:
BASE64CODE x CONTAINS s9Zci6Y4
I haven't thought through all the ways in which a decoder would be useful,
so that exact testname might not be appropriate, but hey, it's a start.
Thanks for reading this all the way through,
Andrew 8)
---
[This E-mail was scanned for viruses by Declude Virus
(http://www.declude.com)]
---
This E-mail came from the Declude.JunkMail mailing list. To
unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
type "unsubscribe Declude.JunkMail". The archives can be found
at http://www.mail-archive.com.
01.orlandocorp.com has no MX or A records [0301].). Action=""
02/23/2004 11:37:05 Q565111ce00b66a85 Filter: Set max weight to 4.
n=LOG.
02/23/2004 11:37:05 Q564611cb00b640f5 Filter: Set max weight to 40.
02/23/2004 11:37:05 Q565311cf00b6740b Triggered CONTAINS filter GIBBERISH on original message [weight->0; Origina02/23/2004 11:37:05 Q56402/23/2004 11:37:05 Q563e186400aa1f82 Tests failed [weight=45]: NOLEGITCONTENT=WARN REV02/23/2004 11:37:05 Q563a186300aa10ec Msg failed MAILPOLICE-BULK (This E-mail came from edas53pha.com, a potential spam source listed 02/23/2004 11:37:05 Q5641186500aa2c24 Msg failed COUNTRY (Message failed COUNTRY test (line 29, weight 0)). Action=""
02/23/2004 11:37:05 Q563411c700b6fa28 Triggered CONTAINS filter BENTALLSPAMUNSUB on 0pt [weight->1; 0pt;
02/23/2004 11:37:05 Q562d27660092dff8 Msg failed NOLEGITCONTENT (No c02/23/2004 11:37:05 Q562e11c600b6e1dd Msg failed NOLEGITCONTENT (No content unique to legitimate E-mail detected.). Action02/23/2004 11:37:05 Q565111ce00b66a85 Filter BADCOUNTRYNOREVDNS: Not skipping E-mail due to current weight of -4.
02/23/2004 11:37:05 Q563a186300aa10ec Msg failed AHBL-PROXY-DYNA ("Open Proxy - http://www.ahbl.org/tools/loo02/23/2004 11:37:05 Q5641186500aa2c24 Msg failed SPAMDOMAINS (Spamdomain 'msn.com' found: Address of [EMAIL PROTECTED] sent from invalid adsl-64-163-241-83.dsl.lsan03.pacbell.net.). Action=""
02/23/2004 11:37:06 Q562a27650092d308 Msg failed BENTALLNEGTEXT (Message failed BENTALLNEGTEXT test (line 51, weight 02/23/2004 11:37:06 02/23/2004 11:37:06 Q565111ce00b66a85 Filter: Set max weight to 13.
ne 28, weight 0)). Action=""
02/23/2004 11:37:06 Q563a186300aa10ec Msg failed BENTALLIPBL (matched 210.205.0.0/16). Action=""
ht of -3.
02/23/2004 11:37:06 Q562d27660092dff8 Msg failed COUNTRY (Message failed COUNTRY test (line 29, weight 0)). Action02/23/2002/23/2004 11:37:06 Q562e11c600b6e1dd Msg failed BENTALLNEGTEXT (Message failed BENTALLNEGTEXT test (line 41, weigh02/23/2004 11:37:06 Q502/23/2004 11:37:06 Q564611cb00b640f5 Filter: Set max weight to 20.
ight of -4.
02/23/2004 11:37:06 Q5641186500aa2c24 Msg failed NOABUSE ("Not supporting [EMAIL PROTECTED]"). Action="" 11:37:06 Q562a27650092d308 R1 Message OK
LNEGTEXT (Message failed BENTALLNEGTEXT test (line 48, weight -10)). Action=""
02/23/2004 11:37:06 Q565111ce00b66a85 Filter: Set max02/23/2004 11:302/23/2004 11:37:06 Q563a186300aa10ec Subject: Get med02/23/2004 11:302/23/2004 11:37:06 Q562d27660092dff8 Msg failed BENTALLSPAMHEAD (Message failed BENTALLSPAMHEAD te02/23/2004 11:37:06 Q562a27650092d308 Su02/23/2004 11:37:06 Q562e11c600b6e1dd Subject: Assigned options
02/23/2004 11:37:06 Q563a186300aa10ec From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] IP: 210.205.188.173 ID:02/02/23/2004 11:37:06 Q562d27660092dff8 R1 Message OK
("Blocked - see http://www.spamcop.net/bl.shtml?64.163.241.83"). Action=""
02/23/2004 11:37:06 Q562e11c600b6e1dd From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] IP: 207.61.221.28 ID:
ID: W1NJ20AH31563
02/23/2004 11:37:06 Q563a186300aa10ec Tests failed [weight=84]: IPNOT02/23/2004 11:37:06 Q5641186500aa2c24 Msg failed XBL-DYNA ("http://www.spamhaus.org/xbl/xbl.lasso?query=64.163.241.83"). Action=""
K=WARN SORBS-HTTP-DYNA=WARN SORBS-DYNA=WARN MAILPOLICE-BULK=WARN AHBL-PROXY-DYNA=WARN BENTALLIPBL=WARN SNIFFER=WARN
