Hi Guys: I recently started reading the archives for this list and I'm naive enough to think that I've found "the" solution to the SPAM problem. I'm presenting my project status, rationale, goals and solution below, free for all, in hopes of eradicating SPAM. (Code and all!) I am rashly asserting that if *everyone* implemented my solution, SPAM would not be long for this world. I *hope* for a thorough critique despite the length of this message.
Current status: Right now, I'm redirecting my own personal mail to a mail server that implements the autowhitelist solution. So far, I've been able to subscribe to mailing lists w/ few or no problems. 100% of my spam has been "identified". There have been a few "false positives" (now corrected), but no mail has been lost or discarded. To explain my own recent situation/rationale: Personally, after having posted to a few mailing lists, a few newsgroups (not many) and registered a few domain names (killer), I find that I get far more SPAM than legitimate mail. I have many automated processes sending me mail that I expect, and desire. I have a small flow of mail from actual people. I have e-mail filters on my mail client that attempt to group this mail into a reasonable set of folders. I have a LOT of SPAM. Unfortunately, I often find myself becoming depressed when I actually get a *real* piece of mail from a *real* live person, because it interferes w/ my day-long task of DELETING SPAM from my mailbox. I also get to hear complaints from my boss, and my users, about how much more spam THEY now get and how disruptive it is for THEIR day. So, w/ this in mind, I'm tossing out any objections about the "cost" and/or justifications for whitelisting as I move on to find techniques to make my mailbox useful once again. Now that I'm in my testing phase, I find that I actually have time to *read* e-mail as opposed to being preoccupied w/ the deletion process. Requirements for my own solution: Low support, No lost mail, Windows clients, Quick for end-usrs. 1) As much as I'd like to shoot first and ask questions later, I can't commit to increasing my administrative overhead (support questions) w/ a solution that's too aggressive. 2) I need to make sure that some sales-guy doesn't accuse me of losing his supposed million dollar order because of a lost e-mail. 3) I need a system to support my existing windows users who mainly use Netscape Messenger and/or MS Outlook as their POP mail clients. (solutions for unix-only clients won't fly) 4) This solution MUST separate the bulk of my end-users legitimate e-mail from the deluge of SPAM. End users MUST find deleting/ignoring SPAM to be an extremely low overhead activity. Goals: Given these requirements I plan to build this system: A sendmail/procmail based automatic whitelist system that will tag subjectlines of mail from unknown senders w/ the string "[UNKNOWN]". The "automatic" part of this system involves detecting mail that is outbound from my mail server, and adding the recipient address(es) to the whitelist database of the sender. An additional system-wide whitelist will allow me to accept mail from certain friendly domains. Filtering capabilities on e-mail clients will deliver whitelisted e-mail to the end-user's "inbox" normally, and the tagged "[UNKNOWN]" mail (presumed SPAM) gets delivered to an "inbucket". End users will be able to retrieve their own SPAM from the "inbucket" in the event that they disagree w/ that classification. Replying to the "offending" From address will "fix" the problem by whitelisting the sender. Remaining Problems: The system appears very useful, but... Calling procmail for non-local mail delivery causes additional overhead. It also *might* introduce some SPAM vulnerabilities. This needs more scrutiny. An automated challenge system would be helpful in case my end-users are lazy about checking their inbuckets. Adding mailing lists to the whitelist is easy for me, but I'm not sure how easy this process will be for the average end-user. Exchanging e-mail w/ some unix users that use a mail-hub and sendmail/mutt enabled workstations shows the same "return-path differs from reply-to" behavior as mailing lists. (reply-to gets added to whitelist, return-path is still regarded as "[UNKNOWN]") -------------------------------------------- The actual Autowhitelist solution!: First, here's a description of the function of the files and the system, after that are the files themselves. There are several files: /etc/mail/sendmail.mc /etc/procmailrcs/outbound.rc /etc/procmailrcs/whitelist /etc/procmailrcs/whitedomains /home/username/.procmailrc /home/username/nsmail/inbound.rc /etc/procmailrcs/username#aeroflex.com That last file is the whitelist itself. ( '#' substitutes for '@' in a local e-mail address) It's a series of lines consisting of whitelisted e-mail addresses. This file gets created automatically when the outbound.rc procmail recipe calls the "whitelist u" bash script. One of these files gets created per local mail user, so each user has their own whitelist. The modifications in sendmail.mc trigger the call to procmail (for non-local delivery) that causes outbound.rc to be processed in the first place. (This technique courtesy of:) http://www.professional.org/procmail/mailer.html The whitedomains file contains a list of domains to be whitelisted system-wide, for all local users. Mail sent from one of those domains will be accepted by any local user. As a result, there is no need to list e-mail addresses from those domains in the individual local user's whitelist. The .procmailrc file of local users who make use of the whitelist system need to include the inbound.rc file. The inbound.rc recipe file calls the whitelist bash script w/ the "c" option. This causes the whitelist program to check the users whitelist rather than update it. If the check is successful, then the inbound.rc recipe delivers the mail unchanged. Otherwise, inbound.rc tags the subjectline of the e-mail w/ the string "[UNKNOWN]" indicating that it came from an unknown address. (If you prefer it, it's a trivial change to make it store mail in a different folder, or delete it) The reason that I chose to tag mail is so that a user w/ outlook, NS-Messenger, or some other remote mail client could download the mail and then use it's built in mail filters to easily drop "[UNKNOWN]" SPAM into an alternate mail folder. In this way, the autowhitelist system is compatible w/ POP, etc. In the event of false positives, I also don't have to hear end-users complaining that I deleted their mail. The end result of all this is that when a user of this system sends mail to someone, the recipient e-mail address automatically gets added to the senders whitelist. When I get mail from people who are on my whitelist, that mail appears in my inbox. When I get (lots of) mail from other people, my mail client is configured to divert it to a folder named "inbucket". At the end of the day, I can take a quick look through the inbucket to see if anything valuable happened to end up there. I've been having a little trouble w/ mailing lists. The reply-to address is often different from the address that is sending the mail. If I view the headers of e-mail on such a piece of mail, the first line is a "From " or "Return-Path: " field. This is what the whitelist system checks and that's what needs to be added to the per-user whitelist file. To make that happen while avoiding having to (unpolitely) send mail to this address, I can send a message to that address w/ a subjectline of "add autowhitelist". This will cause the recipient address to be added to the whitelist, but otherwise the message is discarded and not delivered. Another issue is e-mail aliases. If you have multiple e-mail addresses that get delivered into the same e-mail box, the autowhitelist system will only build a whitelist for the one address that you send with. In order to have the system work for alias addresses, it is necessary to create symbolic links under the alias address, pointing to the "one-true" whitelist file for that user. That's all of my commentary for now. Here (below) are all the individual files. Some of them need to be modified to reflect your own domain name. There's also a lot of debugging code in here that eventually needs to be trimmed or commented out. Comments anyone? -Bill ---------------------------------------------- tail end of /etc/mail/sendmail.mc ---------------------------------------------- LOCAL_RULESETS SProcmailHandler # Local Users R$- .procmail $@ $1 R$- $#Procmail $@/etc/procmailrcs/outbound.rc $: $1 . procmail # Canonical Domains R$* < @ $+ .procmail > $@ $1 < @ $2 . > R$* .procmail < @ $+ . > $@ $1 < @ $2 . > R$* < @ $+ . > $#procmail $@/etc/procmailrcs/outbound.rc $: $1 < @ $2 .procmail > # Unresolvable Canonical Domains R$* < @ $+ .baddomain > $@ $1 < @ $2 > R$* < @ $+ > $#procmail $@/etc/procmailrcs/outbound.rc $: $1 < @ $2 .baddomain > # Duplication of Unresolvable Canonical Domains R$* < @ $+ .xprocmail > $@ $1 < @ $2 > R$* < @ $+ > $#procmail $@/etc/procmailrcs/outbound.rc $: $1 < @ $2 .xprocmail > ---------------------------------------------- /etc/outbound.rc ---------------------------------------------- DEFAULT=| SHELL=/bin/sh LOGFILE=/etc/procmailrcs/outbound.log VERBOSE=ON USER=$1 REC=$2 # copy ALL messages :0ch: /etc/procmailrcs/outbound.header # copy ALL messages :0cb: /etc/procmailrcs/outbound.body :0 ch | echo "$@" >> /etc/procmailrcs/outbound.cmdline :0 ch | env > /etc/procmailrcs/outbound.env # Manually update the whitelist :0 h * ^Subject: add autowhitelist | /etc/procmailrcs/whitelist u # Automatically update the whitelist :0 ch * ^Return-Path: .*@aeroflex.com * !^FROM_MAILER * !^FROM_DAEMON * !^To_.*@aeroflex.com | /etc/procmailrcs/whitelist u # Pass along all mail :0 ! -f "$@" ---------------------------------------------- /etc/procmailrcs/whitelist ---------------------------------------------- #!/bin/sh # whitelist u # update # whitelist c # check SWITCH=$1 TMP=/tmp/procfilter.$$ WORK=/etc/procmailrcs TEST=$WORK/whitelist.test MAILDOMAIN=billtest.aeroflex.com cat > $TMP echo whitelist >$TEST echo $0 "$@" >> $TEST FROM=`grep "^Return-Path: " $TMP |\ head -1|\ (read junk LINE;echo $LINE)|\ tr -d "<>"` if [ -n "$REC" ] then export RECIP=`echo $REC|sed -e 's/\.procmail$//'` echo REC = $REC >> $TEST else export RECIP="$LOGNAME"@$MAILDOMAIN echo LOGNAME = $LOGNAME >> $TEST fi echo RECIP = $RECIP >> $TEST if [ "x$SWITCH" = xc ] then # Checking the database (for incoming mail ) echo c >> $TEST #USER_WHITELIST=`grep "To: " $TMP |\ # head -1|\ # (read junk LINE; echo $LINE) |\ # tr "@" "#"` USER_WHITELIST=$WORK/`echo $RECIP|tr "@" "#"|tr [:upper:] [:lower:]` echo FROM = $FROM >> $TEST echo USER_WHITELIST = $USER_WHITELIST >> $TEST if [ ! -f $USER_WHITELIST ] then echo whitelist database > $USER_WHITELIST chmod a+r $USER_WHITELIST fi env >> $TEST echo if grep $FROM -xq $USER_WHITELIST >> $TEST if grep $FROM -xq $USER_WHITELIST then echo "exit 0 (true)" >> $TEST exit 0 else if echo $FROM|grep -iqf $WORK/whitedomains then echo "exit 0 (true) (in whitedomains)" >> $TEST exit 0 else echo "exit 1 (false)" >> $TEST exit 1 fi fi else echo u >> $TEST if echo $RECIP |grep -iqf $WORK/whitedomains then # Don't add them cause theyre in a whitelisted domain echo whitedomain >> $TEST exit 0 fi # Updating the database (during outgoing mail) # put RECIP into database of FROM USER_WHITELIST=$WORK/`echo $FROM|tr "@" "#"|tr [:upper:] [:lower:]` echo updating >> $TEST echo $RECIP goes into $USER_WHITELIST >> $TEST #echo USER_WHITELIST = $USER_WHITELIST if [ ! -f $USER_WHITELIST ] then echo whitelist database > $USER_WHITELIST fi #grep "^To: " <$TMP |\ #sed 's/^To: //' |\ echo $RECIP |\ grep -xvf $USER_WHITELIST \ > $TMP.3 cat $TMP.3 >> $USER_WHITELIST cat $TMP.3 >>$TEST echo >> $TEST env >> $TEST cat $TMP >> $TEST rm $TMP.3 fi rm $TMP ---------------------------------------------- /etc/procmailrcs/whitedomains (These are domains for which my users will always accept mail. P.S. the "whitelist" program won't add them to individual user whitelists) ---------------------------------------------- @aeroflex.com$ @lintek.aeroflex.com$ ---------------------------------------------- /home/username/.procmailrc ---------------------------------------------- MAILDIR=$HOME/nsmail PMDIR=$MAILDIR INCLUDERC=$PMDIR/inbound.rc ---------------------------------------------- /home/username/nsmail/inbound.rc ---------------------------------------------- # DEFAULT=| SHELL=/bin/sh LOGFILE=$MAILDIR/inbound.log VERBOSE=ON # Copy Headers :0ch: $MAILDIR/inbound.header # Copy Body :0cb: $MAILDIR/inbound.body :0 ch | echo "$@" >> $MAILDIR/inbound.cmdline :0 ch | env > $MAILDIR/inbound.env # Check the whitelist :0 chw * !^Return-Path: .*@aeroflex.com * !^Return-Path: .*@plainview.aeroflex.com | /etc/procmailrcs/whitelist c # Tag mail only if not on whitelist :0 e * ^Subject: \/.* { SUBJECT=$MATCH :0 f | formail -i "Subject: [UNKNOWN] $SUBJECT" ; } ---------------------------------------------- /etc/procmailrcs/username#aeroflex.com ---------------------------------------------- whitelist database [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] _______________________________________________ spamcon-general mailing list [EMAIL PROTECTED] http://mail.spamcon.org/mailman/listinfo/spamcon-general#subscribers Subscribe, unsubscribe, etc: Use the URL above or send "help" in body of message to [EMAIL PROTECTED] Contact administrator: [EMAIL PROTECTED]
