Re: [PHP] Working with overly aggressive anti-spam measures [SOLVED]
On Wed, October 11, 2006 9:14 pm, Google Kreme wrote: > On 10 Oct 2006, at 19:57 , Dave M G wrote: >> It took me a little while to realize that the Spamassassin always >> says that an email is possible scam if it has any score above zero, >> but it takes a score of 5.0 to actually be deleted as spam. > > No no no, this is not at all true. > > First off, Spamassassin does not delete mail. It, in point of fact, > CANNOT delete mail as it has no mechanism to do so. Sorry -- I was very ambiguous. Everywhere I typed "SA" in my post should be replaced with: "The squirrel-mail interface provided by my webhost, combined with a custom PHP IMAP inbox-scanning/filtering script I hacked up for my own unique needs, along with the Spam Assassin scoring also provided by my webhost" My apologies for imprecision. But I get bonus points for brevity in the original post :-) > Anyone who is automatically deleting mail on their own with a score > of 5.0 or higher _will_ miss some legitimate email. Not might, > WILL. Even at a score of 10.0 (where I auto-delete my mail) you will > miss as much as 0.004% legitimate mail, iirc. (so, 4 messages in > 100,000) Yes, but getting 10,000 messages per day[*] means I don't READ email anymore. I do email triage. :-) I'm losing 4 real emails every 10 days to my triage. I can live with that. I cannot live with skimming through 10,000 emails per day searching for the ~1 email every other day that is valid. YMMV [*] It's been several years since I've seen a report on how many unfiltered emails per day I received. 10,000 was the number back then. It is inconceivable that the number has decreased, by any realistic projection. -- Some people have a "gift" link here. Know what I want? I want you to buy a CD from some starving artist. http://cdbaby.com/browse/from/lynch Yeah, I get a buck. So? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Working with overly aggressive anti-spam measures [SOLVED]
On 10 Oct 2006, at 19:57 , Dave M G wrote: It took me a little while to realize that the Spamassassin always says that an email is possible scam if it has any score above zero, but it takes a score of 5.0 to actually be deleted as spam. No no no, this is not at all true. First off, Spamassassin does not delete mail. It, in point of fact, CANNOT delete mail as it has no mechanism to do so. Anyone who is automatically deleting mail on their own with a score of 5.0 or higher _will_ miss some legitimate email. Not might, WILL. Even at a score of 10.0 (where I auto-delete my mail) you will miss as much as 0.004% legitimate mail, iirc. (so, 4 messages in 100,000) most good email will score between -3 and +1 in Spamassassin if you've been running it for any length of time. Your message scored X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_00,DNS_FROM_RFC_ABUSE, SPF_PASS autolearn=no version=3.1.0 So, a -2.4 which, last I checked was something like 99.97% likely to be ham. -- Well I've seen the Heart of Darkness/Read the writing on the wall/and the voice out in the desert/Was the voice out in the hall -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Working with overly aggressive anti-spam measures [SOLVED]
On Tue, October 10, 2006 8:57 pm, Dave M G wrote: > It took me a little while to realize that the Spamassassin always says > that an email is possible scam if it has any score above zero, but it > takes a score of 5.0 to actually be deleted as spam. I checked through > a > bunch of emails in my inbox, and all of them got at least some score, > usually between .5 and 1.5, even though they were all legitimate > emails. That 5.0 is user-configurable -- In fact, for my Inbox, 3.0 or above is sent direct to trash. I spent a month or two with 10 filters sending every message to one of 10 boxes: Inbox.1 Inbox.2 Inbox.3 ... Inbox.10 As time went on, I found that only 1 and 2 scores were worth digging through, and lowered my SA setting to 3 and above is just auto-trash. And a couple mailing lists were going to higher scores, but those were easily white-listed. This would have been much easier had the email client let me sort by SA score, but that feature has apparently not been invented yet... YMMV, but it's amazing how much less spam I got once I let SA kill off all the junk it was pre-configured to accept. Course, I'm sure I'm losing some legit email as well, but it's such a tiny percentage, I just can't get too worked up about it. -- Some people have a "gift" link here. Know what I want? I want you to buy a CD from some starving artist. http://cdbaby.com/browse/from/lynch Yeah, I get a buck. So? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Working with overly aggressive anti-spam measures [SOLVED]
Travis, Richard, Chris, Thank you for your helpful advice. I've managed to set the Return-Path correctly now, and Spamassassin is now only giving me the whitelist error, which I don't think I'm going to get rid of. It took me a little while to realize that the Spamassassin always says that an email is possible scam if it has any score above zero, but it takes a score of 5.0 to actually be deleted as spam. I checked through a bunch of emails in my inbox, and all of them got at least some score, usually between .5 and 1.5, even though they were all legitimate emails. So compared to other emails I'm getting, I'm scoring very low at 0.2. My emails don't use HTML, don't have any common spam key words, and have correct spelling and grammar, so it all helps establish my email's legitimacy. Here is the PHP code I'm using: $fromAddress = "[EMAIL PROTECTED]"; $fromName = "My Name"; $headers = "From: " . $fromName . " <" . $fromAddress . ">\n"; $headers .= "Reply-to: " . $fromName . " <" . $fromAddress . ">\n"; $extra = "-f" . $fromAddress; mail($toAddress, $subject, $content, $headers, $extra); Thank you all for your time and expert information. -- Dave M G Ubuntu 6.06 LTS Kernel 2.6.17.7 Pentium D Dual Core Processor PHP 5, MySQL 5, Apache 2 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Working with overly aggressive anti-spam measures
Dave M G wrote: > 1. Spamassassin says: > > Spam detection software, running on the system "homebase", has > identified this incoming email as possible spam. [...] > > Content analysis details: (0.3 points, 5.0 required) > > pts rule name description > -- > -- > 0.3 AWLAWL: From: address is in the auto white-list > > I looked up about auto white-lists on the net, and it says that it's a > comparison between the current and previous emails. But for testing > purposes, I don't really want it to compare against previous emails, > since previous test emails would be "spammier" and bias spamassassin > the wrong way. > > How can I compensate for this? Yup, as you mentioned the AWL (auto-white-list) is a comparison to previous messages. If that is the only rule that gets hit it looks like you are doing pretty good. You did include the body of the message in the test right? Not just the headers? I expected you would have hit more rules than that. Of course, that is an 'out-of-the-box' install of SA, most people have special rules in there, bayes training done, etc... I believe there is also a website that you can copy paste the message into to perform the same test (see message on this list, "Peter Lauri - PHP Mailer and SMTP = SPAM?") That site might have other rules configured (SARE... etc) that you might be hitting. Worth a shot. Travis Doherty -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Working with overly aggressive anti-spam measures
On Tue, October 10, 2006 3:01 am, Dave M G wrote: > Content analysis details: (0.3 points, 5.0 required) > > pts rule name description > -- > -- > 0.3 AWLAWL: From: address is in the auto > white-list > > I looked up about auto white-lists on the net, and it says that it's a > comparison between the current and previous emails. But for testing > purposes, I don't really want it to compare against previous emails, > since previous test emails would be "spammier" and bias spamassassin > the > wrong way. > > How can I compensate for this? That's gotta be some kind of SA setting... > 2. I've attempted to add the "Return-Path" headers to my headers, like > so: > > $headers = "Return-Path: [EMAIL PROTECTED]"; No, no, no. Return-path: may *LOOK* like a normal header, but it's not, and you cannot just set it in the $headers variable 4th arg to PHP mail(). Return-path is a super-special only-god-can-do-this header, and has to be set in the sendmail_path or with that new FIFTH arg, which only works if you've got permissions set up right in sendmail (or whatever) and PHP (no safe mode). > $headers .= "From: Tokyo Street Hockey Association > <[EMAIL PROTECTED]>\n"; > $headers .= "X-Sender: [EMAIL PROTECTED]"; This is not needed. Reply-to: is good, though, as it looks "more human" > mail($member, $subject, $content, $headers); > > And yet, when I look at the headers of the email that's being > generated, > it says: > > Return-Path: <[EMAIL PROTECTED]> > Received: from server.mywebhost.com (ns1.nameserver.com > [123.123.123.123] (may be forged) > > (Note that I've changed the names of servers and the DNS just to keep > some info private) > > Why are my attempts to set these headers in PHP not taking effect? They're just not handled like normal headers, as they are part of the routing of the email handled by mail software more than they are regular old headers. It's just the way it is, no matter how goofy it seems from our (well, my, anyway) naive view-point. -- Some people have a "gift" link here. Know what I want? I want you to buy a CD from some starving artist. http://cdbaby.com/browse/from/lynch Yeah, I get a buck. So? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Working with overly aggressive anti-spam measures
On Mon, October 9, 2006 9:01 pm, Chris wrote: > You can't change that parameter if safe-mode is on for the server or > if > exim doesn't have the webserver user as a 'trusted-user' (I think only > exim is affected by this particular issue). Almost-for-sure sendmail has the same issue. Or maybe you are all laughing at me because that was obvious to anybody who knows anything about exim and sendmail... :-) -- Some people have a "gift" link here. Know what I want? I want you to buy a CD from some starving artist. http://cdbaby.com/browse/from/lynch Yeah, I get a buck. So? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Working with overly aggressive anti-spam measures
On Mon, October 9, 2006 8:55 pm, Dave M G wrote:
> If possible, can anyone help me with creating the PHP code that will
> make an email as legitimate as it can be? I know I can't totally
> prevent
> my email from being marked as spam (after all, if it were possible,
> all
> the spammers would do it). But as much as I can prevent anti-spam
> measures getting a false positive when testing my email, the better.
Get the problem users to send you valid email that gets through,
headers and all, and compare yours to theirs. :-)
> Here is the PHP code I currently use (trimmed for clarity):
>
> while ( $member = mysql_fetch_row($mysqlResult) )
> {
> $subscriber = $member[0];
> $email = $member[1];
> $subject = "Report for " . date('l jS F Y');
> $mailContent = "This is an email to " . $subscriber . " at " . $email
> . ".";
> $fromAddress = "[EMAIL PROTECTED]"
> mail($email, $subject, $mailContent, $fromAddress);
$fromAddress, by itself, is not a valid header line. That right there
is going to choke many mail agents, before they even try to look at it
for spam content.
$headers = "From: $fromAddress\r\n";
$headers .= "Reply-to: $fromAddress\r\n";
$extra = "-f$fromAddress";
mail($email, $subject, $mailContent, $headers, $extra);
$extra requires PHP 5 (4.x?) and that your mail config "trusts" the
Apache user to forge emails. It may not be possible in your setup.
> }
>
>
> And here is what the headers for an email from that code looks like:
>
> -Account-Key: account5
Hopefully this had an "X" in the front of it...???
> X-UIDL: UID497-1152485402
> X-Mozilla-Status: 0001
> X-Mozilla-Status2:
> Return-path: <[EMAIL PROTECTED]>
This is set by $extra.
If your server won't let you change it, then there is little you can
do other than re-configure sendmail, switch hosts, and/or turn off
safe_mode -- all of which are outside the purvue of PHP and this list.
(Well, okay, safe_mode on/off is within our realm. Turn it off.)
> Envelope-to: [EMAIL PROTECTED]
> Delivery-date: Sun, 03 Sep 2006 14:22:42 -0700
> Received: from nobody by server.myhostingservice.com with local (Exim
> 4.52)
> id 1GJzQQ-0005pA-Mz
> for [EMAIL PROTECTED]; Sun, 03 Sep 2006 14:22:42 -0700
> To: [EMAIL PROTECTED]
> Subject: Report for Monday 4th September 2006
> From: [EMAIL PROTECTED]
I'm AMAZED PHP and/or your MTA got this From: like this, when you
didn't do it right above...
> Message-Id: <[EMAIL PROTECTED]>
> Date: Sun, 03 Sep 2006 14:22:42 -0700
>
> Which parts are key to change, and how?
You're actually very close to a valid email, mainly because you've
kept it so simple...
--
Some people have a "gift" link here.
Know what I want?
I want you to buy a CD from some starving artist.
http://cdbaby.com/browse/from/lynch
Yeah, I get a buck. So?
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Working with overly aggressive anti-spam measures
Dave M G wrote:
Travis, Chris,
Thank you for your advice.
I've installed spamassassin and I'm using it to monitor how well my PHP
generated emails rate as spam.
So far, I'm not doing well. I have two main issues so far:
1. Spamassassin says:
Spam detection software, running on the system "homebase", has
identified this incoming email as possible spam. [...]
Content analysis details: (0.3 points, 5.0 required)
pts rule name description
--
--
0.3 AWLAWL: From: address is in the auto white-list
I looked up about auto white-lists on the net, and it says that it's a
comparison between the current and previous emails. But for testing
purposes, I don't really want it to compare against previous emails,
since previous test emails would be "spammier" and bias spamassassin the
wrong way.
How can I compensate for this?
2. I've attempted to add the "Return-Path" headers to my headers, like so:
$headers = "Return-Path: [EMAIL PROTECTED]";
$headers .= "From: Tokyo Street Hockey Association
<[EMAIL PROTECTED]>\n";
$headers .= "X-Sender: [EMAIL PROTECTED]";
mail($member, $subject, $content, $headers);
And yet, when I look at the headers of the email that's being generated,
it says:
Return-Path: <[EMAIL PROTECTED]>
Received: from server.mywebhost.com (ns1.nameserver.com
[123.123.123.123] (may be forged)
(Note that I've changed the names of servers and the DNS just to keep
some info private)
Why are my attempts to set these headers in PHP not taking effect?
Because the return path can't be set manually.
You need to set the 5th parameter:
mail($to, $subject, $headers, $body, '[EMAIL PROTECTED]');
or
ini_set('sendmail_from', '[EMAIL PROTECTED]');
mail($to, $subject, $headers, $body);
--
Postgresql & php tutorials
http://www.designmagick.com/
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Working with overly aggressive anti-spam measures
Travis, Chris, Thank you for your advice. I've installed spamassassin and I'm using it to monitor how well my PHP generated emails rate as spam. So far, I'm not doing well. I have two main issues so far: 1. Spamassassin says: Spam detection software, running on the system "homebase", has identified this incoming email as possible spam. [...] Content analysis details: (0.3 points, 5.0 required) pts rule name description -- -- 0.3 AWLAWL: From: address is in the auto white-list I looked up about auto white-lists on the net, and it says that it's a comparison between the current and previous emails. But for testing purposes, I don't really want it to compare against previous emails, since previous test emails would be "spammier" and bias spamassassin the wrong way. How can I compensate for this? 2. I've attempted to add the "Return-Path" headers to my headers, like so: $headers = "Return-Path: [EMAIL PROTECTED]"; $headers .= "From: Tokyo Street Hockey Association <[EMAIL PROTECTED]>\n"; $headers .= "X-Sender: [EMAIL PROTECTED]"; mail($member, $subject, $content, $headers); And yet, when I look at the headers of the email that's being generated, it says: Return-Path: <[EMAIL PROTECTED]> Received: from server.mywebhost.com (ns1.nameserver.com [123.123.123.123] (may be forged) (Note that I've changed the names of servers and the DNS just to keep some info private) Why are my attempts to set these headers in PHP not taking effect? -- Dave M G Ubuntu 6.06 LTS Kernel 2.6.17.7 Pentium D Dual Core Processor PHP 5, MySQL 5, Apache 2 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Working with overly aggressive anti-spam measures
Dave M G wrote:
> PHP List,
>
> I run a few various social groups, and with each one I keep in contact
> with members by emailing them short newsletters.
>
> All my user information is stored in a MySQL database. I use PHP to
> get the relevant contact information, and use the mail() command to
> send out the emails one by one, so that each email is a little
> personalized.
>
> I've used this system for many years now with no problems up until now.
>
> What has changed, though, is that in recent years, anti-spam measures
> have become so aggressive that more and more people who sign up to my
> groups complain that they never receive the emails.
>
> A lot of the times, after they alert me to the issue, I can educate
> them a little about the anti-spam measures they most likely have on
> their system, and walk them through how to make it so that my
> newsletters go through.
>
> However, that is clearly not enough.
>
> To shorten a story that has already gone on a little long, it's come
> to my attention that part of the reason that my emails may not be
> getting through are because the headers are not sufficiently
> legitimate looking enough to bypass some server side anti-spam
> measures. Things like "Return-Path" are being set so that they look
> like they come from an email address that begins with the username
> "nobody".
>
> If possible, can anyone help me with creating the PHP code that will
> make an email as legitimate as it can be? I know I can't totally
> prevent my email from being marked as spam (after all, if it were
> possible, all the spammers would do it). But as much as I can prevent
> anti-spam measures getting a false positive when testing my email, the
> better.
>
> Here is the PHP code I currently use (trimmed for clarity):
>
> while ( $member = mysql_fetch_row($mysqlResult) )
> {
> $subscriber = $member[0];
> $email = $member[1];
> $subject = "Report for " . date('l jS F Y');
> $mailContent = "This is an email to " . $subscriber . " at " . $email
> . ".";
> $fromAddress = "[EMAIL PROTECTED]"
> mail($email, $subject, $mailContent, $fromAddress);
> }
>
>
> And here is what the headers for an email from that code looks like:
>
> -Account-Key: account5
> X-UIDL: UID497-1152485402
> X-Mozilla-Status: 0001
> X-Mozilla-Status2:
> Return-path: <[EMAIL PROTECTED]>
> Envelope-to: [EMAIL PROTECTED]
> Delivery-date: Sun, 03 Sep 2006 14:22:42 -0700
> Received: from nobody by server.myhostingservice.com with local (Exim
> 4.52)
> id 1GJzQQ-0005pA-Mz
> for [EMAIL PROTECTED]; Sun, 03 Sep 2006 14:22:42 -0700
> To: [EMAIL PROTECTED]
> Subject: Report for Monday 4th September 2006
> From: [EMAIL PROTECTED]
> Message-Id: <[EMAIL PROTECTED]>
> Date: Sun, 03 Sep 2006 14:22:42 -0700
>
> Which parts are key to change, and how?
>
> Thank you for any and all advice.
>
> --
> Dave M G
> Ubuntu 6.06 LTS
> Kernel 2.6.17.7
> Pentium D Dual Core Processor
> PHP 5, MySQL 5, Apache 2
>
If it is sendmail it would be the -f parameter to set the Envelope-From
address. ie... php.ini would have 'sendmail_path = /pathto/sendmail -f
[EMAIL PROTECTED]'. Since PHP 4.0.5 you can also set the
additional_parameter like php.net/manual/function.mail.php (the example
there is specifically for this use.)
What you actually should be doing: install SpamAssasin on your
workstation and run the message through it in test mode... Figure out
what rules get hit and work to resolve those. Run it like `spamassassin
-t < message_with_headers`
So, you'd copy your complete message (as you have above) into
message_with_headers and run it through SA. Maybe SA is giving you
points for things like too many exclamations, certain ratio of HTML...
I doubt that having a return-path of 'nobody' is the lowest-hanging
fruit you can pick up here to make yourself look less spammy.
Also make sure the sending mail server isn't listed in any blacklists
(yahoo for RBL lookup tool to check.)
Travis Doherty
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Working with overly aggressive anti-spam measures
Dave M G wrote: PHP List, I run a few various social groups, and with each one I keep in contact with members by emailing them short newsletters. All my user information is stored in a MySQL database. I use PHP to get the relevant contact information, and use the mail() command to send out the emails one by one, so that each email is a little personalized. I've used this system for many years now with no problems up until now. What has changed, though, is that in recent years, anti-spam measures have become so aggressive that more and more people who sign up to my groups complain that they never receive the emails. A lot of the times, after they alert me to the issue, I can educate them a little about the anti-spam measures they most likely have on their system, and walk them through how to make it so that my newsletters go through. However, that is clearly not enough. To shorten a story that has already gone on a little long, it's come to my attention that part of the reason that my emails may not be getting through are because the headers are not sufficiently legitimate looking enough to bypass some server side anti-spam measures. Things like "Return-Path" are being set so that they look like they come from an email address that begins with the username "nobody". mail($email, $subject, $mailContent, $fromAddress); You need to set the 5th parameter to change who it comes from (instead of "nobody"). See php.net/mail for more info. You can't change that parameter if safe-mode is on for the server or if exim doesn't have the webserver user as a 'trusted-user' (I think only exim is affected by this particular issue). -- Postgresql & php tutorials http://www.designmagick.com/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Working with overly aggressive anti-spam measures
PHP List,
I run a few various social groups, and with each one I keep in contact
with members by emailing them short newsletters.
All my user information is stored in a MySQL database. I use PHP to get
the relevant contact information, and use the mail() command to send out
the emails one by one, so that each email is a little personalized.
I've used this system for many years now with no problems up until now.
What has changed, though, is that in recent years, anti-spam measures
have become so aggressive that more and more people who sign up to my
groups complain that they never receive the emails.
A lot of the times, after they alert me to the issue, I can educate them
a little about the anti-spam measures they most likely have on their
system, and walk them through how to make it so that my newsletters go
through.
However, that is clearly not enough.
To shorten a story that has already gone on a little long, it's come to
my attention that part of the reason that my emails may not be getting
through are because the headers are not sufficiently legitimate looking
enough to bypass some server side anti-spam measures. Things like
"Return-Path" are being set so that they look like they come from an
email address that begins with the username "nobody".
If possible, can anyone help me with creating the PHP code that will
make an email as legitimate as it can be? I know I can't totally prevent
my email from being marked as spam (after all, if it were possible, all
the spammers would do it). But as much as I can prevent anti-spam
measures getting a false positive when testing my email, the better.
Here is the PHP code I currently use (trimmed for clarity):
while ( $member = mysql_fetch_row($mysqlResult) )
{
$subscriber = $member[0];
$email = $member[1];
$subject = "Report for " . date('l jS F Y');
$mailContent = "This is an email to " . $subscriber . " at " . $email . ".";
$fromAddress = "[EMAIL PROTECTED]"
mail($email, $subject, $mailContent, $fromAddress);
}
And here is what the headers for an email from that code looks like:
-Account-Key: account5
X-UIDL: UID497-1152485402
X-Mozilla-Status: 0001
X-Mozilla-Status2:
Return-path: <[EMAIL PROTECTED]>
Envelope-to: [EMAIL PROTECTED]
Delivery-date: Sun, 03 Sep 2006 14:22:42 -0700
Received: from nobody by server.myhostingservice.com with local (Exim 4.52)
id 1GJzQQ-0005pA-Mz
for [EMAIL PROTECTED]; Sun, 03 Sep 2006 14:22:42 -0700
To: [EMAIL PROTECTED]
Subject: Report for Monday 4th September 2006
From: [EMAIL PROTECTED]
Message-Id: <[EMAIL PROTECTED]>
Date: Sun, 03 Sep 2006 14:22:42 -0700
Which parts are key to change, and how?
Thank you for any and all advice.
--
Dave M G
Ubuntu 6.06 LTS
Kernel 2.6.17.7
Pentium D Dual Core Processor
PHP 5, MySQL 5, Apache 2
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
