Re: replay RBL queries one hour later
Benny, All I know for sure is this - for MANY legit emails - DKIM fails some days later - when it had originally worked/validated at the time the message was sent. I see this often in the real world when I rescan a message to try to verify the impact on a message that a spam filtering change caused - then notice that a very legit email that original passed DKIM at the time the message was received - now suddenly fails DKIM during this days-later rescan - and without ANY changes to the message itself. I think that this is most likely caused by DNS records for that DKIM being changed/updated. But whatever the cause, this is STILL a reality that's worth noting, for anyone who is rescanning messages later. Rob McEwen, invaluement -- Original Message -- From "Benny Pedersen" To users@spamassassin.apache.org Date 2/26/2023 1:37:53 PM Subject Re: replay RBL queries one hour later Rob McEwen skrev den 2023-02-26 19:03: .. sent. This can lead to many egregious false positives. But doing this "one hour later" shouldn't have this problem. message-id is timebased, so why invalidate it ? :) i did that mistake on not dkim sign that header in that regard i now have 2048 kbit size, where 4096 is a bit overkill
Re: replay RBL queries one hour later
Something to keep in mind about this idea of rescanning messages later - once more anti-spam data is available - for use in training/reporting spams - this probably should NOT be done days later because SOME senders aggressively expire/recycle DKIM dns records. I guess that is to minimize the ability for criminals to spoof DKIM? The result is that if you implement this idea on days-old messages, you can end up with some spam scoring that was ONLY due to the DKIM not being valid anymore, where it was valid at the time the message was sent. This can lead to many egregious false positives. But doing this "one hour later" shouldn't have this problem. Rob McEwen, invaluement
Re: May I get to 0 phishing?
I dug a little deeper on this. I'm pretty sure that FROM_PAYPAL_SPOOF is triggered at least in part by __NOT_SPOOFED being set to "false" - and DKIM failing does (or can) cause __NOT_SPOOFED to be false - and so in this case a failed DKIM validation, that most likely worked when the message was originally sent - is what's now causing this chain reaction. It's highly doubtful that this rule would have hit at the time the message was received. --Rob McEwen, invaluement -- Original Message -- From "Rob McEwen" To users@spamassassin.apache.org Date 2/21/2023 4:53:27 PM Subject Re: May I get to 0 phishing? Benny, There are a few holes in your theory/assertions: (1) I know for a fact that this came from PayPal's official transactional servers, in PayPal's IP space. And while the sender (PayPal's customer) was a "bad actor", this wasn't PayPal's actual email server getting hacked. Instead, it was PayPal's deliberate notification they sent on purpose, with all the proper authentication that normally is sent in ALL legit PayPal emails. (2) I'm about 99.9% certain that all the validations that fail now - passed when it was originally sent/received. It's actually common for such large senders to expire DKIM record validation either quickly (to make spoofing harder!) and/or to manually expire it when they find fraud in recently-sent spams. One or the other, or both, likely happened here. I'm very confident that some (probably all!) of the validation failures that caused some portion of your bad scoring - weren't there if SA had been run against this soon after it was sent. (3) I'm using SA 4.x, and a few minutes ago, I ran this against SA, and I ran a legit PayPal notification from this SAME IP address, that was sent today - both against SA. "FROM_PAYPAL_SPOOF" never had a hit on either one - but I also have RBLs and URI-lists set to not run in my SA, since I'm doing all that elsewhere - so maybe that disabled FROM_PAYPAL_SPOOF in my system? Or maybe FROM_PAYPAL_SPOOF isn't in SA4? Nevertheless, if this rule is so great and definitive, why is it only scoring 1.2 points? 1.2 points suggests that it might not be 100% immune to false positives! And if your argument is so great, why was your overall SA score ONLY 1.2 points? Do you really think that everyone using SA should "know" to magically block all messages that ONLY score 1.3 points, but have a hit on this rule? Should other SA users have this magical insight about other such SA rules? I think you destroyed your own argument, with your own evidence. And you seem to be overlooking the fact that these are sent from PayPal servers that also send a MASSIVE amount of legit and transactional emails, including from this actual same IP. For example, in the past 24 hours, my small-ish mail hosting system has 6 legit not-spam PayPal notifications sent from this SAME ip address - all 6 of those were legit. Rob McEwen, invaluement -- Original Message -- From "Benny Pedersen" To users@spamassassin.apache.org Date 2/21/2023 4:03:31 PM Subject Re: May I get to 0 phishing? Rob McEwen skrev den 2023-02-21 20:37: https://pastebin.com/v80qMF99 Content preview: Invoice from Apple. com (0005) xxx...@example.com, here are your invoice details Hello, xxx...@example.com Here's your invoice Content analysis details: (1.2 points, 5.0 required) pts rule name description -- -- -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [173.0.84.227 listed in wl.mailspike.net] 1.8 DKIM_ADSP_DISCARD No valid author signature, domain signs all mail and suggests discarding the rest 0.1 DKIM_SIGNEDMessage has a DKIM or DK signature, not necessarily valid 0.1 DKIM_INVALID DKIM or DK signature exists, but is not valid -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [173.0.84.227 listed in list.dnswl.org] -2.0 RCVD_IN_HOSTKARMA_WRBL: Sender listed in HOSTKARMA-WHITE [173.0.84.227 listed in hostkarma.junkemailfiltercom] 0.0 HTML_MESSAGE BODY: HTML included in message 0.1 MIME_HTML_ONLY BODY: Message only has text/html MIME parts 0.4 KAM_REALLYHUGEIMGSRC RAW: Spam with image tags with ridiculously huge http urls 0.0 MIME_QP_LONG_LINE RAW: Quoted-printable line longer than 76 chars 0.0 LONG_IMG_URI Image URI with very long path component - web bug? 1.0 MSGID_NOFQDN2 Message-ID without a fully-qualified domain name 0.5 LONGLINE Line length exceeds 998 character limit, RFC 5322 1.2 FROM_PAYPAL_SPOOF From PayPal domain but matches SPOOFED 0.2 KAM_HUGEIMGSRC Message co
Re: May I get to 0 phishing?
Benny, There are a few holes in your theory/assertions: (1) I know for a fact that this came from PayPal's official transactional servers, in PayPal's IP space. And while the sender (PayPal's customer) was a "bad actor", this wasn't PayPal's actual email server getting hacked. Instead, it was PayPal's deliberate notification they sent on purpose, with all the proper authentication that normally is sent in ALL legit PayPal emails. (2) I'm about 99.9% certain that all the validations that fail now - passed when it was originally sent/received. It's actually common for such large senders to expire DKIM record validation either quickly (to make spoofing harder!) and/or to manually expire it when they find fraud in recently-sent spams. One or the other, or both, likely happened here. I'm very confident that some (probably all!) of the validation failures that caused some portion of your bad scoring - weren't there if SA had been run against this soon after it was sent. (3) I'm using SA 4.x, and a few minutes ago, I ran this against SA, and I ran a legit PayPal notification from this SAME IP address, that was sent today - both against SA. "FROM_PAYPAL_SPOOF" never had a hit on either one - but I also have RBLs and URI-lists set to not run in my SA, since I'm doing all that elsewhere - so maybe that disabled FROM_PAYPAL_SPOOF in my system? Or maybe FROM_PAYPAL_SPOOF isn't in SA4? Nevertheless, if this rule is so great and definitive, why is it only scoring 1.2 points? 1.2 points suggests that it might not be 100% immune to false positives! And if your argument is so great, why was your overall SA score ONLY 1.2 points? Do you really think that everyone using SA should "know" to magically block all messages that ONLY score 1.3 points, but have a hit on this rule? Should other SA users have this magical insight about other such SA rules? I think you destroyed your own argument, with your own evidence. And you seem to be overlooking the fact that these are sent from PayPal servers that also send a MASSIVE amount of legit and transactional emails, including from this actual same IP. For example, in the past 24 hours, my small-ish mail hosting system has 6 legit not-spam PayPal notifications sent from this SAME ip address - all 6 of those were legit. Rob McEwen, invaluement -- Original Message -- From "Benny Pedersen" To users@spamassassin.apache.org Date 2/21/2023 4:03:31 PM Subject Re: May I get to 0 phishing? Rob McEwen skrev den 2023-02-21 20:37: https://pastebin.com/v80qMF99 Content preview: Invoice from Apple. com (0005) xxx...@example.com, here are your invoice details Hello, xxx...@example.com Here's your invoice Content analysis details: (1.2 points, 5.0 required) pts rule name description -- -- -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [173.0.84.227 listed in wl.mailspike.net] 1.8 DKIM_ADSP_DISCARD No valid author signature, domain signs all mail and suggests discarding the rest 0.1 DKIM_SIGNEDMessage has a DKIM or DK signature, not necessarily valid 0.1 DKIM_INVALID DKIM or DK signature exists, but is not valid -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [173.0.84.227 listed in list.dnswl.org] -2.0 RCVD_IN_HOSTKARMA_WRBL: Sender listed in HOSTKARMA-WHITE [173.0.84.227 listed in hostkarma.junkemailfilter.com] 0.0 HTML_MESSAGE BODY: HTML included in message 0.1 MIME_HTML_ONLY BODY: Message only has text/html MIME parts 0.4 KAM_REALLYHUGEIMGSRC RAW: Spam with image tags with ridiculously huge http urls 0.0 MIME_QP_LONG_LINE RAW: Quoted-printable line longer than 76 chars 0.0 LONG_IMG_URI Image URI with very long path component - web bug? 1.0 MSGID_NOFQDN2 Message-ID without a fully-qualified domain name 0.5 LONGLINE Line length exceeds 998 character limit, RFC 5322 1.2 FROM_PAYPAL_SPOOF From PayPal domain but matches SPOOFED 0.2 KAM_HUGEIMGSRC Message contains many image tags with huge http urls 0.1 DMARC_REJECT DMARC reject policy 0.0 LOTS_OF_MONEY Huge... sums of money 0.0 T_REMOTE_IMAGE Message contains an external image dont know more, but dnswl ? ;) DMARC_REJECT && FROM_PAYPAL_SPOOF why accept it ?
Re: May I get to 0 phishing?
Nope. That was a phishing spam, just maybe not the TYPE of phishing spam you're used to seeing? Calling it a fraud doesn't make it not a phish. When is a phishing spam ever NOT fraud? So what's the deciding factor? The fact that this claimed to be Apple sending an invoice via PayPal - and tried to trick the end user into thinking that they were the real Apple - except it wasn't really Apple - it was a criminal masquerading as Apple, trying to trick the recipient into paying this via thinking that this was the real Apple. THAT is the deciding factor that makes this a phish as well as a fraud. (PayPal should have done better customer vetting on the front end!) Rob McEwen, invaluement -- Original Message -- From "hg user" To "Rob McEwen" Cc users@spamassassin.apache.org Date 2/21/2023 3:10:35 PM Subject Re: May I get to 0 phishing? I think this is not a phishing, more a fraud: it seems a real invoice for something you didn't buy. I'm glad to hear from experts that it's impossible to have 0 phishing, that I'm not missing the "silver bullet" or the magic token. I may perhaps implement ESP plugin, and subscribe to DQS, or add a OCR plugin for those very annoying "pay the fine" scams. Really dubious about enabling razor and pyzor for italian language. Unfortunately my spamassassin, version 3.4.5, is embedded into Zimbra, and it makes me really afraid of adding plugins... Suggestions are always welcome On Tue, Feb 21, 2023 at 8:37 PM Rob McEwen wrote: What Bill Cole said! Agreed. For example, here's an almost impossible phish to block (at least, without blocking legitimate PayPal transactional emails!). This is a PayPal phishing spam, sent from PayPal's own server! It was sent by PayPal. I only changed the intended recipient address (to protect the innocent), and changed the "=" at the end of lines MIME-formatting to regular lines, for better readability when looking through the email body for links. Otherwise, not altered. https://pastebin.com/v80qMF99 However - there are always very helpful improvements that can be made for minimizing the number of phish that get into the inbox. It's a constant battle! Rob McEwen, invaluement -- Original Message -- From "Bill Cole" To users@spamassassin.apache.org Date 2/21/2023 2:11:02 PM Subject Re: May I get to 0 phishing? >On 2023-02-21 at 13:51:09 UTC-0500 (Tue, 21 Feb 2023 19:51:09 +0100) >hg user >is rumored to have said: > >>I was wondering if it is possible to reach the goal of 0 phishing. > >Nope. There are people who find it profitable and they will continue to find ways to trick all the usable programmatic mechanisms deployed to stop it. > >>With 2 layers of paid protection, and a third layer realized with >>spamassassin with a lot of hand made rules, I'm able to catch a lot of spam >>and if some reaches the mailboxes, no problem. >> >>But when phishing is able to reach the mailboxes, it is more dangerous, and >>I'd like to bring it to a minimum. >> >>I'd like to know if you, despite all the barriers, still, although rarely, >>have phishing go through, and how do you handle the situation. > >Eternal vigilance and user education. > >The world is an imperfect place. > > >-- Bill Cole >b...@scconsult.com or billc...@apache.org >(AKA @grumpybozo and many *@billmail.scconsult.com addresses) >Not Currently Available For Hire
Re: May I get to 0 phishing?
What Bill Cole said! Agreed. For example, here's an almost impossible phish to block (at least, without blocking legitimate PayPal transactional emails!). This is a PayPal phishing spam, sent from PayPal's own server! It was sent by PayPal. I only changed the intended recipient address (to protect the innocent), and changed the "=" at the end of lines MIME-formatting to regular lines, for better readability when looking through the email body for links. Otherwise, not altered. https://pastebin.com/v80qMF99 However - there are always very helpful improvements that can be made for minimizing the number of phish that get into the inbox. It's a constant battle! Rob McEwen, invaluement -- Original Message -- From "Bill Cole" To users@spamassassin.apache.org Date 2/21/2023 2:11:02 PM Subject Re: May I get to 0 phishing? On 2023-02-21 at 13:51:09 UTC-0500 (Tue, 21 Feb 2023 19:51:09 +0100) hg user is rumored to have said: I was wondering if it is possible to reach the goal of 0 phishing. Nope. There are people who find it profitable and they will continue to find ways to trick all the usable programmatic mechanisms deployed to stop it. With 2 layers of paid protection, and a third layer realized with spamassassin with a lot of hand made rules, I'm able to catch a lot of spam and if some reaches the mailboxes, no problem. But when phishing is able to reach the mailboxes, it is more dangerous, and I'd like to bring it to a minimum. I'd like to know if you, despite all the barriers, still, although rarely, have phishing go through, and how do you handle the situation. Eternal vigilance and user education. The world is an imperfect place. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not Currently Available For Hire
Re[2]: URIDNSBL full message checking
It's actually just a domain name. This uridnsbl keys off domain names in the body too, I was kinda hoping it would look at the domain names in the headers like the body, guess not. So there's an interesting history here. Back in the early/mid 2000s, when SURBL, URIBL, and invaluement's URI lists were just starting (I was there!) - we didn't have reliable and universally-used/established domain authentication tools like SPF and DKIM and even ESPs were either non-existent or just beginning. Therefore, the vast majority of spammers were sending from their own servers (or bots!) - and both the mail header from and the SMTP-envelope FROM - in spams - was 99+% of the time forged. So trying to run a DNSBL that listed the domains found in the headers was a horrible idea because a massive percentage of spam used forged domains. That was then a losing game of whack-a-mole that would only add much useless one-off data to a dnsbl, as well as providing spammers with intel they could use to find DNSBL spamtrap addresses. Today, so much is radically different since now many spams have their domains authenticated with things like SPF and DKIM. Therefore, SURBL and URIBL and Spamhaus's DBL have since moved more towards purposely including those header and SMTP-envelope domains (as well as the domain at the end of the PTR record) as things that they specifically target with their domain/URI lists. But these are things that "consumed" by SA with OTHER rules, not with URIDNSBL. (also, postfix as some good rules for this too which don't require callouts to content filters like SA. Exim and others probably do, too? At invaluement - we're very very late to this game - and we're going a different route - choosing to target these with a separate list, not our URI list - this will be our SED list, which is currently under development - although, in the meantime, many of our subscribers use our existing URI list in this way, outside of our recommendations, and are happy with those results. The main takeaways are: (1) these require different rules than the URIDNSBL module (since URIDNSBL is for checking domains/IPs inside the clickable links in the body of the message) (2) Any DNSBL trying to do should to pay attention to authentication, and not just throwing every such domain in the list without being sure it really is them and not a forged domain. I hope this helps! Rob McEwen, invaluement
Re: Seeking dhl.com ham samples
I provided a ham sample off-list. Also, I've recently encountered a similar issues with DHL - for example - them, several weeks ago, using an alterate domain in the mail header FROM-address - that didn't actually have ANY DNS records - crazy stuff like that - although I think that they've since stopped using that particular domain name? --Rob McEwen On 8/2/2022 10:50 AM, Bill Cole wrote: Bug 8021 reports breakage in SPF checking for dhl.com mail, due to an inability to resolve the SPF TXT record for dhl.com. That breakage is essentially due to DHL having far too many TXT records (some are clearly stale) and having a SPF record which is right at the limit of complexity, having 10 'include' directives at the top level. If anyone has samples of real legitimate mail from a dhl.com address, please share. I'm seeking a way to reproduce the reported bug, which strikes me as too stupid to be real; we SHOULD have noticed long before now if SPF lookups were not handling UDP truncation of replies. -- Rob McEwen, invaluement
rules for a sneaky SPEAR-VIRUS spam that gets past bayes
rules for a sneaky SPEAR-VIRUS spam that gets past bayes because legit
content from hijacked emails are copied into the spam, making it look
like a follow-up msg of an existing legit conversation. Catch using
these rules below. (Perhaps also add more to this to prevent rare FPs?
But this is a good start!)
FILE SIZE < 50kb
then, on decoded/demime'd msg:
exact match on:
*https://onedrive.live.com/download?cid=**
*
Then a hit on THIS RegEx:
*\b(Fil lösenord|File password): [A-Z]{2}\d{4}\b**
*
(I'll let someone else jump in here and create and share the actual SA
implementation of this, if desired - along with any suggested improvements)
-- Rob McEwen, invaluement
Re: Do these domains merit blocking?
On 12/15/2021 11:39 AM, Bill Cole wrote: Am I missing something special that makes such research spam somehow not spam? Everyone thinks that their own unsolicited bulk email - isn't spam. But a line must be drawn somewhere. In this case, the sender has absolutely no preexisting relationship to the recipient, and Raymond's statement about them sending to "scraped addresses" is, imo, devastating to their case. The closest argument that might have been possible is the idea that the email might potentially be of more benefit to the recipient than it is to the sender (e.g., sort of like a notification about a class action lawsuit) - but I can't find that argument anywhere in this situation either. But even class action lawsuit notifications are rarely sent to scraped addresses. It's on my "to do" list to add those domains as permanent additions to invaluement's URI/domain bl sometime this week, when I get some more time. (I'm in the middle of some intense upgrades, so I barely had time to type this message.) -- Rob McEwen, invaluement
Re: OT: is sorbs.net sleeping ?
On 4/10/2021 6:55 AM, Jared Hall wrote: Rob, I gotta say that I am impressed with the whole Spamhaus-dqs program and their use of customer keyed DNS zone queries. Seems to be the way around the client DNS forwarder issues. How are you guys at Invaluement tracking in that area? I'm not sure I'm understanding what you're saying? Are you referring to the fact that their paid customers doing direct queries (NOT the free stuff!) - use zone names that have a unique key embedded into the actual zone - so that the queries can then be distinguished by this unique key? - thus eliminating the need to use the client's local DNS servers' public IP as the method of allowing/denying direct queries? Is that what you're referring to? Seems to be the way around the client DNS forwarder issues If I'm correct about what you meant - then yes - this eliminates problems that used to happen when trying to track customers, and permission, by IP - because when tracking by an embedded code - then it doesn't matter from WHERE the queries come - and queries that come from public DNS servers (8.8.8.8 or 1.1.1.1) - can be distinguished one from the other - whereas when not doing this - it's impossible to tell distinguish the queries from each other and know who is doing them. This became especially important because so often the default caching DNS server gets auto-flipped to 8.8.8.8, sometimes without the IT person's knowledge! And many IT people think that pointing to 8.8.8.8 is the textbook way to setup DNS - and have never even heard of things like BIND. Is THAT what you're talking about? If so, at invaluement, we've been doing this for 3 years now - but we still have a lot of work to do in migrating many long-time customers over to our new system. And it was developed before I even knew that Spamhaus was doing it this way, and this involved some extremely complex custom modifications of rbldnsd (I couldn't afford to hire an expensive high-quality C++ programmer at the time - so it took me about 100 hours of very intense programming to do that! It didn't help that I'm not very good at C++!). I'm not even sure when Spamhaus started this. Our new system for doing this now involves 86 servers in 43 cities around the world - which enables our clients to get their queries answered much faster due to accessing an invaluement DNS server with an extremely close geolocation. Queries then tend to get answered in a very low number of milliseconds - often <10ms. -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
Re: OT: is sorbs.net sleeping ?
On 4/9/2021 10:34 AM, Benny Pedersen wrote: above ip is not listed yet, with inho is sign of no maintain at all anymore So I noticed that this IP you mentioned is a heavily-listed IP that is currently listed on many DNSBLs, including many of the best and most reliable and accurate ones. (I think that was part of your point.) So you're complaining that SORBS isn't listed this one. Maybe you were providing this as a representative example, correct? So I guess you're saying that there are more like this? But for the sake of clarity, let me just say that no DNSBLs should ever be judged too harshly for "false negatives" - no DNSBL has the exact same view of the worldwide email data - and each DNSBL's false positive prevention filters will always make SOME mistakes that cause "false negatives" - that's a very acceptable price to pay considering that no system can ever be perfect. Low false positives AND overall catch-rates AND overall UNIQUE catch-rates (blocking stuff everyone else is still missing) - are all far more important metrics. (you might be disappointed with SORBS in those areas too? - that's fine - I'm just trying to clarify that overly judging a DNSBL based on /*particular*/ false negatives can be overly harsh and might miss the good things that a DNSBL has to offer) -- Rob McEwen, invaluement +1 (478) 475-9032
Re: Bypass RBL checks for specific address
On 12/22/2020 6:56 PM, Grant Taylor wrote:
Is there a way to bypass RBL checks for a specific address? I've tried
the all_spam_to option, but it looks like it artificially lowers the
score and still runs normal tests. I'd like to disable RBL checks for
one address.
Grant,
First, I'm NOT an expert on all of this - so somebody might be able to
follow up with BETTER information, but this will hopefully point you in
the right direction.
So at some point, I think 10+ years ago, I needed to do this, and
instead of doing this at the spam filter level - I found that it worked
well to do this via BIND - so this works if you're using your own
locally-hosted BIND server for resolving DNSBL queries. So, for example,
if you qualified for use of the free version of SpamCop (as an example),
and you wanted to whitelist the IP 1.2.3.4, you could add the following
to your named.conf file:
zone "4.3.2.1.bl.spamcop.net" in { type master; notify no; file
"master/null.zone"; };
Then add another such record for each DNSBL that you use. Then the
"master/null.zone" file can be the following EMPTY zone (or something
like this - change as desired! This might not even be the best or
"right" way to do this - this is just what I had in that null.zone file
when I was doing this 10+ years ago):
$TTL 52w
@ IN SOA root.localhost. root.localhost (
2005012001 ; serial
52w ; refresh
52w ; retry
52w ; expire
52w ; ttl
)
@ IN NS localhost
localhost IN A 127.0.0.1
So by referencing an empty zone file, that way, each "zone" entry just
points to this one file, for maximum efficiency and caching, and you
don't have to reenter this for each zone. In the named.conf file, I
think each zone statement (my 1st example abovw) would go AFTER the
"options" section, but before the "includes" section. Obviously, if you
suddenly needed to do this for 10s of thousands of IPs or hostnames,
then maybe it would then start to have resource/maintenance issues - but
on a smaller scale, this works great!
HOWEVER - This was so long ago, that I don't know if this STILL works in
more recent version of BIND without causing issues? It is possible that
in more recent version of BIND, DNSSEC might interfere with being able
to do this?
So ANOTHER option might be to use the newer "response-policy" feature -
my first idea was a sort of hack - but this "response-policy" might be
more intended for exactly this purpose. So do a search on the following:
"response-policy" "BIND" "NXDOMAIN"
...with each in quotes, as shown, for instructions on how this is done.
So I think the example above, if implemented using "response-policy",
would be the following:
response-policy {
4.3.2.1.bl.spamcop.net IN CNAME .
};
Or something like that. Double-check my syntax. It might be wrong! So,
again, I've never done the "response-policy", so this is just to get you
started and point you in the right direction. If someone comes along and
corrects my possible mistakes, or provides BETTER info - that is
excellent - in the meantime, hopefully this will point you in the right
direction, or give you some ideas.
-- Rob McEwen, invaluement
Re: Mailchimp support for spamassassin-esp
On 11/30/2020 5:40 PM, Alex wrote: I happened to notice today that the sendgrid spam work being done by Invaluement (https://www.invaluement.com/serviceproviderdnsbl/) and SA developers now apparently supports compromised Mailchimp domains. https://github.com/bigio/spamassassin-esp Is there an ongoing list of compromised mailchimp domains available to be used with this? That info is not included with the man page for this plugin. I also know there's another plugin developed by Paul Stead for this, but has one yet become the defacto version yet? Alex, So yes - this one *is* the official/defacto version for SpamAssassin. This one was developed in coordination with the Apache Foundation and its development was partially funded by invaluement. The problem here is that the main developer (to his credit - this is a GOOD problem!) got ahead of us with the implementations. But we're in the process of catching up on the data-generation side, and hope to have those new types of data released in the next few weeks (for those ones mentioned in those rules, and for other ESPs that will get into those rules eventually). The entire process of developing the engine that produces that SendGrid data - was the equivalent of our entire invaluement staff taking at least a full month of paid leave away from our regular duties. So that got us horribly behind on other things - including getting this data into our regular paid datafeeds with instructions sent to our customers for that - so we're still catching up on all of that - but we hope to get past that soon and to also have those /*other*/ related datafeeds for our "service provider DNSBL" released soon. (it won't be as much work for the other ones, now that the sendgrid anti-spam data engine is already completed - it "blazed the trail") So that explains why a few months have passed since the sendgrid data was released without any additional data being released yet, and how/why the developer of the rules was able to get ahead of us. (again, to his credit!) Thanks for your patience and understanding. -- Rob McEwen https://www.invaluement.com
Re: Apache SpamAssassin and Spammers 1st Amendment Rights
On 11/20/2020 4:59 PM, Charles Sprickman wrote: non-profit stuff in the US is EXEMPT from CAN-SPAM, so they don’t even have to play by the rules still, that doesn't give them a free pass to the inbox, or prevent them from potentially getting listed on an anti-spam list. MANY spammers that are blocked by spam filters and/or listed on anti-spam lists - were already CAN-SPAM compliant. Being *legal* is a very low bar for email, especially in the U.S. -- Rob McEwen, invaluement
Re: Apache SpamAssassin and Spammers 1st Amendment Rights
On 11/20/2020 4:37 PM, Eric Broch wrote: It seems spammers are using political arguments to justify their actions. I'll give them credit, at least they're trying to justify what they do by something greater than (outside of) themselves, albeit wrongly. It seems people on this side of the argument want to jettison politics (and religion) and have no justification (only personal preference) for what they do. Curious! At the core spammers seem more logically consistent than those who oppose them. I have extremely large amounts of spams on file in my spamtrap spam collection from all various political viewpoints, political parties, and moral/ethical/religious viewpoints - MANY of them think that THEIR greater good justifies spamming, and ironically their beliefs are often in 100% contradiction to OTHER spammers who have opposite beliefs, but likewise think that their spam is justified by THEIR "greater good". Thankfully, it isn't my job to determine who is justified and, instead, I believe that NONE of them are justified in sending spam - spam is about *consent* - NOT *content*. -- Rob McEwen, invaluement
Re: Crap getting through
Daryl, Can you please post a copy of the raw email message - with headers - perhaps with your own user's email address (and name?) masked out (change to "") - to pastebin, or to a similar site - then reply here with the link. It is difficult to give specific suggestions without having the raw underlying text of the message (w/headers). But please try to avoid pasting that directly to this list. Thanks! Rob McEwen On 11/8/2020 5:00 PM, Daryl Rose wrote: I'm getting obvious phishing attempts. This one was made to look like it was from Wells Fargo with an obvious spoofed email address. However, when I examined the headers, the From Address was this garbage: *=?utf-8?B?V+G7hWxsc+G4nmFyZ28gQmFuaw==?= * I received another one that was meant to be an Amazon Prime Membership failure. How can I block these? The last time I inquired about phishing, it was suggested to install KAM, which I did, but this crap is still getting through. Any other suggestions? Thank you. Daryl -- Rob McEwen, invaluement
Re: Invaluement sendgrid list
Micah, I haven't heard any other complaints about this - so hopefully the problem you described is rare or at least intermittent. But, nevertheless, I made a change that should GREATLY improve things. Please let me know if this fixes/improves/worsens the situation you described. Thanks for the feedback - and feel free to continue this conversation off-list since the SA list isn't suppose to be the invaluement support list. (or, email me at any time about such things - r...@invaluement.com) - Thanks! Rob McEwen, invaluement.com On 10/13/2020 12:56 PM, micah anderson wrote: Hi all, I've been trying the https://www.invaluement.com/spdata/sendgrid-id-dnsbl.txt list but lately, I've been getting 'Couldn't connect to server' errors, fairly regularly. The site says: 'can set them up for frequent downloads (every minute!) using CURL or WGET - only using the setting that only downloads when the server versions are newer.' I am doing that, once per minute... are others having this issue? thanks -- Rob McEwen, invaluement
Re: blacklisting the likes of sendgrid, mailgun, mailchimp etc.
On 9/18/2020 6:38 AM, Loren Wilton wrote: https://krebsonsecurity.com/2020/08/sendgrid-under-siege-from-hacked-accounts/ also sheds light on the issue too. . SendGrid knows (or should konw) that it has compromised accounts. It could find out what some of them are for free by downloading Rob's list of 25 or so compromised accounts. I strongly suspect that many of those accounts on our 2 Sendgrid lists are just plain 'ol spammers, NOT compromised. So some are compromised, some are spammers. And the list has grown to 594 SendGrid IDs (currently, as I type this) - much more than 25! Also, the list of domains found at the end of the SMTP-FROM that we're also deeming as spam or malicous has likewise grown to 87 domains SEE: https://www.invaluement.com/spdata/sendgrid-id-dnsbl.txt AND: https://www.invaluement.com/spdata/sendgrid-envelopefromdomain-dnsbl.txt https://www.invaluement.com/spdata/sendgrid-envelopefromdomain-dnsbl-rbldnsd.txt (2nd one formatted for rbldnsd) I'm seeing evidence/reports that Sendgrid is likely using this data to greatly improve their system, and that this (maybe combined with their other efforts?) is finally starting to improve things? So that is good news. But I'm also shocked at how many hours go by where certain egregious accounts on our Sendgrid DNSBLs STILL stay in circulation while continuing to send spams, sometimes criminal phishing spams. But I also understand that they have to be careful about overly trusting 3rd party data, to ensure that they don't overreact to what might be an occasional false positive. It shouldn't be too long before they figure out that False Positives in those two Sendgrid lists are very very rare... practically non-existent. They probably should at least PAUSE campaigns pending further investigation. They should at least do that much, imo. (They MIGHT also be suffering from the increasingly common and flawed view in the ESP industry - that not-illegal and CAN-SPAM-compliant mail is always legit and not spam - mistakenly not understanding that spam doesn't have to be illegal and malware, in order to be unsolicited and undesired by the recipient (aka "spam"). Maybe them seeing those types of accounts in our data is confusing them? I don't know - but much of the ESP industry is in great need of a "reset" - and this data is a good first step towards that!) I was planning to spend much time this past week (1) adding this data to my own customer's direct query and rsync feeds, and (2) improving the instructions, including providing more specific instructions for adding this free version to various MTAs - but all that time got put into performance and effectiveness enhancements instead. Therefore, the data has greatly improved in just the past few days. New data sources were added into the mix - and many others of these spams these that were previously getting missed, are now getting caught - and the time from such a spam being first received - to that data getting into the list - has improved from about 1/2 a minute, to just a few seconds! -- Rob McEwen invaluement.com
Re: ANNOUNCEMENT: The NEW invaluement "Service Provider DNSBLs" - 1st one for Sendgrid-spams!
On 8/25/2020 11:04 PM, John Hardin wrote: I just wrote something similar to generate a rule, in case for some reason you don't want to use a plugin. Let me know if there's any interest in it. yes - please share! -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
Re: ANNOUNCEMENT: The NEW invaluement "Service Provider DNSBLs" - 1st one for Sendgrid-spams!
Thanks, John Capo, for the suggestions! Honestly, I'm at the end of my
rope - completely burned out from creating this - desperately needing to
catch up in other areas of my business so that I can pay my bills. And I
have other ideas for how to make this data even better that I'm trying
to get to asap. So help like this is very appreciated!
BTW - does Postfix "know" to refresh the data when the files are
updated? Or is there some kind of command that needs to run to tell
Postfix to reload the files? How does that work? ALSO - would it help if
I created a separate set of files for Postfix that are pre-formatted
this way already?
Thanks!
Rob McEwen, invaluement.com
On 8/25/2020 2:26 PM, John Capo wrote:
On 2020-08-25 11:42, Matus UHLAR - fantomas wrote:
well, do we have anything available now to block at SMTP level?
- postfix policy server?
- milter?
so far I have noticed only SA plugins. Which is not bad, but that HUGE
advantage is not usable now.
Nothing elegant about this but it was easy to implement. You need to
create the software specific to your MX servers to update the files
below from Rob's web site.
Adjust the paths below to your Postfix install
Add these entries to your main.cf:
smtpd_restriction_classes =
sendgrid
# Limit senders that are matched with the regexes in sendgrid-ids
#
sendgrid =
check_sender_access pcre:/usr/local/etc/postfix/maps/sendgrid-ids
smtpd_recipient_restrictions =
check_sender_access hash:/usr/local/etc/postfix/maps/from-sendgrid
Create a file like this from the senders in
https://www.invaluement.com/spdata/sendgrid-envelopefromdomain-dnsbl.txt
sendgrid.net sendgrid
appliedaicourse.com sendgrid
bithumbcorp.email sendgrid
bitline.life sendgrid
bureausveritas.com sendgrid
caractere.ro sendgrid
craftsgenerals.com sendgrid
dalvry.com sendgrid
...
Name it from-sendgrid and place it in your Postfix directory
postmap from-sendgrid
Create a file like this from the ids in
https://www.invaluement.com/spdata/sendgrid-id-dnsbl.txt
/^bounces\+2191708-[0-9a-f]{4}-/ REJECT Phish from compromised
Sendgrid account
/^bounces\+4227563-[0-9a-f]{4}-/ REJECT Phish from compromised
Sendgrid account
/^bounces\+13780591-[0-9a-f]{4}-/ REJECT Phish from compromised
Sendgrid account
/^bounces\+10163588-[0-9a-f]{4}-/ REJECT Phish from compromised
Sendgrid account
/^bounces\+10180020-[0-9a-f]{4}-/ REJECT Phish from compromised
Sendgrid account
...
Name it sendgrid-ids and place it in your Postfix directory
postfix reload
John Capo
Tuffmail.com
--
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032
Re: ANNOUNCEMENT: The NEW invaluement "Service Provider DNSBLs" - 1st one for Sendgrid-spams!
On 8/25/2020 2:29 PM, Benny Pedersen wrote: maybe make clamav sigs ? Benny, Thanks for your other suggestions - those are worth exploring. Also - the Clamav Sigs is not a bad idea - but even besides the fact that (like SA rules), Clamav is content filtering and not at the SMTP-Envelope level - Clamav doesn't tend to have nearly AS fast of a turnaround time as do DNSBLs. In a previous message, someone was disappointed that we missed one, and it turns out our 24-second turnaround time on that message (from the start of the SMTP connection - to being fully deployed in the data) was a contributing factor. We now have a plan to shorten that 24-seconds to about 4 seconds AND (for invaluement subscribers) - we have a "push" technology that is available now where those invaluement subscribers who opt for this feature (no extra charge!) - can get a split second notification to run their RSYNC just 1 second after the file updates - and we do that already for our direct query servers. So there is an option (once implemented!) to potentially get the these FULLY DISTRIBUTED within about 8 seconds from the start of the SMTP connection of the first such spam received - to being FULLY deployed on DNS servers (both our own direct query servers - and our RSYNC subscribers' internal rbldnsd servers) - that will be AMAZING. I expect to be there within a week from now. Something like clamav just can't even begin to compete with that fast of a turnaround. But ClamAv rules may still be a good way to get this implemented for many. Someone else mentioned one that was completely off of our radar - but we're about to double the coverage of these in terms of mailboxes and traps used for this purpose - so that will help further minimize our "blind spots". -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
Re: ANNOUNCEMENT: The NEW invaluement "Service Provider DNSBLs" - 1st one for Sendgrid-spams!
On 8/25/2020 1:20 PM, Rob McEwen wrote: but I can do everything, at least not all at once *can't do -- Rob McEwen https://www.invaluement.com
Re: ANNOUNCEMENT: The NEW invaluement "Service Provider DNSBLs" - 1st one for Sendgrid-spams!
On 8/25/2020 11:42 AM, Matus UHLAR - fantomas wrote: well, do we have anything available now to block at SMTP level? - postfix policy server? - milter? so far I have noticed only SA plugins. Which is not bad, but that HUGE advantage is not usable now. And likewise - 48 hours ago - a SpamAssassin plugin didn't exist either! These things take at least a little bit of time. We're only at the 3rd business day that this tech has been in existence. But I think you and I would both be surprised at how many systems are likely already (quietly) using this at the SMTP-connection level, for certain more custom-programmed systems. I believe adaptation in other public MTAs is inevitable. For example, I have some good contacts at Exim and it's on my "to do" list to ask them about this, but I can do everything, at least not all at once. And those MTAs that don't enable usage of this will be left behind. PRO TIP: Instead of complaining about this problem on this thread - why not go to the discussion list or forum of your preferred MTA - and ask them to implement it? -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
Re: ANNOUNCEMENT: The NEW invaluement "Service Provider DNSBLs" - 1st one for Sendgrid-spams!
On 8/22/2020 3:35 PM, Kenneth Porter wrote: --On Saturday, August 22, 2020 11:15 AM -0400 Jered Floyd wrote: Like most ISPs, they have a feedback loop to remove malicious users. I assume it is too slow, so a SendGrid account ID RBL would provide meaningful value. Would not Pyzor accomplish the same thing? Submit the SendGrid spam to Pyzor to quickly get it blacklisted. (1) Pyzor requires resource-expensive content filtering - whereas the sendgrid list can do the filtering at the SMTP-envelope level - BEFORE the message is even downloaded - for some systems with millions of users - that is a HUGE advantage. (2) being filterable at the SMTP-Envelope level opens up possibilities for things like MTA plugins or feature additions - that enable this filtering at the MTA level - for MTAs that do NOT try to do any content filtering of the message. That creates more options for deployment where many will hopefully be able to make use of this, who don't have Pyzor (for whatever reasons) (3) The strategy you described is SOMETIMES easily defeated with certain variations in the messages, where each message is sufficiently different to NOT be blockable by Pyzor. That is a HUGE loophole in Pyzor technology. This Sendgrid ID list doesn't have that problem. (4) Also, a spammer who sends out many different types of spams - can potentially stay off of Pyzor's radar - but yet ALL of those spams under that Sendgrid ID - will be collectively noticed in our engine. And, likewise, Pyzor's methods could create a game of whack-a-mole. The spammer will just keep coming out with new types of spam - that all get past Pyzor while Pyzor tries to catch up - then Pyzor catches up - then the spammer just reformats the content. Rinse. Repeat. Meanwhile, ALL of those LATER spams are ALREADY blocked by our Sendgrid list BEFORE the next types of spams are sent - ALL OF THEM. (you could argue that we might get into a game of whack-a-mole too with those Sendgrid IDs - but we're FAR less vulnerable to that - it will happen MUCH LESS often!) (5) for these reasons and others - I strongly suspect that our Sendgrid list is going to have a MUCH faster turnaround time on listing the initial spams from a new sendgrid ID - and, as mentioned, their later spams will then ALREADY be caught by this Sendgrid list - while Pyzor is bogged down in that silly whack-a-mole game. Don't get me wrong - Pyzor and other such checksum content filters - are wonderful and have their place - but thinking that they remove the need for this Sendgrid list - is absolutely not even close to true. -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
ANNOUNCEMENT: The NEW invaluement "Service Provider DNSBLs" - 1st one for Sendgrid-spams!
ANNOUNCEMENT: The NEW invaluement "Service Provider DNSBLs" - 1st one for Sendgrid-spams! ...a collection of a new TYPE of DNSBL, with the FIRST of these having a focus on Sendgrid-sent spams. AND - there is a FREE version of this - that can be used NOW! (/well... might need a SpamAssassin rule or two! Your help appreciated!)/: INFO AND INSTRUCTIONS HERE: https://www.invaluement.com/serviceproviderdnsbl/ This provides a way to surgically block Sendgrid's WORST spammers, yet without the massive collateral damage that would happen if blocking Sendgrid domains and IP addresses. But we're NOT stopping at the phishes and viruses - and we're not finished! There will be some well-deserved economic pain, that puts the recipients' best interests at heart. Therefore, flagrant "cold email" spamming to recipients who don't even know the sender - is also being targeted - first with the absolute worst - and then progressing to other offenders as we make adjustments in the coming weeks. -- Rob McEwen https://www.invaluement.com
Re: Bombard by spam source in India that wasn't in any RBL used by spamassassin.
fwiw - this has been blacklisted at invaluement for days. --Rob McEwen, invaluement.com On 11/6/2019 2:33 PM, Mark London wrote: Hi - We got several hours of spam from the IP address 103.136.41.36 in India. When I did a Multi-RBL check, the ip address was in the following databases: bl.emailbasura.org dnsbl.sorbs.net dns.spfbl.net spam.spamrats.com truncate.gbudb.net I think sorbs.net is a paid for service. At least I tried adding rules, but they weren't triggered. I was able to successfully add rules for spamrats and gbudb. Does anyone have experience with those? After about 3 hours, the IP address finally appeared in barracudacentra.org, which spamassassin uses. Given the amount of traffic we were receiving, I'm surprised it didn't show up sooner on the other RBLs. Thanks. - Mark -- Rob McEwen https://www.invaluement.com
announcement about invaluement (or more like a tease?)
announcement about invaluement (or more like a tease?) https://www.linkedin.com/feed/update/urn:li:activity:6571558988201148416/ -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
HostKarma status (was Re: How to block mails from unknown ip addresses?)
On 8/24/2019 6:12 PM, Benny Pedersen wrote: None of the mails is listed at hostkarma.junkemailfilter.com. I also use junkemailfilter to score spam. unmaintained now (BCC'ed to new HostKarma mgmt) Not true. It is under /different/ management now. They are struggling a little because of the large learning curve of abruptly stepping into someone else's shoes on short notice, with lots of proprietary code/processes. Some months ago, I talked to the main tech person over there and he is a smart/good guy who is making MUCH progress. So it looks like HostKarma is going to survive for the long term. -- Rob McEwen https://www.invaluement.com
Re: Freshclam Safebrowsing enabled for SA
On 4/23/2019 11:07 AM, Kevin A. McGrail wrote: I was going to try and run a second daemon or look at hits for Safebrowsing. as a method for scoring, not blocking. The listing and delisting policies are unclear to me and I think there is a good potential for FPs. Probably a nice scoring option - So like Kevin, I'd caution against using this for blocking or high scoring. Why? Because in recent years there has been an epidemic of the following two things: (1) website compromised - hacker installed malicious content (2) email account on the mail server compromised - spammer is sending email from that server HOWEVER - MOST of the time ONLY 1 of these things happened, NOT both. But the Safebrowsing database is mainly focused on the website being compromised. Therefore, this rule is likely fantastic when it comes to hits on content in the body of the message, particularly URLs linking to malicious content on hijacked websites. But if/when this instead has hits on things like ONLY domain name (in the FROM address or elsewhere) - then it might cause a significant number of FPs if/when it hits stuff like that. I'm not very familiar with how this works when implemented in ClamAv - so, for example, if this only has hits on entire URLs going all the way to the malicious content (not merely referencing the domain or home page) - then my FP concerns are likely overstated and this really isn't going to cause many FPs. So I'm just mentioning this so others will be aware and know what to look for when testing this. -- Rob McEwen
Re: How to deel with time limit exceeded
Another thing that helps - is to lighten the load on your SA by putting high quality low-FP DNSBLs in front of SA, that are first called by your MTA, where spams blocked by those aren't even scanned by SA. --Rob McEwen On 11/5/2018 2:48 PM, Andreas Thienemann wrote: Hi, I've got a mailserver dealing with a moderate amount of mails... Say about 20k mails a day. The setup has a spamassassin 3.4.2 with a mysql database for bayes and to do a minimum of logging. I've recently seen a lot of mail scans not completing due to running out of time. The message I am seeing most often is "check: exceeded time limit in Mail::SpamAssassin::Plugin::Check::_eval_tests_type11_prineg90_set3, skipping further tests". This is happening for abour 5% of incoming mail. I am trying to understand what exactly is going on here? It seems that this is happening since I moved bayes to a local mysql database to resolve locking problems I saw in the logs with the on-disk files. I previously had configured spamd to run with --timeout-child=30 which now seems too small. Having looked at the code of SA I understand that this string is built using the following variables: $package_name."::_".$ruletype."_tests_".$clean_priority; But that does not really help me understand _what_ is taking that long and how I can improve the spamassassin runtime. Any ideas? thanks, Andreas -- Rob McEwen
Re: FPs on FORGED_MUA_MOZILLA (for my own hand-typed messages from my latest-version Thunderbird)
The thread has gone somewhat off-topic, which is partly my own fault.
The issues with URIBL misusage is a "side note", NOT the main purpose of
this thread. (again, that is party my fault since I mentioned that to
begin with). Also, I want to make sure that everyone knows that it was
my client (NOT ME!) that was using URIBL incorrectly. I'll educate my
client to hopefully fix that problem soon.
NOW... BACK ON THE MAIN TOPIC:
On 10/2/2018 1:52 PM, Matus UHLAR - fantomas wrote:
Message-ID: <39397904-9830-5010-a3d2-a62af8326...@invaluement.com>
this does seem to match:
MESSAGEID =~
/^<(?:[a-f\d]{8}-(?:[a-f\d]{4}-){3}[a-f\d]{12}|[A-F\d]{8}\.[A-F1-9][A-F\d]{0,7})\@\S+>$/m
8h-4h-4h-4h-12h@
hmmm we need to look at
(__LYRIS_EZLM_REMAILER || __GATED_THROUGH_RCVD_REMOVER ||
__WACKY_SENDMAIL_VERSION || __IPLANET_MESSAGING_SERVER ||
__HOTMAIL_BAYDAV_MSGID || __SYMPATICO_MSGID)
I really don't think I've done anything unusual with my setup of
Thunderbird. Does anyone have other suggestions? Is there anything I can
do with my Thunderbird settings to mitigate this?
Thanks!
--
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032
Re: FPs on FORGED_MUA_MOZILLA (for my own hand-typed messages from my latest-version Thunderbird)
Bill, Even though this part wasn't the main purpose of the thread, that is still very helpful information. I will pass that along to my client so that they can hopefully fix their configuration problem with regards to their usage of URIBL. Thanks! Rob McEwen Sent from my Verizon Motorola Droid On Oct 2, 2018 11:48 AM, Bill Cole wrote: > > On 2 Oct 2018, at 9:36, Rob McEwen wrote: > > > SIDE NOTE: I don't think there was any domain my message that was > > blacklisted on URIBL - so I can't explain the "URIBL_BLOCKED", but > > that only scored 0.001, so that was innocuous. I suspect that that > > rule is malfunctioning on their end, and then they changed the score > > to .001 - so just please ignore that for the purpose of this > > discussion. > > No, "URIBL_BLOCKED" means that the URIBL DNS returned a value that is > supposed to be a message to a mail admin that they are using URIBL wrong > and will nevewr get a useful answer without either (1) paying for a feed > to support their usage volume or (2) using their own recursive resolver > instead of forwarding queries to the likes of Google, OpenDNS, & > CloudFlare. > > A mail filtering system that gets URIBL_BLOCKED hits is broken. A mail > filtering system that gets them chronically is mismanaged.
Re: FPs on FORGED_MUA_MOZILLA (for my own hand-typed messages from my latest-version Thunderbird)
On 10/2/2018 9:59 AM, Matus UHLAR - fantomas wrote: can you post the headers? or at least the Message-Id? Matus... first, THANKS for your help with this! Here is the message as THEIR system saw it (with my client's info masked) - but it looks like their Kerio (or the customer's email client?) might be not be storing everything as it was originally sent? ...but this is what my client sent me, fwiw: Received: from mail.powerviewmail.com <http://mail.powerviewmail.com>([204.9.77.40]) by with ESMTPS (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256 bits)) for ; Mon, 1 Oct 2018 15:17:10 +0200 DKIM-Signature: a=rsa-sha256; t=1538399816; x=1539004616; s=ivm_invaluement; d=invaluement.com <http://invaluement.com>; c=relaxed/relaxed; v=1; bh=C6QzEUsPRf8EoiIEIhSF1hnXxy9JIlmjGFO/079v4QQ=; h=From:Subject:Date:Message-ID:To:MIME-Version:Content-Type:In-Reply-To:References; b=V5Sv2lZUWL4P29pcEVY6r/8uFRcuNL1hR794r6M1TJZcvw+i4vTgrvWf+CKSN/F1f2FS/0CdF4UCux+dS/vFjj3X9fdmwv9jpizZqwvJseyCYEmT2HItdeqo0NfNIoQwziEPDMgYS3f35iWlcb7wqrPjfx5EslHr+oC0eoeGBaA= Received: from [204.9.77.40] ([204.9.77.40]) by mail.powerviewmail.com <http://mail.powerviewmail.com>(IceWarp 12.0.2.1 x64) with ASMTP id 201810010916565985 for ; Mon, 01 Oct 2018 09:16: Here is an excerpt from the headers, copied from the message in my Thunderbird "sent" folder: References: <55521fa7.8080...@invaluement.com> <7c8ad385-8b3d-74d9-7d34-ca2ca9236...@invaluement.com> <1b8ad5ec-18b7-90db-5cad-d86ffa5aa...@invaluement.com> Message-ID: <39397904-9830-5010-a3d2-a62af8326...@invaluement.com> Disposition-Notification-To: Rob McEwen Date: Mon, 1 Oct 2018 09:16:55 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <1b8ad5ec-18b7-90db-5cad-d86ffa5aa...@invaluement.com> Content-Type: multipart/mixed; boundary="54AEB3A413950E8E0A41E1A8" Content-Language: en-US The time difference makes sense because their time zone is 6 hours ahead of mine. -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
FPs on FORGED_MUA_MOZILLA (for my own hand-typed messages from my latest-version Thunderbird)
A client of mine wasn't getting my own hand-typed messages. Unfortunately, they had their SA set to block on a score of 3 (which is aggressive), and this particular rule hit plus a tiny bit of other things put it above 3. But what is weird - is that it was hitting on hand typed-messages from me - that I sent directly from my latest-version of Thunderbird. So this was NOT "forged" at all! (Also, I suspect that the bayes hit was due to previous such messages from me getting blocked and feeding his bayes?) Any suggestions? Could my client be using a very old version of SA - where this is fixed already? (they are using SA from Kerio). Here are the headers: X-Kerio-Anti-Spam: Build: [Engines: 2.15.8.1169, Stamp: 3], Multi: [Enabled, t: (0.12,0.017258)], BW: [Enabled, t: (0.13)], RTDA: [Enabled, t: (0.052863), Hit: No, Details: v2.7.15; Id: 15.1i65djr.1conscun2.ocr1k], total: 0(700) X-Spam-Status: Yes, hits=3.8 required=3.0 tests=KERIO_ANTI_SPAM: -0.000, AWL: -0.000, BAYES_50: 1.567, FORGED_MUA_MOZILLA: 2.309, HTML_MESSAGE: 0.001, URIBL_BLOCKED: 0.001, TOTAL_SCORE: 3.878,autolearn=no Suggestions? SIDE NOTE: I don't think there was any domain my message that was blacklisted on URIBL - so I can't explain the "URIBL_BLOCKED", but that only scored 0.001, so that was innocuous. I suspect that that rule is malfunctioning on their end, and then they changed the score to .001 - so just please ignore that for the purpose of this discussion. -- Rob McEwen https://www.invaluement.com
Re: using URIBL on other headers
to limit false
positives will cause them to defer on the listing - even though it would
have been an excellent and justified ratio of spam-to-ham blocked, with
little collateral damage if the mail systems using that list could have
ONLY blocked using one method or the other, NOT both! DBL likely errs on
the side of less collateral damage - so it is should be safe to use DBL
for blocking based on both methods, as they prescribe, especially
considering Spamhaus' reputation for extremely low false positives.
Then, other URI lists can pick up the slack on the occasional False
Negatives.
(7) Given this information, at invaluement, we have solved this problem
by creating a new domain blacklist ("ivmSED") that is independent of
ivmURI, where ivmSED is a domain blacklist used ONLY for blocking based
on the domains found on the SMTP envelope (FROM, PTR record, HELO) - and
where ivmSED NOT be used for blocking domains in clickable links in the
body of the message, since that is the job of our ivmURI list. That way,
ivmSED and ivmURI are independent, and we then have the flexibility to
block a domain using either method independently, or both together, for
the approach that most surgically targets the spam, keeps collateral
damage to a minimum, and without compromises that lead to more false
negatives. ivmSED has just recently entering beta testing. (SED =
"Sender's Envelope Domain").
--
Rob McEwen
https://www.invaluement.com
Re: DNS and RBL problems
On 9/14/2018 1:36 PM, Alex wrote: Hi, For the past few weeks I've been having problems with queries to many of the common RBLs, including barracuda, mailspike and unsubscore. My logs are filled with "Name service error", SERVFAIL and lame-server messages for RBLs I know to be valid. Alex, Coincidentally, a recent new invaluement subscriber was initially having at least similar problems that didn't make sense. I was stumped. It made no sense that it wasn't working because everything looked correct. But then he figured out that the following bug was the cause, and fixing this bug enabled the queries to start working again: NOTICE: SpamAssassin installations affected by a bug, due to a change Net::DNS made in an earlier version, here is the bug for reference: https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7223 So you should definitely check to see if this is causing your problem? -- Rob McEwen https://www.invaluement.com
Re: CVE-2018-12558: DOS in perl module Email::Address
On 6/20/2018 1:30 PM, Bill Cole wrote: http://www.openwall.com/lists/oss-security/2018/06/19/3 SpamAssassin does not use Email::Address. Thanks, Bill, for clarifying that. I've been concerned about this for hours - but too busy today research it myself. -- Rob McEwen
Re: OT: Congratulations Dianne
On 4/3/2018 1:18 PM, Axb wrote: AppRiver Acquires Roaring Penguin https://globenewswire.com/news-release/2018/03/26/1453063/0/en/AppRiver-Acquires-Roaring-Penguin.html Excellent! Dianne, I hope you benefited greatly in this acquisition! -- Rob McEwen https://www.invaluement.com
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 4/3/2018 9:27 AM, Leandro wrote: We just created an URL signature algorithm to be able to query an entire URL at our URIBL: https://spfbl.net/en/uribl/ Now we are able to blacklist any malicious shortener URL Leandro, Thanks for all you do! And good luck with that. But there are a few potential problems. When I analyzed Google's shortners about a month ago, I found that a VERY large percentage of the most malicious shortened URLs were a situation where the spammers were generating a unique shortner for each individual message/recipient-address. This causes the following HUGE problems (at least for THESE particular shortners) when publishing a full-URL dnsbl: (1) much of what you populate your rbldnsd file with is going to be totally ineffective for anyone since it ONLY applied to whatever single email address where the spam was original sent (where you had trapped it) - everyone else is going to get DIFFERENT shortners for the spam from these same campaigns that are sent to their users. (2) get ready for EXTREME rbldnsd bloat. You're gonna need a LOT of RAM eventually? And if you ever distribute via rsync, those are going to be HUGE rsync files (and then THEY will need a lot of RAM). Sadly, most of that bloat is going to come from entries that are doing absolutely nothing for anyone. (3) You might be revealing your spam traps to the spammers. In cases where the spammers are sending that 1-to-1 spam to single recipient shortners, then all they gave to do is enumerate through their list of shortners, checking them against your list - and they INSTANTLY get a list of every recipient address that triggers a listing on your DNSBL. If you want to destroy the effectiveness of your own DNSBL's spam traps - be my guest. But if you're getting 3rd party spam feeds (paid or free) - then know that you're then screwing over your 3rd party spam feed's spam traps - and those OTHER anti-spam system that rely on such feeds, which will then diminish in quality. (unless you are filtering OUT these MANY 1-to-1 shortner spams) Maybe there is enough OTHER shortners (that are sending the same shortners to multiple recipients) to make this worthwhile? But the bloat from the ones that are uniquely generated could be a challenge, and could potentially cause a MASSIVE amount of useless queries. I'd be very interested to see what PERCENTAGE of such queries generated a hit! Meanwhile, in my analysis I did about a month ago, about 80% of Google's shortners found in egregious spams (that did this one-to-one shorter-to-recipient tactic)... were all banging on one of ONLY a dozen different spammers' domains. Therefore, doing a lookup on these and then checking the domain found at the base of the link it redirects to... is a more effective strategy for these - whereas, for THESE 80% of egregious google shortners, a full URL lookup is worthless, consuming resources without a single hit. Alternatively, you may have found a way to filter out these types of individualized shortners, to prevent that bloat? But even then, everyone should know that while your new list might be helpful, it would be good for others to know your new list isn't applicable to a large percentage of spammy shortners, since it is still useless against these individualized shortners. NOTE: Google has made some improvements recently, and I haven't yet analyzed how much those improvements have changed any of these things I've mentioned? PS - the alphanumeric code at the end of these shortners tend to be case-sensitive, while the rest of the URL is NOT case sensitive (and they also work with both "https" and "http")... so you might want to standardize this on (1) https and (2) everything lower case up until the code at the end of the shortner - before the MD5 is calculated. Otherwise, it could easily break if the spammer just mixes up the capitalization of the shortner URL up until the code at the end of the shortner. -- Rob McEwen https://www.invaluement.com
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 4/1/2018 7:10 PM, Kevin A. McGrail wrote: No, I don't think it's an April Fool's trick though it is possible. They announced this a day or 2 ago. See https://www.cloudconnectcommunity.com/ccc/ls/community/g-suite-feature-ideas/post/6320666165116928 and https://firebase.google.com/docs/dynamic-links/ I have been in talks with Google about this, sharing with them specific data and stats. My contact there sent me an email about this a day or two ago referencing that announcement - I just hadn't had time to review it yet. So this is for real. The data I had compiled and shared with them that I gathered about a month ago... was devastating. I can't make that report public because the spammers were OFTEN sending a one-to-one ratio of a single individual google shortener per recipient address. Therefore, making that data public would have revealed many spamtrap addresses (both mine and those of 3rd party spam feeds). Another problem with that data... is that I was finding 10s of thousands of google shortners that were banging on the same dozens spammer's domains - and it would be so trivial for Google to look for that pattern and then just wipe out all of THOSE many shortners that were redirecting to one of these dozen spammers' domains. At the same time, the vast majority of these shortners were persisting for weeks and weeks with no sign that Google was hardly lifting a finger to terminate them. In fact, in that initial report, 95+% of shortners used by spammers were persisting for weeks on end, and 80% of those were using one of those 12 spammer' domains. HOWEVER... I started compiling new data few days ago, and it looks MUCH better now. I'm not done putting that report together, but early indications show that something has recently changed for the better - but that there is still significant room for improvement, but they are at least headed in the right direction. But that report is still under construction. So I think the pressure on Google, along with some of the data I provided them... might have helped? Or maybe that was just "one straw that broke the camel's back"? Either way, I'm happy that this seems to be getting fixed, or they are at least headed in the right direction. -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
Re: sneaky spams w/zipped URL file, easily caught by "Thread-Index"
On 3/27/2018 9:48 AM, David Jones wrote: Looks like ClamAV UNOFFICIAL sigs are detecting this: Clamd: message was infected: Sanesecurity.Foxhole.Zip_url.UNOFFICIAL David, Excellent... except for one potential problem... this is in their "foxhole_all.cdb" file which they label as "high false positive risk" - which could scare some away! For those who don't score very high on ClamAv and/or who are able to score DIFFERENTLY based on different types of Sanesecurity and/or ClamAv results, this is probably OK. But for others who prefer to either outright block or score high on ClamAv, that MIGHT present a problem. On the other hand, maybe Sanesecurity is just being overly cautious (or considering more theoretical FNs?), and such actual FPs in real world mail flow are actually extremely rare? Any Thoughts? Anyone know? -- Rob McEwen https://www.invaluement.com
sneaky spams w/zipped URL file, easily caught by "Thread-Index"
Today, MUCH sneaky spams are being sent with an attached zipped malicious URL/shortcut file. Most or all of these are easily caught by Thread-Index, as follows: Thread-Index: AdBx5/5UsdSTxflQTPi+FyODmVaqhA== Perhaps someone can make a rule for this and post it here? I already set this in another non-SA part of my anti-spam system, but the rule might help others here. There are also other attributes that could become an SA rule that would cause a hit even if the Thread-Index changed, but that will require a little bit more effort. -- Rob McEwen https://www.invaluement.com
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 3/15/2018 11:13 AM, sha...@shanew.net wrote: You might take a look at https://developers.google.com/url-shortener/v1/getting_started 1 miion requests per day is the default limit. Excellent! Thanks for the suggestion. This should help me MUCH! But, unfortunately, it still leaves a lot to be desired for fixing the problems I described... when it comes to more distribution of automated "plugins" or having spam filters automatically doing this - in a "set it and forget it mode". Still, there is a good argument that if someone has a high enough volume to trigger rate limiting... then they ought to have a large enough staff to handle going the extra mile and setting up API access systems, etc. But I wish it could be more simple. MANY are going to struggle with this... and NOT easily know all the details discussed on this thread! But again, this should help me (and others) much... and it is good to know that there is a proper way to do this at a higher volume that meets Google's approval. -- Rob McEwen https://www.invaluement.com
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 2/20/2018 9:42 PM, Rob McEwen wrote: Google might easily start putting captchas in the way or otherwise consider such lookups to be abusive and/or mistake them for malicious bots... This prediction turned out to be 100% true. Even though others have mentioned that they have been able to do high-volume lookups with no problems... And granted I wasn't implementing a multi-server or multi-ip lookup strategy... But I don't think I was doing nearly as many lookups as others have claimed that they were able to do. I took a batch of 55,000 spams that I had collected from the past 4 weeks where those spams were maliciously using the Google shortener as a way to get their spam delivered via hiding their spammy domain names from spam filters. I started checking those by looking up the redirect from Google's redirector, but without actually visiting the site that the redirector was pointing to. Please note that I was doing the lookups one-at-a-time, not starting the next lookup until the last lookup had completed. After about ONLY 1,400 lookups, ALL of my following lookups started hitting captchas. See attached screenshot. Also, other than not sending from multiple IPs, I was otherwise doing everything correct to make my script look/act like a regular browser. I'll try spreading it out between multiple IPs in order to try to avoid rate limits... However... This is still cause for concern about high-volume lookups in high production systems... those may have to be implemented a little more carefully if they're going to do these kind of lookups! Just because small or medium production systems are able to do this... Or just because somebody went out of their way to get more sophisticated with it to get it to work out... doesn't mean that it's going to work in high production systems that are trying to use "canned" software or plugins. This is a particular challenge for anti-spam blacklists because they typically process a very high volume of spams. Hopefully, the randomness of the ones I process as they come in... will be sufficiently spread out enough to avoid rate limiting? It was my hope to start processing these live with my own DNSBL engine, so that I could start blacklisting the domains that they redirect to... In those cases where they were not already blacklisted... Now I'm going to have to deal with constantly trying to make sure that I'm not hitting this captcha, along with implementing some other strategies to hopefully prevent that. But this brings up a whole other issue... That is more of a policy or legal issue... is Google basically making a statement that automated lookups are not welcome? Or are considered abusive? (btw, I could have collected order of magnitudes more than 55,000 of THESE types of spams, but this was merely what was left over in an after-the-fact search of my archives, after a lot of otherwise redundant spams had already been purged from my system.) PS - Once I gather this information, I will submit more details about the results of this testing. But what is shocking right now is that less than four tenths of 1% of these redirect URLs has been terminated, even though they average two weeks old, with some almost a month old. -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 3/10/2018 11:43 AM, Matus UHLAR - fantomas wrote: On 3/10/2018 11:22 AM, Matus UHLAR - fantomas wrote: this is apparently not the case of one url redirector (shortener) points to another shortener. I really hope that the DecodeShortURLs only checks fopr redirection at those known redirectors (shorteners), not each http->https shortener and only evaluates redirection between them, ignoring http->https redirects On 10.03.18 11:32, Rob McEwen wrote: But also keep in mind that it is NOT rare for the initial shortner found in a spam... to redirect to a spammer's page (that isn't a URL shortner), then THAT page then redirects to the spammer's OTHER page (that isn't a URL shortner)... where the FIRST one is NOT blacklisted very many places... but the second page (that is often the final destination)... *is* heavily blacklisted. Often, the first page is hosted at a legit site that was hacked into by criminal spammers - and is HARDER for blacklists to list due to their good reputation. isn't this thread and the plugin discussed about shorteners? I am aware of other possibilities to do redirects and about consequencies of following them. note that for following the redirect, the destination must be configured via url_shortener directive. what I wonder is, if there's real value in following the redirects. (and if the following stops when the URL is blacklisted, since we're done here). The URL shortner follower plugin that Steve Freegard wrote... ALREADY follows my suggestions about following non-shortner redirects (after the initial shortner redirect). (except I'm not sure it already does my https suggestion?) ...and for good reason. You're tremendously undervaluing or just not understanding my last post. I suggestion you re-read it a couple of times. This tactic I mentioned that spammers use... is COMMON, and not following my last suggestion opens up a big loophole for spammers. Such spammers would be delighted by your last comment... and Steve Freegard went a different route for good reason. Keep in mind, I haven't veered off-topic because we're STILL talking about a possible chain of redirects that was still STARTED by a URL shortner. -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 3/10/2018 11:22 AM, Matus UHLAR - fantomas wrote: this is apparently not the case of one url redirector (shortener) points to another shortener. I really hope that the DecodeShortURLs only checks fopr redirection at those known redirectors (shorteners), not each http->https shortener and only evaluates redirection between them, ignoring http->https redirects But also keep in mind that it is NOT rare for the initial shortner found in a spam... to redirect to a spammer's page (that isn't a URL shortner), then THAT page then redirects to the spammer's OTHER page (that isn't a URL shortner)... where the FIRST one is NOT blacklisted very many places... but the second page (that is often the final destination)... *is* heavily blacklisted. Often, the first page is hosted at a legit site that was hacked into by criminal spammers - and is HARDER for blacklists to list due to their good reputation. Then the 2nd final destination page is just a heavily blacklisted spammer's throwaway domain. Therefore, there is some value to following the redirects a little further than what you suggest, and then collecting all of those host names or domains, checking ALL of them against URI/domain blacklists. (within reason... after too many redirects, it is better to just stop and add points to the spam score) -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 3/10/2018 3:20 AM, Matus UHLAR - fantomas wrote: do you have an example of any chained redirection not suspicious? I haven't examined the code for that plugin very much (yet!) but one type of very common redirect that is very innocent... is the fact that a MASSIVE percentage of web sites have switched to all SSL sites in the past few years, due to Google now punishing http and/or rewarding https... in the search engine rankings. But there are still many links and redirectors pointing to the non-SSL versions of those sites, which is then typically redirected to the SSL version. Therefore, if the code for this plugin (and others using this tactic) doesn't do this already... it should probably not count THAT particular redirect as a spam indicator, when counting the total number of redirects. -- Rob McEwen https://www.invaluement.com
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 2/26/2018 1:00 PM, Kevin A. McGrail wrote: DecodeShortURLs has been on my list of must-have plugins for years, so I was a little surprised it took so long for someone to mention it in this thread. Yeah, my firm is going to look at subsidizing it's addition to SA and Karsten agreed. Just trying to get the time. Kevin, Excellent! Also... for those situations where there are multiple cascading redirecting websites (after the initial URL shortners link found in the message), please consider having an option for *every* non-whitelisted URI in the chain to be added to the checks against blacklists. OFTEN - every single domain in that chain (past the initial URL shortner) is a compromised web site or spammer's website, not just the final destination web site. -- Rob McEwen https://www.invaluement.com
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 2/21/2018 2:41 PM, Amir Caspi wrote: On Feb 21, 2018, at 9:57 AM, Dianne Skoll <d...@roaringpenguin.com> wrote: That's why you only want to do it for URLs that are absolutely known to be shortened URLs. You have to keep a list of known URL-shorteners. On that note -- regardless of what OTHER HW/SW solutions might do, since this is a SpamAssassin mailing list ... is there any facility to implement this in SA? That is, when calling the URIBL plugin, could it check both the shortened URL and the expanded URL (for known shorteners) ? Does that facility already exist and I missed it? If I'm not confusing things, someone answered things earlier in this thread, as follows: On 2/21/2018 11:27 AM, Alex wrote: This is what DecodeShortURLs is for https://github.com/smfreegard/DecodeShortURLs -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 2/21/2018 11:48 AM, Anthony Cartmell wrote: Meanwhile - adding URI lookups (for URIs in the body of the domains) and/or the option to add 3rd party URI list lookups - is STILL is missing from MANY widely used anti-spam systems. If you mean following URLs in messages, you do need to be aware that this can break one-time login links. I have come across one local authority in the UK whose spam filter requests URLs in incoming messages. This meant that the spam filter had already used the one-time login link so that when the intended user tried they got an error saying that the link had already been used. Took a while to work out what was going wrong! Sorry I wasn't more clear. I was just referring to standard things like usage of SURBL, URIBL, ivmURI, and Spamhaus's DBL list - where the domains and IPs that are in the body of the message are collected and checked against those lists - I didn't mean to imply that such links are actually followed. Again, sorry I wasn't more clear. I was just pointing out that if so many systems STILL don't do THAT - or provide an opportunity for adding additional such lists - then I doubt that we're going to see widespread adoption by mail systems of a process where, in real time spam filtering, they check to see where URL shortners lead to, and then factor that destination into the spam filtering. -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))
On 2/21/2018 11:44 AM, Dianne Skoll wrote: On Wed, 21 Feb 2018 16:35:27 + Karol Augustin <ka...@augustin.pl> wrote: I think the point here might be that if Google acted promptly on abuse spammers would stop using shorteners. True, that might happen. OTOH, I see about as many spams with bit.ly shorteners as goo.gl shorteners which is not what one might expect if bit.ly were really that much more proactive than goo.gl. I'm sure mileage may vary - but I'm seeing about 10X the abuse for goo.gl right now as I see from bit.ly. Also, when I do random checks on a handful of abused bitly shortners, a high percentage of them are already terminated. But when I do random checks of abused goo.gl redirectors, most of them are still operational. (I'm referring to redirectors found in spams within the previous few days of when I checked them, with at least hours having gone by since the message was sent - I know that sounds anecdotal - but as I've been researching this in the past weeks, that pattern keeps happening). One thing that could potentially make those numbers different - is when you compare one system that blocks MUCH spam at the perimeter based on the sending IP, such as blocking all Zen-listed spams before DATA while another system might capture ALL messages and process them all. The latter is what my system does. That also might explain the difference in stats? -- Rob McEwen https://www.invaluement.com
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 2/21/2018 11:12 AM, Dianne Skoll wrote: Really? This isn't rocket science. If I thought of it, I'm sure dozens if not hundreds of others have thought of it and implemented it. Meanwhile - adding URI lookups (for URIs in the body of the domains) and/or the option to add 3rd party URI list lookups - is STILL is missing from MANY widely used anti-spam systems. That isn't rocket science either - and is MUCH more needed and beneficial than this feature. That some others have made adjustments to their system to follow and act on redirectors during live spam filtering - again - doesn't alter my original point. The vast majority of anti-spam systems in the real world (1) don't (2) and won't any time soon. That is what I claimed. Please stop nitpicking and please stop arguing with a "straw man". -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))
On 2/21/2018 11:11 AM, Dianne Skoll wrote: I guess I misinterpreted: "...such automated lookups could also put a huge extra burden on Google's servers..." from Message-Id <b229b274-ff75-0109-9510-346b078d9...@invaluement.com> Oh yeah, I'd forgotten about that part. it was a more minor point. But as I think back on my thought processes at the time I typed those words - I was envisioning what would happen if ALL ISPs and hosters and spam filtering vendors, SA installations etc... ALL started doing all of those lookups. But yeah, maybe that would still be a drop in the bucket compared to all that Google does. Nevertheless, it is a shame to have to shift more of the burden onto spam filters to do more work (some of which requires MORE latency) - in order to partly mitigate Google's failure to prevent/correct the abuse. In contrast, at this time, bit.ly is running circles around Google in terms of quickly shutting down their abused redirectors. I know this isn't easy, but there is definitely room for improvement. But my larger point in that overall post you quoted from, was my concern about one organization doing high volume lookups from a single server getting blocked or captcha'd. -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 2/21/2018 10:39 AM, Dianne Skoll wrote: We use HEAD requests to expand known URL-shorteners on a cluster that peaks around 60 msgs/s Thanks for that information. That is good to know! (b) and this isn't going to suddenly become a feature inside of many types of spam filtering hardware and software overnight... that could even take years, It's been part of our practice for about a year now. Excellent! I wish others would be as innovative and on top of things as you are! Unfortunately, your statement doesn't alter my point you were replying to, even one tiny bit. -- Rob McEwen https://www.invaluement.com
Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))
On 2/21/2018 10:37 AM, Dianne Skoll wrote: The concern voiced in another email about overloading Google's infrastructure is quite charming and quaint. My concern was NEVER about overloading google. My concern was about Google auto-blocking or throwing a captcha at very high volume and automated lookups. That is a HUGE difference. -- Rob McEwen https://www.invaluement.com
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 2/21/2018 1:38 AM, @lbutlr wrote: As I suspected, it is possible to get the goo.gl target URL without loading the site, though using curl is probably not realistic in this specific case. That is an idea worth exploring! Some might greatly benefit from that. However: (a) it might not "scale" for high volume mail flows and DNSBLs who, like invaluement, process dozens (or more) spams per second. (b) and this isn't going to suddenly become a feature inside of many types of spam filtering hardware and software overnight... that could even take years, if it could ever even gain traction? It is taking into the decades just to get some of those software and hardware vendors to add support for URI blacklists, or support to for adding custom 3rd party URI blacklists. If that is taking literally decades - they are not going to add this feature within the foreseeable future. So please don't think for a second that this somehow makes the plans I had described as unnecessary. -- Rob McEwen https://www.invaluement.com
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
ic if only idiots ran spam filters and DNSBLs. But, thankfully, I'm able to separate the legit uses - from those spammers who are trying to hide their identities in order to facilitate spamming - a point lost on you. I've NEVER ONCE argued for creating rules that blocked ALL messages that used shortners - but that is the "straw man" that you're arguing against. This is also one of the very reasons that spammers are so happy with this loophole - they know that nobody is going to create a rule that blocks goo.gl (or even scores that high against it). But thankfully, they'll get SOME relief as more of those who abuse it find their IPs blacklisted at invaluement (and hopefully other places), and rightly so! (yet without the collateral damage you keep predicting or implying) If we're going to continue this conversation, can you please STOP putting words in my mouth and arguing against "straw men"? Also, I understand your very valid concerns about collateral damage. I've addressed that numerous times and in numerous ways, in numerous posts. This is getting tiresome. -- Rob McEwen https://www.invaluement.com
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 2/20/2018 6:05 PM, @lbutlr wrote: On 2018-02-20 (08:30 MST), Rob McEwen <r...@invaluement.com> wrote: Spammers are starting to use this to evade spam filters, This is not news. Spammers have been using shortness since 3 seconds after tinyurl.com launched. My "this" was /*specifically*/ referring to Google's shortner, or at least the recent STRONG uptick in the abuse of that shortner. I was already well aware of other similar things from other shortners. But "this" wasn't referring to them. You infused thoughts/meaning into my writing that really wasn't there. (be careful about assuming...) Also, when you see my report further down, you'll understand why those others are not nearly as much of a concern to me at the moment. Keep in mind that, if a marketer is doing things the right way, they should have no need to obfuscate their own domain name. They should instead proudly use it and not feel the need to hide behind Google's shortner. No, that is not at all true. The primary use of a shorter is to shorten a long URL to something that someone can type in. I've acknowledged that there are some good reasons for a shortner - but the vast majority of the times I'm seeing them - in both ham and spam - that is NOT the case! The are shortening things like average-sized domains with either no directory, or with a short directory or page names after the domain. This is the VAST MAJORITY of the shortners I'm seeing in both hams and spams. Clicking a URL in an email is the height of stupidity, so having a short URL that someone can realistically type into a browser is much better. If I spent just a little more time on this, I could collect a large number of Google shortner URLs that are malicious - where my malwarebytes is blocking access to the page to which it is trying to redirect. And these are still "live"! Do you really think that more than a tiny percent of those who saw those in their mailbox (both legit and spams) are manually typing in the URL and not just clicking on it? And in that exceedingly rare occasion where somebody types in the URL and it redirects to a malicious page that tries to install a virus, are they ANY better off than having just clicked on it? Even the best point I can think of that you might have had - that this might help them to better recognize a phishing URL for example - is lost since BOTH the phish and their bank's web site is going to be indistinguishable until AFTER they've launched the shortner (whether by clicking or typing). I think you just mistakenly bolstered my argument against this over-usage (and often inappropriate usage) of shortners! THEREFORE: If you like having NOT-blacklisted IPs, be advised that the invaluement anti-spam DNSBL system is now adding "bad" points to the scoring of all messages that use the "goo.gl" shortner, and we're amplifying other "bad" points. Well, at least you are warning people. However, what you are doing is, frankly, dumb; if you think there's a huge problem, you can simply check the target URLs. That incurs a significant amount of extra resources for DNSBLs and spam filters - and such automated lookups could also put a huge extra burden on Google's servers - and who knows at this point if this is even reliable - Google might easily start putting captchas in the way or otherwise consider such lookups to be abusive and/or mistake them for malicious bots... I'm definitely going to pursue this further - but wow... that you would suggest this... I think spam puts ENOUGH burden on spam filters and mail system as it is! Yes, there are many legitimate uses of Google's shortner, too. However, we are now at a point where a VERY large % (a majority?) of uses of these headed to a typical user's mailbox are egregious spams, and a significant additional portion are likely-spams. Any evidence of this? EVIDENCE/STATS: I ran stats on a sample set of a few thousand mailboxes, over a period of several hours today (mostly during business hours for these particular organizations who use these mailboxes) - and this produced a combined 24K legit messages, and 5K spams (I'm guessing that most systems have more spams per amount of hams? But those were the numbers for this server.) --- NOTE: The sum of individual shortner-hits totals below can be LARGER than the total messages that had hits on ANY shortner - Why? - Because in a few cases, the same message can have hits on more than one of these shortners --- I SEARCHED EACH HAM AND SPAM CORPUS FOR MANY OF HUNDREDS OF URL SHORTNERS HERE ARE THE RESULTS: --- STATS FROM SPAM: 286 total spams blocked that had a shortner, out of hundreds of URL shortners I had searched on (<10 that *MIGHT* be FPs - they were definitely questionable at best - btw, zero of those
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 2/20/2018 12:21 PM, Reindl Harald wrote: we have well working outbound spamfilters Excellent! but just because someone has a google-shortener within a mail says *nothing at all* - frankly i even got a week ago a mail from my boss where the google-shortener was used for a only internal reachable server with a long list of params in the url and hence the google-shortener don't say anything This is a very abused loophole that says MUCH in certain contexts. And I've carefully constructed these change at invaluement to be extremely unlikely to impact those who are using "goo .gl" for legitimate purposes and are not using it to cloak their domain in messages that are UBE or otherwise not desired by the recipients. But, in contrast, marketers and/or ESPs who start doing this routingly, as a purposeful regular practice, and who don't have some kind of real and specific purpose such as what that you described, are essentially giving DNSBLs and spam filters "the middle finger". So I'm giving it back. ANYTHING that facilitates anonymizing identity is VERY BAD for email. Facilitating anonymizing identity causes more spam to be delivered and punishes good senders when bad senders get away with that. Methods that facilitates anonymizing identity for email is not something that anyone should defend or celebrate - even if anonymizing identity wasn't the original intended purpose. I understand your very legitimate concern that this crackdown might lead to collateral damage. That is admirable. But acceptance of a new and pervasive situation in email that anonymizes identity is a HUGE step backwards... like going back to the mid 2000s, or something. So some "push back" measures are exceedingly warranted. -- Rob McEwen https://www.invaluement.com +1 (478) 475-9032
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 2/20/2018 11:45 AM, Rob McEwen wrote: And we ALL have to constantly shift our tactics to deal with emerging realities like this one - or risk getting left behind by our competitors who do keep up. ALSO - Likewise, it was very frustrating that I had to spend hours late last night making adjustments to my spam filter to be able to block more of these egregious spams that are often so difficult for a spam filter to block - particularly since extra careful measures are needed to keep such adjustments from blocking legitimate messages. (I've been trying to get to that for weeks, as this problem has been festering for some time - but I just recently recovered from the flu.) So nobody should cry on my shoulder about the difficulty of ESPs doing such abuse monitoring/prevention. Any such person doing so is likely underestimating the current size and frustration that spam filtering admins are having with this problem! -- Rob McEwen https://www.invaluement.com
Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
On 2/20/2018 10:57 AM, Reindl Harald wrote: and how do you imagine that i prevent paying customers to use whatever url-shortener? Perhaps use the SAME methods that an ESP would use to prevent a customer from sending an egregious phish (or terminate their account for sending a phish). Of course, I also recognize that an egregious phish is much worse. But my point is that such abuse monitoring and prevention is a real thing for ESPs! Yes, some ESPs are more sophisticated than others, where they do a better job at this than others. For example, I've received two egregious phishes to my own email address, from MailGun IP space, within the past several months. I alerted them in both instances and hopefully they are improving their system? In contrast, I don't think I've ever seen such a phish from Exact Target, from example. That isn't by accident! Some do a better job of this than others. And even though no ESP can be perfect - that doesn't mean they can't improve. And we ALL have to constantly shift our tactics to deal with emerging realities like this one - or risk getting left behind by our competitors who do keep up. Also, getting ESPs to pass this message on to their clients, even if just adding this to their instructions for clients, even if just as a "best practices" warning... might also go a long way. when you start list to many legit servers because of that you RBL will be no longer useable for responsible admins which primary job is receiove and deliver email and not to reject it I'm extremely confident that this won't happen. Most likely, a few marginal ESPs and marketers will get blacklisted who were previously just barely avoiding detection. Also, we OFTEN get outliers (such as an occasional VERY bad spam that came from a normally VERY good sender), and "decoys", too! In those cases, if those messages had led to an automatic blacklisting, and we didn't FIRST check those domains and IPs against our very sophisticated "false positive prevention filter" - then what you described - would have happened a long time ago already. But, instead, invaluement's reputation for low False Positives speaks for itself. Given what I know about how invaluement works "under the hood", I can say with confidence that it is practically impossible for this change to put a dent in our hard-earned low-FP reputation. But this COULD cause problems for some already dark-gray-hat ESPs who let this practice run rampant. -- Rob McEwen https://www.invaluement.com
The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)
RE: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response) WARNING FOR ESPs AND MARKETERS: Google's "goo.gl" shortner is OUT OF CONTROL. Spammers are starting to use this to evade spam filters, and Google isn't keeping up with the abuse, nor shutting these down fast enough. Along with blackhat spammers, we're seeing evidence that many gray-ish-hat spammers are jumping on this bandwagon, in the hopes that they will get more mail delivered if they can avoid using their domain in the clickable links. Keep in mind that, if a marketer is doing things the right way, they should have no need to obfuscate their own domain name. They should instead proudly use it and not feel the need to hide behind Google's shortner. Yes, there are many legitimate uses of Google's shortner, too. However, we are now at a point where a VERY large % (a majority?) of uses of these headed to a typical user's mailbox are egregious spams, and a significant additional portion are likely-spams. THEREFORE: If you like having NOT-blacklisted IPs, be advised that the invaluement anti-spam DNSBL system is now adding "bad" points to the scoring of all messages that use the "goo.gl" shortner, and we're amplifying other "bad" points. (We're also doing various sophisticated things to minimize potential resulting FPs, too. But this will still put MANY marginal IPs and domains into our blacklists that normally might have barely avoided an invaluement blacklisting!) YOU HAVE BEEN WARNED! DON'T ALLOW YOUR CLIENTS TO DO THIS! ALSO: We're very seriously evaluating the option of converting each shortner to the URL it redirects to - and then potentially starting to add those domains or IPs within those URLs to our ivmURI domain/URI blacklist. This might not cause other such messages to get blocked, but it will have other negative repercussions for other uses of that domain. -- Rob McEwen https://www.invaluement.com
Re: Blacklist for reply-to?
On 2/18/2018 3:06 PM, Kenneth Porter wrote: Is there a blacklist for domains in the reply-to header? I've noticed a lot of spam with no URL and mutating From but the reply-to domain is always aliyun dot com. I want to add a site-wide blacklist for that. http://msbl.org (I'm not associated with this. Also, it is very high quality and well-run! It should at least make a noticeable improvement, even if it doesn't catch all of them.) -- Rob McEwen https://www.invaluement.com
Re: smtp.centurylink.net 206.152.134.66
On 2/11/2018 2:37 PM, David Jones wrote: This mail server has legit email for centurylink.net and embarqmail.com plus a lot of other spam coming out of it. It's listed on a number of RBLs making this very hard to allow ham through and block spam. http://multirbl.valli.org/lookup/206.152.134.66.html https://pastebin.com/YidWCqp8 I've downgraded the whitelisting entry for this IP at invaluement. It still won't get blacklisted due to the large amount of collateral damage that such a listing would cause. (And others lists having this blacklisted is probably a GOOD thing! I'm not disputing their decision for their list. Different lists serve different purposes, etc.) But with this downgrade at invaluement, future spam that comes from this IP will be examined with greater scrutiny by invaluement, in order to possibly blacklist other domains and IPs related to the spam. Also, the spam sample shows a Google shortner being used as the payload link. I've seen many of those lately - and I think Google needs to work on improving their ability to prevent these, or at least get the shortner terminated faster. At the moment, this one is still "live". I reported this particular one as spam to their shortner abuse form. So, it will be interesting to see how long it persists from this point forward? btw - if anyone ever wants to learn more about one of these google shortners without actually visiting the link (which can be dangerous... for example, some of the more malicious links arrive at a page that tries to install a virus), add ".info" to the end of the google shortner URL and you can then see more info about the shortner, including its intended destination. For example, for this one: https://goo.gl/s7XxhD.info -- Rob McEwen https://www.invaluement.com
potential new SA feature: Direct DNS Querying Per DNSBL Zone
this completed - that would be greatly appreciated! NOTE: Once this feature is completed and bug free, it will then require the approval of the Apache Software Foundation (or something like that - I don't really understand their inner workings very well) - before it can be officially adopted. However, already, it has significant support from multiple important people! Therefore, I have high hopes that it will eventually earn this adoption, but that is not a guarantee and I have no control or say in that decision. But it makes the most sense to FIRST make this potential SA addition bug-free and outstanding - then cross that other bridge when we get there. Thanks to anyone who is willing/able to help with this. Please let me know if you have any questions. I'm also considering putting some kind of modest "bounty" on this - to fund anyone's efforts that either fix the remaining bugs, or at least make significant and measurable progress to that end - send me a private message off-list if that interests you! (I would do this myself, but Perl "looks like Greek" to me!) https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7477 -- Rob McEwen https://www.invaluement.com
Re: Weird new malware
This seems to be catching most of them:
Subject: Invoice [A-Z]{2,3}\d{7}\b
...but it might need to be combined with other things to ensure no false
positives, since there would be a rare legit message that would hit on this?
--Rob McEwen
On 11/8/2017 10:45 AM, Dianne Skoll wrote:
Hi,
Heads-up: We're seeing weird new malware with a subject that looks like
Invoice XXX
where XXX is two or three random upper-case letters and n is a series
of digits. What's weird is that the Content-Type: header looks like this:
Content-Type: multXXXart/mixed
where the XXX is the same as in the subect. That is, a message
with subject "Invoice UUI8187685" has Content-Type "multUUIart/mixed". This
is fooling our MIME parser because it doesn't see the container as a
multipart. Does any client software?
Anyway, might want to make rules for this.
Regards,
Dianne.
--
Rob McEwen
Re: Blocking senders that are whitelisted
Alex, (if you have the time...) You should challenge this sender to provide evidence that your user signed up for their messages. Tell them that it isn't good enough to receive an IP address and date/time-stamp. You want to know what action your user took to get on their distribution list. (then possibly share that information here and with Mail Chimp) It could very well be that your user signed up for their services in some way - but either forgot - or the messages got rebranded in such a way that your user didn't recognize it? But it is also possible that they were added via a purchased list or something bad like that. Rob McEwen invaluement.com On 10/4/2017 11:23 AM, Alex wrote: Hi, we have a user complaining about receiving email from a solar panel company and want us to block it. The problem is that it originates from mailchimp, which is whitelisted. It's my belief that mailchimp is safe to whitelist (mcsv.net). However, what happens when an email is received that needs to be blocked? Do you just report it? Chances are probably good that I could click the "unsubscribe" link and just unsubscribe the user, and will probably do that, but I'm looking for a more general solution. I'll also report the sender to mailchimp, but the email looks like they're following all the rules. The sender is exped...@iowawindandsolar.com. Should I block the From address in postfix? Is it possible to blacklist the From when the sender is whitelisted? -- Rob McEwen http://www.invaluement.com
Re: ramsonware URI list
On 7/15/2017 2:13 PM, David B Funk wrote: How quickly do stale entries get removed from it? I randomly sorted this list, then I tried visiting 10 randomly selected links. I know that isn't a very large sample size, but it is a strong indicator since they were purely randomly chosen. 9 of the 10 links had already been taken down. So there might be much stale data in that list? I also extracted out the host names, deleted duplicates, randomly sorted those, then ran checks of 500 randomly selected host names against SURBL, URIBL, DBL, and ivmURI. The number of hits on all 4 lists of shockingly low. But I think that probably has more to do with stale data on this URL list (and this is really a URL list, not a URI list), rather than with lack of effectiveness of these other domain/URI blacklists. Still, there can be situations where a URI list won't list such a host name due to too much collateral damage - but yet where a URL list that specifically lists the entire URL - can still be effective. Because such a URL list would be LESS efficient (due to being rules-based), it would be preferable that such a list would have much less stale data - and perhaps would focus on the stuff that isn't found on any (or very many) of the 4 major URI lists I mentioned, so as to keep the data small and focused, for maximum processing efficiency. -- Rob McEwen http://www.invaluement.com
Re: URIBL_BLOCKED on 2 Fedora 25 servers with working dnsmasq, w/ NetworkManager service
On 5/18/2017 5:46 PM, David Jones wrote: it should be pretty clear now to not use a forwarding DNS server locally and do not point the server to another DNS server in /etc/resolv.conf. Thanks David! Some may be interested to know at least 15% of my entire labor "overhead" for running invaluement - involves playing "whack a mole" (so to speak) with both testers and existing subscribers - whose DNS settings CONSTANTLY revert back to sending direct queries to invaluement via Google and/or OpenDNS - which are then blocked - even as the instructions were extremely clear about how/why not to do it that way. In many cases, they explain to me that their settings got auto-overwritten by their hoster - who just HAD to switch their resolv.conf file back to 8.8.8.8 In some rare worst case scenarios - I have to "fire the customer", due to many repeated incidents where the labor involved in constantly babysitting their settings - was no longer worth their subscription payment. And unfortunately there is just basically a very sizable portion of IT professionals in the entire world... probably hundreds of thousands of IT people - who have been convinced that pointing all DNS to 8.8.8.8 is standard operating procedure that they think is always the best way. For me, it feels like annoying busy work. Imagine that for at least one hour out of your day - you have to stop what you're doing and dig a hole in your back yard - and then fill it back in. So I'm grateful every time I see thread like this that pushes back against that, and encourages others to run industry standard non-forwarding caching DNS servers. THANKS! -- Rob McEwen http://www.invaluement.com
Razor FP on simple http link (by itself)
I use SA as a "helper app" within my custom written spam filter. So I'll get SA give me an opinion about certain marginal messages, and then my spam filter factors the SA score into my spam filter's scoring. Recently, a prominent law firm for whom I host mail - was complaining about FPs where messages from a prominent real estate company were not making it to them. Interestingly, their messages kept hitting RAZOR, where SA was giving the following response: 1.7 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/) 0.4 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50% [cf: 100] 2.4 RAZOR2_CF_RANGE_E8_51_100 Razor2 gives engine 8 confidence level above 50% [cf: 100] In testing, I narrowed it all the way down to simply the following (alone!) hitting on razor: either http://www.example.com or http://example.com (except with the sender's domain, of course) ...either one was triggering this razor score. I even put that as the ONLY body text of another message (so a totally different header) - and it still triggered. But either variation WITHOUT the "http://; part did not trigger. Interesting... this domain name happens to resolve to an IP that is currently blacklisted on Zen. (I know, that is really really bad!) Unfortunately, that confuses issues! Does RAZOR extract domains from links and checks them against a bad domain database... sort of how SURBL works... and/or check the IP that they resolve to? (I don't think so, but now I have to ask just to be sure!) If not... this seems to go beyond checksum-checking of parts of a message - this seems much more surgical/specific than that. Don't get me wrong... I'm a big fan of razor and of other checksum-technologies. But I'm sort of shaken by this because I always thought a FP for razor would be much more difficult due to larger portions of a message having to match a checksum match in order to have a hit. (sort of like a larger "fingerprint" that is not easily duplicated in another innocent message, allegedly making FPs practically impossible) While this kind of more surgical strike can be beneficial in blocking more spam - it seems like it changes the paradigm of what I (mistakenly?) thought to be RAZOR's potential for collateral damage. Is this "extra curricular activity"? or did I misunderstand RAZOR's checksum technique? -- Rob McEwen
Re: Outgoing email without DMARC
On 5/1/2017 10:30 PM, Marc Perkel wrote: Might be slightly off topic but I've been running into more delivery problems with outgoing email because I don't use DMARC. I don't know a lot about it but is there some simple way I can get around this? Kind of a pain in the rear. Marc, This probably has more to do with DKIM than DMARC? Either way... you're not willing to jump (or haven't yet jumped) though the hoops that the largest ISPs/hosters want us all to jump through... meanwhile... so many of them (and for many many years) have sent such high volumes AND high percentages of outbound spam to all of our SMTPs - to such an extent that you and I would be out of business if our SMTP outbound traffic did that for just one week. I sort of wish they (or many of them... "if the shoe fits...") would clean up their own act FIRST - get the basics done FIRST - before imposing new standards on the rest of us. I'm in the same boat - I'm now having to set aside dozens of hours to get all various domains updated to DKIM so that they'll have more success sending to a certain large/famous hoster - who has sent my server a shitload of spam over the past several years (not just volume-wise - but percentage-wise... I'd be run out of town if I did that) -- Rob McEwen
Re: Fastest listing RBL ?
On 2/14/2017 11:04 PM, Ian Zimmerman wrote: Given a piece of horrible spam, on which RBL is the sending IP address likely to appear first? I want to rationally decide which RBL/s to consult at SMTP time. Afraid to use all of them, not just due to false positives, but also due to negative caching in DNS, which could affect the result when the spam is seen by SA a bit later. Ian, (if possible) Start monitoring the sending IPs of barely missed incoming spam, in real time, as they come in. As soon as possible after the spam comes in, check that IP on multirbl.valli.org (or mxtoolbox), and see which RBLs are listing that IP (recognizing that SOME of those are going to be situations where the IP wasn't blacklisted until seconds after the spam was received - but many were listed before then) Throw out the examples where the IP has a good reputation in places like SenderScore or SenderBase... or where the IP is an MTA of a large/famous hoster/ISP (but take note of which lists were listing those) - However, SOME of those listings with decent scores at SenderScore or SenderBase are appropriately blacklisted, such as a small sender with a massive security hole where they are suddenly spewing out a massive amount of spam. But if you notice an RBL hitting on MANY like this - they might be too aggressive and even worthless - or perhaps more appropriate for low scoring. There are a few lists which are VERY fast-reacting to new spams, but have horrific expiration polices and/or are poorly managed - and would produce many FPs if used in production. But there are some other lists which are just a little too aggressive for outright blocking, but are very useful for scoring a few points (or less). You should then notice about only about ~5-8 lists which rise to the top - in the checking I described, they seem to have very few FPs (you won't know for sure until you test them in production) - and they are very fast reacting. Still, SOME of these are just a little too aggressive for outright blocking (or scoring at/above threshold) - but are great for high scoring. But you may not be able to accurately measure their FP level until you've used them in production for some weeks. A smaller amount of these RBLs (that rose to the top)... in addition to being fast-reacting... block much unique spam missed by the other lists AND have few enough FPs for you to probably feel comfortable outright blocking (or scoring at/above threshold). You might find ~3-5 such lists, including zen.spamhaus.org in that elite group. -- Rob McEwen
Re: The nice thing about standards (was Re: Legit Yahoo mail servers list)
On 2/1/2017 12:56 AM, Dave Warren wrote: They publish SPF records and DKIM sign everything for competent SMTP receivers to handle in real-time, AND they publish a HTML version for humans, and yet someone still finds a reason to complain? Dave, After the initial question was raised, it took about 11 posts and almost 24 hours for someone to notice the discussion who happened to know about the "HTML version for humans" and mention that. During those 11 posts, a well-respected and knowledgeable person was actually defending Yahoo for NOT having such a page, which gave the impression that such didn't exist. (certainly, that was a head-fake that I fell for, even if such was very innocent) So I think there is a strong argument that the existence of this page page isn't exactly common knowledge. Archive.org suggests that this page has only existed for a couple of years. I've been looking for it (occasionally) for the past 10 years - so I think all my memories of past discussions in past years about such a page not existing - were probably accurate. By the time this page existed, I had given up on finding it. (not that I spend every waking hour looking for it - I think I probably looked for it about once every year or two - for some time - and the need for this isn't so great with other senders - because few senders [even large ones] have such a MASSIVE amount of sending IPs that are so particularly hard to find) Regarding your references about such a page not being needed - all I'm going to say is that some systems benefit from having large IP ranges preemptively whitelisted for the sake of efficiency. There are scenarios in certain very high volume systems where this enables the processing of messages at order of magnitudes faster rates than if SPF and DKIM and FCrDNS-confirmation had to be checked on every sending IP. MUCH of that relies on the response times of 3rd party servers - which (even at best!) is order of magnitudes slower than a local rbldnsd query - or than an optimized binary search of an in-memory array - which is even faster than rbldnsd or even a high-end in-memory database. Sometimes, such 3rd party servers can "freeze up" in their responses, or rate limit queries - or firewall such lookups for what is perceived as abuse - causing further complications. Caching only does so much to prevent this! That kind of need for speed is the world in which I live. At invaluement, I'm processing dozens of spams per second - and since much of these are ones where the "low-hanging fruit" - such as ALREADY heavily blacklisted botnet-sent spams are ALREADY filtered out before they get to my system - that means that the processing resources per spam is already much higher for my system than that of a typical ISP or hoster's natural incoming spam. (I process a higher concentration of the more sneaky spams and the newer emitters) With this in mind... if I deleted my IP whitelist, and had to rely on SPF and DKIM and FCrDNS-verification for EVERY message, my queues would back up considerably - and a lot of worthy blacklistings of IPs and domains from new incoming spams would get considerably delayed. (again, inevitably - at this volume - issues come up where such queries/verification suddenly "freeze up" or get rate limited, firewalled, etc) And I think my need for efficiency is probably not much different than some very large hosters and ISPs - who process mail for millions of users? And I think we've already established that there is no possible way to generate "on demand" and remotely efficiently the information on that HTML page just via Yahoo's SPF records. iow - maybe you should have a little more respect and try to be a little less snarky in the future - when you don't necessarily know/understand others' situation/requirements that may be a little different than your particular situation/requirements. -- Rob McEwen
Re: Legit Yahoo mail servers list
On 1/30/2017 8:54 AM, Matus UHLAR - fantomas wrote: they do and it has been mentioned: https://help.yahoo.com/kb/SLN23997.html I wasn't aware of this page. If it was mentioned before in this thread, I missed it. Thanks! -- Rob McEwen
Re: Legit Yahoo mail servers list
On 1/29/2017 7:42 PM, Dianne Skoll wrote: On Sat, 28 Jan 2017 16:33:24 + David Jones <djo...@ena.com> wrote: Read back through this thread. I never said their SPF record is invalid. All I said is their SPF record is not common and it makes it very hard for anyone to know what the official Yahoo outbound mail servers are. Why is that important? Can't you just whitelist the domain yahoo.com if and only if it hits SPF "pass"? We have to work very hard to get our MTAs to whitelist them. It's in their own best interest to make this information easily available to the Internet since so much spam comes out of their platform. Then why would you whitelist them? They are too large to not whitelist. Nobody is too large to not whitelist. They're obviously too large to block, but you'd be foolish to accept any and all mail from a Yahoo server unless you like an awful lot of spam. Regards, Dianne. Dianne, I can't speak for David, but most or all of your answers don't apply to my own anti-spam blacklist's attempt to try to avoid blacklisting Yahoo IPs that are both known for sending much spam, but which also would have a very high rate of collateral damage if blacklisted. (recognizing that some very good DNSBLs, which are more aggressive, are more willing to blacklist Yahoo IPs, and that isn't always a bad thing) ...and/or your answer requires more on-going receiver-side resources. Interestingly, many senders would crawl over broken glass if necessary to provide me their IPs, if said I was seeking those for my whitelist. Also, when David said "whitelist", I can take an educated guess that he isn't allowing Yahoo-sent messages free unfiltered access to the inbox - he is probably just trying to avoid DNSBL checking of those particular IPs - but then he'll probably STILL do other content filtering of those messages. That would be my educated guess. And this would be a SMART strategy. Personally, when I get messages from Yahoo into my hosting business - I have the IPs generally not checked - since I already have most Yahoo IPs whitelisted - then I only content-check the messages - BUT... next I AMPLY any content scoring of such messages since these came from Yahoo and are more likely to be spam - that is, if the sender isn't already in a carefully cultivated exception list of known good Yahoo senders (specific to my mail hosting user base) - I do this for all freemail senders known to send a high volume of spam. I know you mentioned that Yahoo may want to have the flexibility to change their IPs. But instead of providing a list, they could also provide a link to a web page listing the IPs (like what Comcast does) - and then just update that web page whenever their IPs change. This isn't rocket science. As it stands, it is mind boggling just how many Yahoo ranges of sending IPs there are worldwide. Over the years, I've added 53 yahoo entries to my whitelist. Besides the hundreds and hundreds of /24s ranges in there (many are multiple consecutive /24s, showing up as just one line of those 53 entries), there are also several /16s, too. It would be nice to be able to compare that to Yahoo's current list active sending IPs (if such were available?), so that I could EFFICIENTLY update/prune that part of my whitelist. And I strongly suspect that iterating though the millions of IPs to check FCrDNS would take a very, very long time - and might get such probing IPs blacklisted for abuse/intrusion-protection? -- Rob McEwen
Re: Legit Yahoo mail servers list
While I have Yahoo sending IPs extensively covered in my whitelist, I've been trying to get their complete official list of sending IPs for years. I'm amazed that Yahoo doesn't participate in these conversations - or that nobody ever says, "I'll ask my colleague over at Yahoo" seems very odd... -- Rob McEwen
Re: How to create a URIBL
On 10/18/2016 9:09 PM, Alex wrote: How do you then enter ranges? For example, one of the rbldnsd zone examples I've seen have entries such as: 1.168.160.0-255 That does not look to be in reverse order, as the host octet is still last. while there may be a more complicated and unusual answer for this.. the short answer is... you don't, and you shouldn't have to. (1) IPs at the base of clickable links inside the body of the message in spams... is still a little rare... comprising roughly 2% of all such listings. (2) This means that (a) those IPs aren't taking up a lot of space in the dnset files, when compared to the domains and host names there, and (b) of that ~2% of IPs, extremely few of those are even in the same /24 block - so you don't get much mileage out of trying to list ranges having said that... sending-IP lists that use ipset DO have the functionality that you desire. ipset actually has quite a number of acceptable formats to list blocks or ranges of IPs. iptset... not so much. iptset is built for EXTRA speed and EXTRA low-memory usage, but isn't as flexible and generally requires one single IP per line. Based on your question, it could be that you're trying to merge your sending IP blacklist, with your URI/domain blacklists... all into one single dnset rbldnsd file? if so, that is NOT recommended. It causes problems and removes some of rbldnsd best features/strengths. Your service is great, btw. Thanks. Please send me a note off-list as you how/why you think that. I'm not looking for praise... just curious if you're one of my clients (such as at your dayjob?) or if we've crossed paths somewhere and I forgot about it?... or if you have ever testing invaluement? etc (though I know you're a frequent SA discussion participant) -- Rob McEwen http://www.invaluement.com +1 (478) 475-9032
Re: How to create a URIBL
On 10/19/2016 3:51 AM, Matus UHLAR - fantomas wrote: are you REALLY sure the IP has to be reversed? rbldns parses IP and reverses them by itself, if used in ip4* dataset. When used in dnset, it should not be reversed. Your most valid points do not apply to "dnset". they apply to ip4tset and ip4set for sending-IP blacklists. Let me explain... but before I explain, let me say that I'm not arguing for any of this. These standards were put in place long before my time (and are followed by SURBL and URIBL, too). Or, at least I didn't set these standards. I MIGHT have been involved in some of the discussions about this circa 2004, in internal discussions at SURBL - and in SA discussions - but I think this was all set just a little before my time in those forums. So basically, if you look at the anatomy of a domain name... from left to right, you get into a higher hierarchy. So in "foo.example.com" "foo" is drilling into detail. while "example.com" is the bigger picture. And then ".com" is an even bigger picture! In a domain, as you get FURTHER to the right, you go to a HIGHER hierarchy or level. But IPs are the opposite. For an IPv4 IP, the leftmost number is the highest in the hierarchy, and you drill down into more detail as you move to the right. For this reason, it was decided a long time ago... that for URI DNSBL blacklists that use "dnset", the IP should be reversed in the source file. Therefore, in the data file, the test point IP: 127.0.0.1 shows up as 1.0.0.127 And then when the client queries that IP, the query is formatted as follows: 1.0.0.127.example.com (where example.com is the URI blacklist's host name) And, likewise, ALL of the major anti-spam software, (such as SpamAssassin), automatically reverses the IP when that (forward-ordered) IP is extracted from a base of a URL found in the body of a spam, and then this is appended to the beginning of a URI blacklist's hostname, for checking against a URIBL blacklists (such as SURBL, URIBL, or my own ivmURI list) This decision to do it this way PROBABLY had something to do with trying to get rbldnsd engine to NOT have to internally treat IPs and domains/host-names differently. otherwise, it would have had to "know" to reverse IPs, but yet know to NOT reverse domains or host names. (and who knows what TLDs could be coming up in the future?) In contrast, IPs found in sending IP data files (for ip4tset and ip4set) don't have this inconsistency problem. So it make sense to just leave them in forward-order, for EASY readability... and then just allow rbldnsd to reverse order them on-the-fly. (thank God - I'd go nuts if my ip4tset and ip4set were all in reverse order! meanwhile, IPs in URIBL data files are usually a TINY percentage of the listings!) -- Having said all of that, for regular sending0IP blacklists, (just as you said) the IP is NOT in reverse order in the file. But rbldnsd "knows" to reverse order it in memory, before it is compared to the reverse-ordered query that comes in from the client. So you're correct when you say, "rbldns parses IP and reverses them by itself" ... but that only applies to sending-IP blacklists, set up with ip4tset and ip4set in rbldnsd. As shown, dnset operates differently for IP addresses found in URIBL blacklists. ---------- This was a trip down memory lane for me. -- Rob McEwen invaluement
Re: How to create a URIBL
Alex, here are some suggestions: In your rbldnsd-formatted file, put a dot at the beginning, which serves as a wildcard. So your three examples: 109 .73 .134 .241 51steel1 .org amessofblues1 .com (I added spaces here to evade spam filtering, but those spaces shouldn't actually be there) would like like this: .241 .134 .73 .109 .51steel1 .org .amessofblues1 .com (again, the extra spaces shouldn't be there) NOTICE 2 things: (1) The extra dot at the beginning -and- (2) the fact that the IP is in reverse order. The great part about rbldnsd is that a lookup on either example.com OR www.example.com OR foo.bar.foo.example.com ALL of those will get a "hit" when the rbldnsd file has .example.com When it comes to formatting the rbldnsd-formatted file, in addition to my suggestions above, it comes down to a choice... make it a simply list of the domains and (reverse-ordered) IPs? Or provide more information for each individual IP, such as a custom text response, as you did here: foo.example.com:127.0.0.2:Blocked System in my experience, I haven't been able to get this to work unless I put a space just before the first colon, as follows foo.example.com :127.0.0.2:Blocked System But sometimes you don't need that and can simply use just the domain or IP on each line, since much of that can be accomplished with a single line at/near the top of the file, such as this one that I use for the invaluement URI list: :127.0.0.2:Blocked by ivmURI - see http://www.invaluement.com/lookup/?item=$ ...which then causes all following lines of just domains and IPs... to use this line above as if it were on every single line. - and the "$" causes the actual listed item to show up in the SMTP text message. That "$" feature can be very informative and helpful! of course, the most difficult part is not collecting spammy IPs and domains... that part is easy. The most difficult part is knowing when NOT to blacklist a domain--which would be a decoy domain found in a spam, that wasn't the actual "payload" for the spam and is instead an innocent bystander's domain -- and/or generally keeping FPs super low. THAT is the hard part. There are other issues as to WHERE to divide the domain. For example, if you listed .foo.bar.foo.bar.foo.bar.foo.bar.example.com ... but foo.bar.foo.bar.foo.bar.foo.bar. was just decoy material added by the spammer... then... foo.bar.example.com comes in and guess what? your lookup fails to find it. Yet all such variations would be listed if you had simply blacklisted: .example.com (again, with the dot in front) But try this and blacklist: .blogspot.com ...and trigger massive FPs... when you should have listed: .somehorrificspammerfromhell.blogspot.com so that either www.somehorrificspammerfromhell.blogspot.com OR somehorrificspammerfromhell.blogspot.com foo.bar.foo.bar.somehorrificspammerfromhell.blogspot.com would ALL return listing, but blogspot.com ...wouldn't. So it also takes some work determining those boundaries. Some of those are simple domains... while others like blogspot.com or wordpress.com, are more "artificial" (but still critically important). -- Rob McEwen invaluement.com
Re: RCVD_IN_SORBS_SPAM and google IPs
On 9/12/2016 9:37 AM, David Jones wrote: The majority of the junk can be blocked with zen.spamhaus.org and sip.invaluement.com RBLs. Every small mail filtering platform should use zen.spamhaus.org as long as they are under the free usage limit. The sip.invaluement.com is a private RBL but very reasonably priced and is a great complement to zen.spamhaus.org. The major senders should not be listed in these 2 major RBLs so they fit right in with the 6 items above. David, Thanks so much for the recommendation! But I just to make sure that everyone knows that throwing sip.invaluement.com into their spam filter is not going to help anyone as (1) it require a subscription to work -AND- (2) we now use DIFFERENT host names. In fact, eventually, no invaluement users will be using that host name (they'll eventually all be ported over to other alternate host names)... and then sip.invaluement.com will be wildcarded to "list the world"... eventually... (I'll make plenty of announcements before that time comes). Nevertheless, there will be an unfortunately group of users who threw sip.invaluement.com in their filter (unlicensed)... and will leave it there even though they've never gotten a single "hit" from their mis-configuration, and then they'll have a very bad day when that time comes. But, again, thanks for the mention! Perhaps, next time just say "invaluement". -- Rob McEwen invaluement.com
Re: spamassassin and caching nameservers
On 8/22/2016 9:15 PM, Alex wrote: Has anyone configured it as a local caching nameserver, and if so, could you share your config? Correct me if I'm wrong... but... I'm almost positive that rbldnsd acts ONLY as an authoritative name server, and not ever as a caching name server. I don't think there is functionality to either fetch root hints or to do catch-all forwarding to an upstream DNS server for just any host names. Instead, it only serves up the zones that it is specifically told to serve at startup, using the physical source data files to which those zones point. It was designed from the ground up only to serve as a dumbed down locally hosted DNS, only for serving DNSBLs where the data files are found locally. It makes up for the lack of more extensive DNS features with blazing speed and very low memory overhead. -- Rob McEwen
Re: Spoofed Domain
On 8/9/2016 5:56 PM, Anthony Hoppe wrote: Here are the headers as an example: http://pastebin.com/bnU0npLR This particular email has a macro-enabled Word document attached, but I don't want to assume this will be the case every time. Any tips/tricks/suggestions would be greatly appreciated! I think there is a trend now... towards blocking ALL .docm files (if not, there should be!). I think it is EXTREMELY rare for normal human beings to send Word documents in that particularly dangerous format. Most would be send in .doc or .docx format. I'm not sure if there is already a SA rule for scoring against .docm files attachments? Perhaps someone else could help you with that. -- Rob McEwen
Re: Corpus of Spam/Ham headers(Source IP) for research
On 6/29/2016 1:00 AM, Shivram Krishnan wrote: Thank you so much for your views. I agree that your customers would not like it if you share information. But Oliver suggested , I need only the source IP addresses of the Spam and Ham emails , which can even be anonymized in the last octet. Unfortunately, accuracy and credibility goes down since there then isn't any easy way to audit or double-check the root cause of the classification. For example, some people classify spam as "what our filter said was spam" and ham as "what our filter said was ham". For most well-run systems, that is going to be overall very accurate. But there can still be egregious mistakes. And assuming that the existing filter is 100% accurate leaves no room for improvement. It also has the unfortunate side effect of rubber stamping the most elusive spams, sent by the shrewdest of spammers, as ham. If an anti-spam blacklist comes along that is very good at blocking messages that are unsolicited and not desired by end users... but sent by the most shrewd spammer who evade lists like SpamHaus and SURBL (at least for some time)... and where the collateral damage for listing such domains and sending IPs is non-existent... such a blacklist might STILL fare badly in such a rating system... which would then MISTAKENLY assume that such a blacklist has many False Positives. Stats collected from user complaints about False Negatives can also be helpful. However, for snowshoe spam, that is often a lagging indicator... sometimes days behind reality--where the spammer has already moved to new domains/IPs--but such could help such a ratings system to make wise adjustments to past ham/spam stats. Hijacked IP and domains is another sticky issue. Over the past several years, this has become epidemic! If the volume of legit usage is relatively low, and the IP or domain has been hijacked by a spammer... then at SOME point, an anti-spam blacklist should not be penalized for listing such. In fact, Spamhaus does this frequently (lists hijacked domains/IPs where the cost/benefit ratio for that listing is well justified). Some other lists also blacklist hijacked domains/IPs... but are often not as good at making proper cost/benefit ratio decisions... where they list somewhat large senders who had a somewhat small and short-lived spam outbreak. Finding a way to penalize or reward the lists that block hijacked domains/IPs that Spamhaus misses, based on whether they do (or don't do) a good job of making overall good decisions about the cost/benefit ration of a potential listing's collateral damage... is also tricky. My main point is... how to reward blacklists that are more accurate, but without penalizing them for not being a redundant copy of Zen. It isn't as easy as it sounds in a ratings system. (even if real life usage of such by a hoster or ISP can quickly lead to fewer complains from customers about about FP and FNs) -- Rob McEwen
Re: Which DNSBLs do you use?
On 6/16/2016 9:49 AM, Alessio Cecchi wrote: Probably zen.spamhaus.org is the best dnsbl but is too expensive for us. Invaluement SIP is almost comparable to Zen as performance but much less expensive. Thanks, Alessio, for the recommendation. But I need to make one clarification... SIP and SIP24 should not be considered a replacement for ZEN because they purposely do NOT try to "catch every botnet" and instead focus on the more sneaky spams as well as new emitters. If someone tries to replace Zen with SIP and SIP24 (combined) they would usually be very disappointed in their overall spam filtering, unless... as I presume to be the case for Alessio Cecchi ... they had other very good measures in place for blocking botnet spams? But the vast majority of invaluement users use SIP and SIP24 as a supplement to Zen, and find that Zen blocks much spam that invaluement misses. Therefore, as I said, SIP and SIP24 (combined) are intended to be a supplement to Zen, not a replacement of Zen. (just want to make sure this is clear!) -- Rob McEwen http://www.invaluement.com
Re: Spamassassin not capturing obvious Spam
On 5/30/2016 10:24 PM, Shivram Krishnan wrote: I am testing spamassassin on a SPAM/HAM corpus of mails. Spamassassin is not picking up an obvious spam like in this case http://pastebin.com/MbNRNFWy . Your pastebin example didn't show the "last external" sending IP. Could have have been there orginally, but was expunged from this sample? Could there have also been a link in the body of the message that was likewise removed? it would be nice to be able to check those against respected low-FP DNSBLs. Or, if the clickable link really wasn't in the original message, then this particular example was probably a rare malfunctioned spam that will be of no benefit to the spammer, and would then probably not be worth investigating since the spammer then has no incentive to keep sending these types. -- Rob McEwen
Re: A Plan to Stop Violence on Social Media
On 12/15/2015 11:19 PM, Wrolf wrote:
Would it be practical to use the Spamassassin techniques of Bayesian
filtering and RBL lists to block ISIS on social media?
Bayes would have a high potential for false positives... such as
blocking a news story about ISIS, or blocking a discussion about how to
stop ISIS, or blocking an innocent discussion about the middle east, or
an innocent discussion about Islam.
But ISIS isn't an nearly as much of an extremely fast moving and
frequently morphing target as spam. Therefore, the techniques used don't
have to have as much risk of FPs (since they wouldn't have to be as
real-time-reacting to new scenarios) and should therefore be very
surgically targeted towards 100% confirmed ISIS sources.
At the same time, beware of top-down solutions where governments impose
free speech ("thought control") restrictions. Such powers could easily
be abused in the future for nefarious purposes, such as suppressing
criticism of the current party in power, etc.
This could be a "slippery slope".
--
Rob McEwen
+1 478-475-9032
Re: SpamAssassin Rules Regarding Abuse of New Top Level Domains
On 10/20/2015 12:13 PM, sha...@shanew.net wrote: Unlike Larry (and others) I DO want to block the vast majority of the new tlds, because we see nothing but spam from them (and my users tend toward the more false-positives than false-negatives side of the spectrum). Rather than maintain a list of all the problematic tlds, I'd rather have a blanket block rule with the ability whitelist the handful that might be legit. Be careful about doing this for the long term. I think that spammer exploit new TLDs because they know that many anti-spam systems don't account for them correctly at first. (and/or maybe they are cheaper at first?). But in the longer term (years down the road).. they tend to move on to other ones, while the legit TLDs slowly increase. So this strategy can backfire in the long term. (but, of course, MMV... and some smaller hosters don't have to be as concerned about a few extra FPs) -- Rob McEwen +1 478-475-9032
Re: Return Path (TM) whitelists
Also, often, the Return Path certified sender is an ESP who sends for a variety of customers. There is not always an absolute guarantee that every one of that ESP's customer is ethical and truthful. A good ESP will quickly fire such any such bad apple customer... but some do a much better job than others. Some spend endless amounts of time telling blacklists, we're Return Path certified... and we had this bad customer... but we're working with that customer to purge their lists of complainers and bad addresses. (iow, help them listwash, keeping them on as customers) ESPs are economically incentivized to keep marginal customers (or pretenders), and Return Path is economically incentivized to keep those grayhat-ESPs. Yes, at the extremes, customers will be fired in both situations. But there is a lot of gray before those extremes trigger a firing. And there are many situations where those limits are pushed. Having said that, those ESPs who choose to push those limits hurt themselves in the long run as their domains/IPs start getting dragged further and further down in various reputation and anti-spam filtering systems. But some of these are managed by 20-something-year old punk kids who haven't thought that far ahead. I'm sure Return Path stops lots of this stuff but certainly, a significant amount of unsolicited messages can slip through the cracks. Meanwhile, in contrast, DNSWL is NOT economically incentivized to go easy on gray senders. And some on this thread are not realizing that DNSWL has various LEVELS in its ratings of senders... where senders of BOTH legit mail and spam are marked accordingly. That way, you know to not outright block messages from certain mixed ham/spam sender's IPs... but you shouldn't treat them as fully whitelisted either. That is a big difference... therefore, most of the time that a virus-sent spam is sent from an IP in DNSWL, it is from an IP that is marked by DNSWL as a mixed source. -- Rob McEwen http://www.invaluement.com/ +1 478-475-9032
Re: Uptick in spam
On 3/30/2015 11:49 AM, Kris Deugau wrote: Seconded; this is exactly what we've been finding. Invaluement is a great complement to Spamhaus for a fraction of the cost. I wouldn't put it as a front-line reject DNSBL, because some of the things that have been listed are not what I would class, for our customers, as spam - but those entries are distinctly greyhat at best in a lot of cases, and some IP range operators I've flagged as list, delist, and whitelist_from_rcvd as needed due to the mix of legitimate small senders and spammers. Thanks Kris for the compliment. Also, when you say mix of legitimate small senders ...just to clarify, I think that any further analysis will show that (a) MOST of these are situations where very small senders had massive spam-sending outbreaks due to compromised accounts, and (b) the listing was most often very short lived (often mere hours). This is a balancing act... and I think invaluement strikes a great balance. And even in THIS particular area, I think our FP level is still distinctly LESS than UCEProtect, Barracuda, and SORBS (for examples). But if we brought that all the way to zero, MUCH spam that slips past Zen wouldn't be listed on invaluement anymore. (the ham/spam ratios on some of these compromised account situations is horrendous--they send out their usual 400 hams that day, along with 200,000 spams... and the cumulative sum total of those spams from ALL such compromised senders that day, represents MUCH of the spam that gets past filters due to piggybacking on the sender's normally good reputation) Also, what I've found is that many medium-sized ISPs/hosters, with 10s of thousand of mailboxes are very comfortable with outright blocking on invaluement, but will only score on UCEProtect, Barracuda, and SORBS. Much smaller hosters will often block on all of them, because they don't notice those FPs as often. In fact, I see these SAME somewhat rare compromised-sender FPs with Zen, too. It is all about each list's strategies, and aggressiveness, and tolerance levels. As shown, invaluement is in a very strategic spot here... having much of the aggressiveness of these other lists, but with FP levels VERY close to Zen's FP levels. (and then scoring on these other lists... even aggressive, yet still under-threshold, scoring... will help block spams missed by both invaluement and spamhaus) Also, invaluement plays close to the edge with CAN-spam and snowshoe spammers. So invaluement is in a little more dangerous territory...that it can do so and not have a lot more FPs, is not easy. For example, this invaluement may occasionally list the kind of pure ads that, upon further analysis, are arguably not technically spam, but aren't exactly desired by the end users. But these situations tend to sort themselves out over time. The SAME thing happens with invaluement's ivmURI domain blacklist. OFTEN, a normally legit web site has a CURRENT... LIVE spam infestation, where spammers broke into that site and placed spammy content there. This has become epidemic. Sure, it is frustrating for everyone, when such a site that is being used to send phishing and porn spams... causes some of that site's legitimate correspondence to get blocked... but this a necessary lesser of evils. The best part is that such a blacklisting motivates the site owner to fix their site FASTER. In such a situation, the blacklist provided the world a good service, and the resulting collateral damage was well justified. The site owner should be considered at fault for the collateral damage, not the DNSBL. I hope this provides some clarity. -- Rob McEwen +1 478-475-9032
Re: Uptick in spam
On 3/30/2015 1:19 PM, Kris Deugau wrote: The cases I can recall are more along the lines of grey-hat ESPs who pick up a spammer client for a while, Kris, The next time you run across this and think it might be causing a little too much collateral damage (in spite of the spamming), let me know (off-list) and I'll research it. I can then make adjustments accordingly. I'm very responsive to customer feedback. Thanks! -- Rob McEwen +1 478-475-9032
Re: Uptick in spam
On 3/27/2015 10:13 PM, David Jones wrote: The invaluement RBL is not expensive either and it is awesome. We pay thousands per year for a Spamhaus feed because of our volume and mailboxes. The invaluement RBL is only hundreds per year and it's almost as good as Spamhaus Zen. I have Spamhaus in front of invaluement in my postfix configuration but I may try flipping the order just to see if it will start blocking more than Spamhaus. Just to clarify, the two invaluement sender's IP blacklists, ivmSIP and ivmSIP/24, --combined-- is not (and will probably not ever be) an adequate replacement for Spamhaus's Zen list. So please everyone, don't get the idea that you can turn off Zen, add invaluement, and everything will be ok. David Jones was NOT saying that... but i just want to make sure that nobody mistakenly goes too far with this, beyond what David intended. Having said that... thanks, David, (and others) for your mentioning about your success with ivmSIP and ivmSIP/24, where they are helping you block much of the spam that slips past Spamhaus, etc. -- Rob McEwen
Re: Ready to throw in the towel on email providing...
On 7/28/2014 12:10 PM, Ted Mittelstaedt wrote: Small company with about 6 mailboxes who some consultant gave them a song and dance about how Gmail's such a better mail service since they don't get any spam Ted, fwiw, I had a situation last year where a friend (not one of my own clients) called me up asking me about a situation where legit hand-typed messages to their paid business-class gmail service... were getting blocked. So I ask, why are you asking me, ask gmail! He responded by saying that they are a paid business class gmail user, sending from their own domain name... and when he called his paid support line at google (again, this isn't the regular free gmail)... the tech support consultant was not able to lift a finger to help them. No research.. no SMTP logs... nothing. The paid subscriber was told that there must be some kind of problem... you'll have to wait this out In contrast, if one of my mail hosting clients reports that a hand-typed message to them is blocked, i get the details about the message and search through the SMTP logs, and report back to them exactly what happened and fix it if it were something controllable on my end. (usually there is a more innocent explanation, like the sender making a typo in the e-mail address, etc)... but I first ASSUME it is my problem... THEN research it.. then give the client ANSWERS and SOLUTIONS. PLEASE correct me if I'm wrong as my example above is anecdotal.. but from what I understand, Google doesn't provide ANSWERS and SOLUTIONS for situations like this. You just get excuses and delays. Maybe Google has improved since then?... or maybe my report is not accurate (but it came to me first hand from a trustworthy source). Definately double check this. If you can verify that this is true (and continues to be true)... then use this info as a rebuttal the next time you have a client talk about leaving you for gmail. -- Rob McEwen +1 (478) 475-9032
Re: Domain ages (was Re: SPAM from a registrar)
On 6/10/2014 10:21 AM, Axb wrote: All URI BLs I know of (SURBL/URIBL/DBL/Invaluement/etc) check track domain reputation otherwise they'd be unusable. Their listings are not blind - they all have their secret sauce to process before listing a domain. Absolutely. As Axb and KAM and others stated, a very young domain age is too dangerous to outright block or score high on... but might be a good factor or good for combining with other rules. Also, if anyone does see spam that contain domains in the clickable links where that spam should have been blocked, but was not... then check the domain contained within the spam again the lookup found at http://multirbl.valli.org and/or http://mxtoolbox.com/blacklists.aspx (some months ago, MX Toolbox upgraded their system to check domains against URI/domain blacklists. In some cases, this could have been a game of inches where your user caught the tip of the spear and received the very first spams in a spam campaign that otherwise was quickly listed by the well known URI BLs. However, you may find that one or two good URI BLs are simply not implemented in your system--where that would have made all the difference! Those lookup forms will point you in the right direction. The same can also be true for checking sending IPs--then reviewing your current mix of sender's IP dnsbls (aka RBLs). Of course, don't fall into the trap of using a BL that catches much, but has too many FPs. But the list of URI BLs that Axb gave above are all extremely low-FP URI blacklists. -- Rob McEwen +1 (478) 475-9032
Re: Domain ages (was Re: SPAM from a registrar)
On 6/10/2014 10:34 AM, Patrick Domack wrote: So, we are unwilling to look into any new ideas cause there might be an issue? that we haven't scoped or checked into? Patrick, I don't think Axe was arguing against this idea.. I think he was arguing against irrational exuberance by some who may be taking this idea to the point of outright blocking (or high scoring) on it, which would generate significant FPs. His examples were solid real world examples that DO happen and WOULD FP if this idea were taken too far. But using extremely-young-domain-age for low scoring or in combination with other rules could be very helpful. -- Rob McEwen +1 (478) 475-9032
Re: Domain ages (was Re: SPAM from a registrar)
Domain age is a good metric to factor in. But I'm always fascinated with some people's desire to block all messages with extremely new domains. (NOT saying that this applies to everyone who posted on this thread!) Keep in mind that many large and famous businesses... who have fairly good mail sending practices... sometimes launch a new products complete with links to very newly registered domains. Same is often true for advertisments for things like rock concerts, etc. Or web sites that deal with specific events or hot-topic political issues that appeared out of nowhere. Yes, some of these are UBE. But many are NOT! These example provide one of the largest source of FPs for all the major domain/URI blacklists. But the better domain/URI blacklists have good mechanisms in place to (a) PREVENT... MANY of these from ever becoming FPs in the first place, and (b) and where those mechanism failed, they have good triggers/feedback to remove whitelist such FPs VERY QUICKLY if/when they do occur. In contrast, many who might go overboard by outright blocking on newness... and/or scoring too agressively on newness... may find too-high FP problems kicking their butts in the long run. And when such a FP starts happening, they may not have the proper telemetry to catch/fix it until AFTER much FP damage has happened. Personally, I think that the real problem here is that some of the most famous URI/domain blacklists are NOT catching everything and/or NOT catching everything fast enough... combined with many sys admins failing to make use of ALL the good and low-FP URI/domain blacklists... where they 'd see MUCH better results if they were using ALL of the good URI blacklists! ...but I'm a little biased on this point! :) -- Rob McEwen +1 (478) 475-9032
Re: Who wants to trade data?
On 2/6/2014 6:59 PM, Noel Butler wrote: spams an anti-spam list so sharing/discussing data/intel about spammers on an anti-spam list... is spamming? Really? -- Rob McEwen invaluement.com
what is that number at the beginning of .cf files signify?
what is that number at the beginning of .cf files signify? Does that impact SA's actual operation? Or is that just for human organization of files (how they sort when browsing them)? When adding a custom-written .cf file that is made available to the public, should some kind of naming convention be followed, even if just for etiquette? -- Rob McEwen http://dnsbl.invaluement.com/ r...@invaluement.com +1 (478) 475-9032
Re: Uptick in false negatives - filter check?
On 11/7/2013 6:00 PM, Owen Mehegan wrote: Thanks in advance for any advice anyone can offer! fwiw, of the 4 spam examples, ivmURI had blacklisted one or more domains in ALL 4 out of 4 samples at least several minutes BEFORE those spams hit your server (some days or weeks before). In a large portion of those (1/2 or more), I'm fairly sure that ivmURI was the ONLY URI/domain blacklist to have the domain blacklisted at the time the message hit your network. (I'm unable to verify if DBL had caught it at that time and/or some of those could have been a game of inches where ivmURI and other lists had just listed it moments before and it would be somewhat of a propagation issue... but, overall, I think if I provided the date/times that these were blacklisted on ivmURI... that assertion would check out and the raw data would be rather impressive!) If you keep seeing these, check the domains on multirbl.valli.org ...and you'll see in real time what I'm talking about! -- Rob McEwen http://dnsbl.invaluement.com/ r...@invaluement.com +1 (478) 475-9032
Re: KAM pccc URIBL questions
On 10/7/2013 7:42 PM, Raymond Dijkxhoorn wrote: This is harming more then it does good. But its your list so your rules ;) I would not want to use it to filter my mails with it but hey Since this is in its early development, it is probably too early to judge it too much. But from what I've read in this discussion, it is light years away from the current major URI/domain blacklists out there (SURBL, URIBL, ivmURI, DBL)... BUT... Kevin is brilliant so who knows what it might eventually become? ALSO...There is an argument that a more-aggressive-than-normal AND low-scoring URI list may be helpful? In that sense, URIBL.com has traditionally been considered slightly more aggressive than the other lists mentioned above... SLIGHTLY! Maybe something much MORE aggressive, intended for very low scoring... would be useful? (this would be situations where bayes or checksum content filters add points to the spam score combined with such an aggressive URI list putting the message over the top... but then skipping blocking a legit message with this URI because it didn't have the other content points added and thus didn't score high enough--at least that is the idea) But I can't help but think that SOME reading this thread haven't even tried/implemented even all the zero-cost options for the (already matured) lists I mentioned (where applicable)? -- Rob McEwen http://dnsbl.invaluement.com/ r...@invaluement.com +1 (478) 475-9032
