On 2/20/2018 6:05 PM, @lbutlr wrote:
On 2018-02-20 (08:30 MST), Rob McEwen <r...@invaluement.com> wrote:
Spammers are starting to use this to evade spam filters,
This is not news. Spammers have been using shortness since 3 seconds after
tinyurl.com launched.
My "this" was /*specifically*/ referring to Google's shortner, or at
least the recent STRONG uptick in the abuse of that shortner. I was
already well aware of other similar things from other shortners. But
"this" wasn't referring to them. You infused thoughts/meaning into my
writing that really wasn't there. (be careful about assuming...) Also,
when you see my report further down, you'll understand why those others
are not nearly as much of a concern to me at the moment.
Keep in mind that, if a marketer is doing things the right way, they should
have no need to obfuscate their own domain name. They should instead proudly
use it and not feel the need to hide behind Google's shortner.
No, that is not at all true. The primary use of a shorter is to shorten a long
URL to something that someone can type in.
I've acknowledged that there are some good reasons for a shortner - but
the vast majority of the times I'm seeing them - in both ham and spam -
that is NOT the case! The are shortening things like average-sized
domains with either no directory, or with a short directory or page
names after the domain. This is the VAST MAJORITY of the shortners I'm
seeing in both hams and spams.
Clicking a URL in an email is the height of stupidity, so having a short URL
that someone can realistically type into a browser is much better.
If I spent just a little more time on this, I could collect a large
number of Google shortner URLs that are malicious - where my
malwarebytes is blocking access to the page to which it is trying to
redirect. And these are still "live"! Do you really think that more than
a tiny percent of those who saw those in their mailbox (both legit and
spams) are manually typing in the URL and not just clicking on it? And
in that exceedingly rare occasion where somebody types in the URL and it
redirects to a malicious page that tries to install a virus, are they
ANY better off than having just clicked on it? Even the best point I can
think of that you might have had - that this might help them to better
recognize a phishing URL for example - is lost since BOTH the phish and
their bank's web site is going to be indistinguishable until AFTER
they've launched the shortner (whether by clicking or typing). I think
you just mistakenly bolstered my argument against this over-usage (and
often inappropriate usage) of shortners!
THEREFORE: If you like having NOT-blacklisted IPs, be advised that the invaluement anti-spam DNSBL system is
now adding "bad" points to the scoring of all messages that use the "goo.gl" shortner,
and we're amplifying other "bad" points.
Well, at least you are warning people. However, what you are doing is,
frankly, dumb; if you think there's a huge problem, you can simply
check the target URLs.
That incurs a significant amount of extra resources for DNSBLs and spam
filters - and such automated lookups could also put a huge extra burden
on Google's servers - and who knows at this point if this is even
reliable - Google might easily start putting captchas in the way or
otherwise consider such lookups to be abusive and/or mistake them for
malicious bots... I'm definitely going to pursue this further - but
wow... that you would suggest this... I think spam puts ENOUGH burden on
spam filters and mail system as it is!
Yes, there are many legitimate uses of Google's shortner, too. However, we are
now at a point where a VERY large % (a majority?) of uses of these headed to a
typical user's mailbox are egregious spams, and a significant additional
portion are likely-spams.
Any evidence of this?
EVIDENCE/STATS:
I ran stats on a sample set of a few thousand mailboxes, over a period
of several hours today (mostly during business hours for these
particular organizations who use these mailboxes) - and this produced a
combined 24K legit messages, and 5K spams (I'm guessing that most
systems have more spams per amount of hams? But those were the numbers
for this server.)
-----------------------------------
NOTE: The sum of individual shortner-hits totals below can be LARGER
than the total messages that had hits on ANY shortner - Why? - Because
in a few cases, the same message can have hits on more than one of these
shortners
-----------------------------------
I SEARCHED EACH HAM AND SPAM CORPUS FOR MANY OF HUNDREDS OF URL SHORTNERS
HERE ARE THE RESULTS:
-----------------------------------
STATS FROM SPAM:
286 total spams blocked that had a shortner, out of hundreds of URL
shortners I had searched on
(<10 that *MIGHT* be FPs - they were definitely questionable at best -
btw, zero of those questionable ones led to ANY kind of invaluement
blacklisting, even with the adjustments I described)
of that 286 SPAMS...
262 used goo.gl
24 used bit.ly
1 used eepurl.com
0 used tinyurl.com
0 used t.co
ONLY 17 used one of ALL of the others COMBINED
-----------------------------------
STATS FROM HAM:
187 total legit messages had a hit on at least one of hundreds of URL
shortners I had searched on
(a dozen or two of those were "false negatives", most of which used
goo.gl - and that level of missed spams was much higher prior to the
changes I had recently made regarding Google's shortner.)
of that 187
83 used goo.gl
59 used bit.ly
40 used eepurl.com
3 used tinyurl.com
53 used t.co
42 for ALL of the others COMBINED
-----------------------------------
NOTE: eepurl.com was under my radar - but since they predominantly hit
in hams and had extreme few spam hits (just 1), this one is probably
well run.
ALSO: I initially accidentally left out "t.co" in the overall stats, but
they only had 53 ham hits, and zero spam hits - so it didn't seem worth
going back and updating the overall totals to include those messages
that weren't already in those totals.
Clearly, the results here show that, at the moment - Google's shortner
is DOMINATING in its spam usage, where 92% (262 of 286) of ALL spam that
contained shortners used Google. All the rest shared the remaining 8%,
with bit.ly being in the distinct 2nd place.
I should note that I already had SIMILAR rules in place (for YEARS) that
caused higher spam scoring for bit.ly (and a few others) - and I think
that is *one* of the reasons why I'm not picking on them as much at the
present time. (besides the fact that they are not used for spamming
nearly as much at the moment as Google's shortner)
So why am I saying this about Google now - but didn't do a similar post
about bit.ly or TinyURL or others way back when?
(1) the lines never seemed quite as blurred, or at least blurred so
quickly, as they are now with Google. The bit.ly and tinyurl spam usage
seemed more limited in scope, and rules that would hit on it them
without hitting on legit mail usage of those shortners were easier to
create. But part of the problem here is that the recent spam usage of
goo.gl is getting a little bit more sophisticated in their usage of
tactics to evade filters - making the writing of those rules a little
harder. Having said that - this particular analysis has alerted me to
the fact that I'm now overdue to tighten up on bit.ly, too - since some
of these same things are happening more and more with bit.ly - I just
didn't notice it as much since LARGER problems with goo.gl are hogging
my attention.
HOWEVER...
(2) Even so - I'm seeing a distinct pattern whereby bit.ly is QUICK to
dead-end shortners created by spammers... while Google is SLOW!!! And
this is EXACTLY why spammers are currently MUST MORE OFTEN choosing to
use Google's shortner right now. FOR EXAMPLE: For those stats above - I
went back and checked on the 5 oldest spams in the corpus that used
bit.ly shortners - and the 5 oldest spams in that corpus from today that
used goo.gl shortners found in egregious spam. btw - *ALL* of those 5
goo.gl shortner examples were at least 2 hours OLDER than my oldest
bit.ly samples - so Google had a two hour "head start". But all were
about 7-9 hours old, fwiw, when I checked back on those links.
RESULTS:
All 5 of my oldest bit.ly shortners had been disabled :)
All 5 of my oldest goo.gl shortners were fully operational (even if the
redirect was blocked by my malwarebytes... but Google was TRYING to
deliver the spam payloads!) :(
And this is consistent with my recent experiences with these goo.gl
shortners. I've reported several to the Google shortner page as spam
over the past several weeks - and then I'd stop checking on them after a
few days passed and they were STILL operational.
(3) And then another problem with goo.gl is that it seems to be getting
more "institutionalized" with Google's software (etc?). Want a google
map link? Or a link to Google Drive? I'm pretty sure their software will
serve up a shortener version - and for good reason. Some of THOSE links
can be complex and benefit greatly from being shortened. But this ends
of being a "human shield" for spammers if they can't keep up with
policing the abuse.
(4) And as I had mentioned, I'm seeing evidence that gray-hat spammers
and ESPs are jumping on this bandwagon, even MORE so than they ever did
with other shortners - and I have a HUGE HUGE HUGE PROBLEM WITH THAT. It
is a loophole where spammer can evade a spam filter and DNSBL's ability
to FULLY evaluate the *identity* and *reputation* of the sender and the
sender's message - as such senders then hide behind the shortner. This
is OK if this is an innocent and necessary link to Google maps, for
example. This is NOT ok if an ESP says to its clients, "hey guys, we
have this great new trick where you don't ever have to worry about your
domain getting blacklisted and more of your messages will now get
delivered" (more legit senders don't need that help!) And there are
various shades of gray to work out in between. Also, when I was
examining the more legit messages that had this shortner - while I did
find a few example to things like Google maps - I ALSO saw a few
examples where people were putting that as a link in their signature -
for web site addresses which were NOT very long in the first place. That
makes no sense - and makes me wonder, what software is stupidly
auto-generating those links in the signature for them? (more awful human
shields!)
(5) Finally - this is Google!!!! - they carry more *instant* legitimacy
- making them more of a desirable target for spammer to exploit (that is
- if/when the abuse isn't properly policed, as is happening now) -
Google is much more desired than just using some other company that
created a shortner.
I think that spammers and blackhat/grayhat ESPS... are ENJOYING my
pain! And I think they LOVE this loophole. Please don't "carry the
water" for them.
--
Rob McEwen
https://www.invaluement.com
