subject:"Re\: The \"goo.gl\" shortner is OUT OF CONTROL \(\+ invaluement's response\)"

>
> We just created an URL signature algorithm to be able to query an entire
> URL at our URIBL:
>
> https://spfbl.net/en/uribl/
>
> Now we are able to blacklist any malicious shortener URL
>
>
> Leandro,
>
> Thanks for all you do! And good luck with that. But there are a few
> potential problems. When I analyzed Google's shortners about a month ago, I
> found that a VERY large percentage of the most malicious shortened URLs
> were a situation where the spammers were generating a unique shortner for
> each individual message/recipient-address. This causes the following HUGE
> problems (at least for THESE particular shortners) when publishing a
> full-URL dnsbl:
>

Thank you for all those observations!


> (1) much of what you populate your rbldnsd file with is going to be
> totally ineffective for anyone since it ONLY applied to whatever single
> email address where the spam was original sent (where you had trapped it) -
> everyone else is going to get DIFFERENT shortners for the spam from these
> same campaigns that are sent to their users.
>

You are right, but we do not use rbldnsd. We have our own DNSBL
implementation that uses a more efficient data structure. Anyway, I thing
that is not a good idea list each shortener, as you sad. Maybe thrice
complains of same shortener. We will discover what is the best some time.


> (2) get ready for EXTREME rbldnsd bloat. You're gonna need a LOT of RAM
> eventually? And if you ever distribute via rsync, those are going to be
> HUGE rsync files (and then THEY will need a lot of RAM). Sadly, most of
> that bloat is going to come from entries that are doing absolutely nothing
> for anyone.
>

That is it! We use a VM with 16GB and the software is using about 10GB to
keep more than 30 million registers at memory. That is something about 350
bytes per register. Our software have an expiration mechanism, than this
memory occupation is not growing to fast now. But we must keep one eye on
it always.


> (3) You might be revealing your spam traps to the spammers. In cases where
> the spammers are sending that 1-to-1 spam to single recipient shortners,
> then all they gave to do is enumerate through their list of shortners,
> checking them against your list - and they INSTANTLY get a list of every
> recipient address that triggers a listing on your DNSBL. If you want to
> destroy the effectiveness of your own DNSBL's spam traps - be my guest. But
> if you're getting 3rd party spam feeds (paid or free) - then know that
> you're then screwing over your 3rd party spam feed's spam traps - and those
> OTHER anti-spam system that rely on such feeds, which will then diminish in
> quality. (unless you are filtering OUT these MANY 1-to-1 shortner spams)
>

Not only spamtraps will trigger this listing. All active users will do it
too by complains. The spammer will not know who is spamtrap and who is
active user.


> Maybe there is enough OTHER shortners (that are sending the same shortners
> to multiple recipients) to make this worthwhile? But the bloat from the
> ones that are uniquely generated could be a challenge, and could
> potentially cause a MASSIVE amount of useless queries. I'd be very
> interested to see what PERCENTAGE of such queries generated a hit!
>
> Meanwhile, in my analysis I did about a month ago, about 80% of Google's
> shortners found in egregious spams (that did this one-to-one
> shorter-to-recipient tactic)... were all banging on one of ONLY a dozen
> different spammers' domains. Therefore, doing a lookup on these and then
> checking the domain found at the base of the link it redirects to... is a
> more effective strategy for these - whereas, for THESE 80% of egregious
> google shortners, a full URL lookup is worthless, consuming resources
> without a single hit.
>

That is right. We have same situation here.

But check first URL is not only action we do. Our script can follow
shortener redirections and catch the spammer by last URL of redirection
chain:

https://www.dropbox.com/s/5aorrijafw5ygk0/uribl.pl?dl=0

The spammers can be trapped by any shortener they have or by this dozen
domains that shortener hides.

Alternatively, you may have found a way to filter out these types of
> individualized shortners, to prevent that bloat? But even then, everyone
> should know that while your new list might be helpful, it would be good for
> others to know your new list isn't applicable to a large percentage of
> spammy shortners, since it is still useless against these individualized
> shortners.
>

I think that we all must cause to much of work for spammers, as much they
cause to us. If the spammer uses individualized shorteners, we can list
each one by crossing data with listed final chain URL domains. If they uses
individualized URL domains, we can list each one by crossing data with
listed URL equivalent IP (same machine for all spammer domains). We can
make it more and more expensive for spammers. But we must work together to
do it.


> NOTE: Google has made

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

>
> > > Then the frequency is 10 per second, not 100ms. Querying more often
> > > is a higher frequency.
> >
> > That is it! 10 per second or one every 100ms. The first is a flow rate
> and
> > the second is a frequency.
>
> One every 100ms is a frequency, agreed.
>
> Two every 100ms is a higher frequency, and means faster requests.
>
> One every 50ms is the same rate as two every 100ms, therefore it is also a
> higher frequency than one every 100ms.
>

You are right! My mistake. I just fixed website information. Thanks!


>
> Regards,
>
>
> Antony.
>
> --
> I wasn't sure about having a beard at first, but then it grew on me.
>
>

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-04-03 Thread Dianne Skoll

On Tue, 3 Apr 2018 11:21:35 -0400
Rob McEwen  wrote:

> Thanks for all you do! And good luck with that. But there are a few 
> potential problems. When I analyzed Google's shortners about a month 
> ago, I found that a VERY large percentage of the most malicious 
> shortened URLs were a situation where the spammers were generating a 
> unique shortner for each individual message/recipient-address.

We found that too, but in most cases, they generated the unique URLs
by adding query parameters to the same base URL, sort of like this:

http://malware.net/?id=znsjdsjau
http://malware.net/?id=aosu94e
etc...

and then shortening them.

So if you blacklist just the base URL, you cover those all off,
assuming you expand out shortened URLs as part of your processing, of course.

> Meanwhile, in my analysis I did about a month ago, about 80% of
> Google's shortners found in egregious spams (that did this one-to-one 
> shorter-to-recipient tactic)... were all banging on one of ONLY a
> dozen different spammers' domains. Therefore, doing a lookup on these
> and then checking the domain found at the base of the link it
> redirects to... is a more effective strategy for these - whereas, for
> THESE 80% of egregious google shortners, a full URL lookup is
> worthless, consuming resources without a single hit.

Yep, that's what we found too.

Regards,

Dianne.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-04-03 Thread Rob McEwen


On 4/3/2018 9:27 AM, Leandro wrote:
We just created an URL signature algorithm to be able to query an 
entire URL at our URIBL:


https://spfbl.net/en/uribl/

Now we are able to blacklist any malicious shortener URL



Leandro,

Thanks for all you do! And good luck with that. But there are a few 
potential problems. When I analyzed Google's shortners about a month 
ago, I found that a VERY large percentage of the most malicious 
shortened URLs were a situation where the spammers were generating a 
unique shortner for each individual message/recipient-address. This 
causes the following HUGE problems (at least for THESE particular 
shortners) when publishing a full-URL dnsbl:


(1) much of what you populate your rbldnsd file with is going to be 
totally ineffective for anyone since it ONLY applied to whatever single 
email address where the spam was original sent (where you had trapped 
it) - everyone else is going to get DIFFERENT shortners for the spam 
from these same campaigns that are sent to their users.


(2) get ready for EXTREME rbldnsd bloat. You're gonna need a LOT of RAM 
eventually? And if you ever distribute via rsync, those are going to be 
HUGE rsync files (and then THEY will need a lot of RAM). Sadly, most of 
that bloat is going to come from entries that are doing absolutely 
nothing for anyone.


(3) You might be revealing your spam traps to the spammers. In cases 
where the spammers are sending that 1-to-1 spam to single recipient 
shortners, then all they gave to do is enumerate through their list of 
shortners, checking them against your list - and they INSTANTLY get a 
list of every recipient address that triggers a listing on your DNSBL. 
If you want to destroy the effectiveness of your own DNSBL's spam traps 
- be my guest. But if you're getting 3rd party spam feeds (paid or free) 
- then know that you're then screwing over your 3rd party spam feed's 
spam traps - and those OTHER anti-spam system that rely on such feeds, 
which will then diminish in quality. (unless you are filtering OUT these 
MANY 1-to-1 shortner spams)


Maybe there is enough OTHER shortners (that are sending the same 
shortners to multiple recipients) to make this worthwhile? But the bloat 
from the ones that are uniquely generated could be a challenge, and 
could potentially cause a MASSIVE amount of useless queries. I'd be very 
interested to see what PERCENTAGE of such queries generated a hit!


Meanwhile, in my analysis I did about a month ago, about 80% of Google's 
shortners found in egregious spams (that did this one-to-one 
shorter-to-recipient tactic)... were all banging on one of ONLY a dozen 
different spammers' domains. Therefore, doing a lookup on these and then 
checking the domain found at the base of the link it redirects to... is 
a more effective strategy for these - whereas, for THESE 80% of 
egregious google shortners, a full URL lookup is worthless, consuming 
resources without a single hit.


Alternatively, you may have found a way to filter out these types of 
individualized shortners, to prevent that bloat? But even then, everyone 
should know that while your new list might be helpful, it would be good 
for others to know your new list isn't applicable to a large percentage 
of spammy shortners, since it is still useless against these 
individualized shortners.


NOTE: Google has made some improvements recently, and I haven't yet 
analyzed how much those improvements have changed any of these things 
I've mentioned?


PS - the alphanumeric code at the end of these shortners tend to be 
case-sensitive, while the rest of the URL is NOT case sensitive (and 
they also work with both "https" and "http")... so you might want to 
standardize this on (1) https and (2) everything lower case up until the 
code at the end of the shortner - before the MD5 is calculated. 
Otherwise, it could easily break if the spammer just mixes up the 
capitalization of the shortner URL up until the code at the end of the 
shortner.


--
Rob McEwen
https://www.invaluement.com

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-04-03 Thread Antony Stone

On Tuesday 03 April 2018 at 16:43:09, Leandro wrote:

> 2018-04-03 11:35 GMT-03:00 RW:
> > On Tue, 3 Apr 2018 11:09:38 -0300 Leandro wrote:
> > > 2018-04-03 10:34 GMT-03:00 Antony Stone:
> > > > "IMPORTANT: Current limit is 100 ms per IP block. Lower frequencies
> > > > require contribution. Please contact us informing your IP or range,
> > > > for further details."
> > > 
> > > This means, for example, your system do 10 queries at same second,
> > > then the query frequency is 100ms.
> > 
> > Then the frequency is 10 per second, not 100ms. Querying more often
> > is a higher frequency.
> 
> That is it! 10 per second or one every 100ms. The first is a flow rate and
> the second is a frequency.

One every 100ms is a frequency, agreed.

Two every 100ms is a higher frequency, and means faster requests.

One every 50ms is the same rate as two every 100ms, therefore it is also a 
higher frequency than one every 100ms.

Regards,


Antony.

-- 
I wasn't sure about having a beard at first, but then it grew on me.

   Please reply to the list;
 please *don't* CC me.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-04-03 11:35 GMT-03:00 RW :

> On Tue, 3 Apr 2018 11:09:38 -0300
> Leandro wrote:
>
> > 2018-04-03 10:34 GMT-03:00 Antony Stone <
> > antony.st...@spamassassin.open.source.it>:
>
> > > "IMPORTANT: Current limit is 100 ms per IP block. Lower frequencies
> > > require contribution. Please contact us informing your IP or range,
> > > for further details."
> > >
> >
> >
> > This means, for example, your system do 10 queries at same second,
> > then the query frequency is 100ms.
>
> Then the frequency is 10 per second, not 100ms. Querying more often
> is a higher frequency.
>


That is it! 10 per second or one every 100ms. The first is a flow rate and
the second is a frequency.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

>
> > >
> > > "IMPORTANT: Current limit is 100 ms per IP block. Lower frequencies
> > > require contribution. Please contact us informing your IP or range, for
> > > further details."
> >
> > This means, for example, your system do 10 queries at same second, then
> the
> > query frequency is 100ms.
>
> Yes, I got that bit.
>
> How big is an IP block?
>


Maybe I did not understand your question, but our DNSBL lists individual
IPs, even /128 for IPv6.

Sometimes our system lists entire blocks, like /64 for IPv6 and /24 for
IPv4.



>
> > > Please could you explain what this means; what limitations are imposed
> on
> > > use of this service - specifically what is an "IP block", and do you
> really
> > > mean "lower frequencies require contribution"?  Surely that should be
> > > "higher"?
> >
> > Yes, I am sure. Lets use the same example above, but now your system do
> 20
> > queries at same second, then the query frequency becomes 50ms, less than
> > first case.
>
> Ah; I would call 50ms the interval and 20 queries per second the frequency.
>
> Thanks for the explanation.
>
>
You are always welcome!

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-04-03 Thread RW

On Tue, 3 Apr 2018 11:09:38 -0300
Leandro wrote:

> 2018-04-03 10:34 GMT-03:00 Antony Stone <
> antony.st...@spamassassin.open.source.it>:  

> > "IMPORTANT: Current limit is 100 ms per IP block. Lower frequencies
> > require contribution. Please contact us informing your IP or range,
> > for further details."
> >  
> 
> 
> This means, for example, your system do 10 queries at same second,
> then the query frequency is 100ms.

Then the frequency is 10 per second, not 100ms. Querying more often
is a higher frequency.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-04-03 Thread Antony Stone

On Tuesday 03 April 2018 at 16:09:38, Leandro wrote:

> 2018-04-03 10:34 GMT-03:00 Antony Stone:
> > On Tuesday 03 April 2018 at 15:27:11, Leandro wrote:
> > > Hey guys. We just created an URL signature algorithm to be able to
> > > query an entire URL at our URIBL:
> > > 
> > > https://spfbl.net/en/uribl/
> > 
> > I don't think I understand the following statement on that page:
> > 
> > "IMPORTANT: Current limit is 100 ms per IP block. Lower frequencies
> > require contribution. Please contact us informing your IP or range, for
> > further details."
> 
> This means, for example, your system do 10 queries at same second, then the
> query frequency is 100ms.

Yes, I got that bit.

How big is an IP block?

> > Please could you explain what this means; what limitations are imposed on
> > use of this service - specifically what is an "IP block", and do you really
> > mean "lower frequencies require contribution"?  Surely that should be
> > "higher"?
> 
> Yes, I am sure. Lets use the same example above, but now your system do 20
> queries at same second, then the query frequency becomes 50ms, less than
> first case.

Ah; I would call 50ms the interval and 20 queries per second the frequency.

Thanks for the explanation.


Antony.

-- 
90% of networking problems are routing problems.
9 of the remaining 10% are routing problems in the other direction.
The remaining 1% might be something else, but check the routing anyway.

   Please reply to the list;
 please *don't* CC me.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-04-03 10:34 GMT-03:00 Antony Stone <
antony.st...@spamassassin.open.source.it>:

> On Tuesday 03 April 2018 at 15:27:11, Leandro wrote:
>
> > Hey guys. We just created an URL signature algorithm to be able to query
> an
> > entire URL at our URIBL:
> >
> > https://spfbl.net/en/uribl/
>
> I don't think I understand the following statement on that page:
>
> "IMPORTANT: Current limit is 100 ms per IP block. Lower frequencies require
> contribution. Please contact us informing your IP or range, for further
> details."
>


This means, for example, your system do 10 queries at same second, then the
query frequency is 100ms.



>
> Please could you explain what this means; what limitations are imposed on
> use
> of this service - specifically what is an "IP block", and do you really
> mean
> "lower frequencies require contribution"?  Surely that should be "higher"?
>


Yes, I am sure. Lets use the same example above, but now your system do 20
queries at same second, then the query frequency becomes 50ms, less than
first case.



>
>
> Thanks,
>
>
> Antony.
>
> --
> There's a good theatrical performance about puns on in the West End.  It's
> a
> play on words.
>
>Please reply to the
> list;
>  please *don't* CC
> me.
>

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-04-03 Thread Antony Stone

On Tuesday 03 April 2018 at 15:27:11, Leandro wrote:

> Hey guys. We just created an URL signature algorithm to be able to query an
> entire URL at our URIBL:
> 
> https://spfbl.net/en/uribl/

I don't think I understand the following statement on that page:

"IMPORTANT: Current limit is 100 ms per IP block. Lower frequencies require 
contribution. Please contact us informing your IP or range, for further 
details."

Please could you explain what this means; what limitations are imposed on use 
of this service - specifically what is an "IP block", and do you really mean 
"lower frequencies require contribution"?  Surely that should be "higher"?

Thanks,

Antony.

-- 
There's a good theatrical performance about puns on in the West End.  It's a 
play on words.

   Please reply to the list;
 please *don't* CC me.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

Hey guys. We just created an URL signature algorithm to be able to query an
entire URL at our URIBL:

https://spfbl.net/en/uribl/

Now we are able to blacklist any malicious shortener URL. Now I will think
about some public complain interface that automatic lists any correct
malicious sample using some simple AI.

All you have to do now is implement a SA plugin to make this signature and
do the URIBL query.

Regards,

Leandro
SPFBL.net

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-04-01 Thread Rob McEwen


On 4/1/2018 7:10 PM, Kevin A. McGrail wrote:
No, I don't think it's an April Fool's trick though it is possible. 
They announced this a day or 2 ago.
See 
https://www.cloudconnectcommunity.com/ccc/ls/community/g-suite-feature-ideas/post/6320666165116928 
and https://firebase.google.com/docs/dynamic-links/



I have been in talks with Google about this, sharing with them specific 
data and stats. My contact there sent me an email about this a day or 
two ago referencing that announcement - I just hadn't had time to review 
it yet. So this is for real.


The data I had compiled and shared with them that I gathered about a 
month ago... was devastating. I can't make that report public because 
the spammers were OFTEN sending a one-to-one ratio of a single 
individual google shortener per recipient address. Therefore, making 
that data public would have revealed many spamtrap addresses (both mine 
and those of 3rd party spam feeds). Another problem with that data... is 
that I was finding 10s of thousands of google shortners that were 
banging on the same dozens spammer's domains - and it would be so 
trivial for Google to look for that pattern and then just wipe out all 
of THOSE many shortners that were redirecting to one of these dozen 
spammers' domains. At the same time, the vast majority of these 
shortners were persisting for weeks and weeks with no sign that Google 
was hardly lifting a finger to terminate them. In fact, in that initial 
report, 95+% of shortners used by spammers were persisting for weeks on 
end, and 80% of those were using one of those 12 spammer' domains.


HOWEVER... I started compiling new data few days ago, and it looks MUCH 
better now. I'm not done putting that report together, but early 
indications show that something has recently changed for the better - 
but that there is still significant room for improvement, but they are 
at least headed in the right direction. But that report is still under 
construction.


So I think the pressure on Google, along with some of the data I 
provided them... might have helped? Or maybe that was just "one straw 
that broke the camel's back"? Either way, I'm happy that this seems to 
be getting fixed, or they are at least headed in the right direction.


--
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-04-01 Thread Kevin A. McGrail

No, I don't think it's an April Fool's trick though it is possible.  They
announced this a day or 2 ago.

See
https://www.cloudconnectcommunity.com/ccc/ls/community/g-suite-feature-ideas/post/6320666165116928
and https://firebase.google.com/docs/dynamic-links/

Regards,
KAM

--
Kevin A. McGrail
Asst. Treasurer & VP Fundraising, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171

On Sun, Apr 1, 2018 at 5:02 PM, Benny Pedersen  wrote:

> @lbutlr skrev den 2018-04-01 22:46:
>
>> On 2018-02-20 (08:30 MST), Rob McEwen  wrote:
>>
>>>
>>> RE: The "goo.gl
>>> " shortner is OUT OF CONTROL (+ invaluement's response)
>>>
>>
>> 
>>
>
> april fools day :=)
>

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-04-01 Thread Benny Pedersen


@lbutlr skrev den 2018-04-01 22:46:

On 2018-02-20 (08:30 MST), Rob McEwen  wrote:


RE: The "goo.gl
" shortner is OUT OF CONTROL (+ invaluement's response)





april fools day :=)

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-04-01 Thread @lbutlr

On 2018-02-20 (08:30 MST), Rob McEwen  wrote:
> 
> RE: The "goo.gl
> " shortner is OUT OF CONTROL (+ invaluement's response)




-- 
Technically, Aziraphale was a Principality, but people made jokes about
that these days

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-21 Thread David Jones


On 03/18/2018 06:59 PM, RW wrote:

On Sun, 18 Mar 2018 18:24:36 -0500
David Jones wrote:


On 03/18/2018 06:01 PM, Alan Hodgson wrote:

On Sun, 2018-03-18 at 17:14 -0500, David Jones wrote:

I have Steve Freegard's DecodeShortURLs.pm installed but didn't
get any HAS_SHORT_URL hits on this one:




Is it getting any hits? It definitely hits on that one in a test
here.

Note it needs Perl's LWP::UserAgent and DBD::SQLite to get it to
work at all.


I don't think you actually need DBD::SQLite, it just enables caching to
work.

   


I am getting other HAS_SHORT_URL hits.


Probably they are http links and you are missing  LWP::Protocol::https
which is needed to handle the https link in that email.






I was able to get LWP::Protocol::https installed on my mail filters and 
now I am getting HAS_SHORT_URL hits on https links.  Thanks for the tip!


--
David Jones

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-18 Thread RW

On Sun, 18 Mar 2018 18:24:36 -0500
David Jones wrote:

> On 03/18/2018 06:01 PM, Alan Hodgson wrote:
> > On Sun, 2018-03-18 at 17:14 -0500, David Jones wrote:  
> >> I have Steve Freegard's DecodeShortURLs.pm installed but didn't
> >> get any HAS_SHORT_URL hits on this one:


> > Is it getting any hits? It definitely hits on that one in a test
> > here.
> > 
> > Note it needs Perl's LWP::UserAgent and DBD::SQLite to get it to
> > work at all.

I don't think you actually need DBD::SQLite, it just enables caching to
work.

> >   
> 
> I am getting other HAS_SHORT_URL hits. 

Probably they are http links and you are missing  LWP::Protocol::https
which is needed to handle the https link in that email.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-18 Thread David Jones


On 03/18/2018 06:01 PM, Alan Hodgson wrote:

On Sun, 2018-03-18 at 17:14 -0500, David Jones wrote:

I have Steve Freegard's DecodeShortURLs.pm installed but didn't get any
HAS_SHORT_URL hits on this one:

https://pastebin.com/t85b0Bns 


Is it getting any hits? It definitely hits on that one in a test here.

Note it needs Perl's LWP::UserAgent and DBD::SQLite to get it to work at 
all.




I am getting other HAS_SHORT_URL hits.  My 8 SA boxes are setup 
identically with those 2 perl modules.  The original server that 
received this message does have other hits on HAS_SHORT_URL but this 
message still is not hitting.  Now it's scoring a 10.4 and would be 
blocked because of DCC_CHECK and other things have caught up to it.


--
David Jones

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-18 Thread Alan Hodgson

On Sun, 2018-03-18 at 17:14 -0500, David Jones wrote:
> 
I have Steve Freegard's DecodeShortURLs.pm installed but didn't get any 
> HAS_SHORT_URL hits on this one:
> 
> https://pastebin.com/t85b0Bns

Is it getting any hits? It definitely hits on that one in a test here.

Note it needs Perl's LWP::UserAgent and DBD::SQLite to get it to work
at all.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-18 Thread David Jones


On 03/10/2018 10:57 AM, Rob McEwen wrote:

On 3/10/2018 11:43 AM, Matus UHLAR - fantomas wrote:

On 3/10/2018 11:22 AM, Matus UHLAR - fantomas wrote:
this is apparently not the case of one url redirector (shortener) 
points to

another shortener.

I really hope that the DecodeShortURLs only checks fopr redirection 
at those

known redirectors (shorteners), not each http->https shortener and only
evaluates redirection between them, ignoring http->https redirects


On 10.03.18 11:32, Rob McEwen wrote:
But also keep in mind that it is NOT rare for the initial shortner 
found in a spam... to redirect to a spammer's page (that isn't a URL 
shortner), then THAT page then redirects to the spammer's OTHER page 
(that isn't a URL shortner)... where the FIRST one is NOT blacklisted 
very many places... but the second page (that is often the final 
destination)... *is* heavily blacklisted. Often, the first page is 
hosted at a legit site that was hacked into by criminal spammers - 
and is HARDER for blacklists to list due to their good reputation. 


isn't this thread and the plugin discussed about shorteners?
I am aware of other possibilities to do redirects and about 
consequencies of

following them.
note that for following the redirect, the destination must be 
configured via

url_shortener directive.
what I wonder is, if there's real value in following the redirects.
(and if the following stops when the URL is blacklisted, since we're done
here).



The URL shortner follower plugin that Steve Freegard wrote... ALREADY 
follows my suggestions about following non-shortner redirects (after the 
initial shortner redirect). (except I'm not sure it already does my 
https suggestion?) ...and for good reason. You're tremendously 
undervaluing or just not understanding my last post. I suggestion you 
re-read it a couple of times. This tactic I mentioned that spammers 
use... is COMMON, and not following my last suggestion opens up a big 
loophole for spammers. Such spammers would be delighted by your last 
comment... and Steve Freegard went a different route for good reason. 
Keep in mind, I haven't veered off-topic because we're STILL talking 
about a possible chain of redirects that was still STARTED by a URL 
shortner.




I have Steve Freegard's DecodeShortURLs.pm installed but didn't get any 
HAS_SHORT_URL hits on this one:


https://pastebin.com/t85b0Bns

I don't see much that can be done other than training Bayes which will 
probably not help the next variation of this junk.  I have reported to 
Spamcop and Microsoft -- they MUST do a better job of outbound mail 
filtering!


--
David Jones

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-15 Thread Rob McEwen


On 3/15/2018 11:13 AM, sha...@shanew.net wrote:

You might take a look at
https://developers.google.com/url-shortener/v1/getting_started

1 miion requests per day is the default limit.



Excellent! Thanks for the suggestion. This should help me MUCH!

But, unfortunately, it still leaves a lot to be desired for fixing the 
problems I described... when it comes to more distribution of automated 
"plugins" or having spam filters automatically doing this - in a "set it 
and forget it mode".


Still, there is a good argument that if someone has a high enough volume 
to trigger rate limiting... then they ought to have a large enough staff 
to handle going the extra mile and setting up API access systems, etc. 
But I wish it could be more simple. MANY are going to struggle with 
this... and NOT easily know all the details discussed on this thread!


But again, this should help me (and others) much... and it is good to 
know that there is a proper way to do this at a higher volume that meets 
Google's approval.


--
Rob McEwen
https://www.invaluement.com

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-15 Thread shanew


You might take a look at
https://developers.google.com/url-shortener/v1/getting_started

1 miion requests per day is the default limit.

On Wed, 14 Mar 2018, Rob McEwen wrote:


On 2/20/2018 9:42 PM, Rob McEwen wrote:
  Google might easily start putting captchas in the way or
  otherwise consider such lookups to be abusive and/or mistake
  them for malicious bots...

This prediction turned out to be 100% true. Even though others have
mentioned that they have been able to do high-volume lookups with no
problems... And granted I wasn't implementing a multi-server or multi-ip
lookup strategy... But I don't think I was doing nearly as many lookups as
others have claimed that they were able to do. I took a batch of 55,000
spams that I had collected from the past 4 weeks where those spams were
maliciously using the Google shortener as a way to get their spam delivered
via hiding their spammy domain names from spam filters. I started checking
those by looking up the redirect from Google's redirector, but without
actually visiting the site that the redirector was pointing to. Please note
that I was doing the lookups one-at-a-time, not starting the next lookup
until the last lookup had completed. After about ONLY 1,400 lookups, ALL of
my following lookups started hitting captchas. See attached screenshot.
Also, other than not sending from multiple IPs, I was otherwise doing
everything correct to make my script look/act like a regular browser.

I'll try spreading it out between multiple IPs in order to try to avoid rate
limits... However... This is still cause for concern about high-volume
lookups in high production systems... those may have to be implemented a
little more carefully if they're going to do these kind of lookups!

Just because small or medium production systems are able to do this... Or
just because somebody went out of their way to get more sophisticated with
it to get it to work out... doesn't mean that it's going to work in high
production systems that are trying to use "canned" software or plugins. This
is a particular challenge for anti-spam blacklists because they typically
process a very high volume of spams. Hopefully, the randomness of the ones I
process as they come in... will be sufficiently spread out enough to avoid
rate limiting?

It was my hope to start processing these live with my own DNSBL engine, so
that I could start blacklisting the domains that they redirect to... In
those cases where they were not already blacklisted... Now I'm going to have
to deal with constantly trying to make sure that I'm not hitting this
captcha, along with implementing some other strategies to hopefully prevent
that.

But this brings up a whole other issue... That is more of a policy or legal
issue... is Google basically making a statement that automated lookups are
not welcome? Or are considered abusive?

(btw, I could have collected order of magnitudes more than 55,000 of THESE
types of spams, but this was merely what was left over in an after-the-fact
search of my archives, after a lot of otherwise redundant spams had already
been purged from my system.)

PS - Once I gather this information, I will submit more details about the
results of this testing. But what is shocking right now is that less than
four tenths of 1% of these redirect URLs has been terminated, even though
they average two weeks old, with some almost a month old.




--
Public key #7BBC68D9 at| Shane Williams
http://pgp.mit.edu/|  System Admin - UT CompSci
=--+---
All syllogisms contain three lines |  sha...@shanew.net
Therefore this is not a syllogism  | www.ischool.utexas.edu/~shanew

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-14 Thread Rob McEwen


On 2/20/2018 9:42 PM, Rob McEwen wrote:
Google might easily start putting captchas in the way or otherwise 
consider such lookups to be abusive and/or mistake them for malicious 
bots...


This prediction turned out to be 100% true. Even though others have 
mentioned that they have been able to do high-volume lookups with no 
problems... And granted I wasn't implementing a multi-server or multi-ip 
lookup strategy... But I don't think I was doing nearly as many lookups 
as others have claimed that they were able to do. I took a batch of 
55,000 spams that I had collected from the past 4 weeks where those 
spams were maliciously using the Google shortener as a way to get their 
spam delivered via hiding their spammy domain names from spam filters. I 
started checking those by looking up the redirect from Google's 
redirector, but without actually visiting the site that the redirector 
was pointing to. Please note that I was doing the lookups one-at-a-time, 
not starting the next lookup until the last lookup had completed. After 
about ONLY 1,400 lookups, ALL of my following lookups started hitting 
captchas. See attached screenshot. Also, other than not sending from 
multiple IPs, I was otherwise doing everything correct to make my script 
look/act like a regular browser.


I'll try spreading it out between multiple IPs in order to try to avoid 
rate limits... However... This is still cause for concern about 
high-volume lookups in high production systems... those may have to be 
implemented a little more carefully if they're going to do these kind of 
lookups!


Just because small or medium production systems are able to do this... 
Or just because somebody went out of their way to get more sophisticated 
with it to get it to work out... doesn't mean that it's going to work in 
high production systems that are trying to use "canned" software or 
plugins. This is a particular challenge for anti-spam blacklists because 
they typically process a very high volume of spams. Hopefully, the 
randomness of the ones I process as they come in... will be sufficiently 
spread out enough to avoid rate limiting?


It was my hope to start processing these live with my own DNSBL engine, 
so that I could start blacklisting the domains that they redirect to... 
In those cases where they were not already blacklisted... Now I'm going 
to have to deal with constantly trying to make sure that I'm not hitting 
this captcha, along with implementing some other strategies to hopefully 
prevent that.


But this brings up a whole other issue... That is more of a policy or 
legal issue... is Google basically making a statement that automated 
lookups are not welcome? Or are considered abusive?


(btw, I could have collected order of magnitudes more than 55,000 of 
THESE types of spams, but this was merely what was left over in an 
after-the-fact search of my archives, after a lot of otherwise redundant 
spams had already been purged from my system.)


PS - Once I gather this information, I will submit more details about 
the results of this testing. But what is shocking right now is that less 
than four tenths of 1% of these redirect URLs has been terminated, even 
though they average two weeks old, with some almost a month old.


--
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-10 Thread Rob McEwen


On 3/10/2018 11:43 AM, Matus UHLAR - fantomas wrote:

On 3/10/2018 11:22 AM, Matus UHLAR - fantomas wrote:
this is apparently not the case of one url redirector (shortener) 
points to

another shortener.

I really hope that the DecodeShortURLs only checks fopr redirection 
at those

known redirectors (shorteners), not each http->https shortener and only
evaluates redirection between them, ignoring http->https redirects


On 10.03.18 11:32, Rob McEwen wrote:
But also keep in mind that it is NOT rare for the initial shortner 
found in a spam... to redirect to a spammer's page (that isn't a URL 
shortner), then THAT page then redirects to the spammer's OTHER page 
(that isn't a URL shortner)... where the FIRST one is NOT blacklisted 
very many places... but the second page (that is often the final 
destination)... *is* heavily blacklisted. Often, the first page is 
hosted at a legit site that was hacked into by criminal spammers - 
and is HARDER for blacklists to list due to their good reputation. 


isn't this thread and the plugin discussed about shorteners?
I am aware of other possibilities to do redirects and about 
consequencies of

following them.
note that for following the redirect, the destination must be 
configured via

url_shortener directive.
what I wonder is, if there's real value in following the redirects.
(and if the following stops when the URL is blacklisted, since we're done
here).



The URL shortner follower plugin that Steve Freegard wrote... ALREADY 
follows my suggestions about following non-shortner redirects (after the 
initial shortner redirect). (except I'm not sure it already does my 
https suggestion?) ...and for good reason. You're tremendously 
undervaluing or just not understanding my last post. I suggestion you 
re-read it a couple of times. This tactic I mentioned that spammers 
use... is COMMON, and not following my last suggestion opens up a big 
loophole for spammers. Such spammers would be delighted by your last 
comment... and Steve Freegard went a different route for good reason. 
Keep in mind, I haven't veered off-topic because we're STILL talking 
about a possible chain of redirects that was still STARTED by a URL 
shortner.


--
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-10 Thread Matus UHLAR - fantomas


On 3/10/2018 11:22 AM, Matus UHLAR - fantomas wrote:
this is apparently not the case of one url redirector (shortener) 
points to

another shortener.

I really hope that the DecodeShortURLs only checks fopr redirection 
at those

known redirectors (shorteners), not each http->https shortener and only
evaluates redirection between them, ignoring http->https redirects


On 10.03.18 11:32, Rob McEwen wrote:
But also keep in mind that it is NOT rare for the initial shortner 
found in a spam... to redirect to a spammer's page (that isn't a URL 
shortner), then THAT page then redirects to the spammer's OTHER page 
(that isn't a URL shortner)... where the FIRST one is NOT blacklisted 
very many places... but the second page (that is often the final 
destination)... *is* heavily blacklisted. Often, the first page is 
hosted at a legit site that was hacked into by criminal spammers - 
and is HARDER for blacklists to list due to their good reputation. 


isn't this thread and the plugin discussed about shorteners?
I am aware of other possibilities to do redirects and about consequencies of
following them.
note that for following the redirect, the destination must be configured via
url_shortener directive. 


what I wonder is, if there's real value in following the redirects.
(and if the following stops when the URL is blacklisted, since we're done
here).

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Posli tento mail 100 svojim znamim - nech vidia aky si idiot
Send this email to 100 your friends - let them see what an idiot you are

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-10 Thread Rob McEwen


On 3/10/2018 11:22 AM, Matus UHLAR - fantomas wrote:
this is apparently not the case of one url redirector (shortener) 
points to

another shortener.

I really hope that the DecodeShortURLs only checks fopr redirection at 
those

known redirectors (shorteners), not each http->https shortener and only
evaluates redirection between them, ignoring http->https redirects 



But also keep in mind that it is NOT rare for the initial shortner found 
in a spam... to redirect to a spammer's page (that isn't a URL 
shortner), then THAT page then redirects to the spammer's OTHER page 
(that isn't a URL shortner)... where the FIRST one is NOT blacklisted 
very many places... but the second page (that is often the final 
destination)... *is* heavily blacklisted. Often, the first page is 
hosted at a legit site that was hacked into by criminal spammers - and 
is HARDER for blacklists to list due to their good reputation. Then the 
2nd final destination page is just a heavily blacklisted spammer's 
throwaway domain. Therefore, there is some value to following the 
redirects a little further than what you suggest, and then collecting 
all of those host names or domains, checking ALL of them against 
URI/domain blacklists. (within reason... after too many redirects, it is 
better to just stop and add points to the spam score)


--
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-10 Thread Matus UHLAR - fantomas


On 3/10/2018 3:20 AM, Matus UHLAR - fantomas wrote:

do you have an example of any chained redirection not suspicious?


On 10.03.18 11:04, Rob McEwen wrote:
I haven't examined the code for that plugin very much (yet!) but 
one type of very common redirect that is very innocent... is the fact 
that a MASSIVE percentage of web sites have switched to all SSL sites 
in the past few years, due to Google now punishing http and/or 
rewarding https... in the search engine rankings. But there are still 
many links and redirectors pointing to the non-SSL versions of those 
sites, which is then typically redirected to the SSL version. 


this is apparently not the case of one url redirector (shortener) points to
another shortener.

I really hope that the DecodeShortURLs only checks fopr redirection at those
known redirectors (shorteners), not each http->https shortener and only
evaluates redirection between them, ignoring http->https redirects

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I don't have lysdexia. The Dog wouldn't allow that.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-10 Thread Rob McEwen


On 3/10/2018 3:20 AM, Matus UHLAR - fantomas wrote:
do you have an example of any chained redirection not suspicious? 



I haven't examined the code for that plugin very much (yet!) but one 
type of very common redirect that is very innocent... is the fact that a 
MASSIVE percentage of web sites have switched to all SSL sites in the 
past few years, due to Google now punishing http and/or rewarding 
https... in the search engine rankings. But there are still many links 
and redirectors pointing to the non-SSL versions of those sites, which 
is then typically redirected to the SSL version. Therefore, if the code 
for this plugin (and others using this tactic) doesn't do this 
already... it should probably not count THAT particular redirect as a 
spam indicator, when counting the total number of redirects.


--
Rob McEwen
https://www.invaluement.com

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-10 Thread Matus UHLAR - fantomas


On 07.03.18 10:59, sha...@shanew.net wrote:

Just FYI, it does add 3.0 points as soon as it sees any chaining at
all.  The other 5.0 points get added at 10 redirections.  That said,
I think you're guess is right that redirections start to look really
suspicious after just 3 or 4.


do you have an example of any chained redirection not suspicious?


On Sat, 3 Mar 2018, @lbutlr wrote:


On Feb 26, 2018, at 09:55, sha...@shanew.net wrote:


This is why the DecodeShortURLs plugin has an explicit limit of 10
lookups (and penalizes such with a total of 8 points).


I’d guess more than one redirect is highly suspicious and more than two is 
probably a waste of time, just score 5.0 and be done with it.

Has anyone done any analysis on multi-redirects?


--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
How does cat play with mouse? cat /dev/mouse

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-07 Thread shanew


Just FYI, it does add 3.0 points as soon as it sees any chaining at
all.  The other 5.0 points get added at 10 redirections.  That said,
I think you're guess is right that redirections start to look really
suspicious after just 3 or 4.


On Sat, 3 Mar 2018, @lbutlr wrote:


On Feb 26, 2018, at 09:55, sha...@shanew.net wrote:


This is why the DecodeShortURLs plugin has an explicit limit of 10
lookups (and penalizes such with a total of 8 points).


I’d guess more than one redirect is highly suspicious and more than two is 
probably a waste of time, just score 5.0 and be done with it.

Has anyone done any analysis on multi-redirects?




--
Public key #7BBC68D9 at| Shane Williams
http://pgp.mit.edu/|  System Admin - UT CompSci
=--+---
All syllogisms contain three lines |  sha...@shanew.net
Therefore this is not a syllogism  | www.ischool.utexas.edu/~shanew

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-03 Thread Benny Pedersen


John Hardin skrev den 2018-03-03 19:28:


This is why the DecodeShortURLs plugin has an explicit limit of 10
lookups (and penalizes such with a total of 8 points).
I’d guess more than one redirect is highly suspicious and more than 
two is probably a waste of time, just score 5.0 and be done with it.

+1


add blacklist internaly to DecodeShortURLs plugin, and reduce redirector 
list to who support abuse reports only


bit.ly is safe to test

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-03 Thread John Hardin


On Sat, 3 Mar 2018, @lbutlr wrote:


On Feb 26, 2018, at 09:55, sha...@shanew.net wrote:


This is why the DecodeShortURLs plugin has an explicit limit of 10
lookups (and penalizes such with a total of 8 points).


I’d guess more than one redirect is highly suspicious and more than two is 
probably a waste of time, just score 5.0 and be done with it.


+1

--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  Back in 1969 the technology to fake a Moon landing didn't exist,
  but the technology to actually land there did.
  Today, it is the opposite.   -- unknown
---
 10 days until Albert Einstein's 139th Birthday

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-03-03 Thread @lbutlr

On Feb 26, 2018, at 09:55, sha...@shanew.net wrote:
> 
> This is why the DecodeShortURLs plugin has an explicit limit of 10
> lookups (and penalizes such with a total of 8 points).

I’d guess more than one redirect is highly suspicious and more than two is 
probably a waste of time, just score 5.0 and be done with it. 

Has anyone done any analysis on multi-redirects?

-- 
This is my signature. There are many like it, but this one is mine.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-27 Thread Leandro

2018-02-27 9:03 GMT-03:00 Rob McEwen :

> On 2/26/2018 1:00 PM, Kevin A. McGrail wrote:
>
> DecodeShortURLs has been on my list of must-have plugins for years, so
> I was a little surprised it took so long for someone to mention it
> in this thread.
>
> Yeah, my firm is going to look at subsidizing it's addition to SA and
> Karsten agreed.  Just trying to get the time.
>
>
> Kevin,
>
> Excellent! Also... for those situations where there are multiple cascading
> redirecting websites (after the initial URL shortners link found in the
> message), please consider having an option for *every* non-whitelisted URI
> in the chain to be added to the checks against blacklists. OFTEN - every
> single domain in that chain (past the initial URL shortner) is a
> compromised web site or spammer's website, not just the final destination
> web site.
>

Also, you can put more than one shortener (not only hacked websites) inside
same chain:

http://bit.ly/2GLA8Ss => https://goo.gl/5fnqaC => http://spfbl.net/en/uribl/

Our script can already solve this chain:

root@mx-br:/home/ubuntu# ./uribl.pl "http://bit.ly/2GLA8Ss;
(spfbl.net) is not listed in 'uribl.spfbl.net'.

Kevin. You can use our script as model to your own implementation.


> --
> Rob McEwenhttps://www.invaluement.com
>
>
>

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-27 Thread Kevin A. McGrail

I will add it to the list, thanks. 
Regards,
KAM

On February 27, 2018 7:03:16 AM EST, Rob McEwen  wrote:
>On 2/26/2018 1:00 PM, Kevin A. McGrail wrote:
>>> DecodeShortURLs has been on my list of must-have plugins for years,
>so
>>> I was a little surprised it took so long for someone to mention it
>>> in this thread. 
>>
>> Yeah, my firm is going to look at subsidizing it's addition to SA and
>
>> Karsten agreed.  Just trying to get the time.
>>
>
>Kevin,
>
>Excellent! Also... for those situations where there are multiple 
>cascading redirecting websites (after the initial URL shortners link 
>found in the message), please consider having an option for *every* 
>non-whitelisted URI in the chain to be added to the checks against 
>blacklists. OFTEN - every single domain in that chain (past the initial
>
>URL shortner) is a compromised web site or spammer's website, not just 
>the final destination web site.
>
>-- 
>Rob McEwen
>https://www.invaluement.com
>

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-27 Thread Rob McEwen


On 2/26/2018 1:00 PM, Kevin A. McGrail wrote:

DecodeShortURLs has been on my list of must-have plugins for years, so
I was a little surprised it took so long for someone to mention it
in this thread. 


Yeah, my firm is going to look at subsidizing it's addition to SA and 
Karsten agreed.  Just trying to get the time.




Kevin,

Excellent! Also... for those situations where there are multiple 
cascading redirecting websites (after the initial URL shortners link 
found in the message), please consider having an option for *every* 
non-whitelisted URI in the chain to be added to the checks against 
blacklists. OFTEN - every single domain in that chain (past the initial 
URL shortner) is a compromised web site or spammer's website, not just 
the final destination web site.


--
Rob McEwen
https://www.invaluement.com

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-26 Thread Amir Caspi

> On Feb 26, 2018, at 11:00 AM, Kevin A. McGrail  
> wrote:
> 
>> DecodeShortURLs has been on my list of must-have plugins for years, so 
>> I was a little surprised it took so long for someone to mention it 
>> in this thread.
> Yeah, my firm is going to look at subsidizing it's addition to SA and Karsten 
> agreed.  Just trying to get the time.
> 
FWIW, I just implemented this plugin last week after this email thread, and it 
was trivial to include on my (old, very EOL) system.  Hopefully including it in 
the default SA distribution is not too difficult.

(I also finally got DKIM validation working on my virtual hosting system, 
including with the MailScanner setup. Finally, auth rules are working as they 
should!)

Thanks!

--- Amir

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-26 Thread Kevin A. McGrail


On 2/26/2018 12:55 PM, sha...@shanew.net wrote:

On Mon, 26 Feb 2018, David B Funk wrote:

Just be careful how you do that "expand redirections until no more 
redirections" or you may get caught in a spammer trap.


This is why the DecodeShortURLs plugin has an explicit limit of 10
lookups (and penalizes such with a total of 8 points).

DecodeShortURLs has been on my list of must-have plugins for years, so
I was a little surprised it took so long for someone to mention it
in this thread. 


Yeah, my firm is going to look at subsidizing it's addition to SA and 
Karsten agreed.  Just trying to get the time.


Regards,

KAM

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-26 Thread shanew


On Mon, 26 Feb 2018, David B Funk wrote:

Just be careful how you do that "expand redirections until no more 
redirections" or you may get caught in a spammer trap.


This is why the DecodeShortURLs plugin has an explicit limit of 10
lookups (and penalizes such with a total of 8 points).

DecodeShortURLs has been on my list of must-have plugins for years, so
I was a little surprised it took so long for someone to mention it
in this thread.


--
Public key #7BBC68D9 at| Shane Williams
http://pgp.mit.edu/|  System Admin - UT CompSci
=--+---
All syllogisms contain three lines |  sha...@shanew.net
Therefore this is not a syllogism  | www.ischool.utexas.edu/~shanew

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-26 Thread Leandro

2018-02-26 10:41 GMT-03:00 Dianne Skoll :

> On Mon, 26 Feb 2018 00:07:54 -0600 (CST)
> David B Funk  wrote:
>
> > So my bet is that the spammers are crafty enough to check things like
> > browser referrer, cookies, etc to detect/differentiate a browser vs a
> > link-checker.
>
> Yep.  You need to fake your User-Agent (not hard) and put a limit on
> the number of redirections you follow or you can end up in trouble.
>
> I picked a likely-looking agent out of our logs and use this:
>
> $ua->agent('Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64;
> Trident/7.0; SLCC2;');
>
> Might need revising later because it's pretty ancient!
>


That is right Dianne!

Our script is using "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0)
Gecko/20100101 Firefox/57.0":

https://www.dropbox.com/s/5aorrijafw5ygk0/uribl.pl?dl=0

This user agent was most used, some time ago, according to this site:

https://techblog.willshouse.com/2012/01/03/most-common-user-agents/

This is out of date now, than you can update it if you want. We will do it
soon.



>
> Regards,
>
> Dianne.
>

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-26 Thread Dianne Skoll

On Mon, 26 Feb 2018 00:07:54 -0600 (CST)
David B Funk  wrote:

> So my bet is that the spammers are crafty enough to check things like
> browser referrer, cookies, etc to detect/differentiate a browser vs a
> link-checker.

Yep.  You need to fake your User-Agent (not hard) and put a limit on
the number of redirections you follow or you can end up in trouble.

I picked a likely-looking agent out of our logs and use this:

$ua->agent('Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; 
Trident/7.0; SLCC2;');

Might need revising later because it's pretty ancient!

Regards,

Dianne.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-26 Thread Matthias Leisi


> Hi Guys! We provide an URIBL that already have a script in Perl to expand
> redirections until no more redirections:

I would be uneasy to follow such redirections on a production email system (as 
opposed to eg a spamtrap system). You are likely „confirming“ live email 
addresses to the spammer by blindly following links and redirects, which may be 
tied to individual email addresses.

— Matthias

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-26 Thread Leandro

2018-02-26 3:07 GMT-03:00 David B Funk :

>
> Just be careful how you do that "expand redirections until no more
> redirections" or you may get caught in a spammer trap.
>
> If you're going thru a professional redirect site like goo.gl or bit.ly
> you're probably pretty safe but if it's a dedicated spammer site be ware.
>
> I was testing some redirection expantions on URLs from spam and found a
> site that clearly had been crafted to foil this kind of thing.
>
> It was in one of those "check this out" spams which contains one line of
> greeting and then a URL.
>
> When I grabbed it using curl it returned a 301 redirect, so I grabbed that
> target, which lead to another 301, lather-rinse-repeat ad nausium.
> However if you used a browser it went to the target "burn fat pills" site
> in just two redirects.
>
> So my bet is that the spammers are crafty enough to check things like
> browser referrer, cookies, etc to detect/differentiate a browser vs a
> link-checker.
>
>
Great! If the spammer uses same redirection URLs, as a loop, the script
will ignore redirection on first recurrence.

But this script is a beta and it is not is not mature enough to deal with
loop redirects of distinct URLs. Maybe add a count limit resolves this issue,
like 10 times max redirections.

Most likely I will implement this, but my need is for someone to help me
implement this script inside the SA, so that some users here in Brazil can
use our URIBL easily.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-25 Thread David B Funk


On Sun, 25 Feb 2018, LeandroCarlosRodrigues wrote:


Amir Caspi wrote

On that note -- regardless of what OTHER HW/SW solutions might do, since
this is a SpamAssassin mailing list ... is there any facility to implement
this in SA?  That is, when calling the URIBL plugin, could it check both
the shortened URL and the expanded URL (for known shorteners) ?  Does that
facility already exist and I missed it?


Hi Guys! We provide an URIBL that already have a script in Perl to expand
redirections until no more redirections:

[snip..]

Just be careful how you do that "expand redirections until no more redirections" 
or you may get caught in a spammer trap.


If you're going thru a professional redirect site like goo.gl or bit.ly you're 
probably pretty safe but if it's a dedicated spammer site be ware.


I was testing some redirection expantions on URLs from spam and found a site 
that clearly had been crafted to foil this kind of thing.


It was in one of those "check this out" spams which contains one line of 
greeting and then a URL.


When I grabbed it using curl it returned a 301 redirect, so I grabbed that 
target, which lead to another 301, lather-rinse-repeat ad nausium.
However if you used a browser it went to the target "burn fat pills" site in 
just two redirects.


So my bet is that the spammers are crafty enough to check things like browser 
referrer, cookies, etc to detect/differentiate a browser vs a link-checker.



--
Dave Funk  University of Iowa
College of Engineering
319/335-5751   FAX: 319/384-0549   1256 Seamans Center
Sys_admin/Postmaster/cell_adminIowa City, IA 52242-1527
#include 
Better is not better, 'standard' is better. B{

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-25 Thread RW

On Sun, 25 Feb 2018 05:25:06 -0700 (MST)
LeandroCarlosRodrigues wrote:

> Amir Caspi wrote
> > On that note -- regardless of what OTHER HW/SW solutions might do,
> > since this is a SpamAssassin mailing list ... is there any facility
> > to implement this in SA?  That is, when calling the URIBL plugin,
> > could it check both the shortened URL and the expanded URL (for
> > known shorteners) ?  Does that facility already exist and I missed
> > it?  
> 
> Hi Guys! We provide an URIBL that already have a script in Perl to
> expand redirections until no more redirections:
> 

There's already a plugin to do that.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-25 Thread LeandroCarlosRodrigues

Amir Caspi wrote
> On that note -- regardless of what OTHER HW/SW solutions might do, since
> this is a SpamAssassin mailing list ... is there any facility to implement
> this in SA?  That is, when calling the URIBL plugin, could it check both
> the shortened URL and the expanded URL (for known shorteners) ?  Does that
> facility already exist and I missed it?

Hi Guys! We provide an URIBL that already have a script in Perl to expand
redirections until no more redirections:

https://www.dropbox.com/s/5aorrijafw5ygk0/uribl.pl?dl=0
  

Our URIBL ignores the intermediary URLs. The target is the final URL,
that will be viewed by user:

root@mx-br:/home/ubuntu# ./uribl.pl "http://goo.gl/ylUAd;
(www.hollywoodreporter.com) is not listed in 'uribl.spfbl.net'.

More about consumption of our URIBL can be read here:

http://spfbl.net/en/uribl   

Could you guys help me adjust this script to run inside SA?



--
Sent from: http://spamassassin.1065346.n5.nabble.com/SpamAssassin-Users-f3.html

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-21 Thread @lbutlr

On 2018-02-21 (09:27 MST), Alex  wrote:
> 
> This is what DecodeShortURLs is for
> https://github.com/smfreegard/DecodeShortURLs

Aha! I knew something like that must exist! 

-- 
EIR OWN DESTINY. THEY TOUCH THE EARTH LIGHTLY.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-21 Thread Amir Caspi

> On Feb 21, 2018, at 12:45 PM, Dianne Skoll  wrote:
> 
> Someone earlier posted a link to https://github.com/smfreegard/DecodeShortURLs

Oops, I missed that... must have thought it was just about decoding and not 
about SA.  Thanks for clarifying!

--- Amir

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)


On 2/21/2018 2:41 PM, Amir Caspi wrote:

On Feb 21, 2018, at 9:57 AM, Dianne Skoll  wrote:

That's why you only want to do it for URLs that are
absolutely known to be shortened URLs.  You have to keep a list of
known URL-shorteners.

On that note -- regardless of what OTHER HW/SW solutions might do, since this 
is a SpamAssassin mailing list ... is there any facility to implement this in 
SA?  That is, when calling the URIBL plugin, could it check both the shortened 
URL and the expanded URL (for known shorteners) ?  Does that facility already 
exist and I missed it?


If I'm not confusing things, someone answered things earlier in this 
thread, as follows:


On 2/21/2018 11:27 AM, Alex wrote:

This is what DecodeShortURLs is for
https://github.com/smfreegard/DecodeShortURLs



--
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-21 Thread Amir Caspi

> On Feb 21, 2018, at 9:57 AM, Dianne Skoll  wrote:
> 
> That's why you only want to do it for URLs that are
> absolutely known to be shortened URLs.  You have to keep a list of
> known URL-shorteners.

On that note -- regardless of what OTHER HW/SW solutions might do, since this 
is a SpamAssassin mailing list ... is there any facility to implement this in 
SA?  That is, when calling the URIBL plugin, could it check both the shortened 
URL and the expanded URL (for known shorteners) ?  Does that facility already 
exist and I missed it?

It seems like this wouldn't be too horrible to implement within the existing 
URIBL plugin, unless I'm missing something obvious.

Thanks!

--- Amir

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

On Wed, 21 Feb 2018 12:41:05 -0700
Amir Caspi  wrote:

> On that note -- regardless of what OTHER HW/SW solutions might do,
> since this is a SpamAssassin mailing list ... is there any facility
> to implement this in SA?

Someone earlier posted a link to https://github.com/smfreegard/DecodeShortURLs

Regards,

Dianne.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)


On 2/21/2018 11:48 AM, Anthony Cartmell wrote:

Meanwhile - adding URI lookups (for URIs in the body of the domains)
and/or the option to add 3rd party URI list lookups - is STILL is
missing from MANY widely used anti-spam systems.

If you mean following URLs in messages, you do need to be aware that
this can break one-time login links.

I have come across one local authority in the UK whose spam filter
requests URLs in incoming messages. This meant that the spam filter had
already used the one-time login link so that when the intended user
tried they got an error saying that the link had already been used. Took
a while to work out what was going wrong!



Sorry I wasn't more clear. I was just referring to standard things like 
usage of SURBL, URIBL, ivmURI, and Spamhaus's DBL list - where the 
domains and IPs that are in the body of the message are collected and 
checked against those lists - I didn't mean to imply that such links are 
actually followed. Again, sorry I wasn't more clear. I was just pointing 
out that if so many systems STILL don't do THAT - or provide an 
opportunity for adding additional such lists - then I doubt that we're 
going to see widespread adoption by mail systems of a process where, in 
real time spam filtering, they check to see where URL shortners lead to, 
and then factor that destination into the spam filtering.


--
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

On Wed, 21 Feb 2018 16:48:40 +
Anthony Cartmell  wrote:

> If you mean following URLs in messages, you do need to be aware that
> this can break one-time login links.

Big time.  That's why you only want to do it for URLs that are
absolutely known to be shortened URLs.  You have to keep a list of
known URL-shorteners.  It's hard/impossible to get a complete list,
but pretty easy to get a list of the big ones that cover 90% of all
shortened URLs you're likely to see.

Regards,

Dianne.

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))


On 2/21/2018 11:44 AM, Dianne Skoll wrote:

On Wed, 21 Feb 2018 16:35:27 +
Karol Augustin  wrote:

I think the point here might be that if Google acted promptly on abuse
spammers would stop using shorteners.

True, that might happen.  OTOH, I see about as many spams with bit.ly
shorteners as goo.gl shorteners which is not what one might expect if
bit.ly were really that much more proactive than goo.gl.


I'm sure mileage may vary - but I'm seeing about 10X the abuse for 
goo.gl right now as I see from bit.ly. Also, when I do random checks on 
a handful of abused bitly shortners, a high percentage of them are 
already terminated. But when I do random checks of abused goo.gl 
redirectors, most of them are still operational. (I'm referring to 
redirectors found in spams within the previous few days of when I 
checked them, with at least hours having gone by since the message was 
sent - I know that sounds anecdotal - but as I've been researching this 
in the past weeks, that pattern keeps happening). One thing that could 
potentially make those numbers different - is when you compare one 
system that blocks MUCH spam at the perimeter based on the sending IP, 
such as blocking all Zen-listed spams before DATA while another 
system might capture ALL messages and process them all. The latter is 
what my system does. That also might explain the difference in stats?


--
Rob McEwen
https://www.invaluement.com

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-21 Thread Anthony Cartmell

> Meanwhile - adding URI lookups (for URIs in the body of the domains)
> and/or the option to add 3rd party URI list lookups - is STILL is
> missing from MANY widely used anti-spam systems.

If you mean following URLs in messages, you do need to be aware that
this can break one-time login links.

I have come across one local authority in the UK whose spam filter
requests URLs in incoming messages. This meant that the spam filter had
already used the one-time login link so that when the intended user
tried they got an error saying that the link had already been used. Took
a while to work out what was going wrong!

Anthony
-- 
www.fonant.com - Quality web sites
Tel. 01903 867 810
Fonant Ltd is registered in England and Wales, company No. 7006596
Registered office: Amelia House, Crescent Road, Worthing, West Sussex,
BN11 1QR

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))

On Wed, 21 Feb 2018 16:35:27 +
Karol Augustin  wrote:

> I think the point here might be that if Google acted promptly on abuse
> spammers would stop using shorteners.

True, that might happen.  OTOH, I see about as many spams with bit.ly
shorteners as goo.gl shorteners which is not what one might expect if
bit.ly were really that much more proactive than goo.gl.

Regards,

Dianne.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)


On 2/21/2018 11:12 AM, Dianne Skoll wrote:

Really?  This isn't rocket science.  If I thought of it, I'm sure dozens
if not hundreds of others have thought of it and implemented it.


Meanwhile - adding URI lookups (for URIs in the body of the domains) 
and/or the option to add 3rd party URI list lookups - is STILL is 
missing from MANY widely used anti-spam systems. That isn't rocket 
science either - and is MUCH more needed and beneficial than this 
feature. That some others have made adjustments to their system to 
follow and act on redirectors during live spam filtering - again - 
doesn't alter my original point. The vast majority of anti-spam systems 
in the real world (1) don't (2) and won't any time soon. That is what I 
claimed. Please stop nitpicking and please stop arguing with a "straw man".


--
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))

2018-02-21 Thread Karol Augustin

On 2018-02-21 16:31, Dianne Skoll wrote:
> On Wed, 21 Feb 2018 11:29:00 -0500
> Rob McEwen  wrote:
> 
>> Nevertheless, it is a shame to have to shift more of the burden onto
>> spam filters to do more work (some of which requires MORE latency) -
>> in order to partly mitigate Google's failure to prevent/correct the
>> abuse.
> 
> Yes, I agree.  On the other hand, IMO a spam filter should block messages
> that point to a taken-down shortened URL.  Although such messages may
> not be downright dangerous, they are still annoying and are still spam.
> There's no way of avoiding the work.
> 

I think the point here might be that if Google acted promptly on abuse
spammers would stop using shorteners.


Karol

--  
Karol Augustin
ka...@augustin.pl
http://karolaugustin.pl/
+353 85 775 5312

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))

On Wed, 21 Feb 2018 11:29:00 -0500
Rob McEwen  wrote:

> Nevertheless, it is a shame to have to shift more of the burden onto 
> spam filters to do more work (some of which requires MORE latency) -
> in order to partly mitigate Google's failure to prevent/correct the
> abuse.

Yes, I agree.  On the other hand, IMO a spam filter should block messages
that point to a taken-down shortened URL.  Although such messages may
not be downright dangerous, they are still annoying and are still spam.
There's no way of avoiding the work.

Regards,

Dianne.

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))


On 2/21/2018 11:11 AM, Dianne Skoll wrote:

I guess I misinterpreted: "...such automated lookups could also put a
huge extra burden on Google's servers..." from Message-Id



Oh yeah, I'd forgotten about that part. it was a more minor point. But 
as I think back on my thought processes at the time I typed those words 
- I was envisioning what would happen if ALL ISPs and hosters and spam 
filtering vendors, SA installations etc...  ALL started doing all of 
those lookups. But yeah, maybe that would still be a drop in the bucket 
compared to all that Google does.


Nevertheless, it is a shame to have to shift more of the burden onto 
spam filters to do more work (some of which requires MORE latency) - in 
order to partly mitigate Google's failure to prevent/correct the abuse. 
In contrast, at this time, bit.ly is running circles around Google in 
terms of quickly shutting down their abused redirectors. I know this 
isn't easy, but there is definitely room for improvement.


But my larger point in that overall post you quoted from, was my concern 
about one organization doing high volume lookups from a single server 
getting blocked or captcha'd.


--
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-21 Thread Alex

On Wed, Feb 21, 2018 at 1:38 AM, @lbutlr  wrote:
> On 2018-02-20 (22:10 MST), Reindl Harald  wrote:
>>
>> you may hit confirmation-urls (both ham and spam), trigger actions, trigger 
>> *one-time* urls which are invalid for the user after a dumb bot used them 
>> not talking about that it would be illegal in many countries in case of 
>> private ham-mails
>
> As I suspected, it is possible to get the goo.gl target URL without loading 
> the site, though using curl is probably not realistic in this specific case.
>
> $ curl -s "http://goo.gl/ylUAd; | grep -o "http[^\"]*"
> http://www.hollywoodreporter.com/thr-esq/donald-trump-threatens-sue-macy-422135
>
> $ curl -s "http://bit.ly/savecastle; | grep -o "http[^\"]*"
> http://community.livejournal.com/castle_tv/28872.html
>
> Doesn't work with t.co, but that is not surprising since twitter uses that 
> specifically to hide URLs, considering them all their property that must go 
> through their servers.

This is what DecodeShortURLs is for
https://github.com/smfreegard/DecodeShortURLs

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

On Wed, 21 Feb 2018 11:00:48 -0500
Rob McEwen  wrote:

> > [Expanding shorteners] been part of our practice for about a year now.

> Excellent! I wish others would be as innovative and on top of things
> as you are! Unfortunately, your statement doesn't alter my point you
> were replying to, even one tiny bit.

Really?  This isn't rocket science.  If I thought of it, I'm sure dozens
if not hundreds of others have thought of it and implemented it.

Regards,

Dianne.

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))

On Wed, 21 Feb 2018 10:58:17 -0500
Rob McEwen  wrote:

> On 2/21/2018 10:37 AM, Dianne Skoll wrote:
> > The concern voiced in another email about overloading Google's
> > infrastructure is quite charming and quaint.  

> My concern was NEVER about overloading google.

I guess I misinterpreted: "...such automated lookups could also put a
huge extra burden on Google's servers..." from Message-Id

> My concern was about Google auto-blocking or throwing a captcha at
> very high volume and automated lookups. That is a HUGE difference.

That's a concern, but you can mitigate it slightly by caching the
results of URL expansion.  Also, I suspect most URL-shorteners
expect to be bombarded with requests and are engineered accordingly.
I have seen a CAPTCHA thrown up from the search page, but I've never
seen a goo.gl HEAD request reply with anything other than 404 or 301.

Regards,

Dianne.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)


On 2/21/2018 10:39 AM, Dianne Skoll wrote:

We use HEAD requests to expand known URL-shorteners on a cluster that
peaks around 60 msgs/s


Thanks for that information. That is good to know!


(b) and this isn't going to suddenly become a feature inside of many
types of spam filtering hardware and software overnight... that could
even take years,

It's been part of our practice for about a year now.


Excellent! I wish others would be as innovative and on top of things as 
you are! Unfortunately, your statement doesn't alter my point you were 
replying to, even one tiny bit.


--
Rob McEwen
https://www.invaluement.com

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))


On 2/21/2018 10:37 AM, Dianne Skoll wrote:

The concern voiced in another email about overloading Google's
infrastructure is quite charming and quaint.



My concern was NEVER about overloading google. My concern was about 
Google auto-blocking or throwing a captcha at very high volume and 
automated lookups. That is a HUGE difference.


--
Rob McEwen
https://www.invaluement.com

Re: Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))

2018-02-21 Thread Benny Pedersen


Dianne Skoll skrev den 2018-02-21 16:37:


We do a HEAD request and it works on most URL shorteners.

The concern voiced in another email about overloading Google's
infrastructure is quite charming and quaint.


+1

some with icla could add this to spamasssassin with

https://github.com/smfreegard/DecodeShortURLs

so if yes problem is solved imho, but should we consider all _URIHOSTS_ 
as shortners ?


unless thay are confirmed not to do this

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

On Wed, 21 Feb 2018 02:30:40 -0500
Rob McEwen  wrote:

> (a) it might not "scale" for high volume mail flows and DNSBLs who,
> like invaluement, process dozens (or more) spams per second.

We use HEAD requests to expand known URL-shorteners on a cluster that
peaks around 60 msgs/s

> (b) and this isn't going to suddenly become a feature inside of many 
> types of spam filtering hardware and software overnight... that could 
> even take years,

It's been part of our practice for about a year now.

Regards,

Dianne.

Expanding shortened URLs (was Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response))

On Tue, 20 Feb 2018 23:38:53 -0700
"@lbutlr"  wrote:

> As I suspected, it is possible to get the goo.gl target URL without
> loading the site, though using curl is probably not realistic in this
> specific case.

We do a HEAD request and it works on most URL shorteners.

The concern voiced in another email about overloading Google's
infrastructure is quite charming and quaint.

Regards,

Dianne.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-21 Thread @lbutlr

On 2018-02-21 (00:52 MST), Charles Sprickman  wrote:
> 
> You can also see all the analytics by appending “.info” to the URL, eg: 
> http://goo.gl/ylUAd.info

True, but that is a web browser solution, not something that could, for 
example, be scripted (well, not easily or realistically for this sort of task).

Also, appending a "+" I believe does the same thing, and is easier to type. 
http://goo.gl/ylUAd+

Either way, that redirects to a URL in the form

So that could probably be scraped as well if you really wanted to scrape a web 
page.

Still, this isn't a viable solution for spam checking, though I suspect there 
is a low-cost "get redirect URI" library that would have less overhead than 
curl and could grab the Location from the 301 (or 302?) redirect, much like 
curl can do with a -I or -i flags.

Regardless, Reindl was clearly wrong that checking the shortner targets would 
require loading the target URL.

Maybe curl ain't so bad?

# time curl -sI http://goo.gl/ylUAd | grep -o "http.*"
http://www.hollywoodreporter.com/thr-esq/donald-trump-threatens-sue-macy-422135

real0m0.098s
user0m0.000s
sys 0m0.008s

I threw this together in a few minutes and it seems to work:

---
#!/bin/bash

  if [ "$1" = "" ]; then
   echo "usage $0 "
   exit 1
  else 
   URL="$1"
  fi

myInfo=$(curl -sI $URL | grep -o "http.*")
if [ "$myInfo" = "" ]; then
   exit 404
else
   echo $myInfo
fi
---

 # curlri www.covisp.net # This uses rewrite
https://www.covisp.net/
 # curlri www.google.com # No redirect, so returned 404
 # curlri http://goo.gl/ylUAd # Voila!
http://www.hollywoodreporter.com/thr-esq/donald-trump-threatens-sue-macy-422135

It even works for the t.co link:

 # curlri http://t.co/UShCXQG
http://www.youtube.com/watch?v=dfJ6oCdjY9Y

-- 
It is one thing to be mistaken; it is quite another to be willfully
ignorant ~Cecil Adams

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-20 Thread Charles Sprickman


> On Feb 21, 2018, at 1:38 AM, @lbutlr  wrote:
> 
> On 2018-02-20 (22:10 MST), Reindl Harald  wrote:
>> 
>> you may hit confirmation-urls (both ham and spam), trigger actions, trigger 
>> *one-time* urls which are invalid for the user after a dumb bot used them 
>> not talking about that it would be illegal in many countries in case of 
>> private ham-mails
> 
> As I suspected, it is possible to get the goo.gl target URL without loading 
> the site, though using curl is probably not realistic in this specific case.
> 
> $ curl -s "http://goo.gl/ylUAd; | grep -o "http[^\"]*"
> http://www.hollywoodreporter.com/thr-esq/donald-trump-threatens-sue-macy-422135
>  
> 

You can also see all the analytics by appending “.info” to the URL, eg: 
http://goo.gl/ylUAd .info

Charles

> 
> $ curl -s "http://bit.ly/savecastle; | grep -o "http[^\"]*"
> http://community.livejournal.com/castle_tv/28872.html
> 
> Doesn't work with t.co, but that is not surprising since twitter uses that 
> specifically to hide URLs, considering them all their property that must go 
> through their servers.
> 
> -- 
> Mos Eisley spaceport. You will never find a more wretched hive of scum
> and villainy. We must be cautious.
>

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)


On 2/21/2018 1:38 AM, @lbutlr wrote:

As I suspected, it is possible to get the goo.gl target URL without loading the 
site, though using curl is probably not realistic in this specific case.



That is an idea worth exploring! Some might greatly benefit from that.

However:

(a) it might not "scale" for high volume mail flows and DNSBLs who, like 
invaluement, process dozens (or more) spams per second.


(b) and this isn't going to suddenly become a feature inside of many 
types of spam filtering hardware and software overnight... that could 
even take years, if it could ever even gain traction? It is taking into 
the decades just to get some of those software and hardware vendors to 
add support for URI blacklists, or support to for adding custom 3rd 
party URI blacklists. If that is taking literally decades - they are not 
going to add this feature within the foreseeable future.


So please don't think for a second that this somehow makes the plans I 
had described as unnecessary.


--
Rob McEwen
https://www.invaluement.com

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)


On 2/21/2018 1:17 AM, @lbutlr wrote:

goo.gl (and other shorteners) are used for far more than email.
That said, most my incoming email is rejected long before it get to 
any sort of URI lookups based on just the transaction information, 
That is to say, upwards of 90% of incoming mail is rejected before DATA. 


That is besides the point. I'm discussing the more elusive spams that 
often slip past spam filters and are difficult to block. I'm not very 
concerned with the "low hanging fruit" that you're describing - that is 
ANOTHER topic!



286 total spams blocked that had a shortner,

That's not enough to have any sort of reliable statistical data.


This is what was pulled from a corpus of 5K spams. Are you sure you 
didn't get confused and think I said that my total corpus I searched 
only involved 286 spams to start with?


Also, yes, different data batches from different sources can have 
different idiosyncrasies - but to discount this as not reliable due to 
being too small of sample size - is laughable. The server has a good 
variety of dozens of different domains and thousand of different users. 
These are actual "brick and mortar" businesses in the US with real 
people: Schools, law firms, real estate companies, manufacturing 
companies, service-oriented companies - so there is a SUBSTANTIAL 
variety. At the same time, I've had a distinct uptick in complaints 
about spam where the goo.gl shortner was a large factor in recent 
unblocked spams complaints - where those complaints are normally few and 
far between. With changes I've made in the past few days, the number of 
such spams slipping past my filter was been sharply reduced. Also, over 
the years, I've noticed that patterns I seen on my server, like this 
one, often match patterns that others report. It defies Occam's Razor to 
suggestion that I have this magical concentration of egregious goo.gl 
shortner spams that is magically hitting these very diverse and 
unrelated companies for whom I host mail, and not hitting elsewhere. 
That is how statistics tend to work. The next time you read about a poll 
that a newpaper or news show conducted, you may be shocked to learn that 
this poll that gave a nationwide estimate with a small margin of error - 
often only had about a thousand participants. My study isn't as 
scientific because my users aren't perfectly random. But it is diverse 
enough to simulate a sufficient amount of randomness so as to be an 
extremely good indicator. Likewise, whenever I hear about a new spam 
trend that is effecting many people, I pretty much never come up empty 
when searching for examples of that hitting my servers.



187 total legit messages had a hit on at least one of hundreds of URL shortners

So the use of a shortner is a poor spam indicator. Even in your corpus, and a 
negligible indicator even when specifically looking at goo.gl


You have an amazing talent for missing the point. Most often, when used 
in legit mail... it was more of an afterthought - such as someone 
sending a link to a Google maps reference - but yet where the sender was 
NOT using the shortner for obfuscation of spammer's domains, or even for 
their mail hyperlinks. But THAT is what happened when these were used in 
spams, where it was used to HIDE identity by obfuscating their main 
links. So there is a strong correlation right there. But you want an 
"all or none" strawman to argue against, while you conveniently ignore 
many points I've made. Yes - it is frustrating that it is NOT extremely 
easy to differentiate the ones who use it for obfuscation from the ones 
have more appropriate usages. But there are ways. This is why I had to 
spend some hours making adjustments to my system - not some minutes. You 
make a good point that it is unwise to outright block ALL messages due 
to them having a shortner, or having the goo.gl shortner - but I'm light 
years more sophisticated than that - and I haven't argued that even one 
time in this thread.


Another problem with your statement is that if this loophole isn't 
addressed it will only get larger. Also, many certain of these spams are 
already very difficult to block - but would be very easy to block if the 
spammer wasn't hiding behind the shortner. That is a real problem that 
STILL exists regardless of how often shortners are used in legit mail - 
and regardless of how often they are used in spams - but you don't want 
to face that truth, it seems?


The BOTTOM LINE is that anything that (a) helps blacklist those senders 
who are doing this obfuscation in order to facilitate sending spam -AND- 
(b) motivate marketers and ESPs (& their clients) to avoid this tactic - 
is a step in the right direction.



Google's shortner is DOMINATING in its spam usage, where 92% (262 of 286) of 
ALL spam that contained shortners used Google.

But about 25% of goo.gl containing email is not spam, by your own numbers. So, 
a very poor metric.



A poor metric if only idiots ran spam filters and

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-20 Thread @lbutlr

On 2018-02-20 (22:10 MST), Reindl Harald  wrote:
> 
> you may hit confirmation-urls (both ham and spam), trigger actions, trigger 
> *one-time* urls which are invalid for the user after a dumb bot used them not 
> talking about that it would be illegal in many countries in case of private 
> ham-mails

As I suspected, it is possible to get the goo.gl target URL without loading the 
site, though using curl is probably not realistic in this specific case.

$ curl -s "http://goo.gl/ylUAd; | grep -o "http[^\"]*"
http://www.hollywoodreporter.com/thr-esq/donald-trump-threatens-sue-macy-422135

$ curl -s "http://bit.ly/savecastle; | grep -o "http[^\"]*"
http://community.livejournal.com/castle_tv/28872.html

Doesn't work with t.co, but that is not surprising since twitter uses that 
specifically to hide URLs, considering them all their property that must go 
through their servers.

-- 
Mos Eisley spaceport. You will never find a more wretched hive of scum
and villainy. We must be cautious.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-20 Thread @lbutlr

On 2018-02-20 (19:42 MST), Rob McEwen  wrote:
> 
> I ran stats on a sample set of a few thousand mailboxes, over a period of 
> several hours today (mostly during business hours for these particular 
> organizations who use these mailboxes) - and this produced a combined 24K 
> legit messages, and 5K spams (I'm guessing that most systems have more spams 
> per amount of hams? But those were the numbers for this server.)

goo.gl (and other shorteners) are used for far more than email.

That said, most my incoming email is rejected long before it get to any sort of 
URI lookups based on just the transaction information, That is to say, upwards 
of 90% of incoming mail is rejected before DATA.

> 286 total spams blocked that had a shortner,

That's not enough to have any sort of reliable statistical data.

> 187 total legit messages had a hit on at least one of hundreds of URL 
> shortners 

So the use of a shortner is a poor spam indicator. Even in your corpus, and a 
negligible indicator even when specifically looking at goo.gl.

> Google's shortner is DOMINATING in its spam usage, where 92% (262 of 286) of 
> ALL spam that contained shortners used Google. 

But about 25% of goo.gl containing email is not spam, by your own numbers. So, 
a very poor metric.

-- 
"You can speak soon and write like a graduate college if me let you help
for a day of 15 minutes" "1963" Issue #1

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

On 2/20/2018 6:05 PM, @lbutlr wrote:

On 2018-02-20 (08:30 MST), Rob McEwen wrote:

Spammers are starting to use this to evade spam filters,

This is not news. Spammers have been using shortness since 3 seconds after
tinyurl.com launched.

My "this" was /*specifically*/ referring to Google's shortner, or at
least the recent STRONG uptick in the abuse of that shortner. I was
already well aware of other similar things from other shortners. But
"this" wasn't referring to them. You infused thoughts/meaning into my
writing that really wasn't there. (be careful about assuming...) Also,
when you see my report further down, you'll understand why those others
are not nearly as much of a concern to me at the moment.

Keep in mind that, if a marketer is doing things the right way, they should
have no need to obfuscate their own domain name. They should instead proudly
use it and not feel the need to hide behind Google's shortner.

No, that is not at all true. The primary use of a shorter is to shorten a long
URL to something that someone can type in.

I've acknowledged that there are some good reasons for a shortner - but
the vast majority of the times I'm seeing them - in both ham and spam -
that is NOT the case! The are shortening things like average-sized
domains with either no directory, or with a short directory or page
names after the domain. This is the VAST MAJORITY of the shortners I'm
seeing in both hams and spams.

Clicking a URL in an email is the height of stupidity, so having a short URL
that someone can realistically type into a browser is much better.

If I spent just a little more time on this, I could collect a large
number of Google shortner URLs that are malicious - where my
malwarebytes is blocking access to the page to which it is trying to
redirect. And these are still "live"! Do you really think that more than
a tiny percent of those who saw those in their mailbox (both legit and
spams) are manually typing in the URL and not just clicking on it? And
in that exceedingly rare occasion where somebody types in the URL and it
redirects to a malicious page that tries to install a virus, are they
ANY better off than having just clicked on it? Even the best point I can
think of that you might have had - that this might help them to better
recognize a phishing URL for example - is lost since BOTH the phish and
their bank's web site is going to be indistinguishable until AFTER
they've launched the shortner (whether by clicking or typing). I think
you just mistakenly bolstered my argument against this over-usage (and
often inappropriate usage) of shortners!

THEREFORE: If you like having NOT-blacklisted IPs, be advised that the invaluement anti-spam DNSBL system is
now adding "bad" points to the scoring of all messages that use the "goo.gl" shortner,
and we're amplifying other "bad" points.
Well, at least you are warning people. However, what you are doing is,
frankly, dumb; if you think there's a huge problem, you can simply
check the target URLs.

That incurs a significant amount of extra resources for DNSBLs and spam
filters - and such automated lookups could also put a huge extra burden
on Google's servers - and who knows at this point if this is even
reliable - Google might easily start putting captchas in the way or
otherwise consider such lookups to be abusive and/or mistake them for
malicious bots... I'm definitely going to pursue this further - but
wow... that you would suggest this... I think spam puts ENOUGH burden on
spam filters and mail system as it is!

Yes, there are many legitimate uses of Google's shortner, too. However, we are
now at a point where a VERY large % (a majority?) of uses of these headed to a
typical user's mailbox are egregious spams, and a significant additional
portion are likely-spams.

Any evidence of this?

EVIDENCE/STATS:

I ran stats on a sample set of a few thousand mailboxes, over a period
of several hours today (mostly during business hours for these
particular organizations who use these mailboxes) - and this produced a
combined 24K legit messages, and 5K spams (I'm guessing that most
systems have more spams per amount of hams? But those were the numbers
for this server.)

---
NOTE: The sum of individual shortner-hits totals below can be LARGER
than the total messages that had hits on ANY shortner - Why? - Because
in a few cases, the same message can have hits on more than one of these
shortners

---

I SEARCHED EACH HAM AND SPAM CORPUS FOR MANY OF HUNDREDS OF URL SHORTNERS

HERE ARE THE RESULTS:

---
STATS FROM SPAM:
286 total spams blocked that had a shortner, out of hundreds of URL
shortners I had searched on
(<10 that *MIGHT* be FPs - they were definitely questionable at best -
btw, zero of those questionable ones led to ANY kind of invaluement
blacklisting, even with

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)

2018-02-20 Thread @lbutlr

On 2018-02-20 (08:30 MST), Rob McEwen  wrote:
> 
> Spammers are starting to use this to evade spam filters,

This is not news. Spammers have been using shortness since 3 seconds after 
tinyurl.com launched.

> Keep in mind that, if a marketer is doing things the right way, they should 
> have no need to obfuscate their own domain name. They should instead proudly 
> use it and not feel the need to hide behind Google's shortner.

No, that is not at all true. The primary use of a shorter is to shorten a long 
URL to something that someone can type in.

Clicking a URL in an email is the height of stupidity, so having a short URL 
that someone can realistically type into a browser is much better.

> Yes, there are many legitimate uses of Google's shortner, too. However, we 
> are now at a point where a VERY large % (a majority?) of uses of these headed 
> to a typical user's mailbox are egregious spams, and a significant additional 
> portion are likely-spams.

Any evidence of this?

> THEREFORE: If you like having NOT-blacklisted IPs, be advised that the 
> invaluement anti-spam DNSBL system is now adding "bad" points to the scoring 
> of all messages that use the "goo.gl" shortner, and we're amplifying other 
> "bad" points.

Well, at least you are warning people. However, what you are doing is, frankly, 
dumb; if you think there's a huge problem, you can simply check the target URLs.

-- 
I've always had a flair for stage directions.

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)


On 2/20/2018 12:21 PM, Reindl Harald wrote:

we have well working outbound spamfilters


Excellent!

but just because someone has a google-shortener within a mail says 
*nothing at all* - frankly i even got a week ago a mail from my boss 
where the google-shortener was used for a only internal reachable 
server with a long list of params in the url
and hence the google-shortener don't say anything 



This is a very abused loophole that says MUCH in certain contexts. And 
I've carefully constructed these change at invaluement to be extremely 
unlikely to impact those who are using "goo .gl" for legitimate purposes 
and are not using it to cloak their domain in messages that are UBE or 
otherwise not desired by the recipients. But, in contrast, marketers 
and/or ESPs who start doing this routingly, as a purposeful regular 
practice, and who don't have some kind of real and specific purpose such 
as what that you described, are essentially giving DNSBLs and spam 
filters "the middle finger". So I'm giving it back. ANYTHING that 
facilitates anonymizing identity is VERY BAD for email. Facilitating 
anonymizing identity causes more spam to be delivered and punishes good 
senders when bad senders get away with that. Methods that facilitates 
anonymizing identity for email is not something that anyone should 
defend or celebrate - even if anonymizing identity wasn't the original 
intended purpose. I understand your very legitimate concern that this 
crackdown might lead to collateral damage. That is admirable. But 
acceptance of a new and pervasive situation in email that anonymizes 
identity is a HUGE step backwards... like going back to the mid 2000s, 
or something. So some "push back" measures are exceedingly warranted.


--
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)


On 2/20/2018 11:45 AM, Rob McEwen wrote:
And we ALL have to constantly shift our tactics to deal with emerging 
realities like this one - or risk getting left behind by our 
competitors who do keep up.



ALSO - Likewise, it was very frustrating that I had to spend hours late 
last night making adjustments to my spam filter to be able to block more 
of these egregious spams that are often so difficult for a spam filter 
to block - particularly since extra careful measures are needed to keep 
such adjustments from blocking legitimate messages. (I've been trying to 
get to that for weeks, as this problem has been festering for some time 
- but I just recently recovered from the flu.)


So nobody should cry on my shoulder about the difficulty of ESPs doing 
such abuse monitoring/prevention. Any such person doing so is likely 
underestimating the current size and frustration that spam filtering 
admins are having with this problem!


--
Rob McEwen
https://www.invaluement.com

Re: The "goo.gl" shortner is OUT OF CONTROL (+ invaluement's response)