On Wed, 27 Dec 2017 12:47:00 -0500 Alex <mysqlstud...@gmail.com> wrote:
> It [fetching URLs] would also probably lead to inadvertently > unsubscribing people from mailing lists. Yes, if the lists use badly-written mailing list software. At most, I would do a HEAD on a URL and not a GET. HEAD is probably safer and will usually tell you if the link is a redirect. You also want to fake the user-agent to be a common Windows browser because some malware servers look at the User-Agenet and return a 404 if they think the client is not a real Web browser. Even a HEAD can be dangerous; there's an Internet "security" [sic] company out there that shall remain nameless; these geniuses view HEAD requests as attacks and report you to your ISP. It took me 2+ weeks to sort out their BS "abuse" complaints. > I'd like to think some intelligence could be built into such a system, > and know many of the spam companies like Symantec and Mimecast are > doing this to differing degrees. I know of one company that collects URLs and has a central server farm that analyzes them (ie, the URL fetching is done on a completely different set of machines than the spam filtering.) They have all kinds of heuristics and special-case code to make it relatively safe. Regards, Dianne.
