On 6/29/2016 1:00 AM, Shivram Krishnan wrote:
Thank you so much for your views. I agree that your customers would not like it if you share information. But Oliver suggested , I need only the source IP addresses of the Spam and Ham emails , which can even be anonymized in the last octet.
Unfortunately, accuracy and credibility goes down since there then isn't any easy way to audit or double-check the root cause of the classification.
For example, some people classify spam as "what our filter said was spam" and ham as "what our filter said was ham". For most well-run systems, that is going to be overall very accurate. But there can still be egregious mistakes. And assuming that the existing filter is 100% accurate leaves no room for improvement. It also has the unfortunate side effect of rubber stamping the most elusive spams, sent by the shrewdest of spammers, as ham.
If an anti-spam blacklist comes along that is very good at blocking messages that are unsolicited and not desired by end users... but sent by the most shrewd spammer who evade lists like SpamHaus and SURBL (at least for some time)... and where the collateral damage for listing such domains and sending IPs is non-existent... such a blacklist might STILL fare badly in such a rating system... which would then MISTAKENLY assume that such a blacklist has many False Positives.
Stats collected from user complaints about False Negatives can also be helpful. However, for snowshoe spam, that is often a lagging indicator... sometimes days behind reality--where the spammer has already moved to new domains/IPs--but such could help such a ratings system to make wise adjustments to past ham/spam stats.
Hijacked IP and domains is another sticky issue. Over the past several years, this has become epidemic! If the volume of legit usage is relatively low, and the IP or domain has been hijacked by a spammer... then at SOME point, an anti-spam blacklist should not be penalized for listing such. In fact, Spamhaus does this frequently (lists hijacked domains/IPs where the cost/benefit ratio for that listing is well justified). Some other lists also blacklist hijacked domains/IPs... but are often not as good at making proper cost/benefit ratio decisions... where they list somewhat large senders who had a somewhat small and short-lived spam outbreak. Finding a way to penalize or reward the lists that block hijacked domains/IPs that Spamhaus misses, based on whether they do (or don't do) a good job of making overall good decisions about the cost/benefit ration of a potential listing's collateral damage... is also tricky.
My main point is... how to reward blacklists that are more accurate, but without penalizing them for not being a redundant copy of Zen. It isn't as easy as it sounds in a ratings system. (even if real life usage of such by a hoster or ISP can quickly lead to fewer complains from customers about about FP and FNs)
-- Rob McEwen
