> (1) First, I "eat my own dogfood", ... Yes, that was clear.
> (2) A large percentage of invaluement subscribers use SpamAssassin So this should work somewhat. If you have the capacity to let everything be processed by the SA content filter. Earlier you stated that larger setups depended on having blacklists at the gate to keep processing manageable but which results in less weighed filtering. [see #5] But these setups (often) still lack feedback mechanisms that the Yahoo's, Hotmail's and Gmail's of this world have... Feedback from users. It also lacks methods that need large quantities of "historical data". Both add information back into the content filter to improve it. > (3) I'm certain that some portion of invaluement subscribers have BETTER filtering ...- they all have excellent filtering in the areas of their spam filtering that invaluement attempts to improve! :) Hey Rob, I think that you have a very good blacklist. My point is that I think "an excellent blacklist is not good enough". Because you need a better (and faster) integration between the content filter (using e.g. reputation data from you), the traffic limiting mechanisms, the feedback mechanisms (e.g. users marking as spam, or the other way around) and "email history of your domain". > (4) You seem to be very confused about what I mean when I talk about how there has to be some justified level of "collateral damage" these days, due to the very high frequency of hijacked accounts I have seen it all... Up to the level of terrorism. I agree with you on that. > (5) Also, a large percentage of Invaluement subscribers choose to block at the perimeter See #2. > (6) nobody in this thread ever claimed that blacklists-alone are sufficient for having good spam filtering Rob, you as a small set of people, are capable to and have enough access to improve on the current situation. I also think that the Google's, Microsofts and Yahoo's are a systemic issue in the email world. This would include Proofpoint and SpamExperts as well. It is becoming harder and harder to have your own server and not be troubled by spam unless you use this small set of services. "They" are unwilling to share their methods with the rest of us (which I can understand but which results into "the rest" not having that knowledge). I think that we, "the rest", should develop better ways to filter spam. That was my goal of this discussion. To make you conscious of the fact that we need more than IP/domain blacklists. To be able to level up with the proprietary solutions. Instead of being stuck with the idea that we should stick to what we have (because we have it). Anyway, take it as it is. I hope you have a great weekend. Yours, David On 8 June 2018 at 16:27, Rob McEwen <r...@invaluement.com> wrote: > On 6/8/2018 5:49 AM, David Hofstee wrote: > >> > ... score of the sending-IP, which is similar to what you've described, >> correct? >> Correct. >> >> So you have these mechanisms in place. But your customers, who get access >> to the invaluement RBL, do not. Am I correct? If I am, it still results in >> the conclusion that blacklists are not sufficient to have a resulting good >> spam filter. You would be ok, the list would not have false positives, >> but your customers would not be sufficiently covered once bad guys get >> smarter. >> > > > David, > > You've made so many false assumptions to come to these conclusions... and > taken things I've said out of context to get there... I had a hard time > knowing where to begin! > > (1) First, I "eat my own dogfood", even for my own mailbox! In our own > spam filtering system, we score ALL invaluement blacklists "above > threshold". However, in VERY RARE situations, a message will get delivered > in our mail system where (a) it had one hit on one invaluement list, (b) > NOTHING else spammy triggered, (c) and some rules kicked in that lowered > the spam score just barely below threshold -BUT GUESS WHAT?- the vast > majority of the time that happens, it ends up being a FALSE NEGATIVE - then > I'm jealous of my own customers whose systems didn't deliver those spams to > their users' inboxes! > > (2) A large percentage of invaluement subscribers use SpamAssassin, and > likewise use a multi-tiered scoring system where they score blacklists > higher if that blacklist (a) had fewer FPs, -AND- (b) the FPs it generates > are more likely to result in extremely rare and/or extremely justified FPs. > And they score OTHER DNSBLs lower for those which have a higher frequency > of hits on desired mail. And some have similar scoring options with other > spam filtering systems in addition to SpamAssassin. > > (3) I'm certain that some portion of invaluement subscribers have BETTER > filtering than we have for our boutique mail hosting system, but that > circumstance is somewhat rare ONLY because we can afford to put a lot more > resources per mailbox into this, due to our DNSBL effectively subsidizing > our small mail hosting system. (But this helps us to make the quality of > invaluement data even better, which only benefits ALL of our subscribers - > the opposite would be true if we neglected our own filtering system!) But > since our subscribers all use a VARIETY of technologies and approaches to > spam, and since invaluement data is only tacking a subset of all spam, and > isn't trying to be a comprehensive solution for blocking all spam - every > client has a unique situation. And I'm certain that at least SOME of them > have even BETTER spam filtering than what we have for our mail hosting > customers. But they all share one thing in common - they all have excellent > filtering in the areas of their spam filtering that invaluement attempts to > improve! :) > > (4) You seem to be very confused about what I mean when I talk about how > there has to be some justified level of "collateral damage" these days, due > to the very high frequency of hijacked accounts, hijacked websites, and > spamming ESP customers (from ESP that are overall good). Keep in mind that, > ALL of these added together at one time - can be STILL be an astronomically > tiny percentage-wise when it comes to how much the collateral damage > impacts the average end user, yet can still be tremendously harmful to the > company with the security problem, since the problem CONCENTRATES there. To > give an extreme example, suppose a small business with 25 employees, who > averages 500 outbound legit emails a day - had a security lapse and their > server starts attempting to send out 200,000+ egregious spams per day - and > now their 500 outbound legit emails are getting blocked in many places. The > chances that an ISP with 10,000 mailboxes - or even 1M mailboxes - is going > to be impacted by the collateral damage - and have to deal with user > complaints about false positives - is extremely rare - yet this small > business is going to have a very very bad day that day! In a situation like > this, their sending IP is likely to get blacklisted on Invaluement and/or > Spamhaus - but will likely also get delisted fairly soon after they submit > a delist request and/or fix the problem (but that could take longer if they > are doing stupid stuff - like having a poorly formed PTR record that looks > dynamic and doesn't properly convey identity and reputation... but I > digress) And then other higher-FP blacklists will do a lot of similar > listings, except they'll also include situations where the > spam-to-collateral-damage ratios are NOT so clear cut. These other lists > are better for scoring - AND MANY (OR MOST?) INVALUEMENT SUBSCRIBERS HAVE > THE TECHNOLOGY TO DO THAT SCORING. (did you not know that?) > > (5) Also, a large percentage of Invaluement subscribers choose to block at > the perimeter (at connection, without accepting the message) based on Zen > (from Spamhaus), ivmSIP, and ivmSIP/24 - and are extremely pleased with the > extremely low number of FPs - and the way that we score ALL invaluement > DNSBL hits on messages to our own users' mailboxes "above threshold" - is > very similar to that. And in those rare instances where something in our > system caused an invaluement-listed message to get delivered to the inbox, > the vast majority of the time - it ended up being a false negatives in my > system, NOT the avoidance of a false positive. (as it should be - so that > we can have even better telemetry for spotting and quickly fixing potential > invaluement FPs!) But, again, even this exceptional situation. Again, an > invaluement-listed message getting through our own filter, due to a rare > set of circumstances (that usually results in a False Negative!) is > extremely rare when compared to the number of invaluement-listed spams that > our own spam filter routinely blocks. > > Finally, regarding your statement "blacklists are not sufficient to have a > resulting good spam filter" > > (6) nobody in this thread ever claimed that blacklists-alone are > sufficient for having good spam filtering. I certainly have NEVER made such > a statement, or even implied such. But I have stated that it is very > difficult to have high quality filtering AND it is difficult to have > efficient filtering that can keep up with high volumes of spams, without > using blacklists. Also, something not being 100% comprehensive and not > being perfect - doesn't mean it isn't extremely beneficial and often even > critically important. For example, occasionally, we get subscribers who use > MS Exchange, don't subscriber to Microsoft's build-in filtering options, > and don't have any kind of anti-spam exchange add-ons installed. They ONLY > try to filter using RBLs entered into MS Exchange. I ALWAYS try to educate > such a person that their filtering will NEVER have the ability to be very > good if the ONLY thing they are doing is blocking on high-quality low-FP > DNSBLs. (I have this conversation with someone at least a couple of times a > year.) > > -- > Rob McEwen > https://www.invaluement.com > +1 (478) 475-9032 > > > -- -- My opinion is mine.
_______________________________________________ mailop mailing list mailop@mailop.org https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
