[sniffer] Re: How to incorporate a white list?
The F001 bot will be disabled until further notice. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: How to incorporate a white list?
I do not think that anyone was asking the F001 bot to be disabled. Are you doing this for upgrading purposes or because there appeared to be an error with it? A single false positive as described, in my opinion, is no cause for alarm. Any time something changes, there is a potential for error, so please be careful in any attempts to implement suggestions from the community without evaluating all of the possibilities. Personally, I like the way the system is working. However, if it is possible to decrease FPs while maintaining the high level of accuracy in blocking spam, that is always welcome. - Original Message - From: Pete McNeil To: Message Sniffer Community Sent: Wednesday, April 04, 2007 10:26 AM Subject: [sniffer] Re: How to incorporate a white list? The F001 bot will be disabled until further notice. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: How to incorporate a white list?
Hello Jonathan, Wednesday, April 4, 2007, 10:41:19 AM, you wrote: I do not think that anyone was asking the F001 bot to be disabled. Are you doing this for upgrading purposes or because there appeared to be an error with it? A single false positive as described, in my opinion, is no cause for alarm. Any time something changes, there is a potential for error, so please be careful in any attempts to implement suggestions from the community without evaluating all of the possibilities. Personally, I like the way the system is working. However, if it is possible to decrease FPs while maintaining the high level of accuracy in blocking spam, that is always welcome. The F001 bot is facing a no-win scenario. It cannot be upgraded at this time, and even if that decision were to be reversed, there are problems with the whole concept of long-term blocking by IP of the type accomplished by F001: * Increasingly, hacked systems are used to send spam through major ISPs and email systems (even with full authentication hijacked from the hacked bots) - so the source data for the F001 bot (clean spamtraps) is increasingly compromised. That is - even though real spam is being sent through systems like aol, gmail, etc, it is not acceptable to block those systems based on IP data -- that is, after all, why the blackhats are moving in this direction. * IP data constantly changes without notice. More than half of the false positives levied against IP rules are due to older IPs that have been reassigned, or for systems that have since cleaned up their act, or for systems that for IPs that at one point generated spam and now do not. Though these are not generally caused by the F001 bot, they do point out that the IP space has become so dynamic that any kind of long-lived IP blocking will only be increasingly non-viable. * The GBUdb engine will be coming on-line soon enough and it will replace the function of the F001 bot with a better, more dynamic system that follows the real-time activities of IP sources. Since there is no short-term fix for the F001 bot, and since it's functions will continue to be compromised long term, and since even a low rate of false positives like these recent reports are clearly unacceptable. The best choice at this time is to remove the F001 bot from service. The IP rules that are currently in place will remain active, and on occasion additional IP rules may be added through other mechanisms. The IP rule group should not appreciably degrade between now and full deployment of the GBUdb features in SNF - so the net result should be positive. In the end, we (the entire SNF team) want to be responsive and proactive. Given the circumstances - disabling F001 appears to be the best choice. If conditions change then we can always reactivate the device. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: How to incorporate a white list?
Hi Phil, Yes, it seems as if some Sniffer rules, e.g., 1367683, is broadly targeting Google's IPs. I've submitted 3 false positive reports since last night, at least two of them were Google users, one located in the U.S. and the other in the Netherlands! Andy -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Phillip Cohen Sent: Tuesday, April 03, 2007 6:30 AM To: Message Sniffer Community Subject: [sniffer] How to incorporate a white list? I am getting a large number of false positives and not sure why. Mostly mail from newsletters or lists, such as DMXZone, but I am also still unable to receive some mail from my own internal users. I am filtering on a per mailbox right now and I have been sending spam from my mailbox into its own holding directory so I can see what I am missing. It appears that while it gets most spam there are also some real messages getting zapped as well. How do I add a whitelist of domains, or do i send in the false positives in hopes they will somehow be added to the rulebase. I am fairly new at this and it is not real obvious looking at the documentation online as to how this all works. This is running on an old vopmail server. Thanks, Phil # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: How to incorporate a white list?
Hello Andy, Tuesday, April 3, 2007, 9:36:17 AM, you wrote: Hi Phil, Yes, it seems as if some Sniffer rules, e.g., 1367683, is broadly targeting Google's IPs. I've submitted 3 false positive reports since last night, at least two of them were Google users, one located in the U.S. and the other in the Netherlands! This IP rule has been pulled. FP processing will happen shortly. _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: How to incorporate a white list?
Hi, Unless I'm mistaken, rule 1370762 was targeting the same address range. If I may make a suggestion: Before the spam-trap robots are allowed to block major, well-known and easily recognizable email providers, how about the robot script pulls a WHOIS and a Reverse DNS and runs that data against a table of can't block entities - or at least spits those out for human review. If that can't be done, then how about the robots issue an hourly report of suspect IPs. A no-brainer script can pull matching WHOIS and RevDNS for quick human review and overriding (if necessary). I would rather those obvious bad rules are caught before or very quickly after they go live. There is always some delay before I get first reports until I realize that this is a real problem. Then I have to try to get headers from end-users before I can dig into logs... Hours and hours pass (especially if it's overnight events). In the meantime the problem escalates all around me. Thanks, Andy -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Tuesday, April 03, 2007 11:09 AM To: Message Sniffer Community Subject: [sniffer] Re: How to incorporate a white list? Hello Andy, Tuesday, April 3, 2007, 9:36:17 AM, you wrote: Hi Phil, Yes, it seems as if some Sniffer rules, e.g., 1367683, is broadly targeting Google's IPs. I've submitted 3 false positive reports since last night, at least two of them were Google users, one located in the U.S. and the other in the Netherlands! This IP rule has been pulled. FP processing will happen shortly. _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: How to incorporate a white list?
Agreed, however reverse DNS is not a universal solution as things like RR accounts will come from the same base domain as RR spam zombies, and you would otherwise have to track down each unique reverse DNS entry. I would test a connection to the SMTP server instead. Most of these servers will at least respond. So if a domain like yahoo.com, gmail.com, rr.com, etc. is found in the reverse DNS for a new IP rule, you would then check to see if it at least responded to a port 25 connection, and if it did, skip that rule. Note that I score IP rules at half the weight of the others. There are more common issues with international ISP's and webmail providers than with things like yahoo.com, gmail.com, rr.com, etc. Many don't get a lot of international traffic so they don't notice it. Matt Andy Schmidt wrote: Hi, Unless I'm mistaken, rule 1370762 was targeting the same address range. If I may make a suggestion: Before the spam-trap robots are allowed to block major, well-known and easily recognizable email providers, how about the robot script pulls a WHOIS and a Reverse DNS and runs that data against a table of can't block entities - or at least spits those out for human review. If that can't be done, then how about the robots issue an hourly report of suspect IPs. A no-brainer script can pull matching WHOIS and RevDNS for quick human review and overriding (if necessary). I would rather those obvious bad rules are caught before or very quickly after they go live. There is always some delay before I get first reports until I realize that this is a real problem. Then I have to try to get headers from end-users before I can dig into logs... Hours and hours pass (especially if it's overnight events). In the meantime the problem escalates all around me. Thanks, Andy -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Tuesday, April 03, 2007 11:09 AM To: Message Sniffer Community Subject: [sniffer] Re: How to incorporate a white list? Hello Andy, Tuesday, April 3, 2007, 9:36:17 AM, you wrote: Hi Phil, Yes, it seems as if some Sniffer rules, e.g., 1367683, is broadly targeting Google's IPs. I've submitted 3 false positive reports since last night, at least two of them were Google users, one located in the U.S. and the other in the Netherlands! This IP rule has been pulled. FP processing will happen shortly. _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: How to incorporate a white list?
Hello Andy, Tuesday, April 3, 2007, 5:15:12 PM, you wrote: Hi Jonathan: That's exactly the problem. These particular rules were blocking Google mail servers - NOT specific content. To clarify, it was blocking precisely one IP. The F001 bot only tags a single IP at a time (not ranges, ever), and only after repeated appearances at clean spamtraps where the message also fails other tests (often including content problems like bad headers, obfuscation, heuristic scam text matching etc.) The rule was in place from 20070326. The first reported false positives arrived today (just after midnight). The rule was removed just less than 12 hours from that report (due to scheduling and heavy spam activity this morning that requiring my immediate attention). The report was ordinary (not a rule panic). As is the case with all FPs, the rule cannot be repeated (without special actions). Obviously, as already discussed in the past, it IS necessary that these IP-based blocks are put under a higher scrutiny. I'm not suggesting that the automatic bots should be disabled, I'm just proposing that intelligence must be incorporated that will use RevDNS and WHOIS to identify POSSIBLY undesirable blocks and to flag those for human review by Sniffer personnel so that they don't end up poisoning mail severs of all their clients. While interesting, these mechanism are not foolproof nor trivial to implement. Also - our prior research has taught us that direct human involvement in IP rule evaluation leads to far more errors we can allow. Once upon a time, IP rules were created in very much the way you describe -- candidate IPs were generated from spamtraps and the live spam corpus and then reviewed (manually and automatically) against RevDNS, WHOIS, and other tools. At that time, IP rules had the absolute worst reliability of any test mechanism we provided. Upon further RD, we created the F001 bot that is in place and now the error rate has been significantly reduced and our people are able to focus on things that computers can't do better. Please don't get me wrong, I'm definitely not saying that the F001 bot can't be improved - it certainly can, and will if it survives. What I am saying is that it is accurate enough now that any improvements in accuracy would be non-trivial to implement. Our current development focus is on developing the suite of applications and tools that will allow us to complete the alpha and beta testing of the next version of SNF*. That work has priority, and given that these events are rare and easily mitigated we have not deemed it necessary to make enhancements to the F001 bot a higher priority. The following factors make it relatively easy to mitigate these IP FP events (however undesirable): Rule panics can make these rules immediately inert, FP report/response times are sufficiently quick, The IP rule group is sequestered at the lowest priority so that it can easily be weighted lower than other tests. Also, it is likely that the F001 bot and IP rules group will be eliminated once the next SNF version is sufficiently deployed because one of the major enhancements of the new engine is a multi-tier, self-localizing IP reputation system (GBUdb). * A production ready SYNC server is nearing completion. This software will allow large numbers of GBUdb equipped nodes to share near-real-time IP statistics. The new SNF engine itself is nearly complete and has been in alpha testing in production environments for some time to prove that it is stable (and it is). We expect to begin wider alpha testing followed quickly by beta testing within the next few weeks if all goes well. Once the system is deployed, all SNF nodes will cooperate to learn both good and bad IP sources based on content analysis and localized behaviors and they will be able to share what they learn with other nodes within seconds (90 on average) of any significant change or new knowledge. I understand that occasionally some innocent IP can be added accidentally and there is little to avoid that -- but for the top 50 email domains, extra security/intelligence should be in place so that we don't suddenly reject huge volumes of legitimate mail by blocking hotmail, aol, yahoo, google or similar providers! These kind of errors can be caught much earlier. Perhaps; and we do have mechanisms in place to help prevent these events. For example, there is one mechanism were IPs that appear to be at risk are entered into the IP rule group as nokens (Excluded on entry) to prevent manual or automatic processes from creating black rules. As you point out, though, occasionally any system will allow errors from time to time. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL
[sniffer] Re: How to incorporate a white list?
Hi Pete, Thanks for taking the time to respond. The rule was in place from 20070326. The first reported false positives arrived today Except that reports from end users lingered in my email since Friday. Not your fault - but just to better demonstrate the ultimate effect it had. To be certain, I wasn't dissatisfied with your reaction time after I finally got around to looking at the user reports and compiling reports to you. My argument is, that for big email providers, there could be procedures in place to identify possible bad rules and flag them for review without waiting for FP reports. To clarify, it was blocking precisely one IP. The F001 bot only tags a single IP at a time (not ranges, ever) Except that there were multiple rules (I remember seeing at least two) - and each one (if I recall correctly) targeting a different IP in the same block. Thus, the difference is merely technical (whether n rules are needed for n IPs or whether one rule covers multiple). Once upon a time, IP rules were created in very much the way you describe -- candidate IPs were generated from spamtraps and the live spam corpus and then reviewed (manually and automatically) against RevDNS, WHOIS, and other tools. At that time, IP rules had the absolute worst reliability of any test mechanism we provided. I can't follow the logic. If F001 would continue to be used (but certain IPs are reviewed), then this can't possibly increase the false positive rate. At worst, a rule may be prohibited unnecessarily... But that's our job - to err on the save side and let the GOOD mail go through. If we block good mail, then the system has failed the user. Best Regards, Andy -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Tuesday, April 03, 2007 6:31 PM To: Message Sniffer Community Subject: [sniffer] Re: How to incorporate a white list? Hello Andy, Tuesday, April 3, 2007, 5:15:12 PM, you wrote: Hi Jonathan: That's exactly the problem. These particular rules were blocking Google mail servers - NOT specific content. To clarify, it was blocking precisely one IP. The F001 bot only tags a single IP at a time (not ranges, ever), and only after repeated appearances at clean spamtraps where the message also fails other tests (often including content problems like bad headers, obfuscation, heuristic scam text matching etc.) The rule was in place from 20070326. The first reported false positives arrived today (just after midnight). The rule was removed just less than 12 hours from that report (due to scheduling and heavy spam activity this morning that requiring my immediate attention). The report was ordinary (not a rule panic). As is the case with all FPs, the rule cannot be repeated (without special actions). Obviously, as already discussed in the past, it IS necessary that these IP-based blocks are put under a higher scrutiny. I'm not suggesting that the automatic bots should be disabled, I'm just proposing that intelligence must be incorporated that will use RevDNS and WHOIS to identify POSSIBLY undesirable blocks and to flag those for human review by Sniffer personnel so that they don't end up poisoning mail severs of all their clients. While interesting, these mechanism are not foolproof nor trivial to implement. Also - our prior research has taught us that direct human involvement in IP rule evaluation leads to far more errors we can allow. Once upon a time, IP rules were created in very much the way you describe -- candidate IPs were generated from spamtraps and the live spam corpus and then reviewed (manually and automatically) against RevDNS, WHOIS, and other tools. At that time, IP rules had the absolute worst reliability of any test mechanism we provided. Upon further RD, we created the F001 bot that is in place and now the error rate has been significantly reduced and our people are able to focus on things that computers can't do better. Please don't get me wrong, I'm definitely not saying that the F001 bot can't be improved - it certainly can, and will if it survives. What I am saying is that it is accurate enough now that any improvements in accuracy would be non-trivial to implement. Our current development focus is on developing the suite of applications and tools that will allow us to complete the alpha and beta testing of the next version of SNF*. That work has priority, and given that these events are rare and easily mitigated we have not deemed it necessary to make enhancements to the F001 bot a higher priority. The following factors make it relatively easy to mitigate these IP FP events (however undesirable): Rule panics can make these rules immediately inert, FP report/response times are sufficiently quick, The IP rule group is sequestered at the lowest priority so that it can easily be weighted lower than other tests. Also, it is likely that the F001 bot and IP rules group will be eliminated once the next SNF version
[sniffer] Re: How to incorporate a white list?
Pete, CBL has a proven 99.97% accuracy and on some systems over a 40% hit rate on traffic, yet their methods are rather simple and easy to implement. If an IP hits your spamtrap, and it has either no reverse DNS entry or it has a dynamic reverse DNS entry, it is added, if it doesn't, it isn't added. They have a few other mechanisms that I am aware of, but the above will take care of almost everything related to spam zombies. Your current whitelisting method will take care of the few exceptions to this. There is rather simple code that can test for standard types of dynamic reverse DNS entries with both numbers and hex encoded values, and exceptions for names that might include things like mail or mx# in the names. If you want to expand this to static spammers, you merely introduce other pre-qualifications such as having a Mail From domain or HELO that matches the payload domain in the body. I figure that for the most part however that you are tagging static spammers with other rules that take presidence over the IP rules, and that this would be minimally beneficial in comparison to spam zombies. The source of the false positives hitting your spam traps are most likely due to AFF (Advance Fee Fraud) and some phishing, which use free accounts on legitimate servers to send their spam, and an increasing precidence of hacked E-mail accounts being used by zombie spammers. The first method would avoid listing such servers in almost every circumstance, and we certainly wouldn't ever see things like yahoo.com, gmail.com and rr.com mail servers listed like we see with some degree of regularity under the current method. Matt Pete McNeil wrote: Hello Andy, Tuesday, April 3, 2007, 5:15:12 PM, you wrote: Hi Jonathan: That's exactly the problem. These particular rules were blocking Google mail servers - NOT specific content. To clarify, it was blocking precisely one IP. The F001 bot only tags a single IP at a time (not ranges, ever), and only after repeated appearances at clean spamtraps where the message also fails other tests (often including content problems like bad headers, obfuscation, heuristic scam text matching etc.) The rule was in place from 20070326. The first reported false positives arrived today (just after midnight). The rule was removed just less than 12 hours from that report (due to scheduling and heavy spam activity this morning that requiring my immediate attention). The report was ordinary (not a rule panic). As is the case with all FPs, the rule cannot be repeated (without special actions). Obviously, as already discussed in the past, it IS necessary that these IP-based blocks are put under a higher scrutiny. I'm not suggesting that the automatic bots should be disabled, I'm just proposing that intelligence must be incorporated that will use RevDNS and WHOIS to identify POSSIBLY undesirable blocks and to flag those for human review by Sniffer personnel so that they don't end up poisoning mail severs of all their clients. While interesting, these mechanism are not foolproof nor trivial to implement. Also - our prior research has taught us that direct human involvement in IP rule evaluation leads to far more errors we can allow. Once upon a time, IP rules were created in very much the way you describe -- candidate IPs were generated from spamtraps and the live spam corpus and then reviewed (manually and automatically) against RevDNS, WHOIS, and other tools. At that time, IP rules had the absolute worst reliability of any test mechanism we provided. Upon further RD, we created the F001 bot that is in place and now the error rate has been significantly reduced and our people are able to focus on things that computers can't do better. Please don't get me wrong, I'm definitely not saying that the F001 bot can't be improved - it certainly can, and will if it survives. What I am saying is that it is accurate enough now that any improvements in accuracy would be non-trivial to implement. Our current development focus is on developing the suite of applications and tools that will allow us to complete the alpha and beta testing of the next version of SNF*. That work has priority, and given that these events are rare and easily mitigated we have not deemed it necessary to make enhancements to the F001 bot a higher priority. The following factors make it relatively easy to mitigate these IP FP events (however undesirable): Rule panics can make these rules immediately inert, FP report/response times are sufficiently quick, The IP rule group is sequestered at the lowest priority so that it can easily be weighted lower than other tests. Also, it is likely that the F001 bot and IP rules group will be eliminated once the next SNF version is sufficiently deployed because one of the major enhancements of the new engine is a multi-tier, self-localizing IP reputation system (GBUdb). * A production ready SYNC server is nearing completion.
