[Declude.JunkMail] Sniffer Integration - Bad snf_engine.xml
Dave, Pete has helped me figure out that your XML samples, e.g.: http://interim.declude.com/41048/Scanners/SNF/snf_engine.xml is NOT a valid XML file. Specifically, the closing tag for the node element is invalid. It MUST be: /node (Currently it is node/). Consequently, opening this file with an xml parser (even just IE) will result in parser errors. I suppose everyone should double-click that XML file and see if it actually opens (assuming that this bug has been there since day 1). Best Regards, Andy --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to imail...@declude.com, and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Sniffer Integration - Bad snf_engine.xml
Yes you are correct this was reported to us . The file should have been updated with this release. I will ensure this is resolved. To correct this. In the snf_engine.xml change node/ To /node From: supp...@declude.com [mailto:supp...@declude.com] On Behalf Of Andy Schmidt Sent: Wednesday, May 05, 2010 8:57 AM To: declude.junkmail@declude.com Subject: [Declude.JunkMail] Sniffer Integration - Bad snf_engine.xml Importance: High Dave, Pete has helped me figure out that your XML samples, e.g.: http://interim.declude.com/41048/Scanners/SNF/snf_engine.xml is NOT a valid XML file. Specifically, the closing tag for the node element is invalid. It MUST be: /node (Currently it is node/). Consequently, opening this file with an xml parser (even just IE) will result in parser errors. I suppose everyone should double-click that XML file and see if it actually opens (assuming that this bug has been there since day 1). Best Regards, Andy --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to imail...@declude.com, and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to imail...@declude.com, and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Sniffer IP Reputation -- Graduated Weight Scheme
Just a thought. We would have to test it but do you think the same thing could be achieved using: IPREPUTATION-3 SNFIPREP x -3 0 -5 IPREPUTATION-2 SNFIPREP x -2 0 -5 IPREPUTATION-1 SNFIPREP x -1 0 -5 IPREPUTATION-0SNFIPREP x 0 5 -5 IPREPUTATION+1SNFIPREP x 1 5 -5 IPREPUTATION+2SNFIPREP x 2 5 -5 IPREPUTATION+3 SNFIPREP x 3 5 -5 This way the further an IP is on the scale the greater the credit or additional score. This would have to wait till we implement the - negative for the BASEPOINT. David From: supp...@declude.com [mailto:supp...@declude.com] On Behalf Of Andy Schmidt Sent: Monday, May 03, 2010 4:52 PM To: declude.junkmail@declude.com Subject: RE: [Declude.JunkMail] Sniffer IP Reputation -- Graduated Weight Scheme Hi Dave, I'm breaking this into two discussions as they are two different topics. The REAL point of Pete's input (and my suggestion) for SNFIPREP is that the reputation scale of -1 through +1 should NOT just result in either ONE positive or ONE negative weight option. Your example: IPREPUTATIONSNFIPREP x 0 10 -5 only result in either a 10 being added or a 5 being subtracted. So you are turning a continuous scale of -1 to +1 into two discrete values - losing all the key benefits of having the reputation scale in the first place. You already have the SNFIP return codes, if someone wanted a fix value for a particular level of reputation. To really make use of the GBUdb, there should be a continuous weight from 0 to 10 for bad reputation and 0 through -5 for good reputation (using your sample of 10 and -5). Basically, for positive GBUdb values, multiply with the 10 (getting a value from 0 to 10 depending on how bad the reputation is), for negative values multiply with -5 to get a weight from 0 to -5 (depending on how good the IP is). This would make the test really useful because it would only cause BIG weight changes for BIG GBUdb values. Best Regards, Andy From: supp...@declude.com [mailto:supp...@declude.com] On Behalf Of David Barker Sent: Monday, May 03, 2010 3:40 PM To: declude.junkmail@declude.com Subject: RE: [Declude.JunkMail] Sniffer IP Reputation for white listing As Pete already provided input on this. I am not going to prolix the answer other than to say when implementing Message Sniffer we abided by the Pete's advice Since many legitimate ISPs also produce a lot of spam it might be useful to apply a bias to this weight so that these systems appear closer to zero. So currently we do not allow for a negative value as a BASEPOINT, with that said if you think it is really important to be able to use a negative value as you have described in your post, let me know and I can add it to the dev list. David Barker VP Operations Declude Your Email security is our business 978.499.2933 office 978.988.1311 fax mailto:dbar...@declude.com dbar...@declude.com --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to imail...@declude.com, and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to imail...@declude.com, and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Sniffer IP Reputation -- Graduated Weight Scheme
Hi Dave, Hm - yes,I think if you added 21 lines (from -10 to 0 and to +10) to the config file, you would have could cover the reputation range from -1 to +1 in 0.1 step increments. Not elegant - but would have the same effect as multiplying the reputation range with the defined max weight. Best Regards, Andy From: supp...@declude.com [mailto:supp...@declude.com] On Behalf Of David Barker Sent: Wednesday, May 05, 2010 12:12 PM To: declude.junkmail@declude.com Subject: RE: [Declude.JunkMail] Sniffer IP Reputation -- Graduated Weight Scheme Just a thought. We would have to test it but do you think the same thing could be achieved using: IPREPUTATION-3 SNFIPREP x -3 0 -5 IPREPUTATION-2 SNFIPREP x -2 0 -5 IPREPUTATION-1 SNFIPREP x -1 0 -5 IPREPUTATION-0SNFIPREP x 0 5 -5 IPREPUTATION+1SNFIPREP x 1 5 -5 IPREPUTATION+2SNFIPREP x 2 5 -5 IPREPUTATION+3 SNFIPREP x 3 5 -5 This way the further an IP is on the scale the greater the credit or additional score. This would have to wait till we implement the - negative for the BASEPOINT. David From: supp...@declude.com [mailto:supp...@declude.com] On Behalf Of Andy Schmidt Sent: Monday, May 03, 2010 4:52 PM To: declude.junkmail@declude.com Subject: RE: [Declude.JunkMail] Sniffer IP Reputation -- Graduated Weight Scheme Hi Dave, I'm breaking this into two discussions as they are two different topics. The REAL point of Pete's input (and my suggestion) for SNFIPREP is that the reputation scale of -1 through +1 should NOT just result in either ONE positive or ONE negative weight option. Your example: IPREPUTATIONSNFIPREP x 0 10 -5 only result in either a 10 being added or a 5 being subtracted. So you are turning a continuous scale of -1 to +1 into two discrete values - losing all the key benefits of having the reputation scale in the first place. You already have the SNFIP return codes, if someone wanted a fix value for a particular level of reputation. To really make use of the GBUdb, there should be a continuous weight from 0 to 10 for bad reputation and 0 through -5 for good reputation (using your sample of 10 and -5). Basically, for positive GBUdb values, multiply with the 10 (getting a value from 0 to 10 depending on how bad the reputation is), for negative values multiply with -5 to get a weight from 0 to -5 (depending on how good the IP is). This would make the test really useful because it would only cause BIG weight changes for BIG GBUdb values. Best Regards, Andy From: supp...@declude.com [mailto:supp...@declude.com] On Behalf Of David Barker Sent: Monday, May 03, 2010 3:40 PM To: declude.junkmail@declude.com Subject: RE: [Declude.JunkMail] Sniffer IP Reputation for white listing As Pete already provided input on this. I am not going to prolix the answer other than to say when implementing Message Sniffer we abided by the Pete's advice Since many legitimate ISPs also produce a lot of spam it might be useful to apply a bias to this weight so that these systems appear closer to zero. So currently we do not allow for a negative value as a BASEPOINT, with that said if you think it is really important to be able to use a negative value as you have described in your post, let me know and I can add it to the dev list. David Barker VP Operations Declude Your Email security is our business 978.499.2933 office 978.988.1311 fax mailto:dbar...@declude.com dbar...@declude.com --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to imail...@declude.com, and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to imail...@declude.com, and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to imail...@declude.com, and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Sniffer IP Reputation -- Graduated Weight Scheme
On 5/5/2010 1:30 PM, Andy Schmidt wrote: Hi Dave, Hm yes,I think if you added 21 lines (from -10 to 0 and to +10) to the config file, you would have could cover the reputation range from -1 to +1 in 0.1 step increments. Not elegant but would have the same effect as multiplying the reputation range with the defined max weight. I hate to muddy the waters further -- but we solved this problem once when developing the envelope management bit of GBUdb. It might be complicated to explain, but suppose you define the slope at a given point for each line you specify and then have the resulting weight be a linear transform (as was discussed before). Then you would need only two entries by default... One that describes full-scale + and another that defines full scale -. If you find the need to alter the slope then you can add additional points in between. The math works by drawing a straight line from 0 to the next defined point, and from that point to the extreme, and so on. Personally I think it is overkill -- but if you're going to talk about making many many lines for this then the multi-point curve interpolation is the way to go. In practice the best way _seems_ to be to provide only two slopes -- one positive going, one negative going -- and to establish a weight based on those slopes. Theoretically that could be defined on a single Declude test definition line. Is there some constraint that I don't know about causing folks to consider more complexity? Hope this is helpful, _M -- President MicroNeil Research Corporation www.microneil.com ---This E-mail came from the Declude.JunkMail mailing list. Tounsubscribe, just send an E-mail to imail...@declude.com, andtype "unsubscribe Declude.JunkMail". The archives can be foundat http://www.mail-archive.com.
RE: [Declude.JunkMail] Sniffer Integration - Multiple Exit Codes
Hi Dave (just in case this got overlooked - or I missed the answer), Also even though there are multiple entries the test only runs once and the resulted exit code is the triggered. I know that all 18 SNF rule lines only require one invocation of Sniffer - which are then evaluated 18 different way. Fair enough. I also know that the 3 SNFIP rule lines are only one invocation - which is evaluated 3 different ways. And then there is the SNFIPREP rule. So I need to clarify this in my head. Will all 22 SNF. rules (even though they are using 3 different commands) evaluate ONE invocation of Sniffer (just different return fields) or is EACH of these 3 command groups (SNF, SNFIP, SNFIPREPS) a separate entity that requires additional overhead? Since there is some possible overhead between: SNFIPREP (which evaluates the GDUdb) and SNFIP (which also evaluates the GDUdb) and SNF-IP-RULES and SNFTRUNCATE (which also evaluate the GDUdb) - and I'm wondering if eliminating the SNFIP and SNFIPREP and just sticking with the SNF rules (which already has exit codes 20 and 63) will reduce the Sniffer overhead by 2/3? Best Regards, Andy From: supp...@declude.com [mailto:supp...@declude.com] On Behalf Of David Barker Sent: Friday, April 30, 2010 11:14 AM To: declude.junkmail@declude.com Subject: RE: [Declude.JunkMail] Sniffer Integration - Global Exit Code nonzero? The test works as an internal test and not as an external test. The main difference being the location of the exit code. See external is the 1st variable whereas the internal it is the 2nd variable and the NONZERO does not work for that. SNIFFER external nonzero C:\Smartermail\Declude\Sniffer\xxx.exe xxxabc12312 0 SNIFFER-TRAVEL SNFx 47 12 0 Also even though there are multiple entries the test only runs once and the resulted exit code is the triggered. David From: supp...@declude.com [mailto:supp...@declude.com] On Behalf Of Andy Schmidt Sent: Friday, April 30, 2010 10:31 AM To: declude.junkmail@declude.com Subject: RE: [Declude.JunkMail] Sniffer Integration - Global Exit Code nonzero? Hi Dave, Thanks for taking the time to explain it. I see that the sample on your web site has already been corrected to read IPREPUTATIONSNFIPREP and I was simply working off an earlier copy. For the SNF test type, is there a way to have a global match (e.g., NONZERO), instead of having to specify each of the 18 (current) return codes one at a time? The external Sniffer simply allow me to code: SNIFFER external nonzero D:\IMAIL\Declude\SNF\SNFClient.exe10 0 Best Regards, Andy From: supp...@declude.com [mailto:supp...@declude.com] On Behalf Of David Barker Sent: Friday, April 30, 2010 10:05 AM To: declude.junkmail@declude.com Subject: RE: [Declude.JunkMail] Sniffer Integration SNFIPBLACK SNFIP the 2nd variable value is 5 = Block and works as an exit code. IPREPUTATION works differently. Note: IPREPUTATIONSNFIP please update this to IPREPUTATIONSNFIPREP x 0 10 -5 this should be the default. SNFIPREP represents a scale of -1- 0 - 1 when the 2nd variable (BASEPOINT) is set to 0 this will convert the IP reputation to this scale as the examples below: If final score is 0 no score is added to the email dec0430.log1842 04/30/2010 00:01:20.700 49319588 SNFIPRep the Value of Result = 0.00 If final score is + the 3rd variable score is used in this case 10 dec0430.log7351 04/30/2010 00:07:14.043 49319625 SNFIPRep the Value of Result = 0.267262 If final score is - the 4th variable score is used in this case -5 dec0430.log1192604/30/2010 00:08:50.340 49319647 SNFIPRep the Value of Result = -0.267262 The BASEPOINT is the point value at which an email will be considered Good if the result is to the left or Bad if to the right. (SNIFFER RETURN) x 10 - (BASEPOINT) = Result Example: 0.267262 x 10 - 0 = 2 This is positive then the test is triggered for 10 points. 0.267262 x 10 - 1 = 1 This is positive then the test is triggered for 10 points. 0.267262 x 10 - 2 = 0 Not Triggered. 0.267262 x 10 - 3 = -1 This is negative then the test is not-triggered for -5 points. 0.267262 x 10 - 4 = -2 This is negative then the test is not-triggered for -5 points. -0.267262 x 10 - 0 = -2 This is negative then the test is not-triggered for -5 points. -0.267262 x 10 - 1 = -1 This is negative then the test is not-triggered for -5 points. -0.267262 x 10 - 2 = 0 Not Triggered. -0.267262 x 10 - 3 = -1 This is negative then the test is not-triggered for -5 points. -0.267262 x 10 - 4 = -2 This is negative then the test is not-triggered for -5 points. David Barker VP Operations Declude Your Email security is our business
RE: [Declude.JunkMail] Sniffer IP Reputation -- Graduated Weight Scheme
Yes, Declude already has TWO weights associated with SNFIPREP (one for positive, one for negative). Just as you said, but multiplying with the positive or negative weight, as need be, one would get two linear slopes from the center point. On top of that, Dave has a basepoint option that can shift the center point left or right. So - it's 99% there. It just needs to prorate the +/- weights (= multiplying) rather than use them absolute values. From: supp...@declude.com [mailto:supp...@declude.com] On Behalf Of Pete McNeil Sent: Wednesday, May 05, 2010 3:14 PM To: declude.junkmail@declude.com Subject: Re: [Declude.JunkMail] Sniffer IP Reputation -- Graduated Weight Scheme On 5/5/2010 1:30 PM, Andy Schmidt wrote: Hi Dave, Hm - yes,I think if you added 21 lines (from -10 to 0 and to +10) to the config file, you would have could cover the reputation range from -1 to +1 in 0.1 step increments. Not elegant - but would have the same effect as multiplying the reputation range with the defined max weight. I hate to muddy the waters further -- but we solved this problem once when developing the envelope management bit of GBUdb. It might be complicated to explain, but suppose you define the slope at a given point for each line you specify and then have the resulting weight be a linear transform (as was discussed before). Then you would need only two entries by default... One that describes full-scale + and another that defines full scale -. If you find the need to alter the slope then you can add additional points in between. The math works by drawing a straight line from 0 to the next defined point, and from that point to the extreme, and so on. Personally I think it is overkill -- but if you're going to talk about making many many lines for this then the multi-point curve interpolation is the way to go. In practice the best way _seems_ to be to provide only two slopes -- one positive going, one negative going -- and to establish a weight based on those slopes. Theoretically that could be defined on a single Declude test definition line. Is there some constraint that I don't know about causing folks to consider more complexity? Hope this is helpful, _M -- President MicroNeil Research Corporation www.microneil.com --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to imail...@declude.com, and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to imail...@declude.com, and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Sniffer Integration - Multiple Exit Codes
Title: Release 4.10.42 On 5/5/2010 3:24 PM, Andy Schmidt wrote: Hi Dave (just in case this got overlooked or I missed the answer), Also even though there are multiple entries the test only runs once and the resulted exit code is the triggered. I know that all 18 SNF rule lines only require one invocation of Sniffer which are then evaluated 18 different way. Fair enough. I also know that the 3 SNFIP rule lines are only one invocation which is evaluated 3 different ways. And then there is the SNFIPREP rule. So I need to clarify this in my head. Will all 22 SNF rules (even though they are using 3 different commands) evaluate ONE invocation of Sniffer (just different return fields) or is EACH of these 3 command groups (SNF, SNFIP, SNFIPREPS) a separate entity that requires additional overhead? If I may -- I'm not completely sure what you are asking -- but if your concern is that the test for SNFIP and SNFIPREPS represent additional overhead then I can answer that. The amount of code that is run to execute these tests is vanishingly small. You should consider the overhead required to run all three tests as being no more than running the SNF pattern scan. The other two (SNFIP and SNFIPREPS) require so little work that their overhead is virtually impossible to measure. _M -- President MicroNeil Research Corporation www.microneil.com ---This E-mail came from the Declude.JunkMail mailing list. Tounsubscribe, just send an E-mail to imail...@declude.com, andtype "unsubscribe Declude.JunkMail". The archives can be foundat http://www.mail-archive.com.
RE: [Declude.JunkMail] Sniffer Integration - Multiple Exit Codes
Thanks Pete. Hopefully these discussions (and seeing your responsiveness) will convince more folks decide to give Sniffer a try! I'm not completely sure what you are asking The golden rule for external tests and for RBLs is - if you have multiple lines using the SAME command (e.g., the 18 SNF lines), or referring to the same external program (e.g., 5 invURIBL lines), or referring to the same blacklist (10 lines checking different return values), THEN only the FIRST line will actually run the test against that resource (e.g., run the external program, lookup the IP in the RBL). The OTHER lines will just evaluate the return code differently without rerunning the test. Now with the internal Sniffer implementation, we have three DIFFERENT commands (SNF, SNFIP, SNFIPREP). So it's worthwhile confirming whether the same golden rule applies here even though these are NOT multiple lines of the SAME command. From: supp...@declude.com [mailto:supp...@declude.com] On Behalf Of Pete McNeil Sent: Wednesday, May 05, 2010 3:47 PM To: declude.junkmail@declude.com Subject: Re: [Declude.JunkMail] Sniffer Integration - Multiple Exit Codes On 5/5/2010 3:24 PM, Andy Schmidt wrote: Hi Dave (just in case this got overlooked - or I missed the answer), Also even though there are multiple entries the test only runs once and the resulted exit code is the triggered. I know that all 18 SNF rule lines only require one invocation of Sniffer - which are then evaluated 18 different way. Fair enough. I also know that the 3 SNFIP rule lines are only one invocation - which is evaluated 3 different ways. And then there is the SNFIPREP rule. So I need to clarify this in my head. Will all 22 SNF. rules (even though they are using 3 different commands) evaluate ONE invocation of Sniffer (just different return fields) or is EACH of these 3 command groups (SNF, SNFIP, SNFIPREPS) a separate entity that requires additional overhead? If I may -- I'm not completely sure what you are asking -- but if your concern is that the test for SNFIP and SNFIPREPS represent additional overhead then I can answer that. The amount of code that is run to execute these tests is vanishingly small. You should consider the overhead required to run all three tests as being no more than running the SNF pattern scan. The other two (SNFIP and SNFIPREPS) require so little work that their overhead is virtually impossible to measure. _M -- President MicroNeil Research Corporation www.microneil.com --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to imail...@declude.com, and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to imail...@declude.com, and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Sniffer Integration - Multiple Exit Codes
Title: Release 4.10.42 On 5/5/2010 4:05 PM, Andy Schmidt wrote: snip/ The golden rule for external tests and for RBLs is if you have multiple lines using the SAME command (e.g., the 18 SNF lines), or referring to the same external program (e.g., 5 invURIBL lines), or referring to the same blacklist (10 lines checking different return values), THEN only the FIRST line will actually run the test against that resource (e.g., run the external program, lookup the IP in the RBL). The OTHER lines will just evaluate the return code differently without rerunning the test. Now with the internal Sniffer implementation, we have three DIFFERENT commands (SNF, SNFIP, SNFIPREP). So its worthwhile confirming whether the same golden rule applies here even though these are NOT multiple lines of the SAME command. The same rule applies --- Run the test once, use the results of the test many times. However in the case of SNFIP and SNFIPREP the cost of the test is so small that it cannot be measured. The IP reputation database is local (in memory) and immediately accessible (there is no delay or network traffic involved). The only work that gets done is a little bit of math. Best, _M -- President MicroNeil Research Corporation www.microneil.com ---This E-mail came from the Declude.JunkMail mailing list. Tounsubscribe, just send an E-mail to imail...@declude.com, andtype "unsubscribe Declude.JunkMail". The archives can be foundat http://www.mail-archive.com.
