RE: [Declude.JunkMail] Request for additional filtering functionality
Matt: The log file points idea would be a great help - simply show how much each filter contributed. I can't imagine this would be very hard to implement.. would help a lot in figuring out the effectiveness of certain filters.. The idea of END has been discussed before and of course it is more prevalent to those of us that use a lot of filters. I have seen emails that have failed with a weight of 400 and we delete on 60. So the CPU could have taken care of other email instead of wasting time on the email that was deleted. We have seen a lot of help from the IMail delete action with IP4R tests. Deleting on 13 failed tests deletes a lot of spam before it even reaches Declude. Something like: End Weight60 in the global statement could be an effective approach. Of course and END or RETURN statement in the filter would be great too... I think a lot of these ideas are great and will help the product.. I am just wondering how one can explain all these features to someone coming new onboard? I guess that is where the archives come into play.. Regards, Kami -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matthew Bramble Sent: Thursday, November 13, 2003 9:48 PM To: [EMAIL PROTECTED] Subject: [Declude.JunkMail] Request for additional filtering functionality Scott, As I continue to look for new potential in filtering, I have repeatedly come across some limitations which restrict what can be done effectively, difficulty in figuring the scoring of some variable filters, and challenges from the additional processing power required to counterbalance some filters, so I just wanted to request three different things which appear like they might be somewhat reasonable extensions to the current environment. I'm putting these all together in one message because, at least from my perspective, they are all related, and I didn't want to bother you repeatedly with such requests. Those requests are as follows: 1) Provide the score of a test in the logs, WARN function and %TESTSFAILED% variable. This would help with filters that have internal counterbalances or variable scoring so that an admin could quickly determine how many points were assessed. I would imagine that it could be turned on by way of one or three lines in the Global.cfg, i.e. SHOWPOINTSON or SHOWLOGPOINTS ON SHOWWARNPOINTS ON SHOWTESTSFAILEDPOINTS ON When on, this would add the points scored to each of the types of entries as follows: LOG: 11/13/2003 20:43:02 Q331903a90080bcc8 Msg failed IPLINKED ([Score: 7] Message failed IPLINKED test (189)). Action=WARN. WARN: X-RBL-Warning: IPLINKED: [Score: 7] Message failed IPLINKED test (226) TESTSFAILED: X-Weight: 16 (REVDNS [0], IPNOTINMX [0], IPLINKED [7], SPAMCOP [9]) These changes would not only make scoring without custom filters much easier, it would definitely make more advanced configurations much easier to score and therefore make the system as a whole easier to administrate. 2) Provide a method of defeating a custom filter (zero points) based on failing a specially marked test. Having this capability would along with the above requested feature would remove the need to write convoluted systems to counterbalance custom filters with ANTI filters (or whatever you want to call them). So instead of having entries for allowed character strings, base64 encoding, certificates, etc., listed in both a GIBBERISH and ANTI-GIBBERISH filter for instance could be instead be listed in just one file making implementation much easier, more straightforward and saving on processing power that might be required to parse a fully redundant ANTI filter. You might even explore the possibility of using an END function so that tests listed at the top of a custom filter file meant to defeat the test will stop the rest of the filter from being processed and scored as 0 in the event that a trigger is matched. I might suggest a configuration like the following: BODY ENDCONTAINSbase64 I believe that I could probably save between 10% and 30% of my processing power by having the ability to defeat a custom filter or at least not be required to use a combination of filters for counterbalancing. Custom filters with long lists of combinations are very expensive to process, and I have a feeling that my current dual 1 GHz server could only handle about 50,000 messages a day under the current configuration from what I am seeing in task manager (single Declude.exe processes reaching just over 50%). This is after I removed many of the extensive body filters that I was using for a short while. 3) Provide a method of defining a maximum and/or minimum number of points that a particular custom filter can score. This would allow for better use of filters that can produce multiple hits and are scored per hit. There have been a few occasions where I have attempted to code a filter where it increments the
Re: [Declude.JunkMail] Request for additional filtering functionality
As I continue to look for new potential in filtering, I have repeatedly come across some limitations which restrict what can be done effectively, difficulty in figuring the scoring of some variable filters, and challenges from the additional processing power required to counterbalance some filters, so I just wanted to request three different things which appear like they might be somewhat reasonable extensions to the current environment. I'm putting these all together in one message because, at least from my perspective, they are all related, and I didn't want to bother you repeatedly with such requests. Those requests are as follows: Thanks for the suggestions. Giving the number of people that are now using filters in Declude JunkMail, and the size of them, it's about time for us to expand them a bit. LOG: 11/13/2003 20:43:02 Q331903a90080bcc8 Msg failed IPLINKED ([Score: 7] Message failed IPLINKED test (189)). Action=WARN. WARN: X-RBL-Warning: IPLINKED: [Score: 7] Message failed IPLINKED test (226) These two will be changed to use Message failed IPLINKED test (line 189, weight 7). TESTSFAILED: X-Weight: 16 (REVDNS [0], IPNOTINMX [0], IPLINKED [7], SPAMCOP [9]) This can be done in the next release with a new %TESTSFAILEDWITHWEIGHTS% variable. 2) Provide a method of defeating a custom filter (zero points) based on failing a specially marked test. This will be in the next release. END instead of the weight will force the test to end. 3) Provide a method of defining a maximum and/or minimum number of points that a particular custom filter can score. A MAXWEIGHT option will be in the next release, that will allow you to define the maximum weight that the test can add. If the maximum weight is reached, processing will stop (so any negative weights would need to go at the beginning of the test), and the maximum weight will be used instead of the actual weight (IE if you have MAXWEIGHT 60, and the filter is at 55 points with a line that would add 10 points, processing would stop with a weight of 60, not 65). -Scott --- Declude JunkMail: The advanced anti-spam solution for IMail mailservers. Declude Virus: Catches known viruses and is the leader in mailserver vulnerability detection. Find out what you've been missing: Ask about our free 30-day evaluation. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Request for additional filtering functionality
Scott... Let me do a simple test.. Just in case it works.. Could I have a million dollars? Regards, Kami -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of R. Scott Perry Sent: Friday, November 14, 2003 9:44 AM To: [EMAIL PROTECTED] Subject: Re: [Declude.JunkMail] Request for additional filtering functionality As I continue to look for new potential in filtering, I have repeatedly come across some limitations which restrict what can be done effectively, difficulty in figuring the scoring of some variable filters, and challenges from the additional processing power required to counterbalance some filters, so I just wanted to request three different things which appear like they might be somewhat reasonable extensions to the current environment. I'm putting these all together in one message because, at least from my perspective, they are all related, and I didn't want to bother you repeatedly with such requests. Those requests are as follows: Thanks for the suggestions. Giving the number of people that are now using filters in Declude JunkMail, and the size of them, it's about time for us to expand them a bit. LOG: 11/13/2003 20:43:02 Q331903a90080bcc8 Msg failed IPLINKED ([Score: 7] Message failed IPLINKED test (189)). Action=WARN. WARN: X-RBL-Warning: IPLINKED: [Score: 7] Message failed IPLINKED test (226) These two will be changed to use Message failed IPLINKED test (line 189, weight 7). TESTSFAILED: X-Weight: 16 (REVDNS [0], IPNOTINMX [0], IPLINKED [7], SPAMCOP [9]) This can be done in the next release with a new %TESTSFAILEDWITHWEIGHTS% variable. 2) Provide a method of defeating a custom filter (zero points) based on failing a specially marked test. This will be in the next release. END instead of the weight will force the test to end. 3) Provide a method of defining a maximum and/or minimum number of points that a particular custom filter can score. A MAXWEIGHT option will be in the next release, that will allow you to define the maximum weight that the test can add. If the maximum weight is reached, processing will stop (so any negative weights would need to go at the beginning of the test), and the maximum weight will be used instead of the actual weight (IE if you have MAXWEIGHT 60, and the filter is at 55 points with a line that would add 10 points, processing would stop with a weight of 60, not 65). -Scott --- Declude JunkMail: The advanced anti-spam solution for IMail mailservers. Declude Virus: Catches known viruses and is the leader in mailserver vulnerability detection. Find out what you've been missing: Ask about our free 30-day evaluation. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Request for additional filtering functionality
THANK YOU Scott! -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of R. Scott Perry Sent: Friday, November 14, 2003 9:44 AM To: [EMAIL PROTECTED] Subject: Re: [Declude.JunkMail] Request for additional filtering functionality As I continue to look for new potential in filtering, I have repeatedly come across some limitations which restrict what can be done effectively, difficulty in figuring the scoring of some variable filters, and challenges from the additional processing power required to counterbalance some filters, so I just wanted to request three different things which appear like they might be somewhat reasonable extensions to the current environment. I'm putting these all together in one message because, at least from my perspective, they are all related, and I didn't want to bother you repeatedly with such requests. Those requests are as follows: Thanks for the suggestions. Giving the number of people that are now using filters in Declude JunkMail, and the size of them, it's about time for us to expand them a bit. LOG: 11/13/2003 20:43:02 Q331903a90080bcc8 Msg failed IPLINKED ([Score: 7] Message failed IPLINKED test (189)). Action=WARN. WARN: X-RBL-Warning: IPLINKED: [Score: 7] Message failed IPLINKED test (226) These two will be changed to use Message failed IPLINKED test (line 189, weight 7). TESTSFAILED: X-Weight: 16 (REVDNS [0], IPNOTINMX [0], IPLINKED [7], SPAMCOP [9]) This can be done in the next release with a new %TESTSFAILEDWITHWEIGHTS% variable. 2) Provide a method of defeating a custom filter (zero points) based on failing a specially marked test. This will be in the next release. END instead of the weight will force the test to end. 3) Provide a method of defining a maximum and/or minimum number of points that a particular custom filter can score. A MAXWEIGHT option will be in the next release, that will allow you to define the maximum weight that the test can add. If the maximum weight is reached, processing will stop (so any negative weights would need to go at the beginning of the test), and the maximum weight will be used instead of the actual weight (IE if you have MAXWEIGHT 60, and the filter is at 55 points with a line that would add 10 points, processing would stop with a weight of 60, not 65). -Scott --- Declude JunkMail: The advanced anti-spam solution for IMail mailservers. Declude Virus: Catches known viruses and is the leader in mailserver vulnerability detection. Find out what you've been missing: Ask about our free 30-day evaluation. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Request for additional filtering functionality
Scott, EXCELENT!!! Please note the minimum score in addition to the maximum one (I'm not sure if you got that, though it's not nearly as important). Thanks a bunch, Matt R. Scott Perry wrote: As I continue to look for new potential in filtering, I have repeatedly come across some limitations which restrict what can be done effectively, difficulty in figuring the scoring of some variable filters, and challenges from the additional processing power required to counterbalance some filters, so I just wanted to request three different things which appear like they might be somewhat reasonable extensions to the current environment. I'm putting these all together in one message because, at least from my perspective, they are all related, and I didn't want to bother you repeatedly with such requests. Those requests are as follows: Thanks for the suggestions. Giving the number of people that are now using filters in Declude JunkMail, and the size of them, it's about time for us to expand them a bit. LOG: 11/13/2003 20:43:02 Q331903a90080bcc8 Msg failed IPLINKED ([Score: 7] Message failed IPLINKED test (189)). Action=WARN. WARN: X-RBL-Warning: IPLINKED: [Score: 7] Message failed IPLINKED test (226) These two will be changed to use Message failed IPLINKED test (line 189, weight 7). TESTSFAILED: X-Weight: 16 (REVDNS [0], IPNOTINMX [0], IPLINKED [7], SPAMCOP [9]) This can be done in the next release with a new %TESTSFAILEDWITHWEIGHTS% variable. 2) Provide a method of defeating a custom filter (zero points) based on failing a specially marked test. This will be in the next release. END instead of the weight will force the test to end. 3) Provide a method of defining a maximum and/or minimum number of points that a particular custom filter can score. A MAXWEIGHT option will be in the next release, that will allow you to define the maximum weight that the test can add. If the maximum weight is reached, processing will stop (so any negative weights would need to go at the beginning of the test), and the maximum weight will be used instead of the actual weight (IE if you have MAXWEIGHT 60, and the filter is at 55 points with a line that would add 10 points, processing would stop with a weight of 60, not 65). -Scott --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Request for additional filtering functionality
Scott, Elaborating is my favorite pastime :) I mentioned the minimum score choice because while most of what we do is looking for ways to add points, sometimes we also want to subtract them in order to give credit, or alternatively, we sometimes don't want to subtract more than a certain number of points. So for the completeness of options, it made sense to include (though again, it's not nearly as useful as the maximum weight feature because of how filters are mostly used). This might have been useful for instance in my FOREIGN/TLD set, where the TLD-[Region] filters are scored in the Global.cfg as 3 points, and then for each hit (which should be unique in this case) one point is subtracted, like so: - Global.cfg - TLD-ASIAN filter C:\IMail\Declude\Filters\TLD-Asian.txt x 3 0 - TLD-Asian.txt - MAILFROM-1ENDSWITH.af HELO-1ENDSWITH.af REVDNS-1ENDSWITH.af I didn't want to score this too high because there are many cases where a reverse DNS entry is missing from a valid sender, but alternatively, I could have coded it to credit back more points then the score given the Global.cfg and upped the score of the Global.cfg like so: - Global.cfg - TLD-ASIAN filter C:\IMail\Declude\Filters\TLD-Asian.txt x 4 0 - TLD-Asian.txt - MAILFROM-1ENDSWITH.af HELO-2ENDSWITH.af REVDNS-2ENDSWITH.af So a credit of up to 5 points could be deducted, and currently that would give a score of -1 if all three hit, but I might not want to give back 5 points and limit the credit to 4 points with a MINWEIGHT -4 entry (figuring that the points in the Global.cfg would then be added to the score from within the filter). This would allow a sender with a MAILFROM and HELO, or a MAILFROM and REVDNS to net only one point, but I could add 3 points for just a MAILFROM which matched, which might be beneficial in this instance. This would be useful in a many-to-many matching system for both positive and negative scoring. I could see other uses such as pseudo whitelists which make use of negative weighting inside of the filter and may track various types of information, so this would protect from crediting back more points than was desired, but at the same time allow less credit than the total defined by the MINWEIGHT. As far as one-to-one negative weight matches go, these would benefit from the functionality where the filter is stopped from processing after reaching a certain value, thus saving processing time as you described. I don't see any immediate reason why you would need a MAXWEIGHT and MINWEIGHT in the same filter if that helps. I must admit that it's kind of hard to come up with perfect examples since I have been trying to work within the current framework, however I would imagine that over time, there would be even better uses for limiting the negative weights applied within a filter. Regardless of that, I would give my left nut just to have the MAXWEIGHT feature as you expanded on it along with the other things :) I'm pretty confident that this would increase my capacity by 25% with the addition of having the test stop on a MAXWEIGHT as well as an END. Thanks again, Matt R. Scott Perry wrote: Please note the minimum score in addition to the maximum one (I'm not sure if you got that, though it's not nearly as important). I did see that -- could you elaborate on that one a bit? --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Request for additional filtering functionality
I mentioned the minimum score choice because while most of what we do is looking for ways to add points, sometimes we also want to subtract them in order to give credit, or alternatively, we sometimes don't want to subtract more than a certain number of points. So for the completeness of options, it made sense to include (though again, it's not nearly as useful as the maximum weight feature because of how filters are mostly used). The MINWEIGHT option will be added, too. :) Still working on Kami's request for a CASH option. I must admit that it's kind of hard to come up with perfect examples since I have been trying to work within the current framework, however I would imagine that over time, there would be even better uses for limiting the negative weights applied within a filter. That is very often the case. -Scott --- Declude JunkMail: The advanced anti-spam solution for IMail mailservers. Declude Virus: Catches known viruses and is the leader in mailserver vulnerability detection. Find out what you've been missing: Ask about our free 30-day evaluation. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Request for additional filtering functionality
The MINWEIGHT option will be added, too. :) EXCELENT!!! Thanks again. It should be easy to modify the filters to work more effectively under the new features, and my eye strain will subside with the addition of weights in the headers and logs. BTW, another thought...because some of us parse our logs or captured E-mail, it might make it easier to separate the score out if you put it before the line of the filter that is failed since that line will be widely variable. We might want to know for instance how many times a filter assessed points instead of how many times it was hit with or without points. It follows that it would be more difficult to parse/search for weight: Message failed IPLINKED test (line 189, weight 7) Message failed IPLINKED test (line 189, weight 0) than it would be to parse/search for weight: Message failed IPLINKED test (weight 7, line 189) Message failed IPLINKED test (weight 0, line 189) or move it somewhere else all together for those that like to parse the line score as well. Maybe Bill, another power grep-er, or one of the log file analysis guys could suggest the best implementation from their perspectives. I only recommended it as the first entry so it would be easier to spot in my own config which now makes use of WARN actions and since it wouldn't change the format of the line failed portion, but I could deal with almost anything. It isn't of much value to know if something scored 0 points unless you are wondering if a filter actually processed the message, so having a readily available method to search your logs for a combination of filter and weight with a simple text search would be useful. I couldn't do that with a standard text editor if the weight followed the line. Still working on Kami's request for a CASH option. We could turn this into a me too thread :) Matt --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
