[sniffer] Re: [sniffer][Fwd: Re: [sniffer]FP suggestions]
Thunderbird and Netscape just takes the full original source and attaches it as a message/rfc822 attachment. I forwarded this message back to the list by just pressing Forward. Interesting that they include the headers with a simple forward, without specifying forward as attachment. I haven't ever seen that behaviour before in a mail client. Seems like a few forwards would create a very bloated message with all of the old headers. I'm pretty sure that Outlook Express works simply by just pressing Forward As Attachment, or at least it gives me enough of the original, including the full headers, to determine how to block the spam. Yes it does. However you've missed the point. The issue is not how to get the headers. It is how to keep an email client from encoding the message and headers differently, so that Sniffer can properly identify the rule that caught the message. Please excuse me for wanting more detail about the Outlook attachment trick, but would you mind attaching this message to a response so that I could look at the headers and such? Sorry, I don't use Outlook. But I can tell you the steps to take in Outlook 2003 (other versions are almost exactly the same). I have my Outlook users follow these with no problem. 1. Create a new email message 2. Click the arrow beside the paperclip icon, select item instead of file from the dropdown 3. Browse mailboxes from the popup dialog to select the message to attach. 4. Viola, original message and headers attached. There was a discussion about Outlook's behavior with Scott some time ago. Apparently Microsoft was pressured by customers to remove headers when forwarding because they felt that they were a security/privacy risk. No one told them that Outlook was a security/privacy risk on it's own :) ...but that's another story. I would probably feel different if I had the need for groupware though, but digs at Microsoft are irresistible sometimes. I don't remember that discussion, and am not sure we're talking about the same thing. If you attach the original message via the steps above, you get the full original message, headers and body. We have a number of customers who send spam reports this way, mostly on Outlook 2002 and 2003. Darin # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: [sniffer][Fwd: Re: [sniffer]FP suggestions]
Darin, Thunderbird allows you to choose the default forwarding method as either inline or as attachment. It might actually default to inline, I can't remember, but whenever it does message/rfc822 attachments, it is as a whole unlike some other clients that edit it down to the bare minimum of what the consider to be useful like addressing, subject date and MIME stuff if appropriate. I'm definitely guilty of being a Netscape diehard, and I'm very happy that the Mozilla project brought things back to life again. I fully understand the attachment trick with Outlook thanks to the confirmations. This will be easier than having people cut and paste the headers in. This doesn't happen much, but there is nothing worse than getting a spam report without header info. I also understand the encoding issues with forwarding in Outlook/OE. It's a shame that this happens. Maybe having a copy of Thunderbird around for this purpose might fit in where this is an issue. Sounds like adding Sniffer headers would be the best solution for this issue on a wider basis since you definitely can't convince every admin not to submit using Outlook/OE. Soon I'm going to code up my Sniffer FP reports to be automatically triggered when a message is reprocessed from my spam review system, so I won't have to even bother with the source any more. That should only take a couple of hours, and it would be time well spent. I always fix issues and whitelist locally where appropriate, but I also report to Sniffer for the benefit of all in addition to making sure that a FP rule will not tag something outside of the scope of what I whitelisted, and I have to report in order to be able to see what the content of the rule was. Customers do most of the reprocessing now, I just do the back end stuff. Matt Darin Cox wrote: Thunderbird and Netscape just takes the full original source and attaches it as a message/rfc822 attachment. I forwarded this message back to the list by just pressing "Forward". Interesting that they include the headers with a simple forward, without specifying forward as attachment. I haven't ever seen that behaviour before in a mail client. Seems like a few forwards would create a very bloated message with all of the old headers. I'm pretty sure that Outlook Express works simply by just pressing Forward As Attachment, or at least it gives me enough of the original, including the full headers, to determine how to block the spam. Yes it does. However you've missed the point. The issue is not how to get the headers. It is how to keep an email client from encoding the message and headers differently, so that Sniffer can properly identify the rule that caught the message. Please excuse me for wanting more detail about the Outlook attachment trick, but would you mind attaching this message to a response so that I could look at the headers and such? Sorry, I don't use Outlook. But I can tell you the steps to take in Outlook 2003 (other versions are almost exactly the same). I have my Outlook users follow these with no problem. 1. Create a new email message 2. Click the arrow beside the paperclip icon, select item instead of file from the dropdown 3. Browse mailboxes from the popup dialog to select the message to attach. 4. Viola, original message and headers attached. There was a discussion about Outlook's behavior with Scott some time ago. Apparently Microsoft was pressured by customers to remove headers when forwarding because they felt that they were a security/privacy risk. No one told them that Outlook was a security/privacy risk on it's own :) ...but that's another story. I would probably feel different if I had the need for groupware though, but digs at Microsoft are irresistible sometimes. I don't remember that discussion, and am not sure we're talking about the same thing. If you attach the original message via the steps above, you get the full original message, headers and body. We have a number of customers who send spam reports this way, mostly on Outlook 2002 and 2003. Darin # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: [sniffer][Fwd: Re: [sniffer]FP suggestions]
Hello Andrew, Thursday, June 8, 2006, 11:32:47 AM, you wrote: Ditto. I advise people to use Insert, Item. Far easier than explaining how to drag and drop (or tie shoelaces). It might be nice to have a SnagIt of that process to share w/ users. I've noticed that whether the headers survive when they are sent to another Exchange+Outlook company are a crap shoot. Generally speaking, if the message is handled by Outlook, it's not the same message anymore. For example, a BASE64 encoded message becomes plain text, and attached graphics don't show up at all in the View Source version. I just had an interesting FP case like this. By the time the match record got to me along with what was supposed to be the original message, there were at least 9K bytes missing - including the bytes that presumably contained the rule match. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
Re: [sniffer]FP suggestions
The one issue with this I have is 1) Forward full original source to Sniffer with license code. If we could do it without the license code, it would be much easier to automate on our end. I already have a process in place to copy and reroute false positives by rewriting the Q file. I'm hesitant to alter the message itself to add the license code. If we could authenticate the FP report via some other means it would help greatly. How about connecting IP instead? Darin. - Original Message - From: Matt To: Message Sniffer Community Sent: Wednesday, June 07, 2006 12:59 AM Subject: Re: [sniffer]FP suggestions Pete,Regarding suggestions for easing the reporting process, I would recommend the following possible modifications: 1) An E-mail submission tool similar to the one now, but replies would be automated2) Send back links or rather an HTML form with checkboxes in an E-mail auto-response allowing one to block rules.3) Make blocked rules automatic for the submitter, but throw them into a queue for manual review by Sniffer folk in order to determine whether the blocks should become applied to all rulebases.4) Have automatic triggers that lower rule strengths based on users blocking rules regardless of direct Sniffer action.The gist of this is to make it more point and click. The fact that you need full source is cumbersome, so the above recommendations seek ways to make the process easier for both the customer and for Sniffer while dealing with the need to send the full source. No direct customer interaction would be necessary in most cases, and you would have a queue full of items to review and make a determination about that customers have preened for you. To the customer, the process would look like the following: 1) Forward full original source to Sniffer with license code.2) Seconds later there would be an automated reply received in HTML format with a check box for every rule failed (or note that no active rules were found), a text box for optional comments, and submit button.3) Customer checks the boxes for the rules he wants to block, adds notes in a text field if they feel like it, and they press submit. End of story.You could also add a Web interface for this if you wanted to, but E-mail seems the most appropriate for most.I don't think it would be beneficial to rehash a lot of things involving how FP's occur, at least on this list. I know from my system where my customers have single-click reprocessing capability, that they miss about 97% of all FP's either because they don't bother to do review, or they don't bother to reprocess anything but personal E-mail that may get blocked. I would imagine that Sniffer sees a similar rate of customer reported FP's due in part to the difficulty, and in part for the same reasons that relate to my own users.The three biggest sources of false positives are obscure foreign domains/IP's, rules generated from bulk mailings that are too broadly targeted, and things reported to Sniffer that are advertising, but not spam. All three of these things are difficult and time consuming to deal with, particularly the last two. Here's some stats for Sniffer FP's on my system going back about 15 months: SNIFFER-GENERAL 283SNIFFER-EXPERIMENTAL 167 * Excluded 79 FP's from bad rule event on 1/17 - 1/18/2006SNIFFER-IP 61SNIFFER-PHISHING 52SNIFFER-GETRICH 29 * Excluded 115 FP's from bad rule event on 4/18 - 4/19/2006SNIFFER-PHARMACY 25SNIFFER-PORN 24SNIFFER-TRAVEL 13SNIFFER-INSURANCE 7SNIFFER-OBFUSCATION 6SNIFFER-DEBT 6SNIFFER-MALWARE 4SNIFFER-AVSOFT 3SNIFFER-CASINO 2SNIFFER-INK 1SNIFFER-MEDIA 1SNIFFER-SPAMWARE 0It is quite notable how high the FP's are with SNIFFER-GENERAL which is where most bulk-mailers and customer reported spam rules are tagged. This is also what my numbers show even though my customers are much less likely to reprocess bulk mail, and of course they only reprocess a small fraction of my overall FP's. This is almost all customer reported stuff. I score SNIFFER-GENERAL at 53% of my Hold weight. SNIFFER-IP is another standout. I only score SNIFFER-IP at 38% of my Hold weight and it hits less than 2% of all Sniffer hits, yet it scored comparably high so that is worth noting. The FP rate on SNIFFER-IP hasn't really changed since you made adjustments. SNIFFER-EXPERIMENTAL is a top category that caught a lot of zombie spam which is important to many systems, but it did seem to have a high FP rate. SNIFFER-PHISHING was worse for me until around January or February. It seemed to have a lot of FP's on security related newsletters and chain letters. I have mixed feelings about those things. Maybe more efforts on white rules would help with that stuff, and I'm not totally sure if it is appropriate to block chain letters even though I detest this stuff myself.Most FP's do
[sniffer]Re[2]: [sniffer]FP suggestions
Hello Darin, Wednesday, June 7, 2006, 7:31:29 AM, you wrote: The one issue with this I have is 1) Forward full original source to Sniffer with license code. If we could do it without the license code, it would be much easier to automate on our end. I already have a process in place to copy and reroute false positives by rewriting the Q file. I'm hesitant to alter the message itself to add the license code. If we could authenticate the FP report via some other means it would help greatly. How about connecting IP instead? At the moment that is how it's done: a combination of email address and source IP are matched with the license ID. The reason we ask for the license ID is because folks submitting false positives occasionally forget that we authenticate on their registered email address and use some other address. -- The rule is that if the system can't match the email address it should/may drop the message rather than evaluating it. We get a lot of spam and attempts to game the system at our false@ address... so when it's heavy we do drop messages that can't be properly identified. However, in an effort to provide the best service possible, if the license ID is present and we have the time we will look to see if it could be a legit FP submission by researching the source and domain - and if we think it is likely to be legitimate we will process the FP and respond with an additional code reminding the submitter that they must use their registered email address or an authorized alias. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
Re: [sniffer]Re[2]: [sniffer]FP suggestions
Hi Pete, Can I interpret this as email address and matching source IP are sufficient if the correct email address is used to submit? If not, do you have any suggestions on how you would like to see us inserting the license ID in the D file? Darin. - Original Message - From: Pete McNeil [EMAIL PROTECTED] To: Message Sniffer Community sniffer@sortmonster.com Sent: Wednesday, June 07, 2006 8:25 AM Subject: [sniffer]Re[2]: [sniffer]FP suggestions Hello Darin, Wednesday, June 7, 2006, 7:31:29 AM, you wrote: The one issue with this I have is 1) Forward full original source to Sniffer with license code. If we could do it without the license code, it would be much easier to automate on our end. I already have a process in place to copy and reroute false positives by rewriting the Q file. I'm hesitant to alter the message itself to add the license code. If we could authenticate the FP report via some other means it would help greatly. How about connecting IP instead? At the moment that is how it's done: a combination of email address and source IP are matched with the license ID. The reason we ask for the license ID is because folks submitting false positives occasionally forget that we authenticate on their registered email address and use some other address. -- The rule is that if the system can't match the email address it should/may drop the message rather than evaluating it. We get a lot of spam and attempts to game the system at our false@ address... so when it's heavy we do drop messages that can't be properly identified. However, in an effort to provide the best service possible, if the license ID is present and we have the time we will look to see if it could be a legit FP submission by researching the source and domain - and if we think it is likely to be legitimate we will process the FP and respond with an additional code reminding the submitter that they must use their registered email address or an authorized alias. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer]Re[2]: [sniffer]Re[2]: [sniffer]FP suggestions
Hello Darin, Wednesday, June 7, 2006, 8:44:26 AM, you wrote: Hi Pete, Can I interpret this as email address and matching source IP are sufficient if the correct email address is used to submit? Yes. If not, do you have any suggestions on how you would like to see us inserting the license ID in the D file? To clarify, nothing should be inserted in the D file. The original message should be attached as an RFC 822 attachment is as close to the original form as possible. The license id, if included at all, should be in the subject line of the submission message. Remember also, we WILL be responding to the submission message so that we can record a dialogue with you about the false positive in question. Hope this helps, Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer]Re[2]: [sniffer]FP suggestions
Hello Scott, Wednesday, June 7, 2006, 10:08:58 AM, you wrote: For me the pain of false positives submissions is the research that happens when I get a no rule found return. I then need to find the queue-id of the original message and then find the appropriate Sniffer log and pull out the log lines from there and then submit it. Almost always in these cases, a rule is removed. If this process could be improved that would really be a time saver. This depends on the email system you are using. On some systems (MDaemon, and postfix, for example) X- headers from SNF can be emitted into the message. When we see these we can identify the rules directly without asking for the extra research. It would be nice if Declude would offer a mechanism to pick up the optional .xhdr file SNF can generate and include it in the X headers that it already adds to the message. I know this begs the question, why not have SNF add the headers for SmarterMail and IMail platforms, and the reason is that it would require writing an additional copy of the message to disk. Since these systems tend to be io bound already (Declude/IMail anyhow) the performance penalty would be prohibitive. If Declude picks up .xhdr from SNF directly then it can be included in the ONE rewrite Declude makes anyway. I've asked them about this and other improved integration opportunities for a while now (many months), and I get favorable responses, but no action so far. I guess we will see :-) _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
Re: [sniffer]Re[2]: [sniffer]FP suggestions
Pete, An X-Header would be very, very nice to have. I understand the issues related to waiting to see if something comes through, and because of that, I would maybe suggest moving on your own. Sniffer doesn't need to be run on every single message in a Declude system. Through weight based skipping, many administrators (especially the ones that could make the most use of this) could skip processing Sniffer once a certain weight is reached, and in turn that would save enough load that it should easily make up for needing to re-write the message to the disk with the modified headers. On external tests that allow for weight skipping on my system, I was skipping around 50% of messages before lightening the load with pre-scanning. Sniffer could do weight skipping with Declude by accepting the %WEIGHT% variable in the command line. SNIFFER-IP external 063 "C:\IMail\Declude\Sniffer\customer-code.exe license-code WH=26 WL=-5 CW=%WEIGHT%" 5 0 ...etc. The WH setting says don't run if equal to or greater than, the WL says don't run if equal to or less than, and the CW passes in the weight from Declude at the time of calling Sniffer. It still launches Sniffer, but it could be stopped immediately before any heavy lifting is done. The best solution of course would be for Declude to allow for weight-based skipping in the config without calling the executable, but I started asking about that back in the Scott days and I am not holding out hope for that happening soon considering. The most realistic option would seem to then have Sniffer do the heavy lifting of rewriting itself, and save some CPU and disk I/O by improving efficiencies with something as simple as weight-based skipping. I'm pretty sure the net result would be less CPU and disk I/O overall if both were done. Another alternative may be to create a separate executable (with weight-based skipping) that would only deal with adding headers from the text file that Sniffer drops in the directory. There would be less benefit overall to keeping this all in one app, but it would target the primary need. This could easily be written by one of us in _vbscript_ as a proof of concept. I have considered doing this before, but it isn't at the top of my priorities. BTW, you could maybe even encode links in the headers for FP reporting through a Web interface, completely removing the forwarding mechanism from the mix, though you wouldn't have the opportunity to see the messages which may not be good as a whole. Matt Pete McNeil wrote: Hello Scott, Wednesday, June 7, 2006, 10:08:58 AM, you wrote: For me the pain of false positives submissions is the research that happens when I get a "no rule found" return. I then need to find the queue-id of the original message and then find the appropriate Sniffer log and pull out the log lines from there and then submit it. Almost always in these cases, a rule is removed. If this process could be improved that would really be a time saver. This depends on the email system you are using. On some systems (MDaemon, and postfix, for example) X- headers from SNF can be emitted into the message. When we see these we can identify the rules directly without asking for the extra research. It would be nice if Declude would offer a mechanism to pick up the optional .xhdr file SNF can generate and include it in the X headers that it already adds to the message. I know this begs the question, why not have SNF add the headers for SmarterMail and IMail platforms, and the reason is that it would require writing an additional copy of the message to disk. Since these systems tend to be io bound already (Declude/IMail anyhow) the performance penalty would be prohibitive. If Declude picks up .xhdr from SNF directly then it can be included in the ONE rewrite Declude makes anyway. I've asked them about this and other improved integration opportunities for a while now (many months), and I get favorable responses, but no action so far. I guess we will see :-) _M
[sniffer]Re[2]: [sniffer]Re[2]: [sniffer]FP suggestions
Hello Matt, Wednesday, June 7, 2006, 3:37:36 PM, you wrote: Pete, An X-Header would be very, very nice to have. I understand the issues related to waiting to see if something comes through, and because of that, I would maybe suggest moving on your own. I've got it on the list to have a message rewriting option... it's just not as high as some others. I hadn't thought about the weight gating utility - though that seems like something that would be useful in general for external tests... weightgate -5 %WEIGHT% 20 command line to run 5 0 command line to run is executed if %WEIGHT% is in the range [-5,20] and the exit code of command line to run is returned. That seems like a pretty simple utility to knock out - perhaps I will ;-) Also, on the FP reporting links idea, that would break the process - it's important for us to see the message for many reasons, and it's important for the FP resolution process to be interactive. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
Re: [sniffer]FP suggestions
Oh, I assumed the rule had been removed. Are you saying there was a rule in place, but the FP processing somehow failed to find it? If so, I'd say that is a major failing on the part of the FP processing. There's no way thatwe can find time to go through the Sniffer logs after this bounces back with "no rule found". This would have to be automated to have any chance of occurring, but again I would say the FP processing needs to be corrected to identify the rule the message failed since the complete message, headers and body, are included in the report. Darin. - Original Message - From: Scott Fisher To: Message Sniffer Community Sent: Wednesday, June 07, 2006 10:08 AM Subject: Re: [sniffer]FP suggestions For me the pain of false positives submissions is the research that happens when I get a "no rule found" return. I then need to find the queue-id of the original message and then find the appropriate Sniffer log and pull out the log lines from there and then submit it. Almost always in these cases, a rule is removed. If this process could be improved that would really be a time saver.
[sniffer]Re[2]: [sniffer]Re[2]: [sniffer]Re[2]: [sniffer]FP suggestions
Hello Matt, Wednesday, June 7, 2006, 4:22:05 PM, you wrote: Pete, Since the %WEIGHT% variable is added by Declude, it might make sense to have a qualifier instead of making the values space delimited. I don't want to mix delimiters... everything so far is using spaces, so it makes sense to continue that way IMO. Errors in Declude could cause values to not be inserted, and not everyone will want to skip at a low weight. I haven't seen any bugs with %WEIGHT% since shortly after it was introduced, but you never know. I have seen some issues with other Declude inserted variables though. Well, errors are always a possibility, but in this case it _should_ be reasonably safe. For example, if this is used to gate SNF, then a missing %WEIGHT% would result in trying to launch a program with the same name as the authentication string, and it is highly unlikely that would be found, so the result would be the program not found error code. That's not perfect because it's a nonzero result, but it is safe in that it is not likely to launch another program. One other thing that I came across with the way that Declude calls external apps...you can't delimit the data with things like quotes. There is no mechanism for escaping a functional quote from a quote that should appear in the data that you pass to it...so don't use quotes as delimiters :) Not a problem... I just whipped together a utility called WeightGate.exe that can be downloaded here (for now): http://www.messagesniffer.com/Tools/WeightGate.exe Suppose you wanted to use it in Declude to skip running SNF if your weight was already ridiculously low (perhaps white listed) or already so high that you want to save the extra cycles. Then you might do something like this: SNF external nonzero c:\tool\WeightGate.exe -50 %WEIGHT% 30 c:\SNF\sniffer.exe authenticationxx 10 0 (hopefully that didn't wrap, and if it did you will know what I meant ;-) To test this concept out you might first create a copy of WeightGate.exe callled ShowMe.exe (case matters!) and then do something like this: SNF external nonzero c:\tool\ShowMe.exe -50 %WEIGHT% 30 c:\SNF\sniffer.exe authenticationxx 10 0 The result of that would be the creation of a file c:\ShowMe.log that contained all of the parameters ShowMe.exe was called with -- that way you wouldn't have to guess if it was correct. ShowMe.exe ALWAYS returns zero, so this _should_ be safe ;-) If you run WeightGate on the command line without parameters it will tell you all about itself and it's alter ego ShowMe.exe. That description goes like this (I may fix the typo(s) later): WeightGate.exe (C) 2006 ARM Research Labs, LLC. This program is distributed AS-IS, with no warranty of any kind. You are welcome to use this program on your own systems or those that you directly support. Please do not redistribute this program except as noted above, however feel free to recommend this program to others if you wish and direct them to our web site where they can download it for themselves. Thanks! www.armresearch.com. This program is most commonly used to control the activation of external test programs from within Declude (www.declude.com) based on the weigth that has been calculated thus far for a given message. As an added feature, if you rename this program to ShowMe.exe then it will emit all of the command line arguments as it sees them to a file called c:\ShowMe.log so that you can use it as a debugging aid. If you are seeing this message, you have used this program incorrectly. The correct invocation for this program is: WeightGate low weight hight program arg 1, arg 2,... arg n Where: low = a number representing the lowest weight to run progra. weight = a number representing the actual weight to evaluate. high = a number representing the highest weight to run program. program = the program to be activated if weight is in range. arg 1, arg 2, ... arg n = arguments for program. If weight is in the range [low,high] then WeightGate will run program and pass all of arg 1, arg 2,... arg n to it. Then WeightGate will collect the exit code of program and return it as WeightGate's exit code. If WeightGate gets the wrong number of parameters it will display this message and return FAIL_SAFE (zero) as it's exit code. If weight is not in range (less than low or greater than high) then WeightGate will NOT launch program and will return FAIL_SAFE (zero) as it's exit code. As a deubgging aid, I was called with the following arguments: arg[0] me = WeightGate -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer]Re[2]: [sniffer]Re[2]: [sniffer]Re[2]: [sniffer]FP suggestions
Hello Darin, Wednesday, June 7, 2006, 5:05:28 PM, you wrote: snip/ Uh, but the D file contains mime segments corresponding to attachments. That's ok. SNF looks inside those, and w/ the FP scanning software inside the rfc822 atachment also. It's not perfect, but the majority of the time it does pick out the rules that match and having the original helps us put those into context. The license id, if included at all, should be in the subject line of the submission message. Good. Subject line is easier and more reliable to parse out. Not that it's needed per the original question. :-) -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
Re: [sniffer]Re[2]: [sniffer]Re[2]: [sniffer]Re[2]: [sniffer]FP suggestions
(sniff) Aw, cut it out, Matt. You're making me all weepy. p.s. Pete, that's pretty darned amazing! From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of MattSent: Wednesday, June 07, 2006 3:58 PMTo: Message Sniffer CommunitySubject: Re: [sniffer]Re[2]: [sniffer]Re[2]: [sniffer]Re[2]: [sniffer]FP suggestions Pete,I think that you just broke Scott's record with his two hour feature request with your own a two hour program :)Anyone remember those days???Thanks,MattPete McNeil wrote: Hello Matt, Wednesday, June 7, 2006, 4:22:05 PM, you wrote: Pete, Since the %WEIGHT% variable is added by Declude, it might make sense to have a qualifier instead of making the values space delimited. I don't want to mix delimiters... everything so far is using spaces, so it makes sense to continue that way IMO. Errors in Declude could cause values to not be inserted, and not everyone will want to skip at a low weight. I haven't seen any bugs with %WEIGHT% since shortly after it was introduced, but you never know. I have seen some issues with other Declude inserted variables though. Well, errors are always a possibility, but in this case it _should_ be reasonably safe. For example, if this is used to gate SNF, then a missing %WEIGHT% would result in trying to launch a program with the same name as the authentication string, and it is highly unlikely that would be found, so the result would be the "program not found" error code. That's not perfect because it's a nonzero result, but it is safe in that it is not likely to launch another program. One other thing that I came across with the way that Declude calls external apps...you can't delimit the data with things like quotes. There is no mechanism for escaping a functional quote from a quote that should appear in the data that you pass to it...so don't use quotes as delimiters :) Not a problem... I just whipped together a utility called WeightGate.exe that can be downloaded here (for now): http://www.messagesniffer.com/Tools/WeightGate.exe Suppose you wanted to use it in Declude to skip running SNF if your weight was already ridiculously low (perhaps white listed) or already so high that you want to save the extra cycles. Then you might do something like this: SNF external nonzero "c:\tool\WeightGate.exe -50 %WEIGHT% 30 c:\SNF\sniffer.exe authenticationxx" 10 0 (hopefully that didn't wrap, and if it did you will know what I meant ;-) To test this concept out you might first create a copy of WeightGate.exe callled ShowMe.exe (case matters!) and then do something like this: SNF external nonzero "c:\tool\ShowMe.exe -50 %WEIGHT% 30 c:\SNF\sniffer.exe authenticationxx" 10 0 The result of that would be the creation of a file c:\ShowMe.log that contained all of the parameters ShowMe.exe was called with -- that way you wouldn't have to guess if it was correct. ShowMe.exe ALWAYS returns zero, so this _should_ be safe ;-) If you run WeightGate on the command line without parameters it will tell you all about itself and it's alter ego ShowMe.exe. That description goes like this (I may fix the typo(s) later): WeightGate.exe (C) 2006 ARM Research Labs, LLC. This program is distributed AS-IS, with no warranty of any kind. You are welcome to use this program on your own systems or those that you directly support. Please do not redistribute this program except as noted above, however feel free to recommend this program to others if you wish and direct them to our web site where they can download it for themselves. Thanks! www.armresearch.com. This program is most commonly used to control the activation of external test programs from within Declude (www.declude.com) based on the weigth that has been calculated thus far for a given message. As an added feature, if you rename this program to ShowMe.exe then it will emit all of the command line arguments as it sees them to a file called c:\ShowMe.log so that you can use it as a debugging aid. If you are seeing this message, you have used this program incorrectly. The correct invocation for this program is: WeightGate low weight hight program arg 1, arg 2,... arg n Where: low = a number representing the lowest weight to run progra. weight = a number representing the actual weight to evaluate. high = a number representing the highest weight to run program. program = the program to be activated if weight is in range. arg 1, arg 2, ... arg n = arguments for program. If weight is in the range [low,high] then WeightGate will run program and pass all of arg 1, arg 2,... arg n to it. Then WeightGate will collect the exit code of program and return it as WeightGate's exit code. If WeightGate gets the wrong number of parameters it will display this message and return FAIL_SAFE (zero) as it's exit code. If weight is not in range (less than low or greater than high) then WeightGate will NOT
Re: [sniffer]Re[2]: [sniffer]Re[2]: [sniffer]Re[2]: [sniffer]FP suggestions
Awesome. Great job, Pete. Darin. - Original Message - From: Pete McNeil [EMAIL PROTECTED] To: Message Sniffer Community sniffer@sortmonster.com Sent: Wednesday, June 07, 2006 6:49 PM Subject: [sniffer]Re[2]: [sniffer]Re[2]: [sniffer]Re[2]: [sniffer]FP suggestions Hello Matt, Wednesday, June 7, 2006, 4:22:05 PM, you wrote: Pete, Since the %WEIGHT% variable is added by Declude, it might make sense to have a qualifier instead of making the values space delimited. I don't want to mix delimiters... everything so far is using spaces, so it makes sense to continue that way IMO. Errors in Declude could cause values to not be inserted, and not everyone will want to skip at a low weight. I haven't seen any bugs with %WEIGHT% since shortly after it was introduced, but you never know. I have seen some issues with other Declude inserted variables though. Well, errors are always a possibility, but in this case it _should_ be reasonably safe. For example, if this is used to gate SNF, then a missing %WEIGHT% would result in trying to launch a program with the same name as the authentication string, and it is highly unlikely that would be found, so the result would be the program not found error code. That's not perfect because it's a nonzero result, but it is safe in that it is not likely to launch another program. One other thing that I came across with the way that Declude calls external apps...you can't delimit the data with things like quotes. There is no mechanism for escaping a functional quote from a quote that should appear in the data that you pass to it...so don't use quotes as delimiters :) Not a problem... I just whipped together a utility called WeightGate.exe that can be downloaded here (for now): http://www.messagesniffer.com/Tools/WeightGate.exe Suppose you wanted to use it in Declude to skip running SNF if your weight was already ridiculously low (perhaps white listed) or already so high that you want to save the extra cycles. Then you might do something like this: SNF external nonzero c:\tool\WeightGate.exe -50 %WEIGHT% 30 c:\SNF\sniffer.exe authenticationxx 10 0 (hopefully that didn't wrap, and if it did you will know what I meant ;-) To test this concept out you might first create a copy of WeightGate.exe callled ShowMe.exe (case matters!) and then do something like this: SNF external nonzero c:\tool\ShowMe.exe -50 %WEIGHT% 30 c:\SNF\sniffer.exe authenticationxx 10 0 The result of that would be the creation of a file c:\ShowMe.log that contained all of the parameters ShowMe.exe was called with -- that way you wouldn't have to guess if it was correct. ShowMe.exe ALWAYS returns zero, so this _should_ be safe ;-) If you run WeightGate on the command line without parameters it will tell you all about itself and it's alter ego ShowMe.exe. That description goes like this (I may fix the typo(s) later): WeightGate.exe (C) 2006 ARM Research Labs, LLC. This program is distributed AS-IS, with no warranty of any kind. You are welcome to use this program on your own systems or those that you directly support. Please do not redistribute this program except as noted above, however feel free to recommend this program to others if you wish and direct them to our web site where they can download it for themselves. Thanks! www.armresearch.com. This program is most commonly used to control the activation of external test programs from within Declude (www.declude.com) based on the weigth that has been calculated thus far for a given message. As an added feature, if you rename this program to ShowMe.exe then it will emit all of the command line arguments as it sees them to a file called c:\ShowMe.log so that you can use it as a debugging aid. If you are seeing this message, you have used this program incorrectly. The correct invocation for this program is: WeightGate low weight hight program arg 1, arg 2,... arg n Where: low = a number representing the lowest weight to run progra. weight = a number representing the actual weight to evaluate. high = a number representing the highest weight to run program. program = the program to be activated if weight is in range. arg 1, arg 2, ... arg n = arguments for program. If weight is in the range [low,high] then WeightGate will run program and pass all of arg 1, arg 2,... arg n to it. Then WeightGate will collect the exit code of program and return it as WeightGate's exit code. If WeightGate gets the wrong number of parameters it will display this message and return FAIL_SAFE (zero) as it's exit code. If weight is not in range (less than low or greater than high) then WeightGate will NOT launch program and will return FAIL_SAFE (zero) as it's exit code. As a deubgging aid, I was called with the following arguments: arg[0] me = WeightGate -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed
Re: [sniffer]Re[2]: [sniffer]FP suggestions
Unfortunately, by the time the message gets to us it is sometimes just different enough that the original pattern cannot be found. There are some folks who consistently have success, and some who occasionally have problems, and a few who always have a problem. Different in what way? Is the mail client encoding differently in the forwarding process? If so, do you know what clients are altering the messages and how? If there's one that's better for this, we could always use it for forwarding since we currently send it to ourselves first, then forward. If we rewrite the Q file and queue directly from IMail, encoding shouldn't change, correct? If that avoids this issue, we could do that instead. The best solution is to include the headers during the scan since they will travel with the message. What do you mean? The XHDR? We would love that for more several reasons, but Declude is not the same company anymore. The next best is to automate matching the log entries with the message so they can be included with the submission (some do this to prevent the second trip). Yeah, we'd have to automate it. I can't imagine taking the time to manually match for each occurrence of no rule found. Another item for the automation list. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer]Re[2]: [sniffer]Re[2]: [sniffer]FP suggestions
Hello Darin, Wednesday, June 7, 2006, 7:26:48 PM, you wrote: Unfortunately, by the time the message gets to us it is sometimes just different enough that the original pattern cannot be found. There are some folks who consistently have success, and some who occasionally have problems, and a few who always have a problem. Different in what way? Is the mail client encoding differently in the forwarding process? If so, do you know what clients are altering the messages and how? If there's one that's better for this, we could always use it for forwarding since we currently send it to ourselves first, then forward. It is unclear - we receive FPs that have traveled through all sorts of clients, quarantine systems, changed hands various numbers of times, or not (to all of those)... Right now I don't want to make that research project a high priority. If we rewrite the Q file and queue directly from IMail, encoding shouldn't change, correct? If that avoids this issue, we could do that instead. That's true it wouldn't change, but submitting the message directly would not be correct - the dialogue is with you, and in any case, additional trips through the mail server also modify parts of the header and sometimes parts of the message (tag lines, disclaimers, etc)... The best solution is to include the headers during the scan since they will travel with the message. What do you mean? The XHDR? We would love that for more several reasons, but Declude is not the same company anymore. At some point perhaps they will include the SNF engine in DLL form and all of these issues will become simpler. For now there's no definitive answer on that possibility so we will have to find other solutions. I don't like the idea of rewriting the message file more often than absolutely necessary, but that is a feature that is on the todo list and so it may make it into the next heavy update (work in progress). The next best is to automate matching the log entries with the message so they can be included with the submission (some do this to prevent the second trip). Yeah, we'd have to automate it. I can't imagine taking the time to manually match for each occurrence of no rule found. Another item for the automation list. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
Re: [sniffer]FP suggestions
Of course I'm sending the full message as an attachment. You can do that with Outlook byattaching and item, then browsing your mail folders for the message to attach. And yes, that's how you do it with Outlook Express as well. I don't use Thunderbird or Netscape mail, but I would assume you still need to attach the original message to avoid the headers being lost. What I was referring to was a little more involved than that... namely the possibility of it not matching a rule because the attachment was encoded differently. For example, I've seen mail go throughthat baes64 encoded an attached email that was not originally base64 encoded. From Pete's responses, it sounded like "no rule found" really did mean no rule was matched. Especially since he has a separate code for "rule already removed". FPs we send are always from same day, or, at the very least, within 24 hours. Darin. - Original Message - From: Matt To: Message Sniffer Community Sent: Wednesday, June 07, 2006 11:46 PM Subject: Re: [sniffer]FP suggestions Darin,Outlook will strip many of the headers when forwarding. Outlook Express needs to forward the messages using "Forward As Attachment" in order to insert the full original headers. Thunderbird/Netscape Mail will work just by forwarding. If you paste the full source in a message, you should send as plain text.I have many FP's that come back as having no rules found, but these are more likely to be from rules that were already removed. So I wouldn't jump to a conclusion that the rule was not found because of formatting unless you are not sending the full unadulterated original message source. I would imagine that it would mostly be IP rules that aren't found when not forwarding the full original source.MattDarin Cox wrote: It is unclear - we receive FPs that have traveled through all sorts of clients, quarantine systems, changed hands various numbers of times, or not (to all of those)... Right now I don't want to make that research project a high priority. Understood. That's true it wouldn't change, but submitting the message directly would not be correct - the dialogue is with you, and in any case, additional trips through the mail server also modify parts of the header and sometimes parts of the message (tag lines, disclaimers, etc)... Hmmm... with attaching the original message, I guess it still makes more sense to deliver to us first for now. Just looking for an alternative that gets you the message as close as possible to the original form as possible. Maybe we'll write a script to copy and forward the D*.SMD file as an attachment to you for FPs at some point in the future. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer][Fwd: Re: [sniffer]FP suggestions]
Darin, Thunderbird and Netscape just takes the full original source and attaches it as a message/rfc822 attachment. I forwarded this message back to the list by just pressing Forward. I'm pretty sure that Outlook Express works simply by just pressing Forward As Attachment, or at least it gives me enough of the original, including the full headers, to determine how to block the spam. I have been telling Outlook users to copy and paste the headers into a forwarded message. Please excuse me for wanting more detail about the Outlook attachment trick, but would you mind attaching this message to a response so that I could look at the headers and such? There was a discussion about Outlook's behavior with Scott some time ago. Apparently Microsoft was pressured by customers to remove headers when forwarding because they felt that they were a security/privacy risk. No one told them that Outlook was a security/privacy risk on it's own :) ...but that's another story. I would probably feel different if I had the need for groupware though, but digs at Microsoft are irresistible sometimes. Matt ---BeginMessage--- Of course I'm sending the full message as an attachment. You can do that with Outlook byattaching and item, then browsing your mail folders for the message to attach. And yes, that's how you do it with Outlook Express as well. I don't use Thunderbird or Netscape mail, but I would assume you still need to attach the original message to avoid the headers being lost. What I was referring to was a little more involved than that... namely the possibility of it not matching a rule because the attachment was encoded differently. For example, I've seen mail go throughthat baes64 encoded an attached email that was not originally base64 encoded. From Pete's responses, it sounded like "no rule found" really did mean no rule was matched. Especially since he has a separate code for "rule already removed". FPs we send are always from same day, or, at the very least, within 24 hours. Darin. - Original Message - From: Matt To: Message Sniffer Community Sent: Wednesday, June 07, 2006 11:46 PM Subject: Re: [sniffer]FP suggestions Darin,Outlook will strip many of the headers when forwarding. Outlook Express needs to forward the messages using "Forward As Attachment" in order to insert the full original headers. Thunderbird/Netscape Mail will work just by forwarding. If you paste the full source in a message, you should send as plain text.I have many FP's that come back as having no rules found, but these are more likely to be from rules that were already removed. So I wouldn't jump to a conclusion that the rule was not found because of formatting unless you are not sending the full unadulterated original message source. I would imagine that it would mostly be IP rules that aren't found when not forwarding the full original source.MattDarin Cox wrote: It is unclear - we receive FPs that have traveled through all sorts of clients, quarantine systems, changed hands various numbers of times, or not (to all of those)... Right now I don't want to make that research project a high priority. Understood. That's true it wouldn't change, but submitting the message directly would not be correct - the dialogue is with you, and in any case, additional trips through the mail server also modify parts of the header and sometimes parts of the message (tag lines, disclaimers, etc)... Hmmm... with attaching the original message, I guess it still makes more sense to deliver to us first for now. Just looking for an alternative that gets you the message as close as possible to the original form as possible. Maybe we'll write a script to copy and forward the D*.SMD file as an attachment to you for FPs at some point in the future. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] ---End Message--- # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
