[sniffer] Re: [sniffer][Fwd: Re: [sniffer]FP suggestions]
Thunderbird and Netscape just takes the full original source and attaches it as a message/rfc822 attachment. I forwarded this message back to the list by just pressing Forward. Interesting that they include the headers with a simple forward, without specifying forward as attachment. I haven't ever seen that behaviour before in a mail client. Seems like a few forwards would create a very bloated message with all of the old headers. I'm pretty sure that Outlook Express works simply by just pressing Forward As Attachment, or at least it gives me enough of the original, including the full headers, to determine how to block the spam. Yes it does. However you've missed the point. The issue is not how to get the headers. It is how to keep an email client from encoding the message and headers differently, so that Sniffer can properly identify the rule that caught the message. Please excuse me for wanting more detail about the Outlook attachment trick, but would you mind attaching this message to a response so that I could look at the headers and such? Sorry, I don't use Outlook. But I can tell you the steps to take in Outlook 2003 (other versions are almost exactly the same). I have my Outlook users follow these with no problem. 1. Create a new email message 2. Click the arrow beside the paperclip icon, select item instead of file from the dropdown 3. Browse mailboxes from the popup dialog to select the message to attach. 4. Viola, original message and headers attached. There was a discussion about Outlook's behavior with Scott some time ago. Apparently Microsoft was pressured by customers to remove headers when forwarding because they felt that they were a security/privacy risk. No one told them that Outlook was a security/privacy risk on it's own :) ...but that's another story. I would probably feel different if I had the need for groupware though, but digs at Microsoft are irresistible sometimes. I don't remember that discussion, and am not sure we're talking about the same thing. If you attach the original message via the steps above, you get the full original message, headers and body. We have a number of customers who send spam reports this way, mostly on Outlook 2002 and 2003. Darin # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: [sniffer][Fwd: Re: [sniffer]FP suggestions]
Darin, Thunderbird allows you to choose the default forwarding method as either inline or as attachment. It might actually default to inline, I can't remember, but whenever it does message/rfc822 attachments, it is as a whole unlike some other clients that edit it down to the bare minimum of what the consider to be useful like addressing, subject date and MIME stuff if appropriate. I'm definitely guilty of being a Netscape diehard, and I'm very happy that the Mozilla project brought things back to life again. I fully understand the attachment trick with Outlook thanks to the confirmations. This will be easier than having people cut and paste the headers in. This doesn't happen much, but there is nothing worse than getting a spam report without header info. I also understand the encoding issues with forwarding in Outlook/OE. It's a shame that this happens. Maybe having a copy of Thunderbird around for this purpose might fit in where this is an issue. Sounds like adding Sniffer headers would be the best solution for this issue on a wider basis since you definitely can't convince every admin not to submit using Outlook/OE. Soon I'm going to code up my Sniffer FP reports to be automatically triggered when a message is reprocessed from my spam review system, so I won't have to even bother with the source any more. That should only take a couple of hours, and it would be time well spent. I always fix issues and whitelist locally where appropriate, but I also report to Sniffer for the benefit of all in addition to making sure that a FP rule will not tag something outside of the scope of what I whitelisted, and I have to report in order to be able to see what the content of the rule was. Customers do most of the reprocessing now, I just do the back end stuff. Matt Darin Cox wrote: Thunderbird and Netscape just takes the full original source and attaches it as a message/rfc822 attachment. I forwarded this message back to the list by just pressing "Forward". Interesting that they include the headers with a simple forward, without specifying forward as attachment. I haven't ever seen that behaviour before in a mail client. Seems like a few forwards would create a very bloated message with all of the old headers. I'm pretty sure that Outlook Express works simply by just pressing Forward As Attachment, or at least it gives me enough of the original, including the full headers, to determine how to block the spam. Yes it does. However you've missed the point. The issue is not how to get the headers. It is how to keep an email client from encoding the message and headers differently, so that Sniffer can properly identify the rule that caught the message. Please excuse me for wanting more detail about the Outlook attachment trick, but would you mind attaching this message to a response so that I could look at the headers and such? Sorry, I don't use Outlook. But I can tell you the steps to take in Outlook 2003 (other versions are almost exactly the same). I have my Outlook users follow these with no problem. 1. Create a new email message 2. Click the arrow beside the paperclip icon, select item instead of file from the dropdown 3. Browse mailboxes from the popup dialog to select the message to attach. 4. Viola, original message and headers attached. There was a discussion about Outlook's behavior with Scott some time ago. Apparently Microsoft was pressured by customers to remove headers when forwarding because they felt that they were a security/privacy risk. No one told them that Outlook was a security/privacy risk on it's own :) ...but that's another story. I would probably feel different if I had the need for groupware though, but digs at Microsoft are irresistible sometimes. I don't remember that discussion, and am not sure we're talking about the same thing. If you attach the original message via the steps above, you get the full original message, headers and body. We have a number of customers who send spam reports this way, mostly on Outlook 2002 and 2003. Darin # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: [sniffer][Fwd: Re: [sniffer]FP suggestions]
Hello Andrew, Thursday, June 8, 2006, 11:32:47 AM, you wrote: Ditto. I advise people to use Insert, Item. Far easier than explaining how to drag and drop (or tie shoelaces). It might be nice to have a SnagIt of that process to share w/ users. I've noticed that whether the headers survive when they are sent to another Exchange+Outlook company are a crap shoot. Generally speaking, if the message is handled by Outlook, it's not the same message anymore. For example, a BASE64 encoded message becomes plain text, and attached graphics don't show up at all in the View Source version. I just had an interesting FP case like this. By the time the match record got to me along with what was supposed to be the original message, there were at least 9K bytes missing - including the bytes that presumably contained the rule match. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
Re: [sniffer]FP suggestions
The one issue with this I have is 1) Forward full original source to Sniffer with license code. If we could do it without the license code, it would be much easier to automate on our end. I already have a process in place to copy and reroute false positives by rewriting the Q file. I'm hesitant to alter the message itself to add the license code. If we could authenticate the FP report via some other means it would help greatly. How about connecting IP instead? Darin. - Original Message - From: Matt To: Message Sniffer Community Sent: Wednesday, June 07, 2006 12:59 AM Subject: Re: [sniffer]FP suggestions Pete,Regarding suggestions for easing the reporting process, I would recommend the following possible modifications: 1) An E-mail submission tool similar to the one now, but replies would be automated2) Send back links or rather an HTML form with checkboxes in an E-mail auto-response allowing one to block rules.3) Make blocked rules automatic for the submitter, but throw them into a queue for manual review by Sniffer folk in order to determine whether the blocks should become applied to all rulebases.4) Have automatic triggers that lower rule strengths based on users blocking rules regardless of direct Sniffer action.The gist of this is to make it more point and click. The fact that you need full source is cumbersome, so the above recommendations seek ways to make the process easier for both the customer and for Sniffer while dealing with the need to send the full source. No direct customer interaction would be necessary in most cases, and you would have a queue full of items to review and make a determination about that customers have preened for you. To the customer, the process would look like the following: 1) Forward full original source to Sniffer with license code.2) Seconds later there would be an automated reply received in HTML format with a check box for every rule failed (or note that no active rules were found), a text box for optional comments, and submit button.3) Customer checks the boxes for the rules he wants to block, adds notes in a text field if they feel like it, and they press submit. End of story.You could also add a Web interface for this if you wanted to, but E-mail seems the most appropriate for most.I don't think it would be beneficial to rehash a lot of things involving how FP's occur, at least on this list. I know from my system where my customers have single-click reprocessing capability, that they miss about 97% of all FP's either because they don't bother to do review, or they don't bother to reprocess anything but personal E-mail that may get blocked. I would imagine that Sniffer sees a similar rate of customer reported FP's due in part to the difficulty, and in part for the same reasons that relate to my own users.The three biggest sources of false positives are obscure foreign domains/IP's, rules generated from bulk mailings that are too broadly targeted, and things reported to Sniffer that are advertising, but not spam. All three of these things are difficult and time consuming to deal with, particularly the last two. Here's some stats for Sniffer FP's on my system going back about 15 months: SNIFFER-GENERAL 283SNIFFER-EXPERIMENTAL 167 * Excluded 79 FP's from bad rule event on 1/17 - 1/18/2006SNIFFER-IP 61SNIFFER-PHISHING 52SNIFFER-GETRICH 29 * Excluded 115 FP's from bad rule event on 4/18 - 4/19/2006SNIFFER-PHARMACY 25SNIFFER-PORN 24SNIFFER-TRAVEL 13SNIFFER-INSURANCE 7SNIFFER-OBFUSCATION 6SNIFFER-DEBT 6SNIFFER-MALWARE 4SNIFFER-AVSOFT 3SNIFFER-CASINO 2SNIFFER-INK 1SNIFFER-MEDIA 1SNIFFER-SPAMWARE 0It is quite notable how high the FP's are with SNIFFER-GENERAL which is where most bulk-mailers and customer reported spam rules are tagged. This is also what my numbers show even though my customers are much less likely to reprocess bulk mail, and of course they only reprocess a small fraction of my overall FP's. This is almost all customer reported stuff. I score SNIFFER-GENERAL at 53% of my Hold weight. SNIFFER-IP is another standout. I only score SNIFFER-IP at 38% of my Hold weight and it hits less than 2% of all Sniffer hits, yet it scored comparably high so that is worth noting. The FP rate on SNIFFER-IP hasn't really changed since you made adjustments. SNIFFER-EXPERIMENTAL is a top category that caught a lot of zombie spam which is important to many systems, but it did seem to have a high FP rate. SNIFFER-PHISHING was worse for me until around January or February. It seemed to have a lot of FP's on security related newsletters and chain letters. I have mixed feelings about those things. Maybe more efforts on white rules would help with that stuff, and I'm not totally sure if it is appropriate to block chain letters even though I detest this stuff myself.Most FP's do
Re: [sniffer]FP suggestions
Oh, I assumed the rule had been removed. Are you saying there was a rule in place, but the FP processing somehow failed to find it? If so, I'd say that is a major failing on the part of the FP processing. There's no way thatwe can find time to go through the Sniffer logs after this bounces back with "no rule found". This would have to be automated to have any chance of occurring, but again I would say the FP processing needs to be corrected to identify the rule the message failed since the complete message, headers and body, are included in the report. Darin. - Original Message - From: Scott Fisher To: Message Sniffer Community Sent: Wednesday, June 07, 2006 10:08 AM Subject: Re: [sniffer]FP suggestions For me the pain of false positives submissions is the research that happens when I get a "no rule found" return. I then need to find the queue-id of the original message and then find the appropriate Sniffer log and pull out the log lines from there and then submit it. Almost always in these cases, a rule is removed. If this process could be improved that would really be a time saver.
Re: [sniffer]FP suggestions
Of course I'm sending the full message as an attachment. You can do that with Outlook byattaching and item, then browsing your mail folders for the message to attach. And yes, that's how you do it with Outlook Express as well. I don't use Thunderbird or Netscape mail, but I would assume you still need to attach the original message to avoid the headers being lost. What I was referring to was a little more involved than that... namely the possibility of it not matching a rule because the attachment was encoded differently. For example, I've seen mail go throughthat baes64 encoded an attached email that was not originally base64 encoded. From Pete's responses, it sounded like "no rule found" really did mean no rule was matched. Especially since he has a separate code for "rule already removed". FPs we send are always from same day, or, at the very least, within 24 hours. Darin. - Original Message - From: Matt To: Message Sniffer Community Sent: Wednesday, June 07, 2006 11:46 PM Subject: Re: [sniffer]FP suggestions Darin,Outlook will strip many of the headers when forwarding. Outlook Express needs to forward the messages using "Forward As Attachment" in order to insert the full original headers. Thunderbird/Netscape Mail will work just by forwarding. If you paste the full source in a message, you should send as plain text.I have many FP's that come back as having no rules found, but these are more likely to be from rules that were already removed. So I wouldn't jump to a conclusion that the rule was not found because of formatting unless you are not sending the full unadulterated original message source. I would imagine that it would mostly be IP rules that aren't found when not forwarding the full original source.MattDarin Cox wrote: It is unclear - we receive FPs that have traveled through all sorts of clients, quarantine systems, changed hands various numbers of times, or not (to all of those)... Right now I don't want to make that research project a high priority. Understood. That's true it wouldn't change, but submitting the message directly would not be correct - the dialogue is with you, and in any case, additional trips through the mail server also modify parts of the header and sometimes parts of the message (tag lines, disclaimers, etc)... Hmmm... with attaching the original message, I guess it still makes more sense to deliver to us first for now. Just looking for an alternative that gets you the message as close as possible to the original form as possible. Maybe we'll write a script to copy and forward the D*.SMD file as an attachment to you for FPs at some point in the future. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer][Fwd: Re: [sniffer]FP suggestions]
Darin, Thunderbird and Netscape just takes the full original source and attaches it as a message/rfc822 attachment. I forwarded this message back to the list by just pressing Forward. I'm pretty sure that Outlook Express works simply by just pressing Forward As Attachment, or at least it gives me enough of the original, including the full headers, to determine how to block the spam. I have been telling Outlook users to copy and paste the headers into a forwarded message. Please excuse me for wanting more detail about the Outlook attachment trick, but would you mind attaching this message to a response so that I could look at the headers and such? There was a discussion about Outlook's behavior with Scott some time ago. Apparently Microsoft was pressured by customers to remove headers when forwarding because they felt that they were a security/privacy risk. No one told them that Outlook was a security/privacy risk on it's own :) ...but that's another story. I would probably feel different if I had the need for groupware though, but digs at Microsoft are irresistible sometimes. Matt ---BeginMessage--- Of course I'm sending the full message as an attachment. You can do that with Outlook byattaching and item, then browsing your mail folders for the message to attach. And yes, that's how you do it with Outlook Express as well. I don't use Thunderbird or Netscape mail, but I would assume you still need to attach the original message to avoid the headers being lost. What I was referring to was a little more involved than that... namely the possibility of it not matching a rule because the attachment was encoded differently. For example, I've seen mail go throughthat baes64 encoded an attached email that was not originally base64 encoded. From Pete's responses, it sounded like "no rule found" really did mean no rule was matched. Especially since he has a separate code for "rule already removed". FPs we send are always from same day, or, at the very least, within 24 hours. Darin. - Original Message - From: Matt To: Message Sniffer Community Sent: Wednesday, June 07, 2006 11:46 PM Subject: Re: [sniffer]FP suggestions Darin,Outlook will strip many of the headers when forwarding. Outlook Express needs to forward the messages using "Forward As Attachment" in order to insert the full original headers. Thunderbird/Netscape Mail will work just by forwarding. If you paste the full source in a message, you should send as plain text.I have many FP's that come back as having no rules found, but these are more likely to be from rules that were already removed. So I wouldn't jump to a conclusion that the rule was not found because of formatting unless you are not sending the full unadulterated original message source. I would imagine that it would mostly be IP rules that aren't found when not forwarding the full original source.MattDarin Cox wrote: It is unclear - we receive FPs that have traveled through all sorts of clients, quarantine systems, changed hands various numbers of times, or not (to all of those)... Right now I don't want to make that research project a high priority. Understood. That's true it wouldn't change, but submitting the message directly would not be correct - the dialogue is with you, and in any case, additional trips through the mail server also modify parts of the header and sometimes parts of the message (tag lines, disclaimers, etc)... Hmmm... with attaching the original message, I guess it still makes more sense to deliver to us first for now. Just looking for an alternative that gets you the message as close as possible to the original form as possible. Maybe we'll write a script to copy and forward the D*.SMD file as an attachment to you for FPs at some point in the future. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] ---End Message--- # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
