Re: [sniffer] High False Positives
There was a bad rule yesterday. It was removed almost immediately but it looks like you missed the update until 1000pm. It takes a while to compile rulebase updates. Since you mention 4pm and 10pm I'm guessing you have your updates scheduled. A better method would be to trigger updates based on an update notification since this allows us to correct problems like this more quickly. If I've assumed wrong, please disregard. Thanks, _M At 10:27 AM 3/25/2004, you wrote: I had a high number of false positives yesterday starting after my 4:00 PM (CST) Sniffer update. I believe it occurred about the time of the spam storm yesterday, when many spam messages made it through the filter. It appeared to stop at 10:00 PM but I don't know if people quit sending messages for the day or if my Sniffer update fixed the issues. I haven't seen any today (did some spot checks); do I need to submit all the messages that were false positives? Did something happen yesterday? Al Thornberry This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Help
Have you tried a reboot? Checked your error logs? Made sure that DNS and all of your E-mail services are running? Is there even a chance that you will be able to receive this message? Matt Richard Farris wrote: I just did an Windows NT update and now I cant get any email...when I turn sniffer off I at least can send mail to myself but still cant get from outside..any ideas., Richard Farris Ethixs Online 1.270.247. Office 1.800.548.3877 Tech Support - Original Message - From: "Pete McNeil" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, March 24, 2004 2:01 PM Subject: Re: [sniffer] Possible Bad Rule? We had a badly coded rule that matched yahoo. The rule has been removed. About 30 rulebases went out before it was caught. These are being recompiled with the correction right now. I will see if I can push yours to the top. _M At 02:02 PM 3/24/2004, you wrote: I am getting a lot of complaints today from Yahoo users... Sheldon - Original Message - From: "Darrell LaRock" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: "'SnifferSupport'" [EMAIL PROTECTED] Sent: Wednesday, March 24, 2004 10:33 AM Subject: [sniffer] Possible Bad Rule? Pete, I am seeing a ton of false positives for RULE 100543. I sent a few in to you to check out ([EMAIL PROTECTED]). I wanted to post this here as well since it seems to take approx. 24 hours to process false positives. Darrell This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
[sniffer] log upload trouble
I've been having trouble for the last 24 hrs or maybe a bit more with log uploads failing. The FTP either fails to connect, or it does connect and the upload begins and then fails after a small percentage done. Uploads are scheduled every 6 hours. Yesterday afternoon I tried renaming the log files from a couple failures and triggering the upload manually, and it also failed An upload started a few mins ago, at 12:05 PM. It progressed almost to completion, and then ended with a reported failure from WS_FTP. Glenn Z. WCNet - Original Message - From: Pete McNeil [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Saturday, March 20, 2004 1:13 AM Subject: Re: [sniffer] Define Persistent sniffer. At 09:50 PM 3/19/2004, you wrote: Pete, I follow this forum pretty well, however, having been out this week on business it seems I have lost alot with this new feature set. If you don't mind, could you define Persistent Sniffer? We average well over a million emails a day between two servers, what impact might I see on our server if I run this? What is the recommended settings? Thanks for the aid. (Seems I'm in the book writing mode this evening... sorry for the bandwidth) Performance Metrics: Our NT4/SP6a test bed, running IMail/Declude/Sniffer in persistent mode. P2/450, 2x 5400rpm IDE drives, mirrored, 256M Ram (No giggles please - This is an intentionally underpowered server - how better to stress test a program like Sniffer?). Sniffer in persistent mode on this box is able to process 120k msgs / month without issue. Logs show that each message on average now takes about 100ms total. Typical values are 20ms queue, 40ms scan though obviously some messages take longer and occasionally longer queue times do creep in. Prior to testing the persistent version of Sniffer, message scan times varied wildly but averaged about 300ms per message with some messages taking 3-5 seconds while waiting for I/O and other processes (Web Mail, IMAP, etc...). In fact, I intentionally waited until the CPU was at 100% (green line 100%, red line 50%+) before starting the service to see how the creatures would handle the transition under heavy stress - The CPU dropped so much that at first I thought I had broken something (one of those oops moments). The CPU now rests on the floor more often than not and generally runs peaks to about 50% unless something odd is going on - such as a defrag run. YMMV - the above data is based on a very narrow data sample and only loosely calculated - and some of it is anecdotal. However most reports from the field seem to support the general scale of improvement. On the back of the envelope I can calculate something like: 1 million per day is probably on the order of 125000 (1M/8hours) during a peak hour. 125000/3600 = about 35 per second. If message sniffer can scan about 10 per second on an overloaded p2/450, then on a 2.4ghz machine with plenty of memory we might expect at least a linear improvement - approximately 5x, but we will say 4x to be safe - 40/sec covers 35/sec so we have our million based on these assumptions. IO not withstandng I would expect a persistent server version of Sniffer on a well provisioned server with a 2.4ghz processor to handle 1 million per day _IF_ that's all it had to do... since there's always more to do and this would be a maximum load scenario, dividing this across two servers should work nicely - though it would probably be time to start considering a third server. Then again, you are probably not running generic single processor servers if you are handling 1 million messages per day ;-) ___ Definition: Probably the simplest definition of Persistent Sniffer as you put it is a lightweight daemon. It can't actually be launched as a daemon/service on it's own, and it is still compatible with the self-organizing-automata version of Sniffer, but it offers many of the performance savings of a daemon/service - along with some added redundancy and flexibility. For example, if the persistent server instance of Sniffer fails, then the other instances simply return to their normal peer-server mode of operation so there is a drop in performance, but not a loss of service. More Detail: Versions of Message Sniffer prior to 2-2 would always load the rule-base each time a message was to be scanned. Specifically, each instance of Message Sniffer was isolated and did the job itself. Up to 90% of the processing time typically required was bound in loading the rule-base file. On our NT test bed, for example, we would regularly see queue/scan times on the order of 1000/10, though more commonly 360/60 at the time when we developed version 2-2. Beginning with Version 2-2, we implemented a cellular peer-server technology with Message Sniffer. This technology allows instances of Message Sniffer running on the same server to interact and
RE: [sniffer] Spam storm?
That is possible. I'm still looking for an alternate repeatable cause. _M At 08:43 PM 3/24/2004, you wrote: I see over a 1000 of these ERROR_BAD_MATRIX entries in my Sniffer log file today, as well. Is this due to the ruleset issue from earlier today? Bill -Original Message- From: Sheldon Koehler [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 24, 2004 3:19 PM To: [EMAIL PROTECTED] Subject: Re: [sniffer] Spam storm? Well it may not be a spam storm. Log file shows: nsx4b3eh 20040324200108 De90392330028271a.SMD 421 0 ERROR_BAD_MATRIX 71 0 0 2 5 nsx4b3eh 20040324200117 De90c923a00284b5b.SMD 422 0 ERROR_BAD_MATRIX 71 0 0 What is a Bad Matrix? Sheldon Sheldon Koehler, Owner/Partnerhttp://www.tenforward.com Ten Forward Communications 360-457-9023 Nationwide access, neighborhood support! Whenever you find yourself on the side of the majority, it's time to pause and reflect. Mark Twain This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html --- This message and any included attachments are from Siemens Medical Solutions USA, Inc. and are intended only for the addressee(s). The information contained herein may include trade secrets or privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you received this message in error, or have reason to believe you are not authorized to receive it, please promptly delete this message and notify the sender by e-mail with a copy to [EMAIL PROTECTED] Thank you This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Error_Bad_Matrix
I am having the same problem when I download the update and run snf2check H. - Original Message - From: Landry William [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, March 25, 2004 2:57 PM Subject: RE: [sniffer] Error_Bad_Matrix I run snf2check.exe against every .snf file downloaded. I just checked it again manually, and no errors were reported. I now have almost 3500 Error_Bad_Matrix entries in today's log. Bill -Original Message- From: Vivek Khera [mailto:[EMAIL PROTECTED] Sent: Thursday, March 25, 2004 12:52 PM To: [EMAIL PROTECTED] Subject: Re: [sniffer] Error_Bad_Matrix On Mar 25, 2004, at 3:39 PM, Paul Lushinsky wrote: I decided to look in my log files for the past several days because of number of Error_Bad_Matrix related messages. I can't find this message in any of my log files until today starting with the update I auto downloaded at 8:15 this morning, and went until the update at noon. While I was look at the log file, another update notice came, so an update was done and the Error_Bad_Matrix message is back. I'm curious if the people who are seeing these messages are running snf2check.exe before making the rule files live. I do so, and have not seen a single instance of this error. Can you run snf2check.exe on the current bad matrix you have and see if it reports an error? This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- - This message and any included attachments are from Siemens Medical Solutions USA, Inc. and are intended only for the addressee(s). The information contained herein may include trade secrets or privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you received this message in error, or have reason to believe you are not authorized to receive it, please promptly delete this message and notify the sender by e-mail with a copy to [EMAIL PROTECTED] Thank you This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Spam storm?
This has been a bad week here! A big increase in total email volume, a huge increase in false positives as well as a huge increase in spam getting past our filters. Sheldon Sheldon Koehler, Owner/Partnerhttp://www.tenforward.com Ten Forward Communications 360-457-9023 Nationwide access, neighborhood support! Whenever you find yourself on the side of the majority, it's time to pause and reflect. Mark Twain This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Spam storm?
We've found that when we do a manual download, everything works fine. It's the automatic download on the Windows 2000 server that seems to corrupt things. M. Stein Computer House - Original Message - From: Pete McNeil [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, March 25, 2004 6:05 PM Subject: Re: [sniffer] Spam storm? This helps narrow things down. Specifically we know that the rulebase files are not corrupted on the server but during the download. That explains why I haven't been able to recreate a problem in the lab. I have a suspicion that wget may be failing intermittently. Another customer recently had unexplainable, intermittent issues with wget. They replaced wget with code of their own and have had no further problems. Can we narrow this down to wget under heavy traffic conditions perhaps? _M At 10:08 PM 3/24/2004, you wrote: I've noticed that if I do a manual download of the rule base file, it works well, but if it is downloaded automatically via the Windows Task CMD, then sniffer fails and the log fills up with the BAD_MATRIX errors. Anyone else seeing this? Mike - Original Message - From: Landry William [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, March 24, 2004 8:43 PM Subject: RE: [sniffer] Spam storm? I see over a 1000 of these ERROR_BAD_MATRIX entries in my Sniffer log file today, as well. Is this due to the ruleset issue from earlier today? Bill -Original Message- From: Sheldon Koehler [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 24, 2004 3:19 PM To: [EMAIL PROTECTED] Subject: Re: [sniffer] Spam storm? Well it may not be a spam storm. Log file shows: nsx4b3eh 20040324200108 De90392330028271a.SMD 421 0 ERROR_BAD_MATRIX 71 0 0 2 5 nsx4b3eh 20040324200117 De90c923a00284b5b.SMD 422 0 ERROR_BAD_MATRIX 71 0 0 What is a Bad Matrix? Sheldon Sheldon Koehler, Owner/Partnerhttp://www.tenforward.com Ten Forward Communications 360-457-9023 Nationwide access, neighborhood support! Whenever you find yourself on the side of the majority, it's time to pause and reflect. Mark Twain This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- - This message and any included attachments are from Siemens Medical Solutions USA, Inc. and are intended only for the addressee(s). The information contained herein may include trade secrets or privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you received this message in error, or have reason to believe you are not authorized to receive it, please promptly delete this message and notify the sender by e-mail with a copy to [EMAIL PROTECTED] Thank you This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Call for beta testers... snfrv2r3b1
I think the problem is in the file extension. It should not be .com, but rather .cmd. Hope this helps, _M At 12:32 PM 3/25/2004, you wrote: Hi, When I try to run the .com file, I get an error. I have attached the error dialog box and a copy of the .com file (name altered to .co_) that I am using. Can you see what I am doing wrong? The program seems to be running OK in normal mode. Thanks, Bill Morgan -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Wednesday, March 17, 2004 1:05 PM To: [EMAIL PROTECTED] Subject: [sniffer] Call for beta testers... snfrv2r3b1 Hello folks, I know folks are anxious to get their hands on this version so I'm going to play this beta round a little looser than usual. Version 2-3b1 implements a persistent mode feature for our cellular peer-server technology. Launching a persistent instance of Message Sniffer has the effect of creating a daemon so that all other instances will elect to be clients. We observed a DRAMATIC improvement in system performance on our NT4/Imail/Declude test bed. In static tests on my Toshiba 6100 we saw no memory leaks and consistent performance over the past 18+ hours of testing. This included several tests with more than 100+ concurrent client instances - all without failure and without making the system unresponsive (though the WinXP file system did start to show signs of strain). This beta is for the windows platform only... once we're happy with this version will will make the source and *nix versions available as always. Windows platform users who are interested in testing the new beta should download the following file: http://www.sortmonster.com/MessageSniffer/Betas/snfrv2r3b1.zip The file contains an executable and a short readme file. We are going to be extremely busy for the next few hours so we won't be able to provide support on this until later this evening. We have many updates and rulebase mods to attend to at the moment since we shifted resources heavily toward development last evening and through the night... The current spam storm continues to rage with more than 500 core rule-base changes yesterday alone! Be careful. Backup your current production version. Watch carefully. Enjoy :-) _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Help
MicroNeil Voice Line: 703-779-4909 _M At 01:30 PM 3/25/2004, you wrote: I got it.I am on to something so I might figure it outif I dont is there a number I can call.. Richard Farris Ethixs Online 1.270.247. Office 1.800.548.3877 Tech Support - Original Message - From: Matt To: [EMAIL PROTECTED] Sent: Thursday, March 25, 2004 11:27 AM Subject: Re: [sniffer] Help Have you tried a reboot? Checked your error logs? Made sure that DNS and all of your E-mail services are running? Is there even a chance that you will be able to receive this message? Matt Richard Farris wrote: I just did an Windows NT update and now I cant get any email...when I turn sniffer off I at least can send mail to myself but still cant get from outside..any ideas., Richard Farris Ethixs Online 1.270.247. Office 1.800.548.3877 Tech Support - Original Message - From: Pete McNeil [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, March 24, 2004 2:01 PM Subject: Re: [sniffer] Possible Bad Rule? We had a badly coded rule that matched yahoo. The rule has been removed. About 30 rulebases went out before it was caught. These are being recompiled with the correction right now. I will see if I can push yours to the top. _M At 02:02 PM 3/24/2004, you wrote: I am getting a lot of complaints today from Yahoo users... Sheldon - Original Message - From: Darrell LaRock [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: 'SnifferSupport' [EMAIL PROTECTED] Sent: Wednesday, March 24, 2004 10:33 AM Subject: [sniffer] Possible Bad Rule? Pete, I am seeing a ton of false positives for RULE 100543. I sent a few in to you to check out ([EMAIL PROTECTED]). I wanted to post this here as well since it seems to take approx. 24 hours to process false positives. Darrell This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
Re: [sniffer] Error_Bad_Matrix
I've been looking at that. The problem seems to be related to downloads, not generation. That is, every rulebase that I use locally has been clean throughout this episode. Also, folks who manually download the rulebase seem to be able to correct the problem. I'm not sure yet what is different between automated and manual downloads - except perhaps wget. I also don't have any obvious changes on our system recently. I continue to dig. _M At 03:39 PM 3/25/2004, you wrote: Pete, I decided to look in my log files for the past several days because of number of Error_Bad_Matrix related messages. I can't find this message in any of my log files until today starting with the update I auto downloaded at 8:15 this morning, and went until the update at noon. While I was look at the log file, another update notice came, so an update was done and the Error_Bad_Matrix message is back. I am using the latest production version of sniffer. I know you are probably working on this, but I thought you should know for sure that your process for building the rulebase is experiencing some major issues. All times are -0600 GMT. -Original Message- From: Butch Andrews [mailto:[EMAIL PROTECTED] Sent: Thursday, March 25, 2004 10:23 AM To: [EMAIL PROTECTED] Subject: [sniffer] Error_Bad_Matrix I am seeing my log file continue to fill with Error_Bad_Matrix errors and sniffer failing since a lot of spam is getting through. I was running the beta but have gone back to the original version just now. I did amanual update when the program change had no effect and it's back up. I checked last nights log and the problem started with date code 20040325083243 and continued until now. This is for your info since I was using the beta. -Butch This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Error_Bad_Matrix
snf2check.exe will catch a partial download but it will not catch corruption in the middle of the file. _M At 03:57 PM 3/25/2004, you wrote: I run snf2check.exe against every .snf file downloaded. I just checked it again manually, and no errors were reported. I now have almost 3500 Error_Bad_Matrix entries in today's log. Bill -Original Message- From: Vivek Khera [mailto:[EMAIL PROTECTED] Sent: Thursday, March 25, 2004 12:52 PM To: [EMAIL PROTECTED] Subject: Re: [sniffer] Error_Bad_Matrix On Mar 25, 2004, at 3:39 PM, Paul Lushinsky wrote: I decided to look in my log files for the past several days because of number of Error_Bad_Matrix related messages. I can't find this message in any of my log files until today starting with the update I auto downloaded at 8:15 this morning, and went until the update at noon. While I was look at the log file, another update notice came, so an update was done and the Error_Bad_Matrix message is back. I'm curious if the people who are seeing these messages are running snf2check.exe before making the rule files live. I do so, and have not seen a single instance of this error. Can you run snf2check.exe on the current bad matrix you have and see if it reports an error? This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html --- This message and any included attachments are from Siemens Medical Solutions USA, Inc. and are intended only for the addressee(s). The information contained herein may include trade secrets or privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you received this message in error, or have reason to believe you are not authorized to receive it, please promptly delete this message and notify the sender by e-mail with a copy to [EMAIL PROTECTED] Thank you This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Spam storm?
By 8pm we had done at least 6 that I was part of. _M At 04:32 PM 3/25/2004, you wrote: How many updates have happened today...I have only received 1 today.. Richard Farris Ethixs Online 1.270.247. Office 1.800.548.3877 Tech Support - Original Message - From: Pete McNeil [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, March 25, 2004 2:52 PM Subject: Re: [sniffer] Spam storm? Big uptick of new and broken spam. Half way through the day and already at 445 new rules. We may be getting it under control though... (fingers crossed). _M At 06:02 PM 3/24/2004, you wrote: Am I the only one seeing a spam storm today? This is the worst I have EVER seen!!! Sheldon Sheldon Koehler, Owner/Partnerhttp://www.tenforward.com Ten Forward Communications 360-457-9023 Nationwide access, neighborhood support! Whenever you find yourself on the side of the majority, it's time to pause and reflect. Mark Twain This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Error_Bad_Matrix
Pete, FYI, I was trying to set up log uploads yesterday night and it took me a while to figure out that the FTP connection was unreliable from my server. Packets were being dropped/munged somewhere. I also noted a much lower hit rate on SNIFFER-PHARMACY yesterday, but no indication of matrix problems in the logs today (yesterday's were deleted). Every once in a while my colocator's border router goes on the fritz and starts dropping packets. A reboot usually fixes that issue. If your router checks out fine, you might want to take a look at the routes going from your server to the customers that have indicated a problem and those that have indicated that there is none, that might identify something not so obvious if you run out of ideas. I know how these things go and the worst part is not knowing the source while others expect an quick fix. No big deal on my end in the mean time though. Matt Pete McNeil wrote: snf2check.exe will catch a partial download but it will not catch corruption in the middle of the file. _M At 03:57 PM 3/25/2004, you wrote: I run snf2check.exe against every .snf file downloaded. I just checked it again manually, and no errors were reported. I now have almost 3500 Error_Bad_Matrix entries in today's log. Bill -Original Message- From: Vivek Khera [mailto:[EMAIL PROTECTED] Sent: Thursday, March 25, 2004 12:52 PM To: [EMAIL PROTECTED] Subject: Re: [sniffer] Error_Bad_Matrix On Mar 25, 2004, at 3:39 PM, Paul Lushinsky wrote: I decided to look in my log files for the past several days because of number of Error_Bad_Matrix related messages. I can't find this message in any of my log files until today starting with the update I auto downloaded at 8:15 this morning, and went until the update at noon. While I was look at the log file, another update notice came, so an update was done and the Error_Bad_Matrix message is back. I'm curious if the people who are seeing these messages are running snf2check.exe before making the rule files live. I do so, and have not seen a single instance of this error. Can you run snf2check.exe on the current bad matrix you have and see if it reports an error? This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html --- This message and any included attachments are from Siemens Medical Solutions USA, Inc. and are intended only for the addressee(s). The information contained herein may include trade secrets or privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you received this message in error, or have reason to believe you are not authorized to receive it, please promptly delete this message and notify the sender by e-mail with a copy to [EMAIL PROTECTED] Thank you This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Spam storm?
I'm exploring that possibility - though there is nothing in the logs. I've seen some instability on the Sprint T1 though it seems stable now. Sprint made an announcement that they were going to change their routing and that seems to coincide with these new events. Perhaps instability on that part of the network is causing some ftp/wget downloads to become corrupted - though that's not supposed to happen. I've bounced the server just in case something was hung up there that I couldn't see - although some folks are not having trouble so there is nothing conclusive at this time. _M At 06:19 PM 3/25/2004, you wrote: Could it possibly be your FTP server. This morning it timed out 4 times when trying to manually download using my SecureFX program while this afternoon wget has had no problem. Maybe your getting hammered maliciously with outside requests. -Butch *** REPLY SEPARATOR *** On 3/25/2004 at 6:05 PM Pete McNeil wrote: This helps narrow things down. Specifically we know that the rulebase files are not corrupted on the server but during the download. That explains why I haven't been able to recreate a problem in the lab. I have a suspicion that wget may be failing intermittently. Another customer recently had unexplainable, intermittent issues with wget. They replaced wget with code of their own and have had no further problems. Can we narrow this down to wget under heavy traffic conditions perhaps? _M At 10:08 PM 3/24/2004, you wrote: I've noticed that if I do a manual download of the rule base file, it works well, but if it is downloaded automatically via the Windows Task CMD, then sniffer fails and the log fills up with the BAD_MATRIX errors. Anyone else seeing this? Mike - Original Message - From: Landry William [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, March 24, 2004 8:43 PM Subject: RE: [sniffer] Spam storm? I see over a 1000 of these ERROR_BAD_MATRIX entries in my Sniffer log file today, as well. Is this due to the ruleset issue from earlier today? Bill -Original Message- From: Sheldon Koehler [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 24, 2004 3:19 PM To: [EMAIL PROTECTED] Subject: Re: [sniffer] Spam storm? Well it may not be a spam storm. Log file shows: nsx4b3eh 20040324200108 De90392330028271a.SMD 421 0 ERROR_BAD_MATRIX 71 0 0 2 5 nsx4b3eh 20040324200117 De90c923a00284b5b.SMD 422 0 ERROR_BAD_MATRIX 71 0 0 What is a Bad Matrix? Sheldon Sheldon Koehler, Owner/Partnerhttp://www.tenforward.com Ten Forward Communications 360-457-9023 Nationwide access, neighborhood support! Whenever you find yourself on the side of the majority, it's time to pause and reflect. Mark Twain This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- - This message and any included attachments are from Siemens Medical Solutions USA, Inc. and are intended only for the addressee(s). The information contained herein may include trade secrets or privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you received this message in error, or have reason to believe you are not authorized to receive it, please promptly delete this message and notify the sender by e-mail with a copy to [EMAIL PROTECTED] Thank you This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Spam storm?
At 06:25 PM 3/25/2004, you wrote: We also saw many BAD_MATRIX errors last night. If the problem was 'wget', shouldn't the snf2check utility detect a corrupt file? Also, we did a manual update yesterday afternoon and there were no 'wget' error messages. The problem got corrected sometime between last night and this morning. Perhaps though some have had trouble throughout the day. At the very least the verification on snf2check should be improved to catch this issue. Updating with a bad ruleset creates many problems. Agreed. I'm looking for some simple ways to do that without changing the rulebase file format. There aren't any simple mechanisms that come to mind. Perhaps there will be no choice but to change the format in order to prevent this possibility. _M -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Thursday, March 25, 2004 7:06 PM To: [EMAIL PROTECTED] Subject: Re: [sniffer] Spam storm? This helps narrow things down. Specifically we know that the rulebase files are not corrupted on the server but during the download. That explains why I haven't been able to recreate a problem in the lab. I have a suspicion that wget may be failing intermittently. Another customer recently had unexplainable, intermittent issues with wget. They replaced wget with code of their own and have had no further problems. Can we narrow this down to wget under heavy traffic conditions perhaps? _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Spam storm?
snf2check.exe makes the assumption that if the entire file is there and the head and tail of it can be verified that it must have survived the transfer. Clearly something is happening where that is not the case - something new. One possibility that has been suggested is that we could gzip these files. That would be a somewhat radical change - but so would any change to the file format so this may be the best option. On the other hand the system has worked as is for quite some time. I would like to discover what has changed as that clearly represents a problem that must be corrected. _M At 06:35 PM 3/25/2004, you wrote: If that were the case then there is something wrong with either snf2check.exe and/or autosnf.cmd. The autosnf.cmd calls snf2check.exe to validate the downloaded file. If snf2check.exe found the downloaded file invalid, an error is suppose to be returned to keep it from going into production. So if I assume the file does get corrupted during the download, snf2check.exe must not be returning the correct value to indicate the file is bad, snf2check.exe hasn't changed in a long time. So while I can't argue that the file is bad before or after download. I will try to watch the logs more closely and manually test the snf files that begin to generate bad_matrix errors to see if their bad at that time. -Original Message- From: Pete McNeil [EMAIL PROTECTED] To: [EMAIL PROTECTED] Date: Thu, 25 Mar 2004 18:05:39 -0500 Subject: Re: [sniffer] Spam storm? This helps narrow things down. Specifically we know that the rulebase files are not corrupted on the server but during the download. That explains why I haven't been able to recreate a problem in the lab. I have a suspicion that wget may be failing intermittently. Another customer recently had unexplainable, intermittent issues with wget. They replaced wget with code of their own and have had no further problems. Can we narrow this down to wget under heavy traffic conditions perhaps? _M At 10:08 PM 3/24/2004, you wrote: I've noticed that if I do a manual download of the rule base file, it works well, but if it is downloaded automatically via the Windows Task CMD, then sniffer fails and the log fills up with the BAD_MATRIX errors. Anyone else seeing this? Mike - Original Message - From: Landry William [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, March 24, 2004 8:43 PM Subject: RE: [sniffer] Spam storm? I see over a 1000 of these ERROR_BAD_MATRIX entries in my Sniffer log file today, as well. Is this due to the ruleset issue from earlier today? Bill -Original Message- From: Sheldon Koehler [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 24, 2004 3:19 PM To: [EMAIL PROTECTED] Subject: Re: [sniffer] Spam storm? Well it may not be a spam storm. Log file shows: nsx4b3eh 20040324200108 De90392330028271a.SMD 421 0 ERROR_BAD_MATRIX 71 0 0 2 5 nsx4b3eh 20040324200117 De90c923a00284b5b.SMD 422 0 ERROR_BAD_MATRIX 71 0 0 What is a Bad Matrix? Sheldon Sheldon Koehler, Owner/Partnerhttp://www.tenforward.com Ten Forward Communications 360-457-9023 Nationwide access, neighborhood support! Whenever you find yourself on the side of the majority, it's time to pause and reflect. Mark Twain This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html --- --- - This message and any included attachments are from Siemens Medical Solutions USA, Inc. and are intended only for the addressee(s). The information contained herein may include trade secrets or privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you received this message in error, or have reason to believe you are not authorized to receive it, please promptly delete this message and notify the sender by e-mail with a copy to [EMAIL PROTECTED] Thank you This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to
RE: [sniffer] Spam storm?
At 06:51 PM 3/25/2004, you wrote: Looks like a bandwidth issue to me, since even doing the download manually, my connection stalled 5 times before I could complete a successful download. And the download speeds were atrocious, many times in bytes/second rather than even kb/second - and my connection speeds to the Internet are in multiple 100mb connections. Have you considered mirror sites or adding bandwidth? Normally our bandwidth is sufficient. We have considered mirror sites also, and we have plans to move our hosting into a local Equinix facility where we will have similar bandwidth to yours and other benefits. Unfortunately we are not quite up to that level of revenue yet. We currently have two T1s through two networks (Savvis Sprint). More than 90% of the time more than 80% of our bandwidth is avaialable. There are occasional short-lived peaks where this is not the case, but those are rare. Rulebase compilation is metered so that each file is generated in about the same amount of time it takes to download the file through a single T1. Generally this pacing leaves our bandwidth mostly open most of the time. However, it appears that something odd has been going on recently with the Sprint side of the network - I suspect that what you've observed is related to some flapping going on under some heavy load conditions and that this has led to a number of dropped packets. I am investigating this further. An event such as this would reduce our bandwidth by more than half and many packets would be lost. _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Error_Bad_Matrix
I'm getting to be pretty sure it's Sprint. After bouncing the router there have been 109 carrier transitions in 3 hours. That's insane. I will be pounding on them. _M At 11:44 PM 3/25/2004, you wrote: Pete, FYI, I was trying to set up log uploads yesterday night and it took me a while to figure out that the FTP connection was unreliable from my server. Packets were being dropped/munged somewhere. I also noted a much lower hit rate on SNIFFER-PHARMACY yesterday, but no indication of matrix problems in the logs today (yesterday's were deleted). Every once in a while my colocator's border router goes on the fritz and starts dropping packets. A reboot usually fixes that issue. If your router checks out fine, you might want to take a look at the routes going from your server to the customers that have indicated a problem and those that have indicated that there is none, that might identify something not so obvious if you run out of ideas. I know how these things go and the worst part is not knowing the source while others expect an quick fix. No big deal on my end in the mean time though. Matt Pete McNeil wrote: snf2check.exe will catch a partial download but it will not catch corruption in the middle of the file. _M At 03:57 PM 3/25/2004, you wrote: I run snf2check.exe against every .snf file downloaded. I just checked it again manually, and no errors were reported. I now have almost 3500 Error_Bad_Matrix entries in today's log. Bill -Original Message- From: Vivek Khera [mailto:[EMAIL PROTECTED] Sent: Thursday, March 25, 2004 12:52 PM To: [EMAIL PROTECTED] Subject: Re: [sniffer] Error_Bad_Matrix On Mar 25, 2004, at 3:39 PM, Paul Lushinsky wrote: I decided to look in my log files for the past several days because of number of Error_Bad_Matrix related messages. I can't find this message in any of my log files until today starting with the update I auto downloaded at 8:15 this morning, and went until the update at noon. While I was look at the log file, another update notice came, so an update was done and the Error_Bad_Matrix message is back. I'm curious if the people who are seeing these messages are running snf2check.exe before making the rule files live. I do so, and have not seen a single instance of this error. Can you run snf2check.exe on the current bad matrix you have and see if it reports an error? This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html --- This message and any included attachments are from Siemens Medical Solutions USA, Inc. and are intended only for the addressee(s). The information contained herein may include trade secrets or privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you received this message in error, or have reason to believe you are not authorized to receive it, please promptly delete this message and notify the sender by e-mail with a copy to [EMAIL PROTECTED] Thank you This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Spam storm?
snf2check does a byte length and partial checksum by default. The first and last few kbytes of the file are encrypted in sequence using Mangler. If any single bit of those two segments is missing or altered then the file will fail to authenticate. The only thing missing is a CRC for the middle parts of the file. In theory this is covered by TCP - but in practice not so much :-( _M At 12:48 AM 3/26/2004, you wrote: How about a byte length compare or checksum of some sort? Matt Pete McNeil wrote: At 06:25 PM 3/25/2004, you wrote: We also saw many BAD_MATRIX errors last night. If the problem was 'wget', shouldn't the snf2check utility detect a corrupt file? Also, we did a manual update yesterday afternoon and there were no 'wget' error messages. The problem got corrected sometime between last night and this morning. Perhaps though some have had trouble throughout the day. At the very least the verification on snf2check should be improved to catch this issue. Updating with a bad ruleset creates many problems. Agreed. I'm looking for some simple ways to do that without changing the rulebase file format. There aren't any simple mechanisms that come to mind. Perhaps there will be no choice but to change the format in order to prevent this possibility. _M -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Thursday, March 25, 2004 7:06 PM To: [EMAIL PROTECTED] Subject: Re: [sniffer] Spam storm? This helps narrow things down. Specifically we know that the rulebase files are not corrupted on the server but during the download. That explains why I haven't been able to recreate a problem in the lab. I have a suspicion that wget may be failing intermittently. Another customer recently had unexplainable, intermittent issues with wget. They replaced wget with code of their own and have had no further problems. Can we narrow this down to wget under heavy traffic conditions perhaps? _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
