[sniffer] Re: Appriver issue
Inserting my 2 cents here since that is all that it is worth. In backing up what Matt said, let me relate a similar example of a problem that occurred a year and a half ago to a major IT security products vendor: At about 6:15 AM PT on a week day in the middle of a normal busy week, their content filtering servers begin to become unresponsive. At first, it was intermittent and hard to pinpoint. But within about 45 minutes, they stopped responding completely. Well, their appliances did what they were designed to do by default configuration, fail safe. Block all access if the content filtering server does not respond. All one had to do though was to log onto the appliance and change the failsafe block to allow. But this is where the fun (not) began. There are hundreds or more of library's, both public and private, as well as schools, that are using those appliances and that content filtering service. Guess what? They are bound by law to have content filtering in place, meaning they could not turn the fail safe off. Companies and schools and libraries started screaming bloody murder and demanded a resolution an hour ago. The content filtering service was finally restored about 2:30 PM if I recall correctly. So, what happened? I mean this is a big company and it should have things in place to prevent this. Right? They did. As much as some one would expect them to. They had 4 servers. The servers were fine, they were still running. There were no software changes, and in fact their tests showed the servers were still responding. They were located at a location with multiple internet connections, and all tests showed the internet connections were all up and working. Power was flowing fine and all UPSs as well as the generator were all fine. Finally, after about 2 hours, the problem was found: My understanding is that a single module in a enterprise router failed but in a way that was hard to find. Once found, the hardware vendor sent a replacement part by courier to replace. My understanding is that it cost them well over 10 grand to eliminate that one single point of failure. And that was just for the hardware. Just goes to prove once again that in IT, 80% of the result is 20% of the cost. That remain 20% of result is what costs the 80%. John T From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Friday, May 18, 2007 9:44 PM To: Message Sniffer Community Subject: [sniffer] Re: Appriver issue I have something that I would also like to clear up. When I indicated that AppRiver had removed it's contact page, it likely just wasn't operating at the time that I was attempting to access it. Considering their issues, it would not be a surprise to see other issues like this caused, but it seemed suspicious since their home page was working and not their contact page. I did note that it was working by the time that it was pointed out that it was up. In no way did I ever believe that Pete or Sniffer had any direct involvement in the system that created these problems, and in no way should this reflect badly on Pete or Sniffer as far as I am concerned. I was slightly miffed after getting off the phone with them where their reaction quite clearly indicated that they were aware of the issue. I suggested that they take their servers off-line due to the issues that were being caused, but I was probably barking up the wrong tree. The servers weren't taken off line for another hour or so, or maybe this is when the delivery servers caught up with the queued E-mail destined for my client. I'm not sure why they didn't act on this sooner. When you have a loop, it is important to stop it, and their multi-homing made it difficult for others to block. One user received about 500 copies of the same message (and also called them), and there were other examples that we saw which were much more limited. I do hope that they didn't choose to introduce new software at 11 a.m. ET on the busiest E-mail day of the week, and that this was only when the problems surfaced... Everyone that deals with significant volumes of E-mail has issues from time to time, and I wouldn't draw conclusions about AppRiver based on just this one circumstance. I would imagine that it is hard to plan for how to deal with a broad scale looping issue, and I'm sure this was a learning experience for them. Matt
[sniffer] Re: Appriver issue
For those of us in the dark about this, can someone explain who Appriver is, and what is has to do with Message Sniffer? Thank you, Michael Stein Computer House - Original Message - From: Kevin Rogers [EMAIL PROTECTED] To: Message Sniffer Community sniffer@sortmonster.com Sent: Friday, May 18, 2007 6:45 AM Subject: [sniffer] Re: Downloads are not working I sent a message earlier to this list but I'm not sure if it went through. We've been hit by this Appriver issue and it is still going on as far as I can tell. One of our users, call him [EMAIL PROTECTED] sent a message to about 70 people. And this message has been bounced 20 or 30,000 times and counting. At first I thought it was this Exchange issue we experienced last year where a single message was sent over and over. But then I saw that all the headers of the bounced emails contained calls to appriver.com and when I checked here I found this thread. In the end, the only thing I could do was completely remove that user's account and it appears to be OK. But who knows? Things appeared to be OK from 7pm until 1am PST when it all started up again. Anyone have any information on this? Thanks Kevin Pete McNeil wrote: Hello Matt, Thursday, May 17, 2007, 2:22:56 PM, you wrote: Appriver, who is somehow involved with Sniffer, is having a ridicolous problem with sending messages over and over again (once every few seconds). They pulled their contact information from their site but didn't take down their servers. I suspect this is putting strain on them and if Sniffer uses their bandwidth for downloads, that could explain things. I'm not sure what the actual issue is (I will get that data later), however I've just been informed that it should be resolved in the next 20 minutes or so. Our rulebase server is on the same network so it is effected. BTW - they did not take down their contact information. It is right where it always has been. _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Appriver issue
Hello Computer, Friday, May 18, 2007, 8:34:27 AM, you wrote: For those of us in the dark about this, can someone explain who Appriver is, and what is has to do with Message Sniffer? Message Sniffer started out as an incubator project inside of MicroNeil Research Corporation. When it was time for SNF to grow up and become it's own company we went looking for partners who could help it grow. We searched long and hard and ultimately picked AppRiver for a lot of good reasons :-) (Note I can still say that after all this time!) ARM Research Labs, LLC was formed to take over the SNF project. ARM is jointly owned by both AppRiver and MicroNeil. In order to capitalize on economies of scale, asymmetrical bandwidth utilization, and an efficient support infrastructure, a number of servers for ARM are hosted along side AppRiver servers - this includes the servers that provide rulebase updates for SNF. Yesterday's events created a tremendous load on the network and that caused some packet loss that caused rulebase downloads to slow down for a time. For more about ARM see: http://www.armresearch.com/ Hope this helps, Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Appriver issue
Pete - Thanks for the reply, but I guess I don't understand what you're saying. Some packet loss and rulebase downloads to slow down for a time don't reflect what happened to me yesterday and apparently not what happened to one of the other posters either when he said that Appriver was having a problem with sending messages over and over again. I received over (at last count) 35,000 messages (almost all of which were bounced replies, from one email from one of our users who sent an email to about 70 people) yesterday. And I had already gone to http://www.armresearch.com/ yesterday and there was nothing there. There is nothing there today that I can see. What happened? I lost an entire day's worth of email because of bounced messages. I didn't sleep last night. I don't even use Appriver. I would hope someone could explain it a little better than that. Thanks. Kevin Pete McNeil wrote: Hello Computer, Friday, May 18, 2007, 8:34:27 AM, you wrote: For those of us in the dark about this, can someone explain who Appriver is, and what is has to do with Message Sniffer? Message Sniffer started out as an incubator project inside of MicroNeil Research Corporation. When it was time for SNF to grow up and become it's own company we went looking for partners who could help it grow. We searched long and hard and ultimately picked AppRiver for a lot of good reasons :-) (Note I can still say that after all this time!) ARM Research Labs, LLC was formed to take over the SNF project. ARM is jointly owned by both AppRiver and MicroNeil. In order to capitalize on economies of scale, asymmetrical bandwidth utilization, and an efficient support infrastructure, a number of servers for ARM are hosted along side AppRiver servers - this includes the servers that provide rulebase updates for SNF. Yesterday's events created a tremendous load on the network and that caused some packet loss that caused rulebase downloads to slow down for a time. For more about ARM see: http://www.armresearch.com/ Hope this helps, Thanks, _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Appriver issue
Hello Kevin, Friday, May 18, 2007, 8:52:47 PM, you wrote: Pete - Thanks for the reply, but I guess I don't understand what you're saying. Some packet loss and rulebase downloads to slow down for a time don't reflect what happened to me yesterday and apparently not what happened to one of the other posters either when he said that Appriver was having a problem with sending messages over and over again. I received over (at last count) 35,000 messages (almost all of which were bounced replies, from one email from one of our users who sent an email to about 70 people) yesterday. And I had already gone to http://www.armresearch.com/ yesterday and there was nothing there. There is nothing there today that I can see. What happened? I lost an entire day's worth of email because of bounced messages. I didn't sleep last night. I don't even use Appriver. I would hope someone could explain it a little better than that. Thanks. I was answering the question - how is AppRiver related to Message Sniffer. I don't have specifics on the problem at AppRiver yet - they are still picking up the pieces, though operations are back to normal afaik. I do know (preliminarily) that the problem occurred when a new piece of software caused some messages with multiple recipients to loop and as a result to be replicated and resent repeatedly. If you are not a user of AppRiver then you shouldn't have been effected. Perhaps if you sent a message to someone who is a user of AppRiver then that might have gotten your messages involved. The only direct effect I'm aware of for SNF users was that for a time rulebase downloads were slowed due to packet loss. Since we use AppRiver for filtering (they, after all are using SNF) some messages that get sent to us apparently did loop to some lists. Also, some email to our accounts was delayed. I would need to know a lot more about your system and the email you lost before I could make any guesses as to what happened there -- but if you're not using AppRiver then you shouldn't have been effected. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Appriver issue
Maybe I caused the confusion. The problem I had was with my customer using appriver. Not with my customers using message sniffer. How can something that happens with rulebase downloads effect your mail server? It shouldn't. I would expect there's a seperate problem with your mail server that jus happened to occur the same day by coincidence I received a call from appriver today explaining that they released a patch that had acted badly on their servers. Which is why appriver customers had problems. Message sniffer resides on your own server so it should never be effected by any outside outages Thank You, Chris Bunting Lancaster Networks 717-278-6639 Sent by my BlackBerry wireless device -Original Message- From: Pete McNeil [EMAIL PROTECTED] Date: Fri, 18 May 2007 21:44:18 To:Message Sniffer Community sniffer@sortmonster.com Subject: [sniffer] Re: Appriver issue Hello Kevin, Friday, May 18, 2007, 8:52:47 PM, you wrote: Pete - Thanks for the reply, but I guess I don't understand what you're saying. Some packet loss and rulebase downloads to slow down for a time don't reflect what happened to me yesterday and apparently not what happened to one of the other posters either when he said that Appriver was having a problem with sending messages over and over again. I received over (at last count) 35,000 messages (almost all of which were bounced replies, from one email from one of our users who sent an email to about 70 people) yesterday. And I had already gone to http://www.armresearch.com/ yesterday and there was nothing there. There is nothing there today that I can see. What happened? I lost an entire day's worth of email because of bounced messages. I didn't sleep last night. I don't even use Appriver. I would hope someone could explain it a little better than that. Thanks. I was answering the question - how is AppRiver related to Message Sniffer. I don't have specifics on the problem at AppRiver yet - they are still picking up the pieces, though operations are back to normal afaik. I do know (preliminarily) that the problem occurred when a new piece of software caused some messages with multiple recipients to loop and as a result to be replicated and resent repeatedly. If you are not a user of AppRiver then you shouldn't have been effected. Perhaps if you sent a message to someone who is a user of AppRiver then that might have gotten your messages involved. The only direct effect I'm aware of for SNF users was that for a time rulebase downloads were slowed due to packet loss. Since we use AppRiver for filtering (they, after all are using SNF) some messages that get sent to us apparently did loop to some lists. Also, some email to our accounts was delayed. I would need to know a lot more about your system and the email you lost before I could make any guesses as to what happened there -- but if you're not using AppRiver then you shouldn't have been effected. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Appriver issue
I think what Peter is try to say is that Sort monster is hosted at Appriver and Appriver had an issue and therefore so did Sort monster. http://www.dnsstuff.com/tools/dnsreport.ch?domain=sortmonster.com Regards David Moore [EMAIL PROTECTED] J.P. MCP, MCSE, MCSE + INTERNET, CNE. www.adsldirect.com.au for ADSL and Internet www.romtech.com.au for PC sales Office Phone: (+612) 9453 1990 Fax Phone: (+612) 9453 1880 Mobile Phone: +614 18 282 648 POSTAL ADDRESS: PO BOX 190 BELROSE NSW 2085 AUSTRALIA. - This email message is only intended for the addressee(s) and contains information that may be confidential, legally privileged and/or copyright. If you are not the intended recipient please notify the sender by reply email and immediately delete this email. Use, disclosure or reproduction of this email, or taking any action in reliance on its contents by anyone other than the intended recipient(s) is strictly prohibited. No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient. -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Rogers Sent: Saturday, 19 May 2007 11:59 AM To: Message Sniffer Community Subject: [sniffer] Re: Appriver issue Thanks for the explanation, and I wasn't trying to blame you - just wanted more info is all. We use Sniffer, but not Appriver. You said that if we don't use Appriver, we shouldn't have been affected, but you also seemed to say that if one of the recipient's of my user's email uses Appriver that might've caused a problem. And also that *some* of Sniffer users might have experienced the problem as well. It sounds like things are still being worked out. I just wanted some kind of verification that they were aware of the problem, were working on it, that they were in some way sorry about what happened...you know - the usual stuff. And I know that you are not an official rep of Appriver or anything, but presently you're all we have in that role ;) Thanks Kevin Pete McNeil wrote: Hello Kevin, Friday, May 18, 2007, 8:52:47 PM, you wrote: Pete - Thanks for the reply, but I guess I don't understand what you're saying. Some packet loss and rulebase downloads to slow down for a time don't reflect what happened to me yesterday and apparently not what happened to one of the other posters either when he said that Appriver was having a problem with sending messages over and over again. I received over (at last count) 35,000 messages (almost all of which were bounced replies, from one email from one of our users who sent an email to about 70 people) yesterday. And I had already gone to http://www.armresearch.com/ yesterday and there was nothing there. There is nothing there today that I can see. What happened? I lost an entire day's worth of email because of bounced messages. I didn't sleep last night. I don't even use Appriver. I would hope someone could explain it a little better than that. Thanks. I was answering the question - how is AppRiver related to Message Sniffer. I don't have specifics on the problem at AppRiver yet - they are still picking up the pieces, though operations are back to normal afaik. I do know (preliminarily) that the problem occurred when a new piece of software caused some messages with multiple recipients to loop and as a result to be replicated and resent repeatedly. If you are not a user of AppRiver then you shouldn't have been effected. Perhaps if you sent a message to someone who is a user of AppRiver then that might have gotten your messages involved. The only direct effect I'm aware of for SNF users was that for a time rulebase downloads were slowed due to packet loss. Since we use AppRiver for filtering (they, after all are using SNF) some messages that get sent to us apparently did loop to some lists. Also, some email to our accounts was delayed. I would need to know a lot more about your system and the email you lost before I could make any guesses as to what happened there -- but if you're not using AppRiver then you shouldn't have been effected. Hope this helps, _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch
