I sure hope Justin lets us know what the problem really was, after all this..
Jeff Fitzwater OIT Network Systems Princeton University On May 15, 2008, at 3:56 AM, Whisper wrote: > Justin, I have alwasy been under the impression that Network Engineers > primary role was going around constantly proving that the Network is > not the > problem. :) > > Your rant, I suspect, is more or less repeated on daily basis by > Network > Engineers all around the world. > > On Thu, May 15, 2008 at 3:41 PM, Justin Shore <[EMAIL PROTECTED]> > wrote: > >> Nathan wrote: >> >>> Proceed by elimination. If there is someone else in the office (I >>> suppose the T1 is not just for one person) whose Outlook is *not* >>> slow, and especially if "someone else" can be extended to "everybody >>> else" then the problem is not the network. >>> >>> Outlook can have severe speed/response problems when not kept >>> healthy; >>> most notably there's something called PST files that have to be kept >>> at a reasonable size, or re-indexed or something, and people who >>> like >>> to keep all their mail tend to run into that. >> >> Here's a long account of a similar battle over PSTs that I fought. >> >> I fought a 'blame-the-network' battle at a customer's site a couple >> years ago. We built a brand-new GigE greenfield network in a new >> building and help the customer move into their new digs. Shortly >> thereafter a certain group of users started complaining that their >> computers were horribly slow, most especially Outlook. This reached >> upper management before it came back down to us contractors so it >> was a >> huge deal when it landed at our feet. >> >> First thing we did was narrow down exactly who had the problem and >> who >> didn't. 95% of the complaints were "me too!" complaints and weren't >> legitimate. The remaining 5% were isolated to one group of users >> in one >> specific area of the new building. Their IT staff that was working >> on >> this problem with us immediately blamed us again because "it had to >> be >> the network's fault because all the users are in the same physical >> vicinity". I showed them graph after graph of the network I/O from >> the >> Exchange servers through the core and down through the uplinks to >> distribution. In the end we ended up graphing every affected users' >> port. The graphs did not help; we were still to blame. >> >> Finally one day I sat down with the squeakiest user and had her >> show me >> exactly what was slow and the steps she took to make that happen from >> minute 1 of her walking into her office. I had her shut down and >> start >> from a cold boot. She commented that the login process was faster >> than >> normal and asked what I'd done to fix it (grrr). She fired up >> Outlook >> and I noticed that it was very slow. She said that it was faster >> than >> normal. Finally Outlook came up and she started scrolling through >> her >> email. She selected a message and waited 10 seconds or so for the >> message to come up. Then she'd try to save the attachment to the >> desktop and it would take 4-5 minutes (for a 20MB attachment). She >> continued on with her daily routine and started scrolling down >> there her >> Outlook folders. I stopped her when I saw "Inbox, Sent, Drafts, etc" >> scroll by more than once. This was the sign I was looking for. I >> took >> the wheel at this point and started counting. She had 8 (count them, >> EIGHT) sets of default Outlook folders because she had 8 PSTs >> mounted in >> Outlook. She explained that she hits the Exchange PST hard limit >> of 2GB >> every 8-10 months. The company's IT folks would export everything >> to a >> new PST to give her a fresh inbox. Then they'd mount it in Outlook >> so >> she could have access to it (it was tax stuff so Legal wouldn't let >> her >> delete anything, literally). I started hunting for the PSTs and >> found >> them on an old file server, one that we had no idea was related to >> the >> mail system. She was mounting 8 roughly 2GB PSTs across the >> network to >> Outlook on a PC running XP w/ 128MB of RAM. Wonderful. >> >> But it gets better. I noticed that her inbox wasn't on the server >> but >> was instead in a PST on the same file server and her email was set to >> deliver to PST, not Exchange directly. In this situation the way >> Exchange works, email is held on the server for PST users until they >> bring their Outlook online. OL then downloads the queued up email >> and >> stuffs it into the PST. Well, the PST was stored on the server so >> the >> client would have to manipulate the PST on the server. >> >> Oh, but it gets better still. A few days later one of sys admins was >> looking the newly discovered file server that was apparently >> critical to >> the function of the mail server. From across the room we here loud >> profanity and run over to see what happened. He discovered that the >> idiot IT staff set up Windows to compress the non-RAIDed drive that >> contains all the user PSTs and home directories because they ran >> low on >> drive space about a year earlier. Before a user's OL client can >> modify >> the PST the server has to decompress the entire PST, then write the >> changes for the client, and recompress the PST and then write it >> back to >> disk. The server was a low-end MS box with 256MB of RAM with no RAID >> and a backup that usually failed. Oh, and that sys admin also >> discovered shortly thereafter that all of the users created in the >> past >> year and a half were set to deliver to PST because of, you guessed >> it, >> another drive space issue. Isn't that nice. >> >> All the users that reported this problem turned out to be users that >> handled tax data and couldn't delete any email. That's why that >> group >> of users all experienced the problem. Every single one of these >> users >> were mounting 2-8 2GB PSTs across the network. Those that shutdown >> at >> night would come in at 8am and fire up their computers. A couple >> dozen >> different users would all try to pull down their PSTs from the >> compressed file system of the poor server. So it wasn't the >> network's >> fault. The network was running like a champ. The POS server put >> into >> mission critical service by incompetent IT staff was to blame. We >> spent >> weeks troubleshooting the problem and trying to convince management >> that >> the network was fine. In the end I had to sit down with a user, >> watch >> everything that they did and then analyze their steps to figure out >> what >> was causing the problem. Oh, and the reason it was faster the day I >> worked with her was because we did this mid-morning, not at 8am. Did >> anyone ever apologize (even figuratively) to the network folks? >> Nope. >> Of course not. >> >> >> As a network engineer I've found that the vast majority of my job is >> helping other people find their problems. The network seldom >> breaks and >> when it does it's not subtle; it's catastrophic. Even highly skilled >> technical people still blame the network when their stuff doesn't >> work >> right (after all my network is just a bunch of tubes, right?). >> Networking is like mysterious dark magic that no one seems to >> understand. It's the gremlins on the wire that causes Windows to >> crash, >> not poor programming and a lack of QA. Networking is simply not >> understood by most people and it's human nature to fear and loathe >> what >> they don't understand. To be able to do my job effectively I have to >> know my shit and everyone elses' well enough to know how something >> works >> when it inevitably breaks. Had I not come into networking with a >> systems background and were I not a quick study under fire I would >> not >> be good at what I do. Did something "suddenly" break that must have >> been caused by the network maintenance I did last week? No, it's the >> fact that it never worked to begin with and you never actually >> tested it >> when you deployed it a year ago. It wasn't until a user tested it >> for >> you that you became aware of the fact that it wasn't working. It >> just >> happened to come a week after I did maintenance on an unrelated >> device >> on an unrelated network. But I'm going to spend all morning sniffing >> and decoding traffic to help you realize that this device off to the >> side over here couldn't possibly be involved. *sigh* Story of my >> life. >> >> </OT RANT> >> >> Justin >> _______________________________________________ >> cisco-nsp mailing list [email protected] >> https://puck.nether.net/mailman/listinfo/cisco-nsp >> archive at http://puck.nether.net/pipermail/cisco-nsp/ >> > _______________________________________________ > cisco-nsp mailing list [email protected] > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
