I don't know everything that transpired, but I can say that it started when one server was taken out of production, put into an isolated test environment and used for a "smoking crater" DR test scenario, and then brought back into production. That was probably a bad idea. I think the attempts to recover that one server may have contributed to the downfall of all the rest.
It would probably have been better to just rebuild that one server at the outset. From: Michael B. Smith [mailto:[email protected]] Sent: Thursday, September 29, 2011 10:34 AM To: MS-Exchange Admin Issues Subject: RE: Backup of Exchange 2010 I've asked a PFE to review Rob's case and he's asked for additional data from the case owner. After that review, I'll forward the information to Rob and he can choose whether to share here... Now... I've never heard of anything like this before and I've never run into anything like this before. I've been deploying clustering for almost a decade with Exchange and while I've had some issues, nothing like this. I've even had a user remove the cluster object from AD and been able to recover without a rebuild. Clustering is finicky - no question about it. But once you set it up, you leave it alone, and it just WORKS. IMHO. Regards, Michael B. Smith Consultant and Exchange MVP http://TheEssentialExchange.com From: Maglinger, Paul [mailto:[email protected]]<mailto:[mailto:[email protected]]> Sent: Thursday, September 29, 2011 10:16 AM To: MS-Exchange Admin Issues Subject: RE: Backup of Exchange 2010 Wow... After being reassured about how reliable Exchange 2010 was, this is making me step back a bit. I've even had people ask why we still needed tape backup if we were using DAGs. I wasn't ready to let go of my backups of the databases just yet, but was willing to let go of the system states. So if the config files hadn't been copied to the other servers, would this still have been a, pardon the phrase, "cluster-uh... mess"? I'd be interested to hear other thoughts on this from the peanut gallery. From: Campbell, Rob [mailto:[email protected]]<mailto:[mailto:[email protected]]> Sent: Wednesday, September 28, 2011 5:40 PM To: MS-Exchange Admin Issues Subject: RE: Backup of Exchange 2010 I was on vacation when it happened. I can only relate what I came back to. Apparently something bad happened to the cluster config on one of the servers, and it was spread to the other servers, either through replication or by attempts to correct it manually. PSS advised someone to copy the config files from one server to all the rest, and that totally broke everything. Then they said to do a system state restore of all the servers, and it was discovered we didn't have system state backups. I spent a very long night building a new server and copying DBs. After that is was a "rolling rebuild" process, nuking the clustering on each one, uninstalling Exchange, then re-installing and adding it back into a new DAG. From: Michael B. Smith [mailto:[email protected]]<mailto:[mailto:[email protected]]> Sent: Wednesday, September 28, 2011 5:31 PM To: MS-Exchange Admin Issues Subject: RE: Backup of Exchange 2010 I'd love to hear more about that. Care to share? Regards, Michael B. Smith Consultant and Exchange MVP http://TheEssentialExchange.com From: Campbell, Rob [mailto:[email protected]]<mailto:[mailto:[email protected]]> Sent: Wednesday, September 28, 2011 6:20 PM To: MS-Exchange Admin Issues Subject: RE: Backup of Exchange 2010 DAGs are nice, but they need to make the clustering more resilient. We just had to go through a process of completely tearing down and rebuilding all 6 of our mailbox servers because the cluster configuration got corrupted. From: Michael B. Smith [mailto:[email protected]]<mailto:[mailto:[email protected]]> Sent: Wednesday, September 28, 2011 5:12 PM To: MS-Exchange Admin Issues Subject: RE: Backup of Exchange 2010 Yeah, that was the "best practice" preached prior to Exchange 2010 RTM and at RTM. Today, SingleItemRecovery would be recommended instead. We'd still recommend a third DAG copy though. Preferably in a remote data center. Just not lagged. Regards, Michael B. Smith Consultant and Exchange MVP http://TheEssentialExchange.com From: Maglinger, Paul [mailto:[email protected]]<mailto:[mailto:[email protected]]> Sent: Wednesday, September 28, 2011 6:09 PM To: MS-Exchange Admin Issues Subject: RE: Backup of Exchange 2010 I was told in the Exchange 2010 class I told that you could have a 3 member DAG, the third being lagged in case your database became corrupted and replicated to the second DAG, in which case you could roll back. You would have data loss, but at least you would have a recover point. From: Michael B. Smith [mailto:[email protected]]<mailto:[mailto:[email protected]]> Sent: Wednesday, September 28, 2011 5:00 PM To: MS-Exchange Admin Issues Subject: RE: Backup of Exchange 2010 Please elucidate. Regards, Michael B. Smith Consultant and Exchange MVP http://TheEssentialExchange.com From: Maglinger, Paul [mailto:[email protected]]<mailto:[mailto:[email protected]]> Sent: Wednesday, September 28, 2011 5:58 PM To: MS-Exchange Admin Issues Subject: RE: Backup of Exchange 2010 I was told to recover from a corrupt database. From: Michael B. Smith [mailto:[email protected]]<mailto:[mailto:[email protected]]> Sent: Wednesday, September 28, 2011 4:41 PM To: MS-Exchange Admin Issues Subject: RE: Backup of Exchange 2010 Why on earth would you want to lag a DB? Have you ever tried recovering from a lagged DB? Regards, Michael B. Smith Consultant and Exchange MVP http://TheEssentialExchange.com From: Jonathan Link [mailto:[email protected]]<mailto:[mailto:[email protected]]> Sent: Wednesday, September 28, 2011 5:13 PM To: MS-Exchange Admin Issues Subject: Re: Backup of Exchange 2010 Or can you have dbs offsite? And then can you lag those dbs? How frequent are those "oops, where'd that message go?" issues? And then, on average, how far back do you have to go? On Wed, Sep 28, 2011 at 4:55 PM, Steve Ens <[email protected]<mailto:[email protected]>> wrote: Do you have your databases at different sites? On Wed, Sep 28, 2011 at 3:46 PM, Joseph Heaton <[email protected]<mailto:[email protected]>> wrote: If I have my mailbox servers setup in a DAG, is the only reason to backup to disk/tape to cover the "oops, where'd that message go?" issues? If you do backup to disk/tape, what type of retention do you have? And, yes, I know the retention issue is extremely subjective, depending on legal requirements, etc. Just curious. --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe exchangelist --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe exchangelist --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe exchangelist --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe exchangelist --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe exchangelist --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe exchangelist --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe exchangelist --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe exchangelist ************************************************************************************************** Note: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. ************************************************************************************************** --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe exchangelist --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe exchangelist ************************************************************************************************** Note: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. ************************************************************************************************** --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe exchangelist --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe exchangelist --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe exchangelist ************************************************************************************************** Note: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. ************************************************************************************************** --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected] with the body: unsubscribe exchangelist
