I'd suggest contacting Mr. Cue at Lucid8 directly for that. Dane Cue [EMAIL PROTECTED]
Tom -----Original Message----- From: Micheal Espinola Jr [mailto:[EMAIL PROTECTED] Sent: Tuesday, January 22, 2008 11:58 AM To: NT System Admin Issues Subject: Re: Response from GoExchange, Lucid8 Support I'd like to see citations/references with dates to those statistics and quotes. On Jan 22, 2008 11:40 AM, Tom Strader <[EMAIL PROTECTED]> wrote: > > > > A week or more back a discussion about GoExchange was started and several > comments were generated that made me curious about the GoExchange product. I > inquired with Lucid8 to explain the necessity of its GoExchange product in > light of all the comments made. > > Here is a response from Dane Cue, Support Manager for Lucid8. I am curious > to read your responses. > > Thanks, > Tom Strader > NCBPAC > Server/Network Systems Administrator > ------------------------------------------------ > > > Fear of loss compels us to protect ourselves, our cars, homes, families and > friends. Our concerns for potential loss encourage us to take precautions to > help mitigate risks. This is why most of us try to exercise and why we take > our cars into the shop when they make unexplained noises or motions and buy > insurance policies in case of other catastrophic losses. I seriously hope > that no one's life is in danger from their messaging system, but I am fairly > certain that most companies would be wounded should a catastrophic problem > occur within the Exchange environment. > > Exchange Server 2007 is as self-healing as Exchange has ever been, but > without a healthy hardware platform, drivers, Active Directory, DNS, > network, and regular policy management, it can go belly-up faster than you > can say cardiac arrest. To make matters even worse, many Exchange > Administrators are not current with the latest CPR techniques nor have they > been practicing Preventive Maintenance on their Exchange databases in order > to minimize unplanned downtime. > > Database Health and Maintenance > Much has changed with Exchange since its first release and its underlying > database structure has evolved into an incredibly resilient and reliable > system. Microsoft has built mechanisms to protect the JET database from many > types of corruption, but certain conditions can still cause the ESE engine > and databases to fail. It is for this reason that I decided to include not > only procedures to protect the environment from Storage, Network, Component, > Computer and Site failures, but also procedures for preventing software > failure or most importantly database-failures. > > Even with Exchange Server 2007, database maintenance and diagnostic articles > still appear in the top-twenty support issues for Microsoft Exchange Server. > The Exchange Databases are the Achilles Heel of the entire operation and > according to the META Group, 20 percent of unplanned Exchange downtime is > caused by database failure. Should an Exchange database fail, none of the > solutions discussed above can perform as designed, unplanned downtime occurs > and business losses rapidly accelerate. > > Email traffic and message size continue to put greater demands on messaging > systems, and with regulatory compliance requirements becoming more > widespread these systems are becoming more critical to organizations that > use them. Organizations can no longer ignore the importance of proactively > managing and maintaining the Exchange databases. "Disasters" need to be > avoided at all costs. > > Database health is often overlooked in most Exchange environments and many > administrators mistakenly conclude that the database stores will take care > of themselves as part of Exchange's nightly online maintenance process. In > reality, this process is important but is only a precursor to complete > database maintenance. Ignoring the importance of preventative maintenance > can be catastrophic and the fallout cuts across all departments of an > organization. > > According to Microsoft, 90 percent of Exchange Administrators fail to carry > out preventative maintenance of the Exchange Databases until after a > disaster strikes. A few of the reasons for this have been identified as; > > Low understanding of the issues and how to address the problems. > No time, resource, or budget to address the maintenance task. > Non-availability of test equipment (and the high impact of testing in a > production environment). > Incorrect assumption that the risk of doing nothing outweighs the risk of > pro-activity. > Technically capable but see preventative maintenance as boring and time > consuming. > > > When it comes to your messaging systems, an ounce of prevention or in this > case implementing a preventative maintenance solution is worth a pound of > cure aka Disaster Recovery which of course means, unplanned downtime, angry > users and executives, cancelled plans, more stress than anyone needs and > late nights spent in the server room putting humpty dumpty back together > again. Avoid this by implementing an automated tool that will help you and > your Exchange enabled organization be as efficient and effective as > possible. > > Microsoft-provided tools > > What about the tools that Microsoft provides for database maintenance: > ESEutil and ISinteg? These tools are extremely powerful and should not be > handled without a good knowledge of their use and impact. You should test > these tools in a lab environment on backups of your production databases in > order to better gauge their effectiveness and performance. > > ESEutil > It is my opinion that this tool must create some conflict in Redmond. On one > hand, Exchange administrators need access to the tool in order to perform > database maintenance and troubleshooting. On the other hand, ESEUTIL is not > unlike a loaded gun. In short, ESEUTIL is the one tool you can use to verify > the integrity of the Exchange stores, perform hard and soft recoveries of > the stores, copy the database files, repair the files or defrag the files. > Unfortunately, many of the commands require the Exchange stores to be > dismounted so its use is fairly limited in non-disaster scenarios. > > Use of the ESEUTIL command is essential in restoring Exchange databases, and > in repairing them in the event of a catastrophic failure. The tool can also > be used to help diagnose issues and checksum database pages as well as the > database header, but it was not designed for casual use. > > ISinteg > ISinteg does not perform as much "heavy lifting" as the ESEutil does. > ISinteg browses the exchange store tables and indexes for inconsistencies. > For example, attachments are not actually contained in an MAPI message. > Instead they exist in an attachment table and are referenced from the > messages. When this document is deleted from the table, the messages that > contained links to the document should also be removed automatically. > Exchange usually repairs problems such as these online during its off-hours > maintenance schedule but some objects can be missed by the online tools > since the database is still in use while online maintenance occurs. > Inconsistencies such as these can be remedied by using the ISinteg utility. > > http://www.themssforum.com/ExchangeAdmin/GoExchange/ > What is the ongoing need it is filling, besides creating unnecessary > downtime? > You can run ISinteg on a dismounted database - pretty much ANY dismounted > database - and it will give you warnings. > > When running ISinteg against any database you will always find minor > inconsistencies (warnings), this statement is about 99% true, the > implications that the author suggests though are incorrect. The reason for > the above statement being true could be; > Microsoft incorrectly reports possible problems when there are none and > doesn't know what they are talking about (maybe the engineers were bored and > just like to scare people) > Microsoft's database has so many issues that it can never truly be integral > (Hence the reason they have been trying to move away from JET for years and > continue to do so, i.e. next estimate is 2012) > Microsoft reports all problems regardless of apparent triviality because > they could lead to larger issues, they usually don't but they "could" > > Our position through pre-analysis and post-recovery of 100's of damaged or > incorrectly performing databases anecdotally proves that some of the more > minor warnings are "usually" trivial but that they can cascade into more > serious issues, and that some combinations of trivial warnings can also be > more serious taken together than as separate one offs. In other words even > the most minor warnings can be symptoms of potentially larger problems. > > The authors position seems to be A or B, ours is C. Since Microsoft gives no > definitive guidance we attempt to remove all errors and warnings without > regard to category of problem. > > In general we can agree that IF you are ONLY getting minor warnings it > should in general not be a cause for great concern. That being said we > refuse to categorize all warnings into the "unimportant" category and we do > this from experience and perhaps more importantly Microsoft themselves will > not do this either. The argument that you will gain less by performing > maintenance against a database with only minor warnings than if performing > the same maintenance against a database with severe warnings or more > importantly errors is obvious. The problem is that the author assumes you > will only get minor warnings and is implicitly stating that he knows without > looking that the database is 100% healthy, an interesting position to take > to be sure. > > Then again for those that can't be as sure as the author, how are you > supposed to figure this out? Since checking fully transacted integrity or > running a full blown maintenance process are both offline processes I would > personally question any ones knowledge or common sense that says that they > "Know" the absolute state of the database. Nothing in this world is self > maintaining, not your car, your teeth or your home. Take care of them now > because it's ALWAYS less expensive to be proactive rather than reactive. > > NOTE: We understand the internal Exchange db schema, jet engine, and > transaction management within Exchange better than any party outside of the > Microsoft Exchange developers themselves. > > Again we need to stress that the author of the post authoritatively > pre-supposes this will be true (only a few trivial warnings will ever be > generated) if he simply believes in the technology so much then why does > Microsoft provide utilities to fix issues that he apparently refuses to > believe can ever happen. The bottom line is, that no one, including > Microsoft or MVP's will ever know what's going within an Exchange database > until it's inspected, and currently we can only inspect when the EDB is > offline (Online MAPI access etc. cannot tell us anything other than general > accessibility) Anything else is purely a guess, unless of course the EDB > dies then you know you have a problem and the only thing you don't know is > A: how long it's going to take to recover, B: how much data you are going to > lose C: what the impact to your company will be, i.e. lost productivity, > sales etc because this almost always happens at the beginning of the workday > and of course on the worst day possible and of course D: how did this happen > and E: why is Microsoft telling me to run these utilities that they said > were so dangerous F: when will I get to go home G: Why is everyone in the > company blaming me for this downtime and said I should have known...... > > I should mention we do provide an "Analysis" mode where you can actually see > if you are having any issues that are beyond what an individual deems > "trivial" if you get any errors then of course you have issues with the > logical integrity of the database, our position is that if you are going to > take the time to find out if you have any issues then you might as well fix > them (even the most trivial) while you are at it because analysis is an > offline process as well and takes about the same amount of time as fixing > issues while checking. > > This is because ISinteg is a utility and not a database engine itself. > > Again the author is getting a little confused. ISinteg calls into Store.exe > to do its work; Store.exe (Exchange) then uses Jet (the edb engine) to > actually manipulate things. If you think that's not true shut down the > store.exe and see if ISinteg will work. > > So no ISinteg is not a database engine it's a wrapper utility (it basically > is nothing but a utility to pass parameters along to store.exe (exchange) to > tell it to start looking at it database and active directory and figuring > out if anything is wrong at a logical level with the database. > > So although technically a correct statement it's as silly as saying the > command line is "not an operating systems" technically true, it's how you > interact with the operating system, and the ISinteg utility is one way you > can interact with the database engine bits of Exchange. > > You can tell ISinteg to fix those warnings, and run it multiple times until > there aren't any more warnings, and you've just detuned your indexes. I > expect (but admit that I do not know it for sure) > > > ISinteg (actually exchange itself, remember ISinteg is just a command line > processor that talks to Exchange) does not de-tune indexes. > > Performance (tuning) is dependent on exchange and set automatically for the > most part (more to do with buffers and cache sizes really) and many of these > settings the user has no control over, if the author is saying that > correcting problems (even minor ones in temporary indexes) causes > performance degradation then he doesn't know what he is talking about. > > Regardless of the terms used a rebuilt or corrected index, table, or > dictionary by its nature would be better "tuned" than one in which the > corresponding entries were incorrect. > > The author of the statement above obviously does not really understand the > architecture of exchange, the jet engine, the b-tree mechanism, the database > schema or the store application (exchange) > > Db integrity is checked 100% with every single online backup that you take. > 100%. You can't back up a corrupted database with online backup. > > Again the author is confused and has just enough information to make an > incorrect assumption (that physical and logical integrity are the same > things or that lack of problems in one area guarantees a lack of problems in > another area) > > To be precise there are two levels of exchange structure, the physical, > which has to do with pages and internal b-tree layouts, and the logical > (tables, indexes etc) ISinteg does not tell exchange to look for or fix > physical issues, it deals with the logical issues, ESEutil deals with the > physical problems (which are always more serious by nature) > > When doing an online backup (full only, incremental backups don't do this) > if there are physical page failures then an online backup will not complete > (basically if a pages Checksum/ECC is bad) but that is not the same as > "logical" integrity. > > Logical integrity is saying, in essence, the schema (tables and indexes) are > complete and trust worthy and all relate to what they should relate to, its > saying that if a message entry exists, that the headers are there, the body > is there, the attachments are there, and accessible etc. that is "Database > is Integral" the online backup does not do or attempt to do this. > > Physical integrity is saying that the b-tree pages are intact, of course > those pages could be part of tables, or indexes or dictionaries (you don't > know at the Physical level, these are just "chunks" of data to the backup > engine) this is similar to sectors and FAT or NTFS tables on the disk, vs. a > document. The online backup only makes sure the sectors are ok; it doesn't > look at or understand document structure. > > Therefore you can have 100% integral pages with a damaged non-integral > database (table index level) in other words a "Healthy" page does not make a > "Healthy" database, where as a "Damaged" page would always contribute to a > "Damaged" database (how damaged depends on what the page ends up belonging > to, two damaged pages in the primary system dictionaries would be very bad, > in a index easily fixed by an index rebuild etc) > > Physical problem detection would always be handled by running an online > backup which would detect that level of corruption (with DigiVault > integration this happens seamlessly) in which case GOexchange would refuse > to run (backup failure) but only in the time frame where it happens (if you > were running CDP that did a 90 day full, with minute based incremental your > EDB's are not necessarily consistent within that 90 day window because for > incremental this physical page check doesn't happen) > > And in Exchange 2007 sp1, it's regularly checked on an ongoing basis. > > Again the Author is showing some confusion here. The ECC algorithm for page > auto correction is indeed better (it's much more tolerant than previous > versions of the ECC) So yes 2K7 does a better job of maintaining It's > physical integrity (writing pages to disk in a way that is fault tolerant) > logically it also does a better job during online maintenance to carry out > more logical correction operations, but there are still issues that can only > be corrected in offline mode, and although page reuse has been in place for > some time, offline compaction will still do a better job of optimizing the > EDB than online compaction, and during that phase indexes can also be > completely rebuilt and "tuned" if you want to use that word. > > Bottom line is some things can only be done offline, hence Microsoft's > continued inclusion of that functionality in both Jet and Store.exe > (Exchange) which is controlled via ISinteg and ESEutil. > > We agree downtime needs to be managed effectively and we are working on a > hot-swap online version of GOexchange that minimizes this to a few minutes > or seconds, until then it's up to IT to manage this downtime correctly, the > longer time between maintenance cycles the higher the probability that > something could have gone wrong and cascaded into a potential about to be > serious problem. At that point your are going to be in "Cancer" mode, forced > to run off of your backup without up to date data and no idea how long your > patient (database) has to live (because you can't or shouldn't take him > offline for a diagnostic according to the author, just wait until he dies > again and then rewind via backup or bring in the surgeons once you know a > terminal disease has begun ravaging the client) > > * You don't run utilities that are designed to fix problems by slicing > out chunks of data if problems are found, as regular maintenance. That's > just crazy. > > > Here the Author has gone from uninformed to completely out into left field. > We agree, it would be crazy to run utilities that "Slice out chunks of > data". ISinteg and ESEutil are utilities provided by Microsoft to help > maintain your Exchange database. ESEutil has the very serious potential of > causing damage to the database if used incorrectly and using ISinteg out of > sequence without doing other things can make this step non-effective. > > One reason we built GOexchange is because people were using the utilities > incorrectly, this became so prevalent that Microsoft now recommends not > using the utilities unless absolutely necessary and that you have a very > high level of expertise or are being guided by PSS. They don't say they have > no value, or what necessary actually means, they have just determined that > for the average IT person this is too risky unless you are currently > experiencing problems (see cancer reference above) > > For example: when using ESEutil with a /p this runs a hard repair on a > database. There are very few times to run this switch due to the destructive > results it can cause. This switch will slice chucks of data out of the > database if problems are found. (this is a last resort and usually if you > have good backups and complete log sets there are better alternatives) I > ensure you that GOexchange does NOT run this against the database ever the > set of functionality we use is non-destructive (if the data is intact it > stays intact, if it's already damaged beyond access then it may remain > damaged at the worst, but most likely become recoverable in whole or part). > > * You don't intentionally incur unnecessary downtime on a system > designed to be up all the time., as maintenance, when all the critical > maintenance task already run online > > > The key words here are "unnecessary downtime" of course. The problem is > Microsoft doesn't really give anyone a way to know if it's actually > necessary or not until you start seeing performance problems, experience > errors getting mail, or your store will not mount (worst case scenario), and > in reality diagnosis isn't productive when problems have gone too far (false > positives and negative tests can result, in which case index rebuilding etc > might be necessary). In essence they have had so many people misuse the > utilities that it's safer to run blind then risk improper usage. GOexchange > eliminates the issues that Microsoft is worried about in so far as improper > usage and lets you plan preventative maintenance downtime in order to avoid > unplanned/necessary downtime > > We won't tell you (even in diagnostic mode) that you don't need to run the > product for previously stated reasons, If we did this someone would come out > and complain that we said they didn't need to run the product when they > really did (always just after an incident occurred) This is why we call it > "Preventative" Maintenance and don't try and predict outcomes other than > "Issues exist, let's fix them" BEFORE they cause UNPLANNED DOWNTIME, data > and productivity loss etc > > * Even the notion of GOexchange solving something that Microsoft > couldn't is completely ludicrous. > > > Couldn't and don't are two different things. They could provide the same > ease of use expert driven system, and integration with other tools required > to complete the steps safely (backup, notification etc) and make the front > to back process simple, they have chosen not to. > > To be clear we don't claim that we can do "more" than Microsoft could, we > claim we do much more than they provide; in essence we automate this process > so it is flawless. Expert knowledge has been built into the product to do > away with the need for the manual, error prone processes, and the meticulous > attention required to perform what we believe should be routine maintenance. > Please follow the link below for a side by side comparison. > > http://www.lucid8.com/product/products_manual.asp > > > A final note. We have heard from Microsoft themselves which have stated on > various occasions (to us and other customers) > > "With 2K7 you don't need GOexchange anymore like you did with 2K3" > Rather humorously we also previously heard "With 2K3 you don't need > GOexchange anymore like you did with 2K" > > in other words with every generation of Exchange it's admitted we provided a > valuable service for the older product but we were no longer "Needed" for > the newest product, that is until the next generation is released that > solves all the problems that didn't exist. Please note that the Microsoft > utilities that you will never need are still provided, and Lucid8 customers > continue to report that running the process via GOexchange solved 'odd' or > 'performance' issues that Microsoft support was at a loss over. > > Microsoft makes a great product that benefits from occasional maintenance. > Microsoft actions (which speak louder than words) own experience and our > customers experiences around the world agree that GOexchange tunes the EDB > by eliminating logical errors, reclaiming "excessive" slack space, > rebuilding indexes etc. and contributes to a more robust exchange > environment. > > If you run GOexchange against a perfectly health database at worst you will > get peace of mind, if you run it against a sub-par database you will get a > robust and healthy database, it's a win-win situation, alternatively you > have the option of not knowing and hoping instead. > > Not to put too fine a point on it, many experts and others want you to pay > them vs. using an automated and therefore non-revenue generating > process/product. > > By now, you should fully realize the importance of automation in failover, > replication, backup and monitoring. You can attempt to create your own > preventative maintenance solution by creating a manual process utilizing > existing Microsoft Exchange utilities, however; > > The command line interface of ESEutil and ISinteg are not intuitive, nor is > detailed information supplied about when to apply them or in what order, nor > what to do if something goes wrong. > When utilizing these utilities you're essentially performing open-heart > surgery on the Exchange database, so you must know precisely what, when and > how to use each utility. > The process varies between the different versions of Exchange, e.g. if you > have a process worked out on 5.5 it will not work on 2000, 2003 or 2007. > Use of these utilities require manual intervention and must be used after > hours. Therefore, be prepared to spend an inordinate amount time when > implementing a manual solution. > > > > > GOexchange, from Lucid8 is a product that can help complete your > high-availability scenario by providing expert tools to automate the > maintenance of your Exchange environment. GOexchange is a tried, proven and > safe solution that's rich in features like notification, backup integration, > secure clean, as well as detailed and summary reports for every job. > Furthermore, administrators can configure, schedule, and review maintenance > tasks for all Exchange servers within the organization from a single > centralized console. > > So instead of looking up the appropriate ESEutil and ISinteg commands or > notifying all effected users of an impending maintenance, running backups > and manually taking systems offline, GOexchange leverages a series of > Microsoft utilities and API's to create an automated preventative > maintenance solution for optimally maintaining Microsoft Exchange Server. > This allows the administrator to schedule unattended regular maintenance > jobs for any Exchange server, storage group or individual store in their > environment including Exchange 5.5, 2000, 2003, 2007 and even clustered > servers. This is especially important in environments where different > versions of Exchange are used as the syntax for many maintenance commands > are different. More importantly the Exchange administrator can now focus on > moving the organization forward rather than spending inordinate amounts of > time trying to keeps things afloat. > > > Conclusion and Summary > It is important for you to follow basic, proactive maintenance processes to > keep your Exchange environment running smoothly. Those businesses that > require dial-tone availability, regulatory compliance and ease of recovery > will need to put additional processes and technology in place. Server-class > hardware, regular backup procedures, system monitoring, archiving, > eDiscovery, recovery, and preventative database maintenance tools are just > a few of the things needed to help you maintain a rock-solid Exchange > environment. > > > > > > > > > > > > > > > > > > > -- ME2 ~ Upgrade to Next Generation Antispam/Antivirus with Ninja! ~ ~ <http://www.sunbelt-software.com/SunbeltMessagingNinja.cfm> ~ ~ Upgrade to Next Generation Antispam/Antivirus with Ninja! ~ ~ <http://www.sunbelt-software.com/SunbeltMessagingNinja.cfm> ~
