I appreciate your efforts - especially in posting back their response - but I have no desire to shag this ball.
On Jan 22, 2008 12:25 PM, Tom Strader <[EMAIL PROTECTED]> wrote: > I'd suggest contacting Mr. Cue at Lucid8 directly for that. > > Dane Cue [EMAIL PROTECTED] > > Tom > > > > > -----Original Message----- > From: Micheal Espinola Jr [mailto:[EMAIL PROTECTED] > Sent: Tuesday, January 22, 2008 11:58 AM > To: NT System Admin Issues > Subject: Re: Response from GoExchange, Lucid8 Support > > I'd like to see citations/references with dates to those statistics and > quotes. > > On Jan 22, 2008 11:40 AM, Tom Strader <[EMAIL PROTECTED]> wrote: > > > > > > > > A week or more back a discussion about GoExchange was started and > several > > comments were generated that made me curious about the GoExchange > product. I > > inquired with Lucid8 to explain the necessity of its GoExchange > product in > > light of all the comments made. > > > > Here is a response from Dane Cue, Support Manager for Lucid8. I am > curious > > to read your responses. > > > > Thanks, > > Tom Strader > > NCBPAC > > Server/Network Systems Administrator > > ------------------------------------------------ > > > > > > Fear of loss compels us to protect ourselves, our cars, homes, > families and > > friends. Our concerns for potential loss encourage us to take > precautions to > > help mitigate risks. This is why most of us try to exercise and why we > take > > our cars into the shop when they make unexplained noises or motions > and buy > > insurance policies in case of other catastrophic losses. I seriously > hope > > that no one's life is in danger from their messaging system, but I am > fairly > > certain that most companies would be wounded should a catastrophic > problem > > occur within the Exchange environment. > > > > Exchange Server 2007 is as self-healing as Exchange has ever been, but > > without a healthy hardware platform, drivers, Active Directory, DNS, > > network, and regular policy management, it can go belly-up faster than > you > > can say cardiac arrest. To make matters even worse, many Exchange > > Administrators are not current with the latest CPR techniques nor have > they > > been practicing Preventive Maintenance on their Exchange databases in > order > > to minimize unplanned downtime. > > > > Database Health and Maintenance > > Much has changed with Exchange since its first release and its > underlying > > database structure has evolved into an incredibly resilient and > reliable > > system. Microsoft has built mechanisms to protect the JET database > from many > > types of corruption, but certain conditions can still cause the ESE > engine > > and databases to fail. It is for this reason that I decided to > include not > > only procedures to protect the environment from Storage, Network, > Component, > > Computer and Site failures, but also procedures for preventing > software > > failure or most importantly database-failures. > > > > Even with Exchange Server 2007, database maintenance and diagnostic > articles > > still appear in the top-twenty support issues for Microsoft Exchange > Server. > > The Exchange Databases are the Achilles Heel of the entire operation > and > > according to the META Group, 20 percent of unplanned Exchange downtime > is > > caused by database failure. Should an Exchange database fail, none of > the > > solutions discussed above can perform as designed, unplanned downtime > occurs > > and business losses rapidly accelerate. > > > > Email traffic and message size continue to put greater demands on > messaging > > systems, and with regulatory compliance requirements becoming more > > widespread these systems are becoming more critical to organizations > that > > use them. Organizations can no longer ignore the importance of > proactively > > managing and maintaining the Exchange databases. "Disasters" need to > be > > avoided at all costs. > > > > Database health is often overlooked in most Exchange environments and > many > > administrators mistakenly conclude that the database stores will take > care > > of themselves as part of Exchange's nightly online maintenance > process. In > > reality, this process is important but is only a precursor to complete > > database maintenance. Ignoring the importance of preventative > maintenance > > can be catastrophic and the fallout cuts across all departments of an > > organization. > > > > According to Microsoft, 90 percent of Exchange Administrators fail to > carry > > out preventative maintenance of the Exchange Databases until after a > > disaster strikes. A few of the reasons for this have been identified > as; > > > > Low understanding of the issues and how to address the problems. > > No time, resource, or budget to address the maintenance task. > > Non-availability of test equipment (and the high impact of testing in > a > > production environment). > > Incorrect assumption that the risk of doing nothing outweighs the risk > of > > pro-activity. > > Technically capable but see preventative maintenance as boring and > time > > consuming. > > > > > > When it comes to your messaging systems, an ounce of prevention or in > this > > case implementing a preventative maintenance solution is worth a pound > of > > cure aka Disaster Recovery which of course means, unplanned downtime, > angry > > users and executives, cancelled plans, more stress than anyone needs > and > > late nights spent in the server room putting humpty dumpty back > together > > again. Avoid this by implementing an automated tool that will help you > and > > your Exchange enabled organization be as efficient and effective as > > possible. > > > > Microsoft-provided tools > > > > What about the tools that Microsoft provides for database maintenance: > > ESEutil and ISinteg? These tools are extremely powerful and should not > be > > handled without a good knowledge of their use and impact. You should > test > > these tools in a lab environment on backups of your production > databases in > > order to better gauge their effectiveness and performance. > > > > ESEutil > > It is my opinion that this tool must create some conflict in Redmond. > On one > > hand, Exchange administrators need access to the tool in order to > perform > > database maintenance and troubleshooting. On the other hand, ESEUTIL > is not > > unlike a loaded gun. In short, ESEUTIL is the one tool you can use to > verify > > the integrity of the Exchange stores, perform hard and soft recoveries > of > > the stores, copy the database files, repair the files or defrag the > files. > > Unfortunately, many of the commands require the Exchange stores to be > > dismounted so its use is fairly limited in non-disaster scenarios. > > > > Use of the ESEUTIL command is essential in restoring Exchange > databases, and > > in repairing them in the event of a catastrophic failure. The tool can > also > > be used to help diagnose issues and checksum database pages as well as > the > > database header, but it was not designed for casual use. > > > > ISinteg > > ISinteg does not perform as much "heavy lifting" as the ESEutil does. > > ISinteg browses the exchange store tables and indexes for > inconsistencies. > > For example, attachments are not actually contained in an MAPI > message. > > Instead they exist in an attachment table and are referenced from the > > messages. When this document is deleted from the table, the messages > that > > contained links to the document should also be removed automatically. > > Exchange usually repairs problems such as these online during its > off-hours > > maintenance schedule but some objects can be missed by the online > tools > > since the database is still in use while online maintenance occurs. > > Inconsistencies such as these can be remedied by using the ISinteg > utility. > > > > http://www.themssforum.com/ExchangeAdmin/GoExchange/ > > What is the ongoing need it is filling, besides creating unnecessary > > downtime? > > You can run ISinteg on a dismounted database - pretty much ANY > dismounted > > database - and it will give you warnings. > > > > When running ISinteg against any database you will always find minor > > inconsistencies (warnings), this statement is about 99% true, the > > implications that the author suggests though are incorrect. The reason > for > > the above statement being true could be; > > Microsoft incorrectly reports possible problems when there are none > and > > doesn't know what they are talking about (maybe the engineers were > bored and > > just like to scare people) > > Microsoft's database has so many issues that it can never truly be > integral > > (Hence the reason they have been trying to move away from JET for > years and > > continue to do so, i.e. next estimate is 2012) > > Microsoft reports all problems regardless of apparent triviality > because > > they could lead to larger issues, they usually don't but they "could" > > > > Our position through pre-analysis and post-recovery of 100's of > damaged or > > incorrectly performing databases anecdotally proves that some of the > more > > minor warnings are "usually" trivial but that they can cascade into > more > > serious issues, and that some combinations of trivial warnings can > also be > > more serious taken together than as separate one offs. In other words > even > > the most minor warnings can be symptoms of potentially larger > problems. > > > > The authors position seems to be A or B, ours is C. Since Microsoft > gives no > > definitive guidance we attempt to remove all errors and warnings > without > > regard to category of problem. > > > > In general we can agree that IF you are ONLY getting minor warnings it > > should in general not be a cause for great concern. That being said we > > refuse to categorize all warnings into the "unimportant" category and > we do > > this from experience and perhaps more importantly Microsoft themselves > will > > not do this either. The argument that you will gain less by performing > > maintenance against a database with only minor warnings than if > performing > > the same maintenance against a database with severe warnings or more > > importantly errors is obvious. The problem is that the author assumes > you > > will only get minor warnings and is implicitly stating that he knows > without > > looking that the database is 100% healthy, an interesting position to > take > > to be sure. > > > > Then again for those that can't be as sure as the author, how are you > > supposed to figure this out? Since checking fully transacted integrity > or > > running a full blown maintenance process are both offline processes I > would > > personally question any ones knowledge or common sense that says that > they > > "Know" the absolute state of the database. Nothing in this world is > self > > maintaining, not your car, your teeth or your home. Take care of them > now > > because it's ALWAYS less expensive to be proactive rather than > reactive. > > > > NOTE: We understand the internal Exchange db schema, jet engine, and > > transaction management within Exchange better than any party outside > of the > > Microsoft Exchange developers themselves. > > > > Again we need to stress that the author of the post authoritatively > > pre-supposes this will be true (only a few trivial warnings will ever > be > > generated) if he simply believes in the technology so much then why > does > > Microsoft provide utilities to fix issues that he apparently refuses > to > > believe can ever happen. The bottom line is, that no one, including > > Microsoft or MVP's will ever know what's going within an Exchange > database > > until it's inspected, and currently we can only inspect when the EDB > is > > offline (Online MAPI access etc. cannot tell us anything other than > general > > accessibility) Anything else is purely a guess, unless of course the > EDB > > dies then you know you have a problem and the only thing you don't > know is > > A: how long it's going to take to recover, B: how much data you are > going to > > lose C: what the impact to your company will be, i.e. lost > productivity, > > sales etc because this almost always happens at the beginning of the > workday > > and of course on the worst day possible and of course D: how did this > happen > > and E: why is Microsoft telling me to run these utilities that they > said > > were so dangerous F: when will I get to go home G: Why is everyone in > the > > company blaming me for this downtime and said I should have > known...... > > > > I should mention we do provide an "Analysis" mode where you can > actually see > > if you are having any issues that are beyond what an individual deems > > "trivial" if you get any errors then of course you have issues with > the > > logical integrity of the database, our position is that if you are > going to > > take the time to find out if you have any issues then you might as > well fix > > them (even the most trivial) while you are at it because analysis is > an > > offline process as well and takes about the same amount of time as > fixing > > issues while checking. > > > > This is because ISinteg is a utility and not a database engine itself. > > > > Again the author is getting a little confused. ISinteg calls into > Store.exe > > to do its work; Store.exe (Exchange) then uses Jet (the edb engine) to > > actually manipulate things. If you think that's not true shut down the > > store.exe and see if ISinteg will work. > > > > So no ISinteg is not a database engine it's a wrapper utility (it > basically > > is nothing but a utility to pass parameters along to store.exe > (exchange) to > > tell it to start looking at it database and active directory and > figuring > > out if anything is wrong at a logical level with the database. > > > > So although technically a correct statement it's as silly as saying > the > > command line is "not an operating systems" technically true, it's how > you > > interact with the operating system, and the ISinteg utility is one way > you > > can interact with the database engine bits of Exchange. > > > > You can tell ISinteg to fix those warnings, and run it multiple times > until > > there aren't any more warnings, and you've just detuned your indexes. > I > > expect (but admit that I do not know it for sure) > > > > > > ISinteg (actually exchange itself, remember ISinteg is just a command > line > > processor that talks to Exchange) does not de-tune indexes. > > > > Performance (tuning) is dependent on exchange and set automatically > for the > > most part (more to do with buffers and cache sizes really) and many of > these > > settings the user has no control over, if the author is saying that > > correcting problems (even minor ones in temporary indexes) causes > > performance degradation then he doesn't know what he is talking about. > > > > Regardless of the terms used a rebuilt or corrected index, table, or > > dictionary by its nature would be better "tuned" than one in which the > > corresponding entries were incorrect. > > > > The author of the statement above obviously does not really understand > the > > architecture of exchange, the jet engine, the b-tree mechanism, the > database > > schema or the store application (exchange) > > > > Db integrity is checked 100% with every single online backup that you > take. > > 100%. You can't back up a corrupted database with online backup. > > > > Again the author is confused and has just enough information to make > an > > incorrect assumption (that physical and logical integrity are the same > > things or that lack of problems in one area guarantees a lack of > problems in > > another area) > > > > To be precise there are two levels of exchange structure, the > physical, > > which has to do with pages and internal b-tree layouts, and the > logical > > (tables, indexes etc) ISinteg does not tell exchange to look for or > fix > > physical issues, it deals with the logical issues, ESEutil deals with > the > > physical problems (which are always more serious by nature) > > > > When doing an online backup (full only, incremental backups don't do > this) > > if there are physical page failures then an online backup will not > complete > > (basically if a pages Checksum/ECC is bad) but that is not the same as > > "logical" integrity. > > > > Logical integrity is saying, in essence, the schema (tables and > indexes) are > > complete and trust worthy and all relate to what they should relate > to, its > > saying that if a message entry exists, that the headers are there, the > body > > is there, the attachments are there, and accessible etc. that is > "Database > > is Integral" the online backup does not do or attempt to do this. > > > > Physical integrity is saying that the b-tree pages are intact, of > course > > those pages could be part of tables, or indexes or dictionaries (you > don't > > know at the Physical level, these are just "chunks" of data to the > backup > > engine) this is similar to sectors and FAT or NTFS tables on the disk, > vs. a > > document. The online backup only makes sure the sectors are ok; it > doesn't > > look at or understand document structure. > > > > Therefore you can have 100% integral pages with a damaged non-integral > > database (table index level) in other words a "Healthy" page does not > make a > > "Healthy" database, where as a "Damaged" page would always contribute > to a > > "Damaged" database (how damaged depends on what the page ends up > belonging > > to, two damaged pages in the primary system dictionaries would be very > bad, > > in a index easily fixed by an index rebuild etc) > > > > Physical problem detection would always be handled by running an > online > > backup which would detect that level of corruption (with DigiVault > > integration this happens seamlessly) in which case GOexchange would > refuse > > to run (backup failure) but only in the time frame where it happens > (if you > > were running CDP that did a 90 day full, with minute based incremental > your > > EDB's are not necessarily consistent within that 90 day window because > for > > incremental this physical page check doesn't happen) > > > > And in Exchange 2007 sp1, it's regularly checked on an ongoing basis. > > > > Again the Author is showing some confusion here. The ECC algorithm for > page > > auto correction is indeed better (it's much more tolerant than > previous > > versions of the ECC) So yes 2K7 does a better job of maintaining It's > > physical integrity (writing pages to disk in a way that is fault > tolerant) > > logically it also does a better job during online maintenance to carry > out > > more logical correction operations, but there are still issues that > can only > > be corrected in offline mode, and although page reuse has been in > place for > > some time, offline compaction will still do a better job of optimizing > the > > EDB than online compaction, and during that phase indexes can also be > > completely rebuilt and "tuned" if you want to use that word. > > > > Bottom line is some things can only be done offline, hence Microsoft's > > continued inclusion of that functionality in both Jet and Store.exe > > (Exchange) which is controlled via ISinteg and ESEutil. > > > > We agree downtime needs to be managed effectively and we are working > on a > > hot-swap online version of GOexchange that minimizes this to a few > minutes > > or seconds, until then it's up to IT to manage this downtime > correctly, the > > longer time between maintenance cycles the higher the probability that > > something could have gone wrong and cascaded into a potential about to > be > > serious problem. At that point your are going to be in "Cancer" mode, > forced > > to run off of your backup without up to date data and no idea how long > your > > patient (database) has to live (because you can't or shouldn't take > him > > offline for a diagnostic according to the author, just wait until he > dies > > again and then rewind via backup or bring in the surgeons once you > know a > > terminal disease has begun ravaging the client) > > > > * You don't run utilities that are designed to fix problems by > slicing > > out chunks of data if problems are found, as regular maintenance. > That's > > just crazy. > > > > > > Here the Author has gone from uninformed to completely out into left > field. > > We agree, it would be crazy to run utilities that "Slice out chunks of > > data". ISinteg and ESEutil are utilities provided by Microsoft to help > > maintain your Exchange database. ESEutil has the very serious > potential of > > causing damage to the database if used incorrectly and using ISinteg > out of > > sequence without doing other things can make this step non-effective. > > > > One reason we built GOexchange is because people were using the > utilities > > incorrectly, this became so prevalent that Microsoft now recommends > not > > using the utilities unless absolutely necessary and that you have a > very > > high level of expertise or are being guided by PSS. They don't say > they have > > no value, or what necessary actually means, they have just determined > that > > for the average IT person this is too risky unless you are currently > > experiencing problems (see cancer reference above) > > > > For example: when using ESEutil with a /p this runs a hard repair on a > > database. There are very few times to run this switch due to the > destructive > > results it can cause. This switch will slice chucks of data out of the > > database if problems are found. (this is a last resort and usually if > you > > have good backups and complete log sets there are better alternatives) > I > > ensure you that GOexchange does NOT run this against the database ever > the > > set of functionality we use is non-destructive (if the data is intact > it > > stays intact, if it's already damaged beyond access then it may remain > > damaged at the worst, but most likely become recoverable in whole or > part). > > > > * You don't intentionally incur unnecessary downtime on a system > > designed to be up all the time., as maintenance, when all the critical > > maintenance task already run online > > > > > > The key words here are "unnecessary downtime" of course. The problem > is > > Microsoft doesn't really give anyone a way to know if it's actually > > necessary or not until you start seeing performance problems, > experience > > errors getting mail, or your store will not mount (worst case > scenario), and > > in reality diagnosis isn't productive when problems have gone too far > (false > > positives and negative tests can result, in which case index > rebuilding etc > > might be necessary). In essence they have had so many people misuse > the > > utilities that it's safer to run blind then risk improper usage. > GOexchange > > eliminates the issues that Microsoft is worried about in so far as > improper > > usage and lets you plan preventative maintenance downtime in order to > avoid > > unplanned/necessary downtime > > > > We won't tell you (even in diagnostic mode) that you don't need to > run the > > product for previously stated reasons, If we did this someone would > come out > > and complain that we said they didn't need to run the product when > they > > really did (always just after an incident occurred) This is why we > call it > > "Preventative" Maintenance and don't try and predict outcomes other > than > > "Issues exist, let's fix them" BEFORE they cause UNPLANNED DOWNTIME, > data > > and productivity loss etc > > > > * Even the notion of GOexchange solving something that Microsoft > > couldn't is completely ludicrous. > > > > > > Couldn't and don't are two different things. They could provide the > same > > ease of use expert driven system, and integration with other tools > required > > to complete the steps safely (backup, notification etc) and make the > front > > to back process simple, they have chosen not to. > > > > To be clear we don't claim that we can do "more" than Microsoft could, > we > > claim we do much more than they provide; in essence we automate this > process > > so it is flawless. Expert knowledge has been built into the product to > do > > away with the need for the manual, error prone processes, and the > meticulous > > attention required to perform what we believe should be routine > maintenance. > > Please follow the link below for a side by side comparison. > > > > http://www.lucid8.com/product/products_manual.asp > > > > > > A final note. We have heard from Microsoft themselves which have > stated on > > various occasions (to us and other customers) > > > > "With 2K7 you don't need GOexchange anymore like you did with 2K3" > > Rather humorously we also previously heard "With 2K3 you don't need > > GOexchange anymore like you did with 2K" > > > > in other words with every generation of Exchange it's admitted we > provided a > > valuable service for the older product but we were no longer "Needed" > for > > the newest product, that is until the next generation is released that > > solves all the problems that didn't exist. Please note that the > Microsoft > > utilities that you will never need are still provided, and Lucid8 > customers > > continue to report that running the process via GOexchange solved > 'odd' or > > 'performance' issues that Microsoft support was at a loss over. > > > > Microsoft makes a great product that benefits from occasional > maintenance. > > Microsoft actions (which speak louder than words) own experience and > our > > customers experiences around the world agree that GOexchange tunes the > EDB > > by eliminating logical errors, reclaiming "excessive" slack space, > > rebuilding indexes etc. and contributes to a more robust exchange > > environment. > > > > If you run GOexchange against a perfectly health database at worst you > will > > get peace of mind, if you run it against a sub-par database you will > get a > > robust and healthy database, it's a win-win situation, alternatively > you > > have the option of not knowing and hoping instead. > > > > Not to put too fine a point on it, many experts and others want you to > pay > > them vs. using an automated and therefore non-revenue generating > > process/product. > > > > By now, you should fully realize the importance of automation in > failover, > > replication, backup and monitoring. You can attempt to create your own > > preventative maintenance solution by creating a manual process > utilizing > > existing Microsoft Exchange utilities, however; > > > > The command line interface of ESEutil and ISinteg are not intuitive, > nor is > > detailed information supplied about when to apply them or in what > order, nor > > what to do if something goes wrong. > > When utilizing these utilities you're essentially performing > open-heart > > surgery on the Exchange database, so you must know precisely what, > when and > > how to use each utility. > > The process varies between the different versions of Exchange, e.g. if > you > > have a process worked out on 5.5 it will not work on 2000, 2003 or > 2007. > > Use of these utilities require manual intervention and must be used > after > > hours. Therefore, be prepared to spend an inordinate amount time when > > implementing a manual solution. > > > > > > > > > > GOexchange, from Lucid8 is a product that can help complete your > > high-availability scenario by providing expert tools to automate the > > maintenance of your Exchange environment. GOexchange is a tried, > proven and > > safe solution that's rich in features like notification, backup > integration, > > secure clean, as well as detailed and summary reports for every job. > > Furthermore, administrators can configure, schedule, and review > maintenance > > tasks for all Exchange servers within the organization from a single > > centralized console. > > > > So instead of looking up the appropriate ESEutil and ISinteg commands > or > > notifying all effected users of an impending maintenance, running > backups > > and manually taking systems offline, GOexchange leverages a series of > > Microsoft utilities and API's to create an automated preventative > > maintenance solution for optimally maintaining Microsoft Exchange > Server. > > This allows the administrator to schedule unattended regular > maintenance > > jobs for any Exchange server, storage group or individual store in > their > > environment including Exchange 5.5, 2000, 2003, 2007 and even > clustered > > servers. This is especially important in environments where different > > versions of Exchange are used as the syntax for many maintenance > commands > > are different. More importantly the Exchange administrator can now > focus on > > moving the organization forward rather than spending inordinate > amounts of > > time trying to keeps things afloat. > > > > > > Conclusion and Summary > > It is important for you to follow basic, proactive maintenance > processes to > > keep your Exchange environment running smoothly. Those businesses that > > require dial-tone availability, regulatory compliance and ease of > recovery > > will need to put additional processes and technology in place. > Server-class > > hardware, regular backup procedures, system monitoring, archiving, > > eDiscovery, recovery, and preventative database maintenance tools are > just > > a few of the things needed to help you maintain a rock-solid Exchange > > environment. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > ME2 > > ~ Upgrade to Next Generation Antispam/Antivirus with Ninja! ~ > ~ <http://www.sunbelt-software.com/SunbeltMessagingNinja.cfm> ~ > > ~ Upgrade to Next Generation Antispam/Antivirus with Ninja! ~ > ~ <http://www.sunbelt-software.com/SunbeltMessagingNinja.cfm> ~ > -- ME2 ~ Upgrade to Next Generation Antispam/Antivirus with Ninja! ~ ~ <http://www.sunbelt-software.com/SunbeltMessagingNinja.cfm> ~
