Yessir. -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Maglinger, Paul Sent: Tuesday, October 20, 2015 11:06 AM To: '[email protected]' Subject: RE: [Exchange] RE: Microsoft Exchange RPC Client Access runaway
While we're on the subject - the general rule of thumb was to install SP3 and then install the latest UR, which appears to be UR11 now. Does that still hold true? -----Original Message----- From: Maglinger, Paul Sent: Tuesday, October 20, 2015 9:59 AM To: '[email protected]' Subject: RE: [Exchange] RE: Microsoft Exchange RPC Client Access runaway Just the book work to make sure everything is compatible. I know the last time we looked some of our solutions only supported SP2. Hopefully by now SP3 has been out long enough that it's supported. I'll keep y'all posted. Thanks for the input! -Paul -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Michael B. Smith Sent: Tuesday, October 20, 2015 9:47 AM To: [email protected] Subject: RE: [Exchange] RE: Microsoft Exchange RPC Client Access runaway I would guess that this points to Exchange. What does it take in your environment to get current on SP and UR? -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Maglinger, Paul Sent: Tuesday, October 20, 2015 9:27 AM To: '[email protected]' Subject: RE: [Exchange] RE: Microsoft Exchange RPC Client Access runaway Well, it took a few days but it finally reared it's ugly head this morning. The results: Microsoft Windows [Version 6.1.7601] Copyright (c) 2009 Microsoft Corporation. All rights reserved. C:\>vssadmin list writers vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2005 Microsoft Corp. Waiting for responses. These may be delayed if a shadow copy is being prepared. ****** This stayed this way until I killed the RPC service. I waited over ten minutes before doing so and it finished up with this: Writer name: 'Task Scheduler Writer' Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124} Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b} State: [1] Stable Last error: No error Writer name: 'VSS Metadata Store Writer' Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06} Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93} State: [1] Stable Last error: No error Writer name: 'Performance Counters Writer' Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2} Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381} State: [1] Stable Last error: No error Writer name: 'ASR Writer' Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4} Writer Instance Id: {3011b260-3c50-498f-a4c9-3843924e9a65} State: [1] Stable Last error: No error Writer name: 'Shadow Copy Optimization Writer' Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f} Writer Instance Id: {11a3aea2-c80d-425c-84c6-ee52a417bc6e} State: [1] Stable Last error: No error Writer name: 'Registry Writer' Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485} Writer Instance Id: {2dbfc18a-6ad9-4ef3-b84c-57d994d65c90} State: [1] Stable Last error: No error Writer name: 'COM+ REGDB Writer' Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f} Writer Instance Id: {92f499ac-023e-4ee5-9519-c0c3d68399bf} State: [1] Stable Last error: No error Writer name: 'System Writer' Writer Id: {e8132975-6f93-4464-a53e-1050253ae220} Writer Instance Id: {00fcfa03-0072-4417-b643-9d36ce7bc0d6} State: [1] Stable Last error: No error Writer name: 'IIS Config Writer' Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6} Writer Instance Id: {52aedb85-dc16-4e23-87e3-a0a33b4b37a1} State: [1] Stable Last error: No error Writer name: 'BITS Writer' Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0} Writer Instance Id: {e1c0bec1-17c8-41e3-833f-ca37aa5be606} State: [1] Stable Last error: No error Writer name: 'WMI Writer' Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0} Writer Instance Id: {785fda40-1c76-41e6-9f20-b752854e87fd} State: [1] Stable Last error: No error Writer name: 'Microsoft Exchange Writer' Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7} Writer Instance Id: {a7cfb847-ec18-493f-892c-1b81c897ba77} State: [1] Stable Last error: No error Writer name: 'IIS Metabase Writer' Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366} Writer Instance Id: {8ffbc69c-326e-47b3-b2ec-9c07c6fb8a51} State: [1] Stable Last error: No error Writer name: 'Cluster Database' Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e} Writer Instance Id: {0a5a7220-6398-403f-afd7-03782d097efc} State: [1] Stable Last error: No error C:\> ****** I ran the following in a separate elevated command prompt while waiting for the command above to complete. These ran to completion without having to stop the RPC service. Microsoft Windows [Version 6.1.7601] Copyright (c) 2009 Microsoft Corporation. All rights reserved. C:\>vssadmin list providers vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2005 Microsoft Corp. Provider name: 'Microsoft Software Shadow Copy provider 1.0' Provider type: System Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5} Version: 1.0.0.7 C:\>vssadmin list shadows vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2005 Microsoft Corp. No items found that satisfy the query. C:\>vssadmin list volumes vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2005 Microsoft Corp. Volume path: \\?\Volume{b43a2802-da26-11e0-adc1-806e6f6e6963}\ Volume name: \\?\Volume{b43a2802-da26-11e0-adc1-806e6f6e6963}\ Volume path: E:\ Volume name: \\?\Volume{f0724d84-da26-11e0-9fe6-00505681001f}\ Volume path: F:\ Volume name: \\?\Volume{f0724d8c-da26-11e0-9fe6-00505681001f}\ Volume path: C:\ Volume name: \\?\Volume{b43a2803-da26-11e0-adc1-806e6f6e6963}\ C:\> So because the #vssadmin list writers command hung until I stopped RPC and never showed an error, am I still looking at a VSS issue or a RPC issue? -Paul -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Michael B. Smith Sent: Thursday, October 15, 2015 9:21 AM To: [email protected] Subject: RE: [Exchange] RE: Microsoft Exchange RPC Client Access runaway All of this is normal. This is your baseline. You should not defrag mailbox database volumes. OS volume is ok. -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Maglinger, Paul Sent: Thursday, October 15, 2015 10:11 AM To: '[email protected]' Subject: RE: [Exchange] RE: Microsoft Exchange RPC Client Access runaway Will do. Chances are good that it may happen tomorrow morning. I went ahead and checked the defrag task and it's disabled - I thought disabling was recommended for a mailbox server. I checked the other tasks while I was in there and didn't see anything obvious that coincided with the problem. For giggles I went ahead and ran the vssadmin command this morning even though the problem wasn't occurring and got the following: C:\>vssadmin list volumes vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2005 Microsoft Corp. Volume path: \\?\Volume{b43a2802-da26-11e0-adc1-806e6f6e6963}\ Volume name: \\?\Volume{b43a2802-da26-11e0-adc1-806e6f6e6963}\ Volume path: E:\ Volume name: \\?\Volume{f0724d84-da26-11e0-9fe6-00505681001f}\ Volume path: F:\ Volume name: \\?\Volume{f0724d8c-da26-11e0-9fe6-00505681001f}\ Volume path: C:\ Volume name: \\?\Volume{b43a2803-da26-11e0-adc1-806e6f6e6963}\ C:\>vssadmin list writers vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2005 Microsoft Corp. Writer name: 'Task Scheduler Writer' Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124} Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b} State: [1] Stable Last error: No error Writer name: 'VSS Metadata Store Writer' Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06} Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93} State: [1] Stable Last error: No error Writer name: 'Performance Counters Writer' Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2} Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381} State: [1] Stable Last error: No error Writer name: 'System Writer' Writer Id: {e8132975-6f93-4464-a53e-1050253ae220} Writer Instance Id: {8886d435-a3a9-4d20-b492-c55bf10dc4cd} State: [1] Stable Last error: No error Writer name: 'Microsoft Exchange Replica Writer' Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7} Writer Instance Id: {d005ce2f-1675-40bd-ac1d-f653fe1c1ab6} State: [1] Stable Last error: No error Writer name: 'ASR Writer' Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4} Writer Instance Id: {45364100-0e2f-4301-8bc1-0d0e1e3a3bc7} State: [1] Stable Last error: No error Writer name: 'IIS Metabase Writer' Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366} Writer Instance Id: {91239ea1-cbcc-4335-bf0f-412a3dfe7c39} State: [1] Stable Last error: No error Writer name: 'IIS Config Writer' Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6} Writer Instance Id: {775dbd83-4ee8-457e-8b89-3393a13c9ed5} State: [1] Stable Last error: No error Writer name: 'WMI Writer' Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0} Writer Instance Id: {8a9456b3-c5e8-4995-89f5-92b391332578} State: [1] Stable Last error: No error Writer name: 'Microsoft Exchange Writer' Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7} Writer Instance Id: {fbc25473-80ce-485f-a66a-e9ab4e5bb458} State: [1] Stable Last error: No error Writer name: 'Shadow Copy Optimization Writer' Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f} Writer Instance Id: {38e079e2-85b2-42c6-a03f-7fab6f7dfd68} State: [1] Stable Last error: No error Writer name: 'Registry Writer' Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485} Writer Instance Id: {0edd3c2f-db30-4d30-afb9-b66aec0d4c3e} State: [1] Stable Last error: No error Writer name: 'COM+ REGDB Writer' Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f} Writer Instance Id: {6eed2efe-fb80-418e-8f63-659671cec847} State: [1] Stable Last error: No error Writer name: 'Cluster Database' Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e} Writer Instance Id: {27566fdf-eae2-48a5-9fd1-5b5a4a68a11c} State: [1] Stable Last error: No error C:\>vssadmin list providers vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2005 Microsoft Corp. Provider name: 'Microsoft Software Shadow Copy provider 1.0' Provider type: System Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5} Version: 1.0.0.7 C:\>vssadmin list shadows vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2005 Microsoft Corp. No items found that satisfy the query. -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Michael B. Smith Sent: Wednesday, October 14, 2015 6:49 PM To: [email protected] Subject: RE: [Exchange] RE: Microsoft Exchange RPC Client Access runaway Let's run an experiment. :) The next time it happens, run these 3 commands: Vssadmin list writers Vssadmin list providers Vssadmin list shadows It has to be from an elevated session I believe. See if the word "error" is in either of the first two (I'm guessing yes) and whether or not you have a current shadow copy from the third (I'm also guessing yes). But these are just guesses. Oh, let's also run "vssadmin list volumes". Let's also look in Task Scheduler and see when the defrag is scheduled to run and how often. I'm aware of issues in 2010 SP2 with RPC Client Access crashing - but not it consuming lots of proc time. I'm not saying there isn't an issue - I just don't know of one. I do know that lots of weirdness shows up with VSS in 2008 and 2008 R2. I can't imagine a link between RPC Client Access and Search. But regardless of that, depending on the size of your databases, it can take a couple of days to rebuild a search index. I know that users at my clients would be pretty irritated to not have search for a couple of days. So much so that I probably wouldn't re-create a new search index, but instead create a new mailbox database and move all the users to that (the search index is built as part of the move mailbox process). -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Maglinger, Paul Sent: Wednesday, October 14, 2015 7:34 PM To: <[email protected]> Subject: Re: [Exchange] RE: Microsoft Exchange RPC Client Access runaway I assumed that the VSS errors were caused by the RPC service chewing up the CPU. You're thinking the RPC utilization was caused by a problem with VSS? > On Oct 14, 2015, at 5:08 PM, Michael B. Smith <[email protected]> wrote: > > There are a couple of VSS rollups available for Server 2008 R2. They weren't > available on WU/MU so you may not have them installed. > > -----Original Message----- > From: [email protected] [mailto:[email protected]] > On Behalf Of Maglinger, Paul > Sent: Wednesday, October 14, 2015 5:52 PM > To: '[email protected]' > Subject: [Exchange] RE: Microsoft Exchange RPC Client Access runaway > > Veritas BackupExec 2014 R2 to tape. > Windows 2008 R2 Enterprise. > > -----Original Message----- > From: [email protected] [mailto:[email protected]] > On Behalf Of Michael B. Smith > Sent: Wednesday, October 14, 2015 4:44 PM > To: [email protected] > Subject: [Exchange] RE: Microsoft Exchange RPC Client Access runaway > > How are you doing backups? What OS? > > -----Original Message----- > From: [email protected] [mailto:[email protected]] > On Behalf Of Maglinger, Paul > Sent: Wednesday, October 14, 2015 11:25 AM > To: New Exchange List ([email protected]) > Subject: [Exchange] Microsoft Exchange RPC Client Access runaway > > I'm hoping someone can give me some insight on this. > Running Update Rollup 6 Exchange Server 2010 SP2 on the mailbox servers set > up as a DAG. > For the past couple of weeks I'm arriving in the morning (usually on > Wednesdays and Fridays) seeing the Microsoft Exchange RPC Client Access > service with 80+ cpu utilization. This generally starts sometime between 4 > and 4:30am. I'm not showing any problems in the event logs during that time. > The only going on around that time is the database health checks at around > 4:44am and they all are passing. When people start showing up and logging at > around 7:30am I start getting: > > Log Name: Application > Source: VSS > Date: 10/14/2015 7:20:06 AM > Event ID: 11 > Task Category: None > Level: Error > Keywords: Classic > User: N/A > Computer: COMSTAR1.scvl.com > Description: > Volume Shadow Copy Service information: The COM Server with CLSID > {e579ab5f-1cc4-44b4-bed9-de0991ff0623} and name IVssCoordinatorEx2 cannot be > started. Most likely the CPU is under heavy load. [0x80080005, Server > execution failed] > > Log Name: Application > Source: VSS > Date: 10/14/2015 7:20:07 AM > Event ID: 8193 > Task Category: None > Level: Error > Keywords: Classic > User: N/A > Computer: COMSTAR1.scvl.com > Description: > Volume Shadow Copy Service error: Unexpected error calling routine > CoCreateInstance. hr = 0x80080005, Server execution failed. > > That's obvious. And later on, and I believe because the RPC service is > banging the CPU so hard, I start getting these for each of the databases: > > Log Name: Application > Source: MSExchangeRepl > Date: 10/14/2015 8:03:23 AM > Event ID: 2153 > Task Category: Service > Level: Error > Keywords: Classic > User: N/A > Computer: COMSTAR1.scvl.com > Description: > The log copier was unable to communicate with server 'COMSTAR2.scvl.com'. The > copy of database 'HQMBDB\COMSTAR1' is in a disconnected state. The > communication error was: An error occurred while communicating with server > 'COMSTAR2.scvl.com'. Error: Unable to read data from the transport > connection: An existing connection was forcibly closed by the remote host. > The copier will automatically retry after a short delay. > > And these for various RPC counters when I stop the service: > > Log Name: Application > Source: MSExchange Common > Date: 10/14/2015 8:14:13 AM > Event ID: 106 > Task Category: General > Level: Error > Keywords: Classic > User: N/A > Computer: COMSTAR1.scvl.com > Description: > Performance counter updating error. Counter name is Client: Foreground RPCs > succeeded, category name is MSExchange RpcClientAccess. Optional code: 3. > Exception: The exception thrown is : System.InvalidOperationException: The > requested Performance Counter is not a custom counter, it has to be > initialized as ReadOnly. > > After restarting the service everything is fine. > Backups start at 7:00pm and finish up by 10pm. The only scheduled tasks that > run anywhere near that time is the AitAgent at 2:30am. > The first time it happened I did some Googling and ran a > PublicFolderDatabaseRepairRequest and MailboxRepairRequest. They found > issues on the public folders and I corrected them. Nothing was found with > the mailboxes. I ran the checks again today and can't find anything wrong. > > Additional searching came up with stopping and placing the Exchange Search > Indexer and Search services in manual, rebooting the server and see if the > problem occurs again. If not, delete and rebuild the catalog index. The > article I found said that they ran without the search services for several > days and the users couldn't use search, and they were okay with that. Not > sure how much flak I'd get for doing this, so before going down this road I > wanted to get some input from you guys. Anyone run across this and care to > share past experience, or offer another suggestion? > > Thanks, > > Paul
