Will do. Chances are good that it may happen tomorrow morning.
I went ahead and checked the defrag task and it's disabled - I thought
disabling was recommended for a mailbox server. I checked the other tasks
while I was in there and didn't see anything obvious that coincided with the
problem.
For giggles I went ahead and ran the vssadmin command this morning even though
the problem wasn't occurring and got the following:
C:\>vssadmin list volumes
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
Volume path: \\?\Volume{b43a2802-da26-11e0-adc1-806e6f6e6963}\
Volume name: \\?\Volume{b43a2802-da26-11e0-adc1-806e6f6e6963}\
Volume path: E:\
Volume name: \\?\Volume{f0724d84-da26-11e0-9fe6-00505681001f}\
Volume path: F:\
Volume name: \\?\Volume{f0724d8c-da26-11e0-9fe6-00505681001f}\
Volume path: C:\
Volume name: \\?\Volume{b43a2803-da26-11e0-adc1-806e6f6e6963}\
C:\>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
Writer name: 'Task Scheduler Writer'
Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
State: [1] Stable
Last error: No error
Writer name: 'VSS Metadata Store Writer'
Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
State: [1] Stable
Last error: No error
Writer name: 'Performance Counters Writer'
Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
State: [1] Stable
Last error: No error
Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {8886d435-a3a9-4d20-b492-c55bf10dc4cd}
State: [1] Stable
Last error: No error
Writer name: 'Microsoft Exchange Replica Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {d005ce2f-1675-40bd-ac1d-f653fe1c1ab6}
State: [1] Stable
Last error: No error
Writer name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {45364100-0e2f-4301-8bc1-0d0e1e3a3bc7}
State: [1] Stable
Last error: No error
Writer name: 'IIS Metabase Writer'
Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
Writer Instance Id: {91239ea1-cbcc-4335-bf0f-412a3dfe7c39}
State: [1] Stable
Last error: No error
Writer name: 'IIS Config Writer'
Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
Writer Instance Id: {775dbd83-4ee8-457e-8b89-3393a13c9ed5}
State: [1] Stable
Last error: No error
Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {8a9456b3-c5e8-4995-89f5-92b391332578}
State: [1] Stable
Last error: No error
Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {fbc25473-80ce-485f-a66a-e9ab4e5bb458}
State: [1] Stable
Last error: No error
Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {38e079e2-85b2-42c6-a03f-7fab6f7dfd68}
State: [1] Stable
Last error: No error
Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {0edd3c2f-db30-4d30-afb9-b66aec0d4c3e}
State: [1] Stable
Last error: No error
Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {6eed2efe-fb80-418e-8f63-659671cec847}
State: [1] Stable
Last error: No error
Writer name: 'Cluster Database'
Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
Writer Instance Id: {27566fdf-eae2-48a5-9fd1-5b5a4a68a11c}
State: [1] Stable
Last error: No error
C:\>vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
Version: 1.0.0.7
C:\>vssadmin list shadows
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
No items found that satisfy the query.
-----Original Message-----
From: [email protected] [mailto:[email protected]] On
Behalf Of Michael B. Smith
Sent: Wednesday, October 14, 2015 6:49 PM
To: [email protected]
Subject: RE: [Exchange] RE: Microsoft Exchange RPC Client Access runaway
Let's run an experiment. :)
The next time it happens, run these 3 commands:
Vssadmin list writers
Vssadmin list providers
Vssadmin list shadows
It has to be from an elevated session I believe. See if the word "error" is in
either of the first two (I'm guessing yes) and whether or not you have a
current shadow copy from the third (I'm also guessing yes).
But these are just guesses.
Oh, let's also run "vssadmin list volumes".
Let's also look in Task Scheduler and see when the defrag is scheduled to run
and how often.
I'm aware of issues in 2010 SP2 with RPC Client Access crashing - but not it
consuming lots of proc time. I'm not saying there isn't an issue - I just don't
know of one. I do know that lots of weirdness shows up with VSS in 2008 and
2008 R2.
I can't imagine a link between RPC Client Access and Search. But regardless of
that, depending on the size of your databases, it can take a couple of days to
rebuild a search index. I know that users at my clients would be pretty
irritated to not have search for a couple of days. So much so that I probably
wouldn't re-create a new search index, but instead create a new mailbox
database and move all the users to that (the search index is built as part of
the move mailbox process).
-----Original Message-----
From: [email protected] [mailto:[email protected]] On
Behalf Of Maglinger, Paul
Sent: Wednesday, October 14, 2015 7:34 PM
To: <[email protected]>
Subject: Re: [Exchange] RE: Microsoft Exchange RPC Client Access runaway
I assumed that the VSS errors were caused by the RPC service chewing up the
CPU. You're thinking the RPC utilization was caused by a problem with VSS?
> On Oct 14, 2015, at 5:08 PM, Michael B. Smith <[email protected]> wrote:
>
> There are a couple of VSS rollups available for Server 2008 R2. They weren't
> available on WU/MU so you may not have them installed.
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]]
> On Behalf Of Maglinger, Paul
> Sent: Wednesday, October 14, 2015 5:52 PM
> To: '[email protected]'
> Subject: [Exchange] RE: Microsoft Exchange RPC Client Access runaway
>
> Veritas BackupExec 2014 R2 to tape.
> Windows 2008 R2 Enterprise.
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]]
> On Behalf Of Michael B. Smith
> Sent: Wednesday, October 14, 2015 4:44 PM
> To: [email protected]
> Subject: [Exchange] RE: Microsoft Exchange RPC Client Access runaway
>
> How are you doing backups? What OS?
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]]
> On Behalf Of Maglinger, Paul
> Sent: Wednesday, October 14, 2015 11:25 AM
> To: New Exchange List ([email protected])
> Subject: [Exchange] Microsoft Exchange RPC Client Access runaway
>
> I'm hoping someone can give me some insight on this.
> Running Update Rollup 6 Exchange Server 2010 SP2 on the mailbox servers set
> up as a DAG.
> For the past couple of weeks I'm arriving in the morning (usually on
> Wednesdays and Fridays) seeing the Microsoft Exchange RPC Client Access
> service with 80+ cpu utilization. This generally starts sometime between 4
> and 4:30am. I'm not showing any problems in the event logs during that time.
> The only going on around that time is the database health checks at around
> 4:44am and they all are passing. When people start showing up and logging at
> around 7:30am I start getting:
>
> Log Name: Application
> Source: VSS
> Date: 10/14/2015 7:20:06 AM
> Event ID: 11
> Task Category: None
> Level: Error
> Keywords: Classic
> User: N/A
> Computer: COMSTAR1.scvl.com
> Description:
> Volume Shadow Copy Service information: The COM Server with CLSID
> {e579ab5f-1cc4-44b4-bed9-de0991ff0623} and name IVssCoordinatorEx2 cannot be
> started. Most likely the CPU is under heavy load. [0x80080005, Server
> execution failed]
>
> Log Name: Application
> Source: VSS
> Date: 10/14/2015 7:20:07 AM
> Event ID: 8193
> Task Category: None
> Level: Error
> Keywords: Classic
> User: N/A
> Computer: COMSTAR1.scvl.com
> Description:
> Volume Shadow Copy Service error: Unexpected error calling routine
> CoCreateInstance. hr = 0x80080005, Server execution failed.
>
> That's obvious. And later on, and I believe because the RPC service is
> banging the CPU so hard, I start getting these for each of the databases:
>
> Log Name: Application
> Source: MSExchangeRepl
> Date: 10/14/2015 8:03:23 AM
> Event ID: 2153
> Task Category: Service
> Level: Error
> Keywords: Classic
> User: N/A
> Computer: COMSTAR1.scvl.com
> Description:
> The log copier was unable to communicate with server 'COMSTAR2.scvl.com'. The
> copy of database 'HQMBDB\COMSTAR1' is in a disconnected state. The
> communication error was: An error occurred while communicating with server
> 'COMSTAR2.scvl.com'. Error: Unable to read data from the transport
> connection: An existing connection was forcibly closed by the remote host.
> The copier will automatically retry after a short delay.
>
> And these for various RPC counters when I stop the service:
>
> Log Name: Application
> Source: MSExchange Common
> Date: 10/14/2015 8:14:13 AM
> Event ID: 106
> Task Category: General
> Level: Error
> Keywords: Classic
> User: N/A
> Computer: COMSTAR1.scvl.com
> Description:
> Performance counter updating error. Counter name is Client: Foreground RPCs
> succeeded, category name is MSExchange RpcClientAccess. Optional code: 3.
> Exception: The exception thrown is : System.InvalidOperationException: The
> requested Performance Counter is not a custom counter, it has to be
> initialized as ReadOnly.
>
> After restarting the service everything is fine.
> Backups start at 7:00pm and finish up by 10pm. The only scheduled tasks that
> run anywhere near that time is the AitAgent at 2:30am.
> The first time it happened I did some Googling and ran a
> PublicFolderDatabaseRepairRequest and MailboxRepairRequest. They found
> issues on the public folders and I corrected them. Nothing was found with
> the mailboxes. I ran the checks again today and can't find anything wrong.
>
> Additional searching came up with stopping and placing the Exchange Search
> Indexer and Search services in manual, rebooting the server and see if the
> problem occurs again. If not, delete and rebuild the catalog index. The
> article I found said that they ran without the search services for several
> days and the users couldn't use search, and they were okay with that. Not
> sure how much flak I'd get for doing this, so before going down this road I
> wanted to get some input from you guys. Anyone run across this and care to
> share past experience, or offer another suggestion?
>
> Thanks,
>
> Paul
