{2] and [3] are known issues, currently without a resolution.
Can you please let me know your case number, off list? A TAP is experiencing
this problem and senior TAP engineer is interested.
From: listsad...@lists.myitforum.com [mailto:listsad...@lists.myitforum.com] On
Behalf Of Senter, John
Sent: Monday, September 18, 2017 10:15 AM
To: exchange@lists.myitforum.com
Subject: [Exchange] RE: KB4036108 on Exchange 2016 CU6 issue
So let me start with the disappointment that I have with Microsoft Premier
Support with Tier 3 on Exchange. Since these servers were pre-prod I opened
the case as a sev C, thinking the fix would be quick and I will have it up on
Thursday. The Tech was not available until late Thursday night or Friday close
to noon, so I took Friday. After 3 hours on the phone and remote session, all
they did was collect some screen shots of app logs; and they got a eng to
figure out why the patch did not install. That eng asked what the patch
extension was and when I told him .msp, he informed us that was another team.
With the weekend fast approaching and knowing these servers were going into
prod on Tuesday night I told them to bump to a sev A so we could continue work
on it, as a sev C only has to work on it every couple of days. So now I am a
sev A, and still have the same Exchange eng on the ticket. This eng is still
not doing anything to get the servers up, remember I have 2 servers that will
not start Exchange services. We run a procmon to see what is going on when the
service starts. While they were looking at their lab environment I review my
server logs to see that Microsoft.Exchange.Common.dll is showing as missing the
procmon calls. Check the Bin directory and well and behold it is missing. Do
a compare of the Bin folder on a working server and find 36 files are missing.
Looks like the security patch did not restore those when the install failed and
it backed out the patch. So we tried running the CU6 upgrade to see if it
would replace the missing files, this failed. So I decided to copy the missing
files from a working server to the broken server, and wow the services start.
I re-run the CU6 upgrade just to make sure files are correct.
The eng starts to tell me that this is unsupported and I would need to do a CU6
recover, which means removing the server from the dag, which I did not want to
do. I did the same process on the other down server to get it up. I then ran
the security patch on both servers and they worked as expected. Then the eng
come back and stated that another long term MS eng, came back that the process
I did would be supported and should be fine. Great news, now my servers are
back up and patched. So I do some health checks on the servers to fine that
most of the servers showed ecp and owa as unhealthy. Again I do some searches
to find that this happened in 2013 after a CU had been applied so started
checking what they said to look at and found the same issue. When I told the
MS eng this they searched on this and found the internal article stating the
same thing, so they took notes to see about getting the doc updated to include
2016. So here is what was broken and the fix with this patch:
1) On 2 of the 14 servers, missing DLL files in Bin. Copied over the
missing files from a good server to get it back
2) On all the servers the content indexes showed unhealthy after reboot.
The following services were set to "disabled" during the patch so I set them
back to "automatic" and started them
a. Microsoft Exchange Search Host Controller
b. Tracing Service for Search in Exchange
3) On all the servers the ecp and one of the owa... monitors showed as
"Unhealthy". The blog I found on the service issue also mentioned this as a
problem. Had to do with the web.config in the ...\V15\ClientAccess\ecp folder
had the environment label "%ExchangeInstallDir%" in the file instead of the
full path. Looks like this was an older environment label that has been
renamed. So on each server I had to update the web.config file to replace that
environment label with "C:\Program Files\Microsoft\Exchange Server\V15\".
4) The last step was to run Updatecas.ps1 in the $exscripts folder to fix
the broken image links in ecp and owa.
After doing all of this all the servers look to be solid. I have contacted MS
to do a once over to make sure all monitors are showing as they should.
This is where I really have a issue on the MS support. After getting the
servers running late Friday night the eng informs me they have only being MS
for 2 months and this was the first sev A they have worked. They only worked
sev C tickets until now. I know that I started with a sev C, so I understand
why I go them, but once I bumped to a sev A, I would expect they put on a more
senior eng due to the nature of the issue. I know that the Exchange eng team
needs the experience, but they should be shadowing a senior eng on a few sev A
issues, just to understand the way to engage. I feel I did most of the
discoveries and fixes because this was over the eng's experience level. Also,
never found a root cause for the patch to crash, so it may not happen to
everyone.
So be leery on this patch because if it does not fail and break Exch it will
still cause you to do steps 2-4. I hope MS will either correct the patch or
make sure the CU7 does not have the same problem and is released quickly.
Luckily I only had 14 servers. Would really suck to have to do all of this in
an environment with 30+ servers. Also slows down the patching process since
you have to spend more time on each server.
From: listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com>
[mailto:listsad...@lists.myitforum.com] On Behalf Of Michael B. Smith
Sent: Thursday, September 14, 2017 10:33 AM
To: exchange@lists.myitforum.com<mailto:exchange@lists.myitforum.com>
Subject: [Exchange] RE: KB4036108 on Exchange 2016 CU6 issue
Please keep us (well, at least me) informed. I'm quite curious.
From: listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com>
[mailto:listsad...@lists.myitforum.com] On Behalf Of Senter, John
Sent: Thursday, September 14, 2017 10:17 AM
To: exchange@lists.myitforum.com<mailto:exchange@lists.myitforum.com>
Subject: [Exchange] RE: KB4036108 on Exchange 2016 CU6 issue
So applied this to 7 servers and one failed. Now the Exchange services are all
failing. I have opened a case with MS to figure out what is going on and why
the failed patch did not back out the changes correctly.
From: listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com>
[mailto:listsad...@lists.myitforum.com] On Behalf Of Michael B. Smith
Sent: Wednesday, September 13, 2017 7:31 PM
To: exchange@lists.myitforum.com<mailto:exchange@lists.myitforum.com>
Subject: [Exchange] RE: KB4036108 on Exchange 2016 CU6 issue
Thanks for this.
I've reported it, but since CU7 is coming soon, I really don't expect action on
it.
From: listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com>
[mailto:listsad...@lists.myitforum.com] On Behalf Of Senter, John
Sent: Wednesday, September 13, 2017 11:13 AM
To: exchange@lists.myitforum.com<mailto:exchange@lists.myitforum.com>
Subject: [Exchange] KB4036108 on Exchange 2016 CU6 issue
Just a heads up on what I have found while applying this security fix on
Exchange 2016 CU6. After installing and rebooting I found that the Content
Index State was failed on all the DB's on a server. When I did ran this
command:
Get-MailboxDatabaseCopyStatus -Server $server | fl name,*index*
It showed that service "Microsoft Exchange Search Host Controller" was not
running. Looking at that service it was set to disabled, but the servers that
had not been patched showed it automatic and running. So I set this back to
automatic and started it. After a few mins the indexes went back to healthy
and I was able to move the DB's back to mounted state.
Doing some searching it looks like other CU's have done this in the past. Also
found that this service was also set to disabled by the patch so it needs to be
set back to automatic.
Tracing Service for Search in Exchange
Looks like MS has a bug in the installer that it does not set the state back to
the correct setting after the patch is applied.
js
