{2] and [3] are known issues, currently without a resolution. Can you please let me know your case number, off list? A TAP is experiencing this problem and senior TAP engineer is interested.
From: listsad...@lists.myitforum.com [mailto:listsad...@lists.myitforum.com] On Behalf Of Senter, John Sent: Monday, September 18, 2017 10:15 AM To: exchange@lists.myitforum.com Subject: [Exchange] RE: KB4036108 on Exchange 2016 CU6 issue So let me start with the disappointment that I have with Microsoft Premier Support with Tier 3 on Exchange. Since these servers were pre-prod I opened the case as a sev C, thinking the fix would be quick and I will have it up on Thursday. The Tech was not available until late Thursday night or Friday close to noon, so I took Friday. After 3 hours on the phone and remote session, all they did was collect some screen shots of app logs; and they got a eng to figure out why the patch did not install. That eng asked what the patch extension was and when I told him .msp, he informed us that was another team. With the weekend fast approaching and knowing these servers were going into prod on Tuesday night I told them to bump to a sev A so we could continue work on it, as a sev C only has to work on it every couple of days. So now I am a sev A, and still have the same Exchange eng on the ticket. This eng is still not doing anything to get the servers up, remember I have 2 servers that will not start Exchange services. We run a procmon to see what is going on when the service starts. While they were looking at their lab environment I review my server logs to see that Microsoft.Exchange.Common.dll is showing as missing the procmon calls. Check the Bin directory and well and behold it is missing. Do a compare of the Bin folder on a working server and find 36 files are missing. Looks like the security patch did not restore those when the install failed and it backed out the patch. So we tried running the CU6 upgrade to see if it would replace the missing files, this failed. So I decided to copy the missing files from a working server to the broken server, and wow the services start. I re-run the CU6 upgrade just to make sure files are correct. The eng starts to tell me that this is unsupported and I would need to do a CU6 recover, which means removing the server from the dag, which I did not want to do. I did the same process on the other down server to get it up. I then ran the security patch on both servers and they worked as expected. Then the eng come back and stated that another long term MS eng, came back that the process I did would be supported and should be fine. Great news, now my servers are back up and patched. So I do some health checks on the servers to fine that most of the servers showed ecp and owa as unhealthy. Again I do some searches to find that this happened in 2013 after a CU had been applied so started checking what they said to look at and found the same issue. When I told the MS eng this they searched on this and found the internal article stating the same thing, so they took notes to see about getting the doc updated to include 2016. So here is what was broken and the fix with this patch: 1) On 2 of the 14 servers, missing DLL files in Bin. Copied over the missing files from a good server to get it back 2) On all the servers the content indexes showed unhealthy after reboot. The following services were set to "disabled" during the patch so I set them back to "automatic" and started them a. Microsoft Exchange Search Host Controller b. Tracing Service for Search in Exchange 3) On all the servers the ecp and one of the owa... monitors showed as "Unhealthy". The blog I found on the service issue also mentioned this as a problem. Had to do with the web.config in the ...\V15\ClientAccess\ecp folder had the environment label "%ExchangeInstallDir%" in the file instead of the full path. Looks like this was an older environment label that has been renamed. So on each server I had to update the web.config file to replace that environment label with "C:\Program Files\Microsoft\Exchange Server\V15\". 4) The last step was to run Updatecas.ps1 in the $exscripts folder to fix the broken image links in ecp and owa. After doing all of this all the servers look to be solid. I have contacted MS to do a once over to make sure all monitors are showing as they should. This is where I really have a issue on the MS support. After getting the servers running late Friday night the eng informs me they have only being MS for 2 months and this was the first sev A they have worked. They only worked sev C tickets until now. I know that I started with a sev C, so I understand why I go them, but once I bumped to a sev A, I would expect they put on a more senior eng due to the nature of the issue. I know that the Exchange eng team needs the experience, but they should be shadowing a senior eng on a few sev A issues, just to understand the way to engage. I feel I did most of the discoveries and fixes because this was over the eng's experience level. Also, never found a root cause for the patch to crash, so it may not happen to everyone. So be leery on this patch because if it does not fail and break Exch it will still cause you to do steps 2-4. I hope MS will either correct the patch or make sure the CU7 does not have the same problem and is released quickly. Luckily I only had 14 servers. Would really suck to have to do all of this in an environment with 30+ servers. Also slows down the patching process since you have to spend more time on each server. From: listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com> [mailto:listsad...@lists.myitforum.com] On Behalf Of Michael B. Smith Sent: Thursday, September 14, 2017 10:33 AM To: exchange@lists.myitforum.com<mailto:exchange@lists.myitforum.com> Subject: [Exchange] RE: KB4036108 on Exchange 2016 CU6 issue Please keep us (well, at least me) informed. I'm quite curious. From: listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com> [mailto:listsad...@lists.myitforum.com] On Behalf Of Senter, John Sent: Thursday, September 14, 2017 10:17 AM To: exchange@lists.myitforum.com<mailto:exchange@lists.myitforum.com> Subject: [Exchange] RE: KB4036108 on Exchange 2016 CU6 issue So applied this to 7 servers and one failed. Now the Exchange services are all failing. I have opened a case with MS to figure out what is going on and why the failed patch did not back out the changes correctly. From: listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com> [mailto:listsad...@lists.myitforum.com] On Behalf Of Michael B. Smith Sent: Wednesday, September 13, 2017 7:31 PM To: exchange@lists.myitforum.com<mailto:exchange@lists.myitforum.com> Subject: [Exchange] RE: KB4036108 on Exchange 2016 CU6 issue Thanks for this. I've reported it, but since CU7 is coming soon, I really don't expect action on it. From: listsad...@lists.myitforum.com<mailto:listsad...@lists.myitforum.com> [mailto:listsad...@lists.myitforum.com] On Behalf Of Senter, John Sent: Wednesday, September 13, 2017 11:13 AM To: exchange@lists.myitforum.com<mailto:exchange@lists.myitforum.com> Subject: [Exchange] KB4036108 on Exchange 2016 CU6 issue Just a heads up on what I have found while applying this security fix on Exchange 2016 CU6. After installing and rebooting I found that the Content Index State was failed on all the DB's on a server. When I did ran this command: Get-MailboxDatabaseCopyStatus -Server $server | fl name,*index* It showed that service "Microsoft Exchange Search Host Controller" was not running. Looking at that service it was set to disabled, but the servers that had not been patched showed it automatic and running. So I set this back to automatic and started it. After a few mins the indexes went back to healthy and I was able to move the DB's back to mounted state. Doing some searching it looks like other CU's have done this in the past. Also found that this service was also set to disabled by the patch so it needs to be set back to automatic. Tracing Service for Search in Exchange Looks like MS has a bug in the installer that it does not set the state back to the correct setting after the patch is applied. js