Microsoft's Security Response Center: How Little Patches Are Made June 8, 2005 By Ryan Naraine
http://www.eweek.com/print_article2/0,2533,a=153744,00.asp ORLANDO, Fla.�Anxious to shed the company's image as having a lax attitude about software security, officials at the Microsoft Security Response Center are using the Tech Ed conference here to provide a rare glimpse at the step-by-step process used to create, test and roll out security patches. The software maker trained the spotlight on the operations of the MSRC during breakout sessions and one-on-one discussions with customers, stressing that all publicly�and privately�reported vulnerabilities are thoroughly investigated to determine whether customers are at risk. ADVERTISEMENT "We're on all the [security mailing] lists, just like you are, and we investigate everything, even if it's a post about a simple weird behavior in a product," said MSRC program manager Stephen Toulouse. By monitoring the public lists and underground hacker sites, Toulouse said the company is able to keep track of discussions about vulnerabilities that may not have been reported to Microsoft. In addition, he said, a "real person" monitoring the [EMAIL PROTECTED] e-mail address keeps close watch on all vulnerability reports sent in by private researchers. "Once we determine it's a real security issue, it goes into the triage phase. We immediately get in touch with the researcher and ask for more information so we can recreate the attack scenario internally. We also look for different attack scenarios," he said. Researchers have complained in the past that Microsoft routinely ignores threat warnings, which contributes to the underlying distrust, but Toulouse said the company's mission is to improve its relationship and "create a community" with grey hat hackers. "We respond immediately to the initial vulnerability report and provide the researcher with contact names, e-mail addresses and phone numbers. We make it clear we want to work closely with the researcher to pinpoint the problem and get it fixed. We commit to providing [researchers] with a progress report on the Microsoft investigation every time they ask for one," he said. "We may not have enough information from the initial report and we may not know the configurations of the [researcher's] vulnerable system, so we always work closely to get all the relevant details." Armed with the vulnerability information, the triage team kicks into high gear. On every product team within Microsoft, a staff member is on call to coordinate with the MSRC and join the investigation, Toulouse said. "It becomes a detailed, complicated process because we are looking at all supported versions of Windows to understand the complete angle and we also have to figure out the impact it could have on customers," he said. "Once we are talking to a product team, we've already confirmed it's an issue that needs to be fixed and we're already working on the bulletin that will eventually be released [on Patch Tuesday]." In the midst of investigations, especially for flaws in network-facing products that could cause code execution attacks, he said, the MSRC continues to scour the Internet to determine whether public exploits are circulating. Windows security ranks high on this year's Tech Ed agenda. Click here to read more. At this stage, Toulouse said, "parallel investigations" are done by the MSRC and the product teams to "widen the circle" and reproduce the attack vector. "We are now thinking like the hacker. We look at the vulnerability from the hacker's perspective. If it's a buffer overflow and it's Internet-facing, the alarms go off and we have to figure out if it's easy to exploit. The product team is also investigating from the code point of view. They want to know if there are additional vulnerabilities in the associated code. So it becomes a complete audit," he said. This, he admitted, leads to long delays in getting a comprehensive fix rolled out, but it's a risk the company is willing to take to ensure the quality of patches. "We find that the [audit] process gives us more depth, and we keep the researcher in the loop throughout the process to let them know what we're doing." The testing process also puts a monkey wrench in the patch release timetable. Even though Microsoft has recruited external patch testers as part of a formalized Security Update Validation Program, Toulouse said the quality assurance process has become "very, very complex," especially for products that are closely tied to the operating system. Click here to read more about Microsoft's decision to use external patch testers. "In theory, we can release an update with a patch very quickly, but that's a big mistake. One of the things customers demand is quality patches. They don't want to deal with faulty patches that break their applications and they don't want to deal with all the associated trouble," he said. It's never a perfect science, and Toulouse said the MSRC learned the hard way in 2003 when it rolled out the MS03-026 bulletin to fix a code execution hole in a DCOM RPC (Windows Distributed Component Object Model Remote Procedure Call) interface and subsequently discovered that it was a substandard patch. A few weeks later, the Blaster worm ripped through the Internet and Microsoft released MS03-039 with an admission that additional ports involving RPC remained unpatched. "That was an experience that taught us a valuable lesson. It's better if we find it before the bad guys figure it out," he said. In some cases, particularly when the Internet Explorer browser is involved, the testing process "becomes a significant undertaking," Toulouse said. "It's not easy to test an IE update. There are six or seven supported versions and then we're dealing with all the different languages. Our commitment is to protect all customers in all languages on all supported products at the same time, so it becomes a huge undertaking." "This is exactly why it can take a long time to ship an IE patch. We're dealing with about 440 different updates that have to be tested. We have to test thoroughly to make sure it doesn't introduce a new problem. We have to make sure it doesn't break the Internet. We have to make sure online banking sites work and third-party applications aren't affected," he added. Internet Explorer updates are also cumulative, meaning that they address several newly discovered vulnerabilities and all previously released patches, causing even more delays when the new fixes are bundled into older updates. "This is why it takes so long, but that's not to say that if there's an exploit, we won't accelerate testing and get it out there as fast as we can. But if we find problems in the testing phase, it could trigger a restart and cause even more delays," Toulouse said. Next Page: The job doesn't end on Patch Day. Once the tests are completed, the process shifts to actually creating content for the bulletins, which are scheduled for release on the second Tuesday of every month. During this phase, the MSRC again coordinates closely with the product teams to make sure that there's a perfect balance between providing accurate guidance without offering enough information for malicious hackers to reverse-engineer the patch. "We have one author for each bulletin. Then, once it's done, the entire team works on it to put in the correct mitigation guidance and available workarounds," he said. Three working days before the scheduled bulletin release, an advance notice is published with brief details of products affected and the severity rating. Around 9:00 a.m. PDT on Patch Day, the bulletins are uploaded to Microsoft's security Web site and the accompanying e-mail alerts are sent out; but the job doesn't end there, Toulouse said. Once the patches are shipped, the MSRC goes into "watch mode" to monitor the way researchers release their own alerts. In most cases, those alerts are accompanied by proof-of-concept code, a practice that researchers favor but Microsoft frowns on. Private researchers say proof-of-concepts are valuable for testing patch quality, but Microsoft believes the information can put customers at risk because malicious hackers are skilled enough to use the published code to create actual exploits to target unpatched systems. If exploit code starts to circulate, the MSRC shifts into "activation mode" to determine if follow-up action is necessary to thwart a worm outbreak. In one case earlier this year, Microsoft activated its incident response mechanism and made an overnight decision to make an MSN Messenger patch a mandatory upgrade after the research firm that initially reported the flaw published code that could be used in a widespread attack. "Several dozen times over the past year, we've activated the security incident response system to deal with an issue. You may not know about it because it didn't escalate into a major incident, but we're quietly doing a lot of things to protect customers," he said. Toulouse described patch creation and bulletin release as a "very complex and interesting process" that "starts long before you see the end result and � never ends because we have to keep monitoring and updating the bulletins when new information becomes available." You are a subscribed member of the infowarrior list. Visit www.infowarrior.org for list information or to unsubscribe. This message may be redistributed freely in its entirety. Any and all copyrights appearing in list messages are maintained by their respective owners.
