Inside Microsoft's Zotob Situation Room August 26, 2005 By Ryan Naraine http://www.eweek.com/print_article2/0,1217,a=158848,00.asp
When Microsoft Corp. shipped the MS05-039 bulletin on Aug. 9 to patch a "critical" flaw in the Windows Plug and Play service, there was general feeling of trepidation within the Microsoft Security Response Center. Software engineers at the Redmond, Wash., company smelled trouble right off the bat. It had been more than a year since the Slammer and Sasser worm outbreaks and, to MSRC Program Manager Stephen Toulouse, the severity of this PnP vulnerability brought back a mixture of memories ranging from chaos and confusion to outright pride in the way those threats were handled. During the Slammer outbreak, Toulouse was at a service station when he learned of the attacks over his car radio. He recalls buzzing pagers, screeching tires and puzzled faces as he scrambled to get to Redmond to start the process of containing the worm. This time around, it would be different. "This has been a very disciplined week," Toulouse said in an interview from the MSRC's specially created "Situation Room" at the height of the recent Zotob worm attacks. "This is something we had created an entire process around and we were much better prepared this time," he said. "Our process is working, and it's working very well." That process, Toulouse explained, started long before Patch Tuesday. "Whenever we're dealing with critical updates, one of the things we do is really look very hard at the attack vectors. What are the ways people will try to exploit this? How easy is it to create and unleash a worm? We attack the flaw just like the attacker would, and we knew up front that this one would be trouble. "We had three critical bulletins in August but, in the case of the Plug and Play vulnerability, we knew there was a remote, unauthenticated attack vector affecting Windows 2000. Whenever there's a remote, unauthenticated attack vector, it sends up major red flags," Toulouse said. Click here to read more about "critical" PnP fix in the latest MS Patch Day. As is customary, Toulouse and others within the MSRC began making the media rounds, underscoring the severity of that particular vulnerability. "At that stage, we're worried about this one. Our guidance immediately after the patches are released was for Windows 2000 users to apply MS05-039 as the highest priority update. We wanted to stress that upfront. If you're running Windows 2000, you need to pay attention to this one." Then, a hiccup on the Download Center that caused a big distraction. One of the "critical" bulletinswith patches for a code execution Internet Explorer flawgot corrupted, breaking the digital signatures and preventing them from installing. The MSRC was forced to pull the patches, investigate the cause of the problem and rerelease the bulletin. "As soon as we push the button and the bulletins get published, we watch to make sure everyone can get them. We had to cope with the IE problems, but everything was fine for everything else, including the PnP issue. Then, we have to watch the discussion lists to see how the security community is reacting," he explained. The immediate chatter around MS05-039 was no surprise. On the security mailing lists, hackers were openly discussing the severity of the Plug and Play hole and the ways in which it could be exploited. Microsoft was watching and taking notes, keeping a wary eye out for the first proof-of-concept exploit to be released. By Thursday, Aug. 11, the first sign of exploit code appeared on the FrSIRT (French Security Incident Response Team) Web site. In all, five Windows exploits were posted, including two for the PnP flaw. The MSRC mobilized and started testing the public exploits. The code provided a footpath to create a destructive worm, and a decision was immediately made to publish a fresh advisory with new warnings about the potential for danger. Microsoft's advisory went out late on Thursday with a very blunt message to Windows 2000 users: Patch, or else. Toulouse and his colleagues, meanwhile, prepared for a long, testing weekend. "We saw the exploit code and our Security Windows Reaction Team tested it against the patch, and we were convinced we would see an attack. It was only a matter of time," Toulouse said. "We knew we would want to have our guidance and protection content published on Microsoft.com, so we alerted the folks there about what we were expecting. We wanted to have an advisory and a separate incident page if an attack happened over the weekend. This is a process we have tested and refined with every incident. "We mobilized the product support folks and discussed what kinds of calls to expect in the event of an attack. We wanted to make sure everyone had their cell phones charged; pagers had batteries. We made sure everyone understood this was going to be a long weekend," Toulouse added. "If something happened, we needed to move very quickly." Unlike Blaster and Sassernetwork worms that hit Windows XP machinesthis attack could not successfully impact the general public. The affected Windows 2000 operating system is already out of mainstream support and is not considered a consumer operating system. "A lot of things have changed since Slammer," Toulouse said. "Customers are more aware of the need to move into a maintenance mindset. Customers using Windows 2003 Server SP1 [Service Pack 1] weren't impacted by the vulnerability because of changes we made. This is best example of learning how to make product more resilient to attack and have it be secure by default." On Saturday, the MSRC staff checked the lists again and found that the proof-of-concept code was being modified. "People were looking at it, changing it, making it more dangerous," he said. "We're watching these discussions, watching the PSS [Product Support Services] calls to see if people were being impacted." eWEEK.com takes a look back at some of the biggest security attacks against Windows 95 since its launch a decade ago. Click here to read more. In the wee hours of Sunday morning, an enterprise customer contacted the MSRC with the first positive identification of what would become the Zotob attack. Toulouse declined to name the customer. "They came to us with a sample of a new attack that they believed was exploiting the Plug and Play vulnerability," he said. "We took the code and started our own investigation. We also passed it to our VIA [Virus Information Alliance] partners to make sure everyone can get their signatures updated to provide protection." The MSRC's investigation confirmed that an actual attack exploiting MS05-039 was under way and would only get worse. "Early Sunday morning, our investigators tell us to get started on our process. We weren't seeing a widespread attack, and the anti-virus vendors weren't seeing anything major yet. But, with everything we knew, we decided to activate our security response process." By 10 a.m. Sunday, pagers started buzzing. The Situation Room was set up in Building 27 at Microsoft's Redmond campus. "This is considered a major incident, so we want to have all the right people in one room," Toulouse said. "The people responsible for the update were there. The product team guys were there. The internal investigators who were working through the night were there to brief us on how the code worked. Our communication staff was there along with the PSS guys. We're all in one place going over the response plan." Next Page: Gates, Ballmer are notified. By midday, senior executives including Microsoft Chairman Bill Gates and CEO Steve Ballmer were notified. The "Executive e-mail" is a key part of the response process, and it includes the use of a very specific, high-priority subject line to make sure the mail is read by the senior executives. The security advisory that was first issued to warn of the attack was updated to confirm an attack was under way. Toulouse himself placed a warning on the MSRC Blog. A stand-alone Zotob incident page was created while Microsoft's virus encyclopedia was updated to reflect the new threat. "The stand-alone incident page is important," he said. "Once the word got out that an attack was under way, we need to have specific instructions to help people understand what was going on and how they could protect themselves. If someone got infected, they could find help to clean up." Banner headlines were placed on the front page of Micrsoft.com. The warning was duplicated on the company's security portal and on the Windows 2000 product page. E-mail blasts were sent with links to the incident page, patch download locations and other mitigation guidance. Zotob was still a very low threat but, with businesses opening for work Monday, there was a likelihood that things would escalate. "Although infection rates are low, it doesn't mean it's not a bad situation," Toulouse said. "We want to make sure, not only are we providing information to make sure customers aren't impacted, but to make sure they know how to get back to an operational state." By Monday morning, the variants started squirming, refining the original Zotob code to get around anti-virus detections. The internal investigation team was back at work, analyzing the code, rushing to keep up with the virus writers. By Monday evening, the virus encyclopedia was updated to add entries for Zotob.C and Zotob.C. "They [the virus writers] were changing the executables and changing the way they scanned for networks. As we find the new variants, we're updating the stand-alone incident page," Toulouse said. Next Page: CNN gets hit. "We're literally in a meeting going over our plans, keeping track of things, when we got word on Tuesday that CNN was reporting they had been hit. At the time, we knew a high-profile target was reporting they had been hit and they didn't know what it was. Their computers were shutting down and restarting," Toulouse recalled. Microsoft would use the mainstream media interest to its advantage. The MSRC got in touch with CNN officials to discuss the attack and help contain the threat, but the television network would offer more value. "We invited them to the Situation Room, and we let them help us get the word out. This attack against CNN was not a new attack. It was the same thing we were seeing since Sunday, but it became a major story because some big media companies got infected. "At this stage, there are two things we want people to know. This affects Windows 2000 only, and the available patch provides protection. We also want them to know we're working to help those who were impacted." By this time, there were about a dozen Zotob mutants and evidence of rival virus writers deleting each other's malware. The MSRC made the decision to ship an out-of-cycle update of the malicious software removal tool to offer detection and disinfection. Click here to read more about Microsoft's Zotob worm remover. The utility is normally updated only on the second Tuesday of every month, but with Zotob squirming and the mainstream media reporting a major outbreak, Microsoft wanted customers to find immediate help. "In the Situation Room, everything is happening simultaneously. While we were providing updates to our incident page, we were working on getting that information to the malware removal tool. We weren't seeing, from our end, a massive sudden change in situation. It was the same low level of impact but with new variants and some different customers were being impacted," Toulouse said. For insights on security coverage around the Web, check out eWEEK.com Security Center Editor Larry Seltzer's Weblog. Once the malware remover shipped, all the guidance pages needed to be updated to reflect the fact that it was available and to point customers to the download location. The plan was to add the tool to Automatic Updates as part of the September patch cycle. "At the moment, we're still mobilized, but things have settled down," he said. "We're continuing to investigate the variants. As we see new variants, we'll add detections in the tool and make that available broadly on the next patch day." Check out eWEEK.com's Windows Center for Microsoft and Windows news, views and analysis. You are a subscribed member of the infowarrior list. Visit www.infowarrior.org for list information or to unsubscribe. This message may be redistributed freely in its entirety. Any and all copyrights appearing in list messages are maintained by their respective owners.
