Debugging: OK to Outsource? http://www.redherring.com/article.aspx?a=14475#
Cash rewards for freelance researchers is a cheap and easy way to fix security holes. But critics say the expanding practice could hurt the software security industry. November 14, 2005 Print Issue Tom Ferris is a bounty hunter. In the Wild West of software security, he¹s a cowboy who hunts bugssecurity loopholes in software that can be exploited to launch malicious attacks. On nights and weekends, Mr. Ferris combs through popular software products searching for the gaps. Every discovery has the potential of a financial payoff, anywhere from $500 to a few thousand dollars for a ³valuable² bug. Mr. Ferris, 27, is one of the hundreds of independent security researchers who hunt bugs as a hobby. They are passionate about their work, but their loyalty is tenuousonce driven by the idea of fame, money is now their incentive. Their patrons are security companies that have started bug bounty programs, created to farm out bug hunting to freelance security researchers. Since money-for-bugs programs gained popularity a few years agosparked by Symantec¹s 2002 acquisition of SecurityFocus, which hosted an online community where bug hunters could post their findingsfreelance researchers have found and reported hundreds of security holes in software products, helping security companies improve their offerings and fattening their own wallets a little in the process. But the relationship between freelance researchers like Mr. Ferris and the security companies isn¹t always harmonious and symbiotic. Security companies don¹t always respond amicably when researchers find a flaw; some researchers say they aren¹t rewarded accordingly when they do find something important. And even when the relations go smoothly, some say this form of commercializing and outsourcing vulnerability research could lead to a public relations nightmare for security companies, orin the worst-case scenarioa rogue bounty hunter selling vulnerabilities to hackers who will exploit the hole. In the long run, these reward programs can do more damage than good, warns Pete Lindstrom, director of research at SpireSecurity. ³Their contribution to the profession is at best ambivalent, and at worst negative and destructive,² he says. Finding the Holes As long as the rewards are offered, freelance researchers like Mr. Ferris will keep looking for and finding holes. A security researcher for nine years, Mr. Ferris has a day job as a software engineer for a security company that he declines to name. When he searches for bugs outside of work, he starts by picking a popular product and profiling it. He checks the features, and learns the functionalities and protocols that it uses. Then he goes about ³fuzzing² itsending random or malformed data to the program, which causes it to crash or overflow. It¹s an easy way to pick the low-hanging fruitthe security bugs that can be found through automated tools, he says. If a fuzzer doesn¹t do the trick, Mr. Ferris will try to reverse engineer the product. Either way, by the end of his effort, he will hopefully have found a bug or two. His favorite targets are Microsoft products, if only because they are so ubiquitous. ³A flaw in it affects the most people, instead of, say, a Joe Pablo¹s server that might not affect anybody,² says Mr. Ferris. But he will also take a shot at any product that claims to be secure, such as the Mozilla Foundation¹s Firefox web browser, whose selling point is security. In September, Mr. Ferris publicized an advisory that notified users of a flaw in the Firefox browser. The flaw attracted widespread media attention and forced Mozilla to post a fix to the problem within two days of it being made public. Mr. Ferris says he told Mozilla about the flaw, but the company did not respond to his request, and the Mozilla employee he dealt with was rude to him. Though Mozilla may be loath to admit it, the incident highlighted the uneasy relationship between freelance researchers and security companies. Mike Schroepfer, director of engineering for Mozilla, says that the run-in with Mr. Ferris was an exception. Overall, Mozilla has had an excellent relationship with independent researchers who bring bugs to the company¹s attention in return for a bounty, he says. As for the public disclosure of the bug before the release of a patch, Mr. Schroepfer shrugs it off. ³In an ideal world, the two would coincide, but that doesn¹t always happen,² he says. The Economics of Software Bugs Bug bounty programs aren¹t new, but the tensions and the ethical questions that they pose are now coming to center stage. In the past, bug hunters have usually posted notes about security vulnerabilities just for glory or as a contribution to the community. But that changed in 2002 when Symantec acquired SecurityFocus for $75 million in cash. SecurityFocus¹ biggest selling point was its Bugtraq mailing list, where security researchers exchanged notes about bugs in popular software products. That acquisition made bug tracking a big business. A few months later, security intelligence company iDEFENSE (acquired by VeriSign in July of this year) created its money-for-vulnerabilities program. ³We realized that security vulnerabilities are not typically found by corporations or software vendors. They are discovered by independent security researchers,² says Michael Sutton, director of iDEFENSE Labs. The action around bug bounty programs truly started heating up this year. Mozilla promised to pay $500 and a T-shirt for a ³reasonably important² bug found in its software products. In July, TippingPoint, a division of 3Com, started its Zero Day Initiative program. In the case of iDEFENSE and TippingPoint, the programs were a way to gather research that they could either sell to their customers or implement into their antidote products. So far, these money-for-bugs programs have been reasonably successful. iDEFENSE has received about 1,200 submissions over three years, though the company says it rejected about 50 percent of those. Still, it has managed to notch up some successes. TippingPoint found its first big bug in Veritas¹ software, though it has had about 100 submissions so far. Mozilla says it has paid out 40 bounties to 16 separate researchers over the past year. With the exception of Mozilla, all security companies interviewed declined to reveal the money they pay for bugs turned in, but some independent researchers say they have been offered anywhere between $500 to a few thousand dollars. This kind of commercialism could ultimately prove to be dangerous for the security business, says Dan Ingevaldson, director of professional services at enterprise security company Internet Security Systems (ISS). ISS has an elite unit called the ³X-Force,² comprised of 100 engineers who are among the highest paid in the business. Most of them earn a premium of 20 to 50 percent over an average software developer. The X-Force members are tough to find and hold on toone reason why many companies attempt to outsource security research, says Mr. Ingevaldson. And with most independent researchers holding a day job, their focus on finding bugs is just a hobby, however passionate it may be. Mr. Ingevaldson says ISS would rather have people on staff and get 100 percent of their attention and effort. Building an exclusive club is not the name of the game, say bounty creators. As more and more security vulnerabilities are discovered by independent researchers, it becomes important to encourage them to come to security companies with their knowledge, says David Endler, director of security research for Tipping Point. ³Our position is, why shouldn¹t do-gooders be rewarded for what they find?² he says. Rewards, say the security companies, are based on the criticality of the bug raised. But Mr. Ferris says that often the money paid out is ³peanuts² compared to the efforts that go into finding a bug. ³Companies like Microsoft are offering $250,000 for information about guys who write a worm,² he says, referring to Microsoft¹s bounty paid out for information regarding creators of the Sasser worm. ³If they were to pay us something like that, a whole lot of people would come out from under the rocks and submit flaws.² That is but one of the pitfalls of reward programs, say critics. In a competitive market, the price for a bug could be driven up. And if an independent bug hunter isn¹t paid the price he wants, he could disclose it publicly, or sell it to those who will pay the asking price for it, says Mr. Ingevaldson. Free Market Paying for vulnerabilities also brings into question the role of security companies. ³Security companies, who are chartered to protect the people, are creating a market around the information that could hurt them,² says Mr. Ingevaldson. ³Is a security company¹s job to drum up a market around dangerous tools or to protect users from those tools?² Having an army of freelance contributors is not the most efficient way to solve the problem of vulnerabilities, agrees SpireSecurity¹s Mr. Lindstrom. Outsourcing research may be cheaper and easier, but it is not what customers expect out of a security company, says ISS¹ Mr. Ingevaldson. ³Customers don¹t want to work with a security company that buys its research in an a la carte fashion from the market,² he says. It is the big picture that freelance security researchers help capture, and it is what their clients want, insist security companies. Customers want to know about potential bugs first, says iDEFENSE, which counts financial companies like MassMutual and government organizations like the U.S. Department of Health and Human Services among its clients. Steve Manzuik, moderator of Vulnwatch, a community web site, says that bug bounty programs give those with the knowledge an incentive to share their findings. ³Programs like this also give the independent guys a way to make a small income,² he says. And that¹s what it all boils down to. Despite what the critics may say, and the frictions that exist between freelance researchers and security companies, bug bounty programs are thriving because in a free market everything has value. At least with the bug bounty programs, it¹s the good guys who are paying for the bugs. You are a subscribed member of the infowarrior list. Visit www.infowarrior.org for list information or to unsubscribe. This message may be redistributed freely in its entirety. Any and all copyrights appearing in list messages are maintained by their respective owners.
