Debugging: OK to Outsource?

Cash rewards for freelance researchers is a cheap and easy way to fix
security holes. But critics say the expanding practice could hurt the
software security industry. November 14, 2005 Print Issue

Tom Ferris is a bounty hunter. In the Wild West of software security, he¹s a
cowboy who hunts bugs‹security loopholes in software that can be exploited
to launch malicious attacks.

On nights and weekends, Mr. Ferris combs through popular software products
searching for the gaps. Every discovery has the potential of a financial
payoff, anywhere from $500 to a few thousand dollars for a ³valuable² bug.

Mr. Ferris, 27, is one of the hundreds of independent security researchers
who hunt bugs as a hobby. They are passionate about their work, but their
loyalty is tenuous‹once driven by the idea of fame, money is now their
incentive. Their patrons are security companies that have started bug bounty
programs, created to farm out bug hunting to freelance security researchers.

Since money-for-bugs programs gained popularity a few years ago‹sparked by
Symantec¹s 2002 acquisition of SecurityFocus, which hosted an online
community where bug hunters could post their findings‹freelance researchers
have found and reported hundreds of security holes in software products,
helping security companies improve their offerings and fattening their own
wallets a little in the process.

But the relationship between freelance researchers like Mr. Ferris and the
security companies isn¹t always harmonious and symbiotic. Security companies
don¹t always respond amicably when researchers find a flaw; some researchers
say they aren¹t rewarded accordingly when they do find something important.
And even when the relations go smoothly, some say this form of
commercializing and outsourcing vulnerability research could lead to a
public relations nightmare for security companies, or‹in the worst-case
scenario‹a rogue bounty hunter selling vulnerabilities to hackers who will
exploit the hole.

In the long run, these reward programs can do more damage than good, warns
Pete Lindstrom, director of research at SpireSecurity. ³Their contribution
to the profession is at best ambivalent, and at worst negative and
destructive,² he says.

Finding the Holes

As long as the rewards are offered, freelance researchers like Mr. Ferris
will keep looking for and finding holes. A security researcher for nine
years, Mr. Ferris has a day job as a software engineer for a security
company that he declines to name. When he searches for bugs outside of work,
he starts by picking a popular product and profiling it. He checks the
features, and learns the functionalities and protocols that it uses. Then he
goes about ³fuzzing² it‹sending random or malformed data to the program,
which causes it to crash or overflow. It¹s an easy way to pick the
low-hanging fruit‹the security bugs that can be found through automated
tools, he says.

If a fuzzer doesn¹t do the trick, Mr. Ferris will try to reverse engineer
the product. Either way, by the end of his effort, he will hopefully have
found a bug or two. His favorite targets are Microsoft products, if only
because they are so ubiquitous. ³A flaw in it affects the most people,
instead of, say, a Joe Pablo¹s server that might not affect anybody,² says
Mr. Ferris.

But he will also take a shot at any product that claims to be secure, such
as the Mozilla Foundation¹s Firefox web browser, whose selling point is
security. In September, Mr. Ferris publicized an advisory that notified
users of a flaw in the Firefox browser. The flaw attracted widespread media
attention and forced Mozilla to post a fix to the problem within two days of
it being made public. Mr. Ferris says he told Mozilla about the flaw, but
the company did not respond to his request, and the Mozilla employee he
dealt with was rude to him.

Though Mozilla may be loath to admit it, the incident highlighted the uneasy
relationship between freelance researchers and security companies.

Mike Schroepfer, director of engineering for Mozilla, says that the run-in
with Mr. Ferris was an exception. Overall, Mozilla has had an excellent
relationship with independent researchers who bring bugs to the company¹s
attention in return for a bounty, he says. As for the public disclosure of
the bug before the release of a patch, Mr. Schroepfer shrugs it off. ³In an
ideal world, the two would coincide, but that doesn¹t always happen,² he

The Economics of Software Bugs

Bug bounty programs aren¹t new, but the tensions and the ethical questions
that they pose are now coming to center stage. In the past, bug hunters have
usually posted notes about security vulnerabilities just for glory or as a
contribution to the community. But that changed in 2002 when Symantec
acquired SecurityFocus for $75 million in cash. SecurityFocus¹ biggest
selling point was its Bugtraq mailing list, where security researchers
exchanged notes about bugs in popular software products. That acquisition
made bug tracking a big business.

A few months later, security intelligence company iDEFENSE (acquired by
VeriSign in July of this year) created its money-for-vulnerabilities
program. ³We realized that security vulnerabilities are not typically found
by corporations or software vendors. They are discovered by independent
security researchers,² says Michael Sutton, director of iDEFENSE Labs.

The action around bug bounty programs truly started heating up this year.
Mozilla promised to pay $500 and a T-shirt for a ³reasonably important² bug
found in its software products. In July, TippingPoint, a division of 3Com,
started its Zero Day Initiative program. In the case of iDEFENSE and
TippingPoint, the programs were a way to gather research that they could
either sell to their customers or implement into their antidote products.

So far, these money-for-bugs programs have been reasonably successful.
iDEFENSE has received about 1,200 submissions over three years, though the
company says it rejected about 50 percent of those. Still, it has managed to
notch up some successes. TippingPoint found its first big bug in Veritas¹
software, though it has had about 100 submissions so far. Mozilla says it
has paid out 40 bounties to 16 separate researchers over the past year.

With the exception of Mozilla, all security companies interviewed declined
to reveal the money they pay for bugs turned in, but some independent
researchers say they have been offered anywhere between $500 to a few
thousand dollars.

This kind of commercialism could ultimately prove to be dangerous for the
security business, says Dan Ingevaldson, director of professional services
at enterprise security company Internet Security Systems (ISS). ISS has an
elite unit called the ³X-Force,² comprised of 100 engineers who are among
the highest paid in the business. Most of them earn a premium of 20 to 50
percent over an average software developer.

The X-Force members are tough to find and hold on to‹one reason why many
companies attempt to outsource security research, says Mr. Ingevaldson. And
with most independent researchers holding a day job, their focus on finding
bugs is just a hobby, however passionate it may be. Mr. Ingevaldson says ISS
would rather have people on staff and get 100 percent of their attention and

Building an exclusive club is not the name of the game, say bounty creators.
As more and more security vulnerabilities are discovered by independent
researchers, it becomes important to encourage them to come to security
companies with their knowledge, says David Endler, director of security
research for Tipping Point. ³Our position is, why shouldn¹t do-gooders be
rewarded for what they find?² he says.

Rewards, say the security companies, are based on the criticality of the bug

But Mr. Ferris says that often the money paid out is ³peanuts² compared to
the efforts that go into finding a bug. ³Companies like Microsoft are
offering $250,000 for information about guys who write a worm,² he says,
referring to Microsoft¹s bounty paid out for information regarding creators
of the Sasser worm. ³If they were to pay us something like that, a whole lot
of people would come out from under the rocks and submit flaws.²

That is but one of the pitfalls of reward programs, say critics. In a
competitive market, the price for a bug could be driven up. And if an
independent bug hunter isn¹t paid the price he wants, he could disclose it
publicly, or sell it to those who will pay the asking price for it, says Mr.

Free Market

Paying for vulnerabilities also brings into question the role of security

³Security companies, who are chartered to protect the people, are creating a
market around the information that could hurt them,² says Mr. Ingevaldson.
³Is a security company¹s job to drum up a market around dangerous tools or
to protect users from those tools?²

Having an army of freelance contributors is not the most efficient way to
solve the problem of vulnerabilities, agrees SpireSecurity¹s Mr. Lindstrom.
Outsourcing research may be cheaper and easier, but it is not what customers
expect out of a security company, says ISS¹ Mr. Ingevaldson. ³Customers
don¹t want to work with a security company that buys its research in an a la
carte fashion from the market,² he says.

It is the big picture that freelance security researchers help capture, and
it is what their clients want, insist security companies. Customers want to
know about potential bugs first, says iDEFENSE, which counts financial
companies like MassMutual and government organizations like the U.S.
Department of Health and Human Services among its clients.

Steve Manzuik, moderator of Vulnwatch, a community web site, says that bug
bounty programs give those with the knowledge an incentive to share their
findings. ³Programs like this also give the independent guys a way to make a
small income,² he says.

And that¹s what it all boils down to. Despite what the critics may say, and
the frictions that exist between freelance researchers and security
companies, bug bounty programs are thriving because in a free market
everything has value. At least with the bug bounty programs, it¹s the good
guys who are paying for the bugs.  

You are a subscribed member of the infowarrior list. Visit for list information or to unsubscribe. This message
may be redistributed freely in its entirety. Any and all copyrights
appearing in list messages are maintained by their respective owners.

Reply via email to