On 12-06-05 03:48 PM, Brett Watson wrote:
On Jun 5, 2012, at 9:52 AM, Peter Kristolaitis wrote:As far as horror stories... yeah. My most memorable experience was a guy (with a CISSP designation, working for a company who came highly recommended) who: - Spent a day trying to get his Backtrack CD to "work properly". When I looked at it, it was just a color depth issue in X that took about 45 seconds from "why is this broken?" to "hey look, I fixed it!". - Completely missed the honeypot machine I set up for the test. I had logs from the machine showing that his scanning had hit the machine and had found several of the vulnerabilities, but the entire machine was absent from the report. - Called us complaining that a certain behavior that "he'd never seen before" was happening when he tried to nmap our network. The "certain behavior" was a firewall with some IPS functionality, along with him not knowing how to read nmap output. - Completely messed up the report -- three times. His report had the wrong ports& vulnerabilities listed on the wrong IPs, so according to the report, we apparently had FreeBSD boxes running IOS or MS SQL... - Stopped taking our calls when we asked why the honeypot machine was completely missing from the report. In general, my experience with most "pen testers" is a severe disappointment, and isn't anything that couldn't be done in-house by taking the person in your department who has the most ingrained hacker/geek personality, giving them Nessus/Metasploit/nmap/etc, pizza and a big ass pot of coffee, and saying "Find stuff we don't know about. Go.". There is the occasional pen tester who is absolutely phenomenal and does the job properly (i.e. the guys who actually write their own shellcode, etc), but the vast majority of "pen testers" just use automated tools and call it a day. Like everything else in IT, security has been "commercialized" to the point where finding really good vendors/people is hard, because everyone and their mom has CEH, CISSP, and whatever other alphabet soup certifications you can imagine.I agree with a lot of what you've said, but there are absolutely good security guys (pen tester, vulnerability assessors, etc) that use both open source and commercial automated tools, but still do a fantastic job because they understand the underlying technologies and protocols. I used to do a lot of this in the past, had lots of automated tools, and only occasionally wrote some assessment modules or exploit code if necessary. But again, a person in that position has to understand technology holistically (network, systems, software, protocols, etc). -b
I completely agree. I didn't mean to imply that using automated tools is a bad thing -- simply that running an automated tool to pump out a report with no further investigation isn't really a useful pen test. I've seen vendors whose "comprehensive penetration testing" was basically "We'll run Nessus against your network, write up an executive summary and email you the scan results. Quite the bargain for $20K!"
Automated tools are definitely good to provide a first pass over a network, but even then multiple tools should be used, and an experienced eye should review the results for anomalies (whether that's a vulnerability that has a chance for false positives, discrepancies between the results of two or more automated tools, etc). That kind of work, along with more aggressive pen tests and exploit development, need a "guru meditation"-level understanding of the involved technologies, protocols, etc, as you mentioned.
Like everything else IT, the specific tools used are more or less immaterial to an excellent practitioner -- a good programmer can hack code in any language, a good network engineer can use any brand of network equipment, etc -- because these types of people truly understand the systems they're dealing with, and use tools to accomplish a specific task which fits into part of the "big picture" they have in their heads. Poor practitioners in a field use tools for the sake of using the tool ("I'm scanning a network with Nessus because that's what the certification course told me to do") without that deep level of understanding, and therefore don't provide any real value to the process.
- Pete
