On 14 Apr 2010, at 16:24, Wall, Kevin wrote:
> I just reread your Dark Reading post and I must say I agree with it
> almost 100%. The only part where I disagree with it is where you wrote:
>        The multiple choice test itself is one of the problems. I
>        have discussed the idea of using multiple choice to
>        discriminate knowledgeable developers from clueless
>        developers (like the SANS test does) with many professors
>        of computer science. Not one of them thought it was possible.

This is the part of the article I disagree with most, as well. Asking whether 
multiple choice exams can discriminate between clueful and clueless developers 
is a valid and important question to ask.  However, I believe few professors of 
computer science could discriminate between clueful and clueless developers if 
"developer" and "clue" have industry-relevant definitions.  What passes for 
"development" in an academic sense and what is required for "clue" in an 
academic sense are usually defined on very different axes than the axes used in 

So, I think asking college professors whether standardised tests are valid in 
this respect is posing the important question to the wrong people. There are 
notorious disconnects between what academics and industry value. Perhaps if you 
asked the folks who hire, promote, and evaluate developers, they could give a 
better opinion as to whether clue and standardised test performance correlate. 
Even then, I'd prefer to see something somewhat objective, like months between 
promotions versus certifications held, as opposed to calling a bunch of CIOs or 
VPs of Engineering and asking how well they think tests work.

Having said this, I am a CSSLP and I have helped write a ton of questions for 
the exam. I can tell you we struggle long and hard to write meaningful 
questions that actually discriminate a practitioner who has experience from a 
random, unqualified candidate. We use follow well-established psychometric 
principles when designing the questions. The whole test creation/maintenance 
process is ANSI-approved and audited. Careful statistics are kept on the 
pass/fail rates on individual questions to discard questions that do not 
discriminate well. Over time, the question bank is maintained to remove 
questions that don't test well and to write new questions that represent 
changes in the landscape. Some of you will undoubtedly dismiss this, saying 
"garbage in, garbage out, regardless of how pristine the pipes are." I believe 
that's too simplistic a view.

Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates

Reply via email to