On 14 Apr 2010, at 16:24, Wall, Kevin wrote: > I just reread your Dark Reading post and I must say I agree with it > almost 100%. The only part where I disagree with it is where you wrote: > > The multiple choice test itself is one of the problems. I > have discussed the idea of using multiple choice to > discriminate knowledgeable developers from clueless > developers (like the SANS test does) with many professors > of computer science. Not one of them thought it was possible.
This is the part of the article I disagree with most, as well. Asking whether multiple choice exams can discriminate between clueful and clueless developers is a valid and important question to ask. However, I believe few professors of computer science could discriminate between clueful and clueless developers if "developer" and "clue" have industry-relevant definitions. What passes for "development" in an academic sense and what is required for "clue" in an academic sense are usually defined on very different axes than the axes used in industry. So, I think asking college professors whether standardised tests are valid in this respect is posing the important question to the wrong people. There are notorious disconnects between what academics and industry value. Perhaps if you asked the folks who hire, promote, and evaluate developers, they could give a better opinion as to whether clue and standardised test performance correlate. Even then, I'd prefer to see something somewhat objective, like months between promotions versus certifications held, as opposed to calling a bunch of CIOs or VPs of Engineering and asking how well they think tests work. Having said this, I am a CSSLP and I have helped write a ton of questions for the exam. I can tell you we struggle long and hard to write meaningful questions that actually discriminate a practitioner who has experience from a random, unqualified candidate. We use follow well-established psychometric principles when designing the questions. The whole test creation/maintenance process is ANSI-approved and audited. Careful statistics are kept on the pass/fail rates on individual questions to discard questions that do not discriminate well. Over time, the question bank is maintained to remove questions that don't test well and to write new questions that represent changes in the landscape. Some of you will undoubtedly dismiss this, saying "garbage in, garbage out, regardless of how pristine the pipes are." I believe that's too simplistic a view. Paco _______________________________________________ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates _______________________________________________
