scott
Thu, 9 Sep 1999 04:39:34 -0700
On Thu, 9 Sep 1999, Alan & Susan Mead wrote: > >We have definitely been discussing written tests. I think there is a lot > of > >interest in psychometricians and others in more performance-based testing > >and a lot of certification folks (i.e., sys admins) have mentioned it in > >discussions but I don't know anyone that is doing it validly and cost > >effectively. Alan message touches upon all of the major limitations of using "hands-on" assessment instead of traditional testing. It all boils down to this: It might appear obvious that hands-on assessment, where for example the examinee actually sits in front of a live Linux box and manipulates it, is a better measure of competency that regular testing, but research does not back this assumption up. Most studies that have been done comparing the two methodologies find that there is a stronger correlation between test score and future job performance than between evaluation of hands-on experience and future job performance. The truth is we (psychometricians) have not figured out how to extract reliable and valid measures from performance tests. Until (and unless) advances are made that permit this, we are better off sticking with traditional tests. One of the greatest benefits of traditional Q&A testing over hands-on experiences is that the former allows broader and deeper content sampling than the latter. By this, I mean that it is possible to cover much more material on a traditional test than on a hands-on assessment (assuming time is held constant.) For example, in 2 hours candidates could answer as many as 90 questions, covering a broad range of topics. But if hands-on assessment were used, they would be limited to just a few different assessments, and it would be difficult to draw general conclusions about their overall level of competence. As a result of these factors, and other mentioned by Alan, we have decided to use tests rather than hands-on assessments for our basic certification levels. The key to making this work is to include many different kinds of questions, particularly those that require the examinee to apply high level problem solving skills. ------------------------------------------------------------------------ Scott Murray Director of Exam Development / Psychometrician [EMAIL PROTECTED] Linux Professional Institute http://www.lpi.org/ ------------------------------------------------------------------------ ________________________________________________________________________ This message was sent by the linux-cert-program mailing list. To unsubscribe: echo unsubscribe | mail -s '' [EMAIL PROTECTED]