On Friday 13 June 2008, Tobias Crefeld wrote: > <[EMAIL PROTECTED]> wrote: > > On Friday 13 June 2008, Grant Sewell wrote: > > > Cisco used to have the same scheme on their Academy courses. They > > > recently (3 years?) changed their system to accommodate the > > > flexibility. Essentially their system now recognises that if a > > > question has 3 marks then there should be an opportunity for the > > > examinee to get 0, 1, 2 or 3 marks for it. > > > > It also encourages guessing and thereby completely screws up the > > statistical analysis of the answers. > > The support of guessing is a disadvantage of multiple choice tests in > general no matter how many answers you offer. > The only advantage is the reduced efforts for automated scoring.
Guessing does introduce an X factor, which can be (trivially) statistically removed: Problem: Offer a multiple choice exam with 100 questions, 4 choices each, one correct answer each. Answering randomly gives a statistical average of a 25% score. Solution: Move the baseline. A score of 25% gives you a rating of 0, your *actual* marks are whatever you scored in excess of 25%. That's the highly simplified version of course, but it illustrates the general principle > > Multi-answers questions test the candidates ability and knowledge > > on a topic that has more than one element. The whole point of the > > question is to identify if the candidate does in fact know all the > > elements. > > I think it depends on the type of answer and it depends on how close > to practical problems the test should operate: > > If you ask how to solve a problem it should be enough if you have one > correct solution. If you have more solutions this is nice but > without any practical benefit so you get just one point and none if > one or more wrong answers got marked. This is probably not viable for a written exam, so these kinds of question are simply not asked. In contrast, Red Hat's exam can use this method as they only check for the end result. (I'm an active RHCI so I have some insight into this) As a simple example, suppose the question is "Setup and configure a mail server to receive mail for example.com". The marking scripts may connect to port 25, and check for the expected response after "RCPT To:" and do very little else. The candidate is entirely free to use whatever MTA they choose. If the candidate has superior l33t c0ding skillz and thinks they can write and build an MTA from scratch in the three hours allocated, and does so *and* it works, he will get full marks. This is of course highly unlikely, but it would be perfectly acceptable if it were ever accomplished. LPI cannot use this marking method for practical reasons, so we avoid the problem entirely by simply not asking questions with several equally correct answers. > On the other hand if you ask e.g. for some effects of a command every > correctly marked item should count as it proves a slightly increased > level of knowledge and every wrong answer marked should bring one > minus point down to a limit of zero points per question. In practice, questions like this are horrendously difficult to formulate. Once upon a time I ran several quite large item labs and a few attendees tried to come up with questions like this just for fun. What actually happened is that there was always some ambiguity in te wording. Fixing that always introduced an explosion of edge cases that had to be accounted for. Trying to deal with that always ended up with a stupid simplistic question where the very wording of the question gave away it's own answer. It was quite an eye-opener to everyone, and quite hilarious afterwards. We all got significant insight into the highly technical nature of exam questions. In the end we concentrated on what was recommended to us in the beginning anyway - questions deal with one known fact, which has one unambiguous answer. If the answer has two separate components then you still treat the answer as one complete whole (with two parts). > The third possibility might be a security question, e.g. you have to > list all points you have to care about if you setup a secure host. > As there is no half secure host only a complete list should get > scored. This is a possibility that does work in practice, and for more things than just security. The only trick is to make sure that all the points that have to be taken into account belong to the same published Objective. You shouldn't ask a question that requires a candidate to answer that port 143 must be open on the firewall and dovecot must be running for the user to read their mail using imap for example - that is two completely different exam Objectives > This is just my personal point of view. Don't care about it. ;) Nice discussion though :-) In the past 4 years I have learnt that education and testing are much further removed from real life than I first thought. The classroom is always an artificial environment, so is the exam room. This is true for any exam room, and for any type of exam you could ever dream up. So an exam you have passed never proves anything in the scientific sense, but it does give a fairly reliable indicator that you have a fair amount of knowledge in the area tested. Nothing would make me happier than to have a thorough Linux exam method that tests sysadmins to the same throughness as airline pilots and brain surgeons. Have the candidate demonstrate over and over again that they can perform every required action consistently, and that they know how to do it and why it's done this way. But that's just ludicrous - few sysadmins on the planet could afford to pay the fees :-) -- Alan McKinnon alan dot mckinnon at gmail dot com _______________________________________________ lpi-discuss mailing list [email protected] http://list.lpi.org/cgi-bin/mailman/listinfo/lpi-discuss
