Hey all, guess I've sat this one out long enough to see the different
opinions, not always my method, usually I just throw it out there and
see who salutes...
First off, as the lead fly in the ointment for hands-on exams, I
think that Felipe has made some very valid points, and I want to
supplement, not supplant those below.
On Jun 13, 2006, at 4:05 PM, Felipe Salum wrote:
Answering your email in the break of Brazil and Croatia game :)
On 6/13/06, Dimitrios Bogiatzoules, Product Developer
<[EMAIL PROTECTED]> wrote:
Hi Felipe,
Felipe Salum said the following on 13.06.2006 19:19:
> Hello Dimitrios.
>
> First, thanks for changing the subject. Yes, I'm talking about
hands on
> exams.
>
> Well, let me try to explain what I think about hands on exams.
>
> 1. L3 is to be the senior certification for LPI, so I think that
to it
> get more credibility, a hands on exam is perfect. (I'm not
saying that
> L1 and L2 has no credibility, or any other paper based exam).
I've heard that before. Why is hands on giving more credibility in
your
opinion?
In hands on if you dont know how to do a task you can't close your
eyes and choose one option to mark, many times you'll see things that
only who is in linux administration all time knows how to finish it. I
hope that LPIC-3 is for experienced administrators, not those who just
have read some linux book, bought the last testking and get certified.
In my experience, which as many know includes a LOT of LPI bootcamps
for Level 1 (there's the data points for CBT exams) and Novell/SUSE
CLP's (there's the data points for Hands-On) I feel it's important to
not just focus on the actual objectives that are being tested, though
they are damned important. There are really three things that affect
the test taker when they present themselves at the testing center for
their examination, whether it's physically or virtually across the
wide area network:
1. The stated objectives and awareness of the actual difficulty of
the exams
2. The attendees actual preparation and skill set prior to the exam
3. The wording and format of the exam, environmental issues and
physical stamina
1) The first is our best friend in the testing industry, it's a
combination of the perceptions of the people who the examinee knows,
has interacted with and seen/had conversations with in any format,
that lets the attendee know what they are getting themselves in
for. Just reading the objectives online doesn't really prepare you
for how the questions will be asked, or the tested scenarios will be
formatted. Many an examinee will have prepared with multiple books,
a class or online webinar, lots of banging about trying things, a
brain dump or two and plenty of lurking/participating in forums about
the topic, all of which is environmental to their perception of the
exam's difficulty and what they personally will need to prepare for
that exam. If a number of people are apparently succeeding at brain
dumping a question-based exam, then the ROI for some people will be
to concentrate their limited amount of time and dedication to
"CLEPping" out of that exam by effectively memorizing the possible
questions and answers.
I tell people getting started on a certification that most exams are
the collision of real-world experience and your ability to problem-
solve, and the first thing they should do is read through the entire
set of objectives and in their mind visualize if they can do the
task, or try it, and then gauge how far off of the mark they are,
then work to fill that gap. We all do our best to appropriately
represent the exams and organizations we have passed the mark for,
and we contribute to the public perception of those exams through our
actions, word and contributions online.
2) I can't count the people who I have conversed/advised/lectured/
cautioned/scolded gently about their preparation for a given exam. I
call it a "No-Shitter" and it's designed to put them into the right
frame of mind about what they are getting themselves into. The good
souls that really approach a certification as it should be are the
ones who treat it as a merit badge of ability gained, not an end
point. We have all done a lot to publicize and evangelize what is
needed to adequately prepare for the LPI exams, and there are
countless success stories that have happened, and many disappointed
people who have done what can only be described as half-assed
preparation, such as reading a book that's 5 years out of date and
not even bothering to note if the objectives have been updated in
that time before showing up at the exam center. For every person who
such people turn off of the cert because they are bitter, and there
are a few victims of this, there are a multitude of others who look
at that person as a personal warning sign to have their stuff
together before they attempt the exam.
3) I have seen many a person who prepared adequately for a CBT exam
experience complete meltdown on a Lab-based exam. They just don't
have the framework or structure to know where to begin when faced
with an actual real-world scenario, they NEED the focused and
artificial format of question/response to actually get through the
exam. This stress and the shock of reality that happens (the Oh Shit
Factor) when you find out just what's being asked and how hard it
will actually be to do the task that happens in a Hands On exam is in
my experience and opinion nearly impossible to replicate in a CBT
format.
>
> 2. We all know about braindumps, testkings, boson, etc, where some
> people just memorize the answers.
Don't forget that hands on exams handle with a very limited number of
different. If you know that you will be asked to complete one of,
lets
say, 10 scenarios on a running machine then it becomes rather
easier...
I don't agree that a scenario is easier to memorize or to guess, if
there were only one way to do things that would be true, but for
example the CLP exams allow for ANY way to accomplish the task, as
long as the task is accomplished and verifiable, it's a pass. In the
beginning there were only a few limited scenarios, but they are not
hard to build out and check for any broken dependencies, it's just
like storyboarding a movie or multimedia event, continuity is
important but so is modularity of the examined tasks. There are SO
MANY things that are difficult enough to adequately test an
examinee's ability that are effectively self-contained that I don't
think it's a real obstacle, in fact the CLP's are an example of how
it can be done, but I'm not saying they are better than anything,
just that they do work and work well.
Yes, I know. In hands on exams you have limited time too, so who just
knows what will be asked and tested at home, if anything is different
in the exam will fail, different from the experts one.
This is a very important point: If you have a broad enough set of
objectives, you effectively FORCE the examinee to prepare for ALL OF
THEM. The actual method of testing is not as important as the fact
that you are letting the examinee know they could be faced with ANY
of the objectives, and they can't just depend on a given limited set
of scenarios happening. So far I haven't seen anyone from the
Question-based exam camp really GET this fact, and that it affects
all of the types of exam methods. It's the perception of difficulty
that is communicated about the exam that will cause adequate
preparation, simple and clear.
>
> 3. Just for example, I was taking a look at Sans GSE certification,
> which they did too much difficult to have just a little people
> certified. I don't want that for LPIC3, but looking at GSE, you see
> that people have to demonstrate really that they own what they are
> being tested. It is what I see in a hands on exam. Just
memorizing the
> solution is not sufficient to put your hands there and make it
> working.
I do not know that certification so I can't answer here...
It is GIAC Security Expert certification, nice requisites to take it.
Take a look at http://www.giac.org
>
> 4. RHCE, CCIE and others are successfull examples of hands on
exams.
> You see many people with CCNA (paper based with some hands on
> simulator) and just a few with CCIE.
I think there are enough hands-on exams that have a perception of
being real tough nuts to crack that we can use as inspiration for
anything we would create.
Don't forget that both are based on a very specific environment. RH
tests only RH and CISCO even uses its own hardware. Try to do that on
different hardware and with different distributions and a possible
solution becomes very sophisticated.
BTW: I've added some notes at the end of
https://group.lpi.org/cgi-bin/publicwiki/view/Examdev/LPIC-3Samba
before we started this discussion.
I saw the notes. I agree the solution becomes very sophisticated.
I agree completely, this would be the hard part of the hands-on
angle, but not impossible. This would not be a showstopper, but
would require some finesse to overcome.
[...]
> Thinking more there are many others arguments for using hands on
exam,
> but first we need to know if LPI has interest in this type of
exam. I
> have seen this same thread in lpi-discuss mailing list before
and LPI
> always say, the price to make a infrastructure for hands on exam
will
> increase the exam price, there is no infra, etc.
And those issues may well cause us to NOT do a hands-on exam, but at
least we are talking about it and considering it as an option, so
that's better than the sometimes dismissal that has occurred, almost
all of which happened in the previous, ah, "regime". It's not polite
to talk of those who have passed on to other things, so I'll let that
rest forever.
The discussion before was not about Level 3 but Level 1 and 2 and for
that we have very good reasons not to make them hands on.
Yes L1 and L2, that because I have not joined in the thread. I think
that in the case just the senior cert could be hands on.
And with the perceived and actual level of difficulty we want to
project, a hands-on might be JUST the thing for THIS level. I agree
that a hands-on for the previous levels would be counter to the goals
of being affordable, accessible and easily administerable.
The final proof that results of hands on exams are providing more
information that a candidate deserves a certificate than "normal"
forms
is still missing.
As for Level 3 we do start with the same expectation but while
development we may face a situation where "normal" exams wouldn't
provide us the information we need.
The only reason IMHO why LPI should/could deploy a hands on exam
is if
candidates knowledge couldn't get certified else.
I agree as long as you count the factors I mentioned above, the
totality of the effect of those factors on the examinee, as part of
why they couldn't get certified with that particular level and with
the corresponding amount of credibility and bragging rights. I
submit that we could build a formidable and deliverable exam that was
hands on, then deliver it either in person or as a proctored CBT/
Remote session exam from certified testing centers.
This is a great thread, thanks for letting me get this all out of my
system, I'll support whatever we come up with, but it's important
that we investigate all the options, and let's not kid ourselves, as
this level, no one should balk at the expense for a hands-on exam,
they're getting a premium level of certification, and great value for
their cost.
Ross
_______________________________________________
lpi-examdev mailing list
[email protected]
http://list.lpi.org/mailman/listinfo/lpi-examdev
