On Wed, 26 Jan 2000 [EMAIL PROTECTED] wrote:
> To summarize we will take the following steps:
>
> 1) Improve the job description via discussion on this mailing list.
> 2) Collect relevant tasks amongst members of this mailing list.
> 3) Complete a Job Analysis Survey of the collected tasks.
> 4) Complete the Job Analysis Report.
> 5) Write Objectives based on the JA results with a small team of writers.
> 6) Improve/Review/Comment on the written objectives via mailing list.
> 7) Write test items via both a team of writers and public reviewed input.
> 8) Screen and review written and submitted test items.
> 9) Construct Exam
> 10) Deploy Exam, starting with a Validation Period.
>[snip]
> The purpose of the JA is to identify the low-levels tasks that are performed
> by job incumbents indented to be covered by the examination program. The JA
> involves surveying subject matter experts (SME's) about the frequency and
> importance of tasks they perform.
Some of the discussion of, I assume, the "job description" seems to me to
be misplaced. What we are working towards is a way to distinguish L1 from
L2 (from L3) sysadmins. An actual job description would logically FOLLOW
the job analysis and never precede it (imagine coding something and then
designing it--sure it happens but it's still ass-backwards).
So shouldn't we shelve the "Shouldn't we include XXX?" comments? Their
proper place is in the task gathering phase of the job analysis. Then in
the surveying part of the job analysis, we'll see who uses what and that
should be the determinant of whether it goes into L2.
I propose that we concentrate on a much shorter and more straight-forward
definition (or example) of what is L2 based on a list of elements that
distinguish between L1 and L2 and between L2 and L3. I think the
admirable and skillfully-crafted job description is far too long to serve
as a practical definition for the purposes of... well, for any purpose
that I can imagine. I guess our "purposes" and (1) to let potential
consumers of the exams benchmark where their level of acumen fits in our
system; (2) to allow would-be subject matter experts to self-select
themselves into or out of our job analysis; and (3) to have a good
definition for the judges in the cut-score analysis.
>[snip]
> One (.5) week should be allocated to this sub-step.[analysis of JA data]
I suggest one (1) week.
> [snip]
> One-third (.33) week should be allocated for this sub-step. [convert to
VUE format]
Is this realistic? About a day and a half? Scott knows best (?) but I
would have guessed this is either a 5 minute job or a long, tedious labor.
Is there time budgeted for proofing the test after VUE gets done? I think
this will take more than 12 hours. This point in the schedule is not the
right time to rush things.
> About 3 weeks will be required by the publishing vender to deploy. The beta
> will run about 4 weeks.
I'll just note that we have no idea (that I know of) how long a beta will
be needed. It really depends on how many people take the beta exam. That
said, I think four weeks is a bit optimistic.
> 6.2 Conduct Beta and Cut-score Analysis
>
> After the beta exam forms have been administered to a sufficient number of
> examinees, psychometricians will use the itemmetric data generated to
> conduct validation research on the exams. Poor performing items will be
> dropped from the forms at this time. The surviving items will be
> republished, as described in the previous sub-step.
>
> Also at this time, a study will be performed to determine the required
> passing score for each exam.
>
> About 2 weeks will be required for this sub-step.
>
How about this paragraph:
After the beta exam forms have been administered to at least 100
examinees, LPI staff will conduct item analyses to determine which items
are sufficiently precise and valid. Items deemed poor will be
dropped. The surviving items will be assembled into two exam forms.
These forms will be converted into the format required by VUE. The
VUE-formatted exams will be proofed against the original.
In parallel with the conversion and proofing, these forms will be subject
to a cut-score study to determine passing scores for each form.
About FOUR weeks will be required for this sub-step.
(I think that all this work will take more than two weeks, the cut-score
study might take two weeks by itself.)
-Alan Mead
--
This message was sent from the lpi-examdev mailing list.
Send `unsubscribe lpi-examdev' in the subject to [EMAIL PROTECTED]
to leave the list.
