I have a problem evaluating a Grading System. I have data from a grading system. The problem is to determine if the system is fair. There are applications to be graded and there are hundreds. There are graders and there are about a dozen of them. Exams are held several times a year and this process has been going on for years. The applicants fall into groups depending on who they will be working for if they pass their exam. The Graders groups differ in terms of the kind of work the applicant is applying to do. The assignment to the grading group is made by an expert panel. When an application comes in it can either be thrown out or scored. In either case I know which happens. If it's scored then I have the score. If the score is really bad the applicant can apply for a later exam. An applicant can come back up to several times. I know which ones come back and can link the records. Graders are pretty stable over time but they have their careers with start and end times. My questions; Are applicants treated differently by the graders depending on which type of work they applying for? Is there any way to distinguish a poor group of applicants from a bias against an applicant group in the graders? Does it matter how many times the applicant comes back for follow up exams. The scores generally get better but there is no guarantee. The scores are not normally distributed and I have tried several approaches. I have looked at the risk ratio for the initial turn-down. I have looked at the Mann Whitney Wilcoxon for detecting shifts in the rank of the scores. I don't know if the variance of the scores is constant across applicant groups and I know that is confounded with the wilcoxon test. If the risk ratio isn't at least 1.8:1 people won't notice a difference. How do I estimate and correct for the lack of power in that test. The examiners want to use average scores to evaluate the grading system but I don't know how to properly test the averages considering I don't know the distributions. Are there standard ways to approach this problem? Problems like this come up in cases of employment discrimination, What sort of model should I use? Stan
_________________________________________________________________ Send and receive Hotmail on your mobile device: http://mobile.msn.com . . ================================================================= Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at: . http://jse.stat.ncsu.edu/ . =================================================================
