while gene has read these documents carefully (someone has to right?) ... i
have not BUT, the following tidbit struck my interest
========
Comparisons of School- and District-Level Scores
The statistical significance of these comparisons is based on the number of
students tested. Whether
making comparisons within a single year or across years, the table below
will assist in comparing
the scores of two or more schools or districts, comparing the scores of a
school to the district and/or
state, or the scores of a district to the state. The table shows the
smallest differences in scores that
represent a statistically significant difference in performance, based on
the number of students tested
in the school and/or district.
SINCE WHEN DOES STATISTICAL SIGNIFICANCE REALLY ENTER INTO THIS GIVEN ...
THIS IS A STATEWIDE ASSESSMENT ... AND, PRESUMABLY ... ALL STUDENTS IN A
SCHOOL ... AND ALL SCHOOLS ... AND ALL DISTRICTS ARE ASSESSED?
IF MY SCHOOL HAS 50 KIDS IN GRADE 4 ... BUT ANOTHER SCHOOL HAS 140 KIDS IN
GRADE 4 ... WHY SHOULD COMPARISONS BETWEEN MEANS OF THESE SCHOOLS ... BE
TREATED DIFFERENTLY IN THE SENSE OF STATISTICAL SIGNIFICANCE? ARE EACH OF
THESE REALLY SAMPLES THAT ARE BEING USED TO ESTIMATE SOME POPULATION (ALL
4TH GRADERS IN THE STATE) PARAMETER??
I THINK THIS IS A MISUSE OF STATISTICAL SIGNIFICANCE
THE USEFULNESS OF THE SCORES IS A FUNCTION HERE OF THE RELIABILITY OF THE
TESTS ...
It is not appropriate or possible to compare a school's or district's
scaled scores across two subject
areas since the scaled scores in each subject area were determined by
separate standard-setting
processes.
THIS IS INDEED INTERESTING ... WHILE IT IS PERFECTLY FINE TO COMPARE ACROSS
SCHOOLS ... OR DISTRICTS ... FOR THE same TEST ... IT IS NOT APPROPRIATE TO
COMPARE within THE SAME SCHOOL OR SAME DISTRICT ... TWO DIFFERENT TESTS ...
SO I ASK ... HOW WOULD YOU KNOW (and this seems a perfectly legitimate ? to
ask about your school or district) IF YOU WERE DOING RELATIVELY BETTER IN
MATH ... OR ENGLISH?
IF YOU CAN'T REALLY TELL IF A SCHOOL OR DISTRICT IS DOING BETTER/WORSE
ACROSS TEST AREAS ... HOW CAN WE TRUST AND BE CONVINCED THAT IT IS OK TO
COMPARE SCHOOLS/DISTRICTS ON THE same TEST AREA?
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
