On Tue, 17 Aug 2010 07:01:05 -0700, Scott O Lilienfeld wrote: >Hi Mike et al. - I'd long heard "through the grapevine" from university >folks in the know that US News and World Report likes to change the >criteria for its rankings each year so that it can sell magazines (apparently, >they make a lot of their revenue from sales of these rankings, both in >magazine form and online).
Let me make some points: (1) Many magazines have "Best of" issues which are probably their best selling issues because it provides perhaps the only "objective" measure available in an area (e.g., New York Magazine has a "Best Doctors" issue that is well-advertised and is probably used by folks to determine which doctor to go in a particular area; given the readership of New York Magazine -- well-heeled New Yorkers -- they probably don't rely upon their medical insurance to serve as their main consideration in what doctor to go to). Although there are other guides to college, the U.S. News report appears to make the information much more accessible to the casual reader though it may be inaccurate or misleading or not relevant. NOTE: New York Magazine is not the same as the New Yorker magazine. (2) I don't really know much about U.S. News actual methodology but it seems to me that they are using what would be referred to as "improper linear models" as defined by Robyn Dawes in his famous American Psychologist article "The Robust Beauty of Improper Linear Models". In these models, arbitrary weights are used (instead of optimal weights derived, say, from regression analysis) as an indicator of the relative importance (or some other indicator) of the associated variable. Within this context, if the weights are fixed across years, the outcomes (i.e., final score or ranking) would still vary from year to year because of (a) random error in the variables used, (b) systematic error in the variables used, and (c) systematic changes in the level of variables from year to year (e.g., increase in graduation rate, decrease in full-time faculty, etc.). So, changes in rank *should* show some changes from year to year even if the weights remain constant. It could be the case, however, that relevant variables used in the equation are stable over time, thus producing similar patterns of ranking over the years (if error in the measurement is relatively low). >If the criteria remained the same each year, the rankings probably >wouldn't change much - so by altering the variable weights a bit each >year (and occasionally introducing new variables into the equation, like >high school counselors' ratings of reputation this year), the rankings >bounce around and hence generate more interest. Perhaps but has anyone calculated the test-retest correlation coefficients for the rankings across years? I would assume that, even with jiggling the weights a little, that the coefficient will be very high, implying a high degree of reliability in the stability over time sense. For example, what is the range of ranks for Harvard over the past decade or so? I doubt that it is greater than 5 (i.e., it is consistently in the top 5 highest ranked schools). Perhaps U.S. News does jiggle the weights but I don't think this will result in in much change in rank, perhaps 1-2 places (I am open to correction if other have more specific knowledge). Adding variables can also change rankings but I assume that U.S. News adds new variables in order to deal with criticisms from some source it considers important or relevant to sales. Even so, what is the Pearson r between last year's and this year's rankings? I would assume it would be high (anyone willing to do the analysis?). > But I don't know if this view is too cynical; I'd be curious to hear >others' thoughts. I wouldn't characterize what you said as cynical because, at the heart of the matter we're talking about business practices, not research. Changing weights and variables keeps the product fresh but it may not be clear how much these changes produce signficant variations from year to year -- but one always claim "new and improved" as part of the "fluff" used to sell the product. Again, the rankings provided by U.S. News can be accessed here: http://colleges.usnews.rankingsandreviews.com/best-colleges -Mike Palij New York University [email protected] --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=4266 or send a blank email to leave-4266-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
