> In the first instance I leaned away from pairwise because a) it > doesn't show you how good your prediction skills are compared with > other players b) I thought it would take more plays than my spec to to > get 217k images usable ratings. Can someone mathematical tell me how > many 'plays' of a pairwise game it would take to get any useful > results (according to any definition of useful you choose)? I am no > mathmo and might have got this quite wrong.
Let's assume each picture has a unique intrinsic prettiness value, and that we're trying to sort the last image into the deck. Whilst using a binary search, it'll take about 19 matches to get its exact position in the sorted deck, this is overkill. Instead, assuming we have the sorted list of images, we can find the image's position to an accuracy of an eight-point scale within three matches, and sixteen within four. However, this doesn't allow for taste; each individual is going to rate pictures differently. I'm uncertain about the right way of going about handling this, it is suggested on the net [http://stackoverflow.com/questions/164831/how-to-rank-a-million-images-with-a-crowdsourced-sort] that using the system used in chess rankings might help. Perhaps having a 'mis-ranked' value (how often, recently, has this been voted better/worse than the other picture unexpectedly) and focussing on images which might be mis-ranked might help. But I'd anticipate that you would want to compare pictures to pictures at least several thousand points away, at least initially, to try to get the list broadly sorted. The biggest question I can see is how to measure the reliability of the data. The 1..10 scale has a number of systematic issues already identified (changing expectations of what a 1 or 10 is over time; how people's inputs are distributed across the scale), but does allow simple statistical tests. I'm not sure how to verify how much the list is sorted, other than perhaps the stability of the rankings. Dave. PS: IANA Mathematician, but I do have a pair of sandals somewhere. _______________________________________________ Mailing list [email protected] Archive, settings, or unsubscribe: https://secure.mysociety.org/admin/lists/mailman/listinfo/developers-public
