Folks, For a bunch of decision theorists, we sure do fall easily into alternative-focused thinking!!!
Let's try some value-focused thinking instead. What are the values we are attempting to serve with the structure of our review process? Values I have heard people articulate are: - Quality. We want the set of accepted papers to be of uniformly high quality. - Fairness. We want authors to have a fair chance of having their work accepted on its merits whether or not they are well known to the insiders in the UAI community. - Openness to new ideas. We want outsiders with interesting new ideas to have a fair hearing and a chance to have their papers accepted. - Cost in time and effort. We do not want the process to be overly burdensome on authors, reviewers, and the program chairs. - Cost in dollars. Obviously, cost in time and effort to volunteer reviewers and chairs can be reduced by paying professionals to manage certain aspects of the conference. This cost would have to be borne either by conference registrants, by our corporate sponsors, or by charging dues to join AUAI. Can anyone add to the list? We might consider creating a matrix with major values as the rows and options as the columns. One option obviously is no change to current processes. Other options would involve different modifications aimed at improving our score on some of the rows, but possibly at the expense of the score on other rows. The proponent of a given modification might describe his/her modified process in enough detail to provide an assessment of how it would score on each of the rows. In particular, can proponents of double-blind reviewing describe processes that we could implement at relatively low cost and would provide significant improvement on some of the key attributes of value? Then we could focus our discussion on what our relative weights are as a community on the different attributes of value, and on our assessments of how well each of the options under consideration scores on the different attributes of value. Once we have structured the problem, we might want to consider taking a survey to assess the community's views on the subject. Putting a survey form up on the web is a relatively easy matter. Of course we wouldn't have a scientific sampling process, but it would still be a more reliable indicator of community views than the opinions of those of us who tend to pipe up on email listservs. :-) Kathy
