[ The Types Forum (announcements only), http://lists.seas.upenn.edu/mailman/listinfo/types-announce ]
Gentle colleagues There has been a vigorous debate on the Types mailing list about acceptance rates and criteria for POPL. Phil Wadler, as SIGPLAN Chair, and a member of the POPL steering committee, asked me to present a concrete proposal for discussion at the POPL community meeting tomorrow. Thank you for the opportunity, Phil. The proposal appears below. I am sorry that I am missing the meeting this year. Enjoy POPL! Simon Reviewing for POPL: a concrete proposal ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ I will not repeat the details of the debate here, since you all have access to it: http://lists.seas.upenn.edu/pipermail/types-announce/2010/thread.html#1740 Instead, responding to Phil's request, I want to make single proposal That we use a "quality bar" rather than a "quantity bar" to govern acceptance for POPL. It is difficult to quantify just what "POPL-publishable quality" is, but I propose that it should be a level that, if it had been applied in recent years, would have resulted in an acceptance range in the region of 30%. The current norm is 16-23%. One data point is that ICFP typically accepts rather more than 30%, but the quality IMHO is still very high. (Historical figures for POPL, PLDI, and ICFP appear below) An alternative would to continue with a quantity bar, but increase it substantially, say from its current 35 to 50 papers. Personally I prefer a target acceptance rate because my gut feel is that the average quality does not change much year to year, whereas submission volume does. Regardless of the exact figure, I am advocating a sea change in our attitude to the POPL evaluation process, not just an incremental shift in policy. It is worth noting anecdotal evidence that individual POPL program chairs have tried and failed to get their committee to accept more papers. The idea is discussed, potential papers are brought up, but they are ultimately rejected. To achieve this change, if we want it, will take a broad community decision that gives a clear mandate to the program committee. With all that said, we can't tie the program committee's hands completely, by requiring them to accept N papers or X% of submissions; in the end we have to trust the PC. This proposal is not to restrict their discretion, but to give them a mandate. Reasons for this change (in brief) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * Fine, publishable papers are being rejected, which is bad for both authors and audience. Those papers are recycled at other conferences and workshop, where they increase the reviewing load (by being re-reviewed) and crowd out workshop-y papers. * Acceptance or rejection has a significant element of chance: it is very difficult for program committees to choose 35 out of 70 very good papers. Yet, partly because it is so competitive, acceptance at POPL has a strong effect on promotion and tenure committees. Having career-important decisions based on a chancy process seems wrong. * The pressure for slots makes it hard for a program committee to accept a ground-breaking but flawed paper over a more incremental but well-executed one. This is not a clear-cut issue, but many people (including me) think that the relentless pressure for slots forces program committees to err towards more conservative conference programmes. * It is a change that we can deliver. In contrast, arguing that journal publications should be valued more highly might be desirable, but is a cultural change that no one can guarantee to deliver. (However if POPL starts accepting more papers, a cultural change may well follow in due course.) Consequences of the change ~~~~~~~~~~~~~~~~~~~~~~~~~~ If this proposal were to be accepted, we would need to figure out how to accommodate many more papers at the physical meeting. How to achieve this is secondary to my main proposal, but a number of proposals have been floated, including * Parallel sessions * A lottery among accepted papers * Voting by conference registrants * Program committee decision I suggest that we invite the POPL steering committee to consider these and other possibilities, and make a proposal in due course. Background acceptance rates for POPL, PLDI, ICFP ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Source: http://people.engr.ncsu.edu/txie/seconferences.htm --------------------------------------------------- Year POPL PLDI ICFP --------------------------------------------------- 2008 35/212(16%) 34/184(18%) 29/87(33%) 2007 36/198(18%) 45/178(25%) 26/103(25%) 2006 33/167(20%) 36/169(21%) 24/74(32%) 2005 31/172(18%) 28/135(21%) 26/87(30%) 2004 29/176(16%) ?/?(20%) 21/80(26%) 2003 24/126(19%) 28/131(21%) 24/95(25%) 2002 28/128(22%) 28/169(17%) 24/76(32%) 2001 24/126(19%) 30/144(21%) 23/66(35%) 2000 30/151(20%) 30/173(17%) 24/110(22%) 1999 24/136(18%) 26/130(20%) 25/81(31%) 1998 31/175(18%) 31/136(23%) 30/70(39%) 1997 36/225(16%) 31/158(20%) 25/78(32%) 1996 34/148(23%) 28/112(25%) 25/83(30%) 1995 ? 28/105(27%) ? ---------------------------------------------------