[ The Types Forum (announcements only), http://lists.seas.upenn.edu/mailman/listinfo/types-announce ]
I am in complete agreement with Simon, and strongly opposed to the proposed plan to change the POPL reviewing process.
Bob Harper On Jan 11, 2010, at 5:46 PM, Simon Peyton-Jones wrote:
[ The Types Forum (announcements only), http://lists.seas.upenn.edu/mailman/listinfo/types-announce ] Colleagues | Request for comments: Two-phase reviewing for POPL ... | The POPL Steering Committee has formulated the following proposal, | which we are circulating for discussion and feedback from the| community. The proposal aims to improve the decision process for POPL | while still working in a fixed time frame and with bounded resources.Thank you for broadcasting the proposal, and offering the opportunity for feedback. I can't come to POPL this year, but I do have opinions about this proposal, so I thought I would put them in writing. I'm sending this response only to the TYPES mailing list. Many people are concerned about the publication norms that have developed in our field [1,2,3,4]. In particular, we have evolved a somewhat bizarre system in which we place tremendous weight on publication in premier conferences with extremely low acceptance rates. Promotion and tenure can depend on publication in these venues. Yet anyone who has served on a program committee knows that (a) the evaluation is fairly rough and ready, and (b) it is hard to avoid a tendency to pick well-executed but incremental papers over more adventurous but flawed work. The current proposal for POPL is presumably a direct response to this situation. But I believe its main thrust, to invest yet more effort inthe selection process, is addressing the wrong problem. The problem isnot that program committees are selecting the *wrong* papers. The problem is that they are selecting too *few* papers.Before developing these claims, I want to mention some real advantages ofthe current conference system. * It is quick -- and *predictably* quick. There is a delay of only a few months between submission and presentation; and there is never any slippage, because the conference itself is immoveable.* It is a *fantastic* deal for authors. The most precious commodity forany author is the focused attention of other experts in the field. When I began my academic career an author would be lucky to get three scrawled sentences of review, on physical scraps of paper. Nowadays authors get between three and six substantial, thoughtful reviews. That is gold dust. * Reviewing is recognised to be rough and ready. Everyone knows that there is no time to hunt for the perfect reviewer. The reviewers know they have limited time for their work, and cut their cloth accordingly. For that very reason they are more inclined to agree to write a review than if they are asked to review a 60-page journal paper when they are supposed to do a bang-up thorough job. Program committee members review 20-30 papers, and simply cannot spend days on each; and the universal acceptance of this fact is what makes people willing to serve on PCs I regard this limited time-budget for each review as a major advantage. 80% of the benefit of a review comes from the first 20% of investment. Yes, individual injustices are sometimes done, and all of us have been on the receiving end, but in the aggregate it is a very efficient evaluation mechanism. That is, it is not perfectly accurate, but it is a *very effective use of reviewing bandwidth*. * Much has been written about the evils of banging out papers to meet conference deadlines, and no one would defend salami-slicing incremental papers instead of working in a sustained way on adventurous research. Less has been written about the intellectual *advantages* of writing frequently. My own experience is that the act of writing a paper is tremendously enlightening. I learn that I do not understand what I though I understood. The act of putting ideas onto paper forces clarity, or at least exposes muddy thinking. It puts thoughts into a form when they can be shared with others. Since I am a weak mortal, the incentive of a conference deadline is often just what I need to force me to action.In short, there are really good things about our current system that wedo not want to lose. All that said, clearly something is wrong at the moment. POPL is getting 250 submissions, and accepting 30-40. That means that many fine papers are being rejected, and among the best 60 papers there is a strong element of chance about which ones end up being accepted. The same is true of PLDI, and perhaps to a lesser extent, of ICFP. (I don't have personal experience of the OOPSLA program committee.)We cannot fix this, as some would wish, by changing the culture to makejournal publications be regarded as more valuable than conference ones. If this happened, the spotlight would just shift to journals, which would be overwhelmed with submissions; and we would lose many of the advantages I outline above. But in any case it's a non-starter. No one can wave such a magic wand: cultures are *hard* to shift. Nor can we fix the problem by investing more effort in the review process, as the POPL committee is apparently suggesting. We are already investing quite enough! I'm all for careful reviewing. Double-blind reviewing (if done with a light touch, so that it does not cramp the authors style), and the opportunity for authors to rebut factual errors in reviews, both seem to have a good power-to-weight ratio. But adding a whole new round of reviewing would represent an enormous new investment on the part of both authors and reviewer, and to what end? Perhaps the published papers would be a little bit better, and the decisions would be a little bit more just. But the costs are heavy, the benefits are marginal, and it addresses none of the fundamental problems. I for one would think three times about agreeing to serve on such a PC. (I already think twice.) No, the trouble is that POPL and conferences like it simply rejects too many fine, publishable papers. This is bad because - Authors are badly served, obviously - Readers are badly served, because they don't get to read those papers - The papers get recycled at other conferences and workshops, where they increase reviewing load (by being reviewed a second time), and crowd out the truly workshop-y work in progress that should be showing up at workshops In short, we should just accept more papers at all our premier conferences, using a *quality* bar (is this paper good enough?) not a *quantity* bar (is it one of the best 30?). How can we do this? The "fat proceedings" problem is getting less and less important as we increasingly use digital media. Really the only difficulty is how to accommodate the presentations at the physical meeting itself. But this is a problem that could be dealt with in many ways. One straightforward one is to have parallel tracks. Another is to have more days. Still another, which I am rather fond of, is to accept (say) 60 papers, and then hold a lottery for 20 presentation slots. [That's fewer than usual, so there'd be longer breaks for mingling, which is actually the real reason most people go to conferences in the first place.] I would argue *against* choosing the "best" papers for presentation, because that will just re-introduce the ills we are currently struggling with. Make it clearly a matter of luck, then no one will read anything into the "chosen for presentation" badge. The lottery selection could even done at the conference itself. I'm only half joking; that way, no one could be denied travel funding on the grounds that his or her paper had not been chosen for presentation. Or perhaps participants registering for the conference could vote in advance for which accepted papers they'd like to see presented, so the programme is partly created by those attending the conference? A big advantage of this approach (simply accepting more papers) is that it is something we can simply choose to do. It does not require every conference to make the same choice simultaneously, and it doesn't require a magical cultural change. However, if we did take this path, then a significant cultural change would follow, over time. If publication at POPL was no longer an extraordinary achievement, but rather a recognition for a fine piece of work, appointment committees would in due course adjust their evaluation criteria. And that in turn might actually reduce the overwhelming number of submissions to top-drawer conferences. Simon Peyton Jones  J Wing, "CS woes: deadline-driven research, academic inequality", CACM 52(12) Dec 2009, p8  J Crowcroft, S Keshav and N McKeown, "Scaling the academic publication process to internet scale", CACM 52(1), Jan 2009, pp27-30.  M Vardi, "Conferences vs journals", CACM 52(5), May 2009, p5  K Bierman, FB Schneider "Program committee overload in systems", CACM 52(5), May 2009
Description: S/MIME cryptographic signature