[ The Types Forum (announcements only),
http://lists.seas.upenn.edu/mailman/listinfo/types-announce ]
Colleagues
| Request for comments: Two-phase reviewing for POPL
...
| The POPL Steering Committee has formulated the following proposal,
| which we are circulating for discussion and feedback from the
| community. The proposal aims to improve the decision process for POPL
| while still working in a fixed time frame and with bounded resources.
Thank you for broadcasting the proposal, and offering the opportunity
for feedback. I can't come to POPL this year, but I do have opinions
about this proposal, so I thought I would put them in writing. I'm
sending this response only to the TYPES mailing list.
Many people are concerned about the publication norms that have
developed in our field [1,2,3,4]. In particular, we have evolved a
somewhat bizarre system in which we place tremendous weight on
publication in premier conferences with extremely low acceptance
rates. Promotion and tenure can depend on publication in these
venues. Yet anyone who has served on a program committee knows that
(a) the evaluation is fairly rough and ready, and (b) it is hard to
avoid a tendency to pick well-executed but incremental papers over
more adventurous but flawed work.
The current proposal for POPL is presumably a direct response to this
situation. But I believe its main thrust, to invest yet more effort in
the selection process, is addressing the wrong problem. The problem is
not that program committees are selecting the *wrong* papers. The
problem is that they are selecting too *few* papers.
Before developing these claims, I want to mention some real advantages of
the current conference system.
* It is quick -- and *predictably* quick. There is a delay of only a
few months between submission and presentation; and there is never
any slippage, because the conference itself is immoveable.
* It is a *fantastic* deal for authors. The most precious commodity for
any author is the focused attention of other experts in the field.
When I began my academic career an author would be lucky to get
three scrawled sentences of review, on physical scraps of paper.
Nowadays authors get between three and six substantial, thoughtful
reviews. That is gold dust.
* Reviewing is recognised to be rough and ready. Everyone knows that
there is no time to hunt for the perfect reviewer. The reviewers
know they have limited time for their work, and cut their cloth
accordingly. For that very reason they are more inclined to agree
to write a review than if they are asked to review a 60-page journal
paper when they are supposed to do a bang-up thorough job. Program
committee members review 20-30 papers, and simply cannot spend days
on each; and the universal acceptance of this fact is what makes
people willing to serve on PCs
I regard this limited time-budget for each review as a major
advantage. 80% of the benefit of a review comes from the first 20%
of investment. Yes, individual injustices are sometimes done, and
all of us have been on the receiving end, but in the aggregate it is
a very efficient evaluation mechanism. That is, it is not
perfectly accurate, but it is a *very effective use of reviewing
bandwidth*.
* Much has been written about the evils of banging out papers to meet
conference deadlines, and no one would defend salami-slicing
incremental papers instead of working in a sustained way on
adventurous research.
Less has been written about the intellectual *advantages* of writing
frequently. My own experience is that the act of writing a paper is
tremendously enlightening. I learn that I do not understand what I
though I understood. The act of putting ideas onto paper forces
clarity, or at least exposes muddy thinking. It puts thoughts into
a form when they can be shared with others.
Since I am a weak mortal, the incentive of a conference deadline is
often just what I need to force me to action.
In short, there are really good things about our current system that we
do not want to lose.
All that said, clearly something is wrong at the moment. POPL is
getting 250 submissions, and accepting 30-40. That means that many
fine papers are being rejected, and among the best 60 papers there is
a strong element of chance about which ones end up being accepted.
The same is true of PLDI, and perhaps to a lesser extent, of ICFP.
(I don't have personal experience of the OOPSLA program committee.)
We cannot fix this, as some would wish, by changing the culture to make
journal publications be regarded as more valuable than conference
ones. If this happened, the spotlight would just shift to journals,
which would be overwhelmed with submissions; and we would lose many
of the advantages I outline above. But in any case it's a
non-starter. No one can wave such a magic wand: cultures are *hard* to
shift.
Nor can we fix the problem by investing more
