On 11/3/2011 7:00 AM, Pars Mutaf wrote:
...
> The reviews should be publicly available to everyone.
There have been attempts to explore this and other models, e.g., in no
particular order:
A- author rebuttal of reviews
B- blind reviews
C- double-blind process
where the paper authors are hidden during review
D- public reviews
where reviews are published with the paper
E- open reviews
where the author sees the reviewer's names
F- adding a venue for papers on the 'borderline' of the
main conference
Speaking as someone who has participated as a PC member in these in
various places (as an individual, not as TCCC Chair):
A was tried at Infocom (and elsewhere). The goal was to avoid a paper
being discarded because of an incorrect review. The result was a
substantial increase in review time (actually, it ended up resulting in
less time for reviewers to complete their reviews due to a fixed yearly
cycle), but no substantial change in paper handling. Most of the
rebuttals did not point out review errors, but rather disagreed with
review opinion.
B is currently typical.
C is used at Sigcomm and more recently at ICNP. It is intended to avoid
favoritism, but IMO it also tends to work against systems work that has
been vetted in workshops and symposia in parts.
D has been tried for some CCR papers, where a single review or summary
of the reviews is presented.
E was tried at Global Internet a number of years ago, and nearly killed
the meeting. Submissions went down over 50%. The result was much more
pleasantly-written reviews, but the reviews were (IMO) less useful.
F was introduced at Infocom several years ago. IMO, it simply introduced
a second borderline, and made it very difficult to distinguish between
full accepts and "consolation prize" accepts.
All of the above were introduced to address a perceived or real concern.
None of them was tested in a true experiment (e.g., with a control group
during the same year). Most of them (IMO) were introduced because chairs
believe that mechanism can address review process problems. IMO, there
is only one good solution for all such problems:
PC chairs MUST review the reviews. EVERY review. EVERY year.
Reviews whose ranks are not substantiated by
meaningful comment must be both discarded and
replaced.
Overall, IMO, it is useful to understand that:
- reviewing is an imperfect process
- a paper's quality is determined by what the reader
receives (goodput), not what is sent (offered load) ;-)
- papers are rejected because of the lack of positive comments,
not for any single negative comment
(so arguing each negative comment in a review
won't fix a paper - many reviewers simply provide
sufficient negatives to justify a decision, but
could provide other negatives if asked)
- at large conferences, papers are rejected after substantial
decision
e.g., at Infocom, a paper is either a unanimous reject
by three reviewers, OR is then considered by at least
an additional 8-10 people during the PC meeting
I see none of these changing in an open process.
Joe
_______________________________________________
IEEE Communications Society Tech. Committee on Computer Communications
(TCCC) - for discussions on computer networking and communication.
[email protected]
https://lists.cs.columbia.edu/cucslists/listinfo/tccc
