Hi Joe, Rather than more guidelines, I would prefer to see (from IEEE and others) a stronger commitment to data-driven understanding of the review process and support for (semi-) controlled experiments on improving it.
I'd bet that EDAS knows things about the review process that we don't know ourselves... Serious work probably requires better consensus about data handling and sharing than we have now, especially if we want to correlate over multiple conferences. But I think it would be helpful to begin this effort and also to raise expectations on Chairs for scientific approach to the problem. For specific comments on the guidelines: > 2. involvement in CFP promotion E/A/D It's not clear that this a significant indicator of quality. (At least, I'm not lacking CFP spam.) > 3. paper assignment for review E/A/D What is best practice?: Even with double blinding, giving reviewers too much role in selection can be vulnerable to collusion or (more likely) just inbreeding among circles of people who have similar ideas about what's interesting. Is there data? Do papers tend to have correlated sets of reviewers and does it affect variability in review scores? If correlation exists, does it persist across conferences? Can assignment algorithms be designed to mitigate this? > 5. TPC meeting E/A/D Disagree. TPC meetings are expensive, environmentally unfriendly, and reduce TPC diversity. Phone-only meetings are better, but are still unavoidably at 2am in either America, Europe or Asia. > 6. paper review process E/A/D > > E = considers average rank AND outlier info, discussion points > also based on natural 'gap' in evaluation > A = considers average rank based on natural gap in evaluation > D = considers rank only I'm somewhat confused about the idea of 'natural gap'. Conventional wisdom is that most conferences have a few clear accepts, many clear rejects, and a certain amount of randomness (along with careful evaluation, of course) in the middle. N) I was quite surprised that rules for delegating reviews weren't considered relevant. I was slightly surprised that review load wasn't considered important. N) I was disappointed that reviewer diversity (gender/national /institutional/year-on-year turnover) wasn't considered relevant. Regards, Laura _______________________________________________ IEEE Communications Society Tech. Committee on Computer Communications (TCCC) - for discussions on computer networking and communication. Tccc@lists.cs.columbia.edu https://lists.cs.columbia.edu/cucslists/listinfo/tccc
