Re: [TYPES/announce] Proposed changes to POPL review process
[ The Types Forum (announcements only), http://lists.seas.upenn.edu/mailman/listinfo/types-announce ] On Sat, Jan 16, 2010 at 03:39:10PM +, Philip Wadler wrote: > My experience matches Rajeev's. The process of a physical PC meeting > necessarily requires quick decisions, often based on who is most vocal > or an overnight reading of a paper. My experience of electronic > meetings is that the discussions can be more considered. My experience is quite opposite. The physical PC meetings that I participated in were quite interesting and every paper was considered and discussed by the whole PC. During electronic meetings, on the other hand, I feel that I have no global understanding of what is going on, just a limited view of the papers that I have reviewed. The argument of cost (in time, money and carbon) is important, but otherwise I think a physical meeting wins hands down. Of course, it might be possible to use a combination of both modes (e.g. hold a preliminary electronic discussion before the physical meeting takes place; make sure every paper has received enough expert reviews, for instance). Best regards, -- François Pottier francois.pott...@inria.fr http://gallium.inria.fr/~fpottier/
Re: [TYPES/announce] Proposed changes to POPL review process
[ The Types Forum (announcements only), http://lists.seas.upenn.edu/mailman/listinfo/types-announce ] Hi, Phil. FWIW, the POPL'11 PC also discussed this issue in detail when deciding what kind of meeting we are going to hold this year. As you say, the issue split people down the middle. I thought it might be useful to summarize the main arguments that were given for physical vs. electronic meetings. (To clarify, by "electronic", I'm talking here about a week-long, online discussion through the conference management website, NOT a phone discussion, which most of the PC members pooh-poohed.) Main benefits of physical meeting: - Easier to get a "global" view of the submissions - PC members are more likely to contribute to the discussion of papers they didn't review - More effective use of PC members' "bandwidth", no delays in communication due to time zone differences - Excellent networking opportunity, esp. for more junior PC members - Much more fun, esp. for extroverts Main benefits of electronic meeting: - No travel costs, no jet lag, no carbon footprint - Longer, more careful and thorough discussion of papers (with the possibility of requesting additional expert reviews during the discussion) - Easier, more fun for non-native English speakers (as well as introverts) to participate In the end, the POPL'11 PC decided to have a 10-day electronic meeting followed by a 2-day physical meeting. Personally, I started the discussion strongly in favor of physical meetings, for all the reasons given above, but I found the arguments for electronic meetings to be compelling, and I sense that the future is electronic. Phil, concerning your comment that the advantages of electronic meetings become clearer in a context where more papers are accepted, I understand the argument but I don't think it's so clear-cut. You're right that there's less of a need for a "global" view in this case, because determining the "cut-off" point is less important. OTOH, getting a global view is still important for the purpose of normalizing the reviewing process. That is, one paper might get a B and 2 C's (and get rejected), and another might get an A and two B's (and get accepted), with the only difference being that the first paper had reviewers with low average scores and the second paper had reviewers with high average scores. To detect this problem, one cannot simply consider each paper independently, and so I think it is easier to detect in a physical meeting, given that more people pay attention to the discussion of each paper. But there may, of course, be purely electronic solutions. Best regards, Derek On Sat, Jan 16, 2010 at 4:39 PM, Philip Wadler wrote: > [ The Types Forum (announcements only), > http://lists.seas.upenn.edu/mailman/listinfo/types-announce ] > > Following on from Rajeev Alur's e-mail, I want to make the following proposal: > > - Replace the POPL physical PC meeting with an electronic one. > > This proposal was discussed by the POPL Steering Committee, which was > split on the matter. Input from the community would be helpful. > > My experience matches Rajeev's. The process of a physical PC meeting > necessarily requires quick decisions, often based on who is most vocal > or an overnight reading of a paper. My experience of electronic > meetings is that the discussions can be more considered. I believe > the advantages of electronic meetings become clearer in a context > where more papers are accepted, so the decisions centre on 'Is this > paper of good quality?' rather than 'Is it better than the other > papers we've accepted?' > > In addition to the cogent points raised by Rajeev, I'll mention that > there is also the cost, particularly to the climate, of a physical > meeting. Typically, POPL has about 200 attendees and a PC of 25, so a > physical PC meeting increases the carbon footprint of the conference > by on the order of 10%. > > How does the community feel about a move to electronic PC meetings? > Yours, -- P > > > > On Thu, Jan 14, 2010 at 3:32 PM, Rajeev Alur wrote: >> [ The Types Forum (announcements only), >> http://lists.seas.upenn.edu/mailman/listinfo/types-announce ] >> >> Dear Phil, >> It has been interesting to see all these responses to the two-phase >> proposal. I also think two-phase reviewing is not that great. >> One additional drawback I see is that getting rejection after the >> second phase will amplify the frustration (particularly for starting >> graduate students). >> I feel that low acceptance ratio of POPL is a desirable feature, it is >> critical >> to its reputation, and I would hate to see that go up significantly. >> As an aside, some people have essentially suggested that "since number >> of POPL papers impacts tenure, we should make it easy for researchers to >> publish in POPL". I do not find this argument compelling. In fact, I am >> not aware >> any committee counting this number. What matters is whether senior POPL >> researc
Re: [TYPES/announce] Proposed changes to POPL review process
[ The Types Forum (announcements only), http://lists.seas.upenn.edu/mailman/listinfo/types-announce ] Following on from Rajeev Alur's e-mail, I want to make the following proposal: - Replace the POPL physical PC meeting with an electronic one. This proposal was discussed by the POPL Steering Committee, which was split on the matter. Input from the community would be helpful. My experience matches Rajeev's. The process of a physical PC meeting necessarily requires quick decisions, often based on who is most vocal or an overnight reading of a paper. My experience of electronic meetings is that the discussions can be more considered. I believe the advantages of electronic meetings become clearer in a context where more papers are accepted, so the decisions centre on 'Is this paper of good quality?' rather than 'Is it better than the other papers we've accepted?' In addition to the cogent points raised by Rajeev, I'll mention that there is also the cost, particularly to the climate, of a physical meeting. Typically, POPL has about 200 attendees and a PC of 25, so a physical PC meeting increases the carbon footprint of the conference by on the order of 10%. How does the community feel about a move to electronic PC meetings? Yours, -- P On Thu, Jan 14, 2010 at 3:32 PM, Rajeev Alur wrote: > [ The Types Forum (announcements only), > http://lists.seas.upenn.edu/mailman/listinfo/types-announce ] > > Dear Phil, > It has been interesting to see all these responses to the two-phase > proposal. I also think two-phase reviewing is not that great. > One additional drawback I see is that getting rejection after the > second phase will amplify the frustration (particularly for starting > graduate students). > I feel that low acceptance ratio of POPL is a desirable feature, it is > critical > to its reputation, and I would hate to see that go up significantly. > As an aside, some people have essentially suggested that "since number > of POPL papers impacts tenure, we should make it easy for researchers to > publish in POPL". I do not find this argument compelling. In fact, I am > not aware > any committee counting this number. What matters is whether senior POPL > researchers are impressed by your work, and given that POPL is selective, > easiest way for someone to gain attention is by publishing > papers in POPL. Fortunately, selection is based on merit, so this presents > a clear recipe to draw attention to your work. If POPL is not selective, > then the only way would be to be a student of a famous advisor. > In fields such as control theory, top conferences such as CDC have > high acceptance rates, and indeed good pedigree is a necessity. > In any case, POPL review process should focus on selecting best papers > and maintaining high quality, without worrying about other factors. > > More constructively: > conferences such as LICS and CAV use electronic PC meeting. > I have been on POPL PC once and PLDI PC once, but I have a lot more > experience with LICS and CAV (also as PC Chair for both). > The problem with POPL (or LICS/CAV for that matter) initial reviews is > not the > quality (with some exceptions, most papers' contribution is very clear > from the reviews), > and also not the selective biases of individuals (which are a given, and > also, useful, > otherwise no evaluation would be possible), but rather that assignment > of letter grades > A/B/C/D in a distributed manner. For example, a clear technical advance > on a well-studied > problem may get a B or a C depending on the reviewer. This can make a huge > difference, and thus, the same paper rejected from one conference may > get accepted the > next time, making the process unpredictable. The goal of the PC meeting > is to correct > for this bias. But the physical meeting is not conducive to correcting > this. > For a given paper, the opinion of the person whose interests match most > closely with the paper, > counts more (but it should not: experts' reviews are useful, > not his/her biases on what to do with supposedly incremental, or supposedly > theoretical-that-will-never-work, or supposedly > practical-but-not-conceptually-deep papers). > Also, more vocal people get more influence. Time pressure impacts > decisions. > In practice, PC members are actively involved only in papers they have > been assigned, > maintaining the distributed nature of the process. > What one says on the spur of moment weighs more than what one writes > after careful > thought, editing, and sanity checks. > Thus, physical PC meeting adds unpredictable noise in the selection > process. > These are less of a problem in an electronic PC meeting. I think every PC > member needs to look at all the papers, and focus on selecting best X > submissions > based on reviews by applying his/her bias uniformly (and not just to > one's own pile). > This is easier to do on a longer time scale of electronic PC meeting. > Bottomline: not clear why POPL does not switch to e
[TYPES/announce] Proposed changes to POPL review process
[ The Types Forum (announcements only), http://lists.seas.upenn.edu/mailman/listinfo/types-announce ] Dear Phil, It has been interesting to see all these responses to the two-phase proposal. I also think two-phase reviewing is not that great. One additional drawback I see is that getting rejection after the second phase will amplify the frustration (particularly for starting graduate students). I feel that low acceptance ratio of POPL is a desirable feature, it is critical to its reputation, and I would hate to see that go up significantly. As an aside, some people have essentially suggested that "since number of POPL papers impacts tenure, we should make it easy for researchers to publish in POPL". I do not find this argument compelling. In fact, I am not aware any committee counting this number. What matters is whether senior POPL researchers are impressed by your work, and given that POPL is selective, easiest way for someone to gain attention is by publishing papers in POPL. Fortunately, selection is based on merit, so this presents a clear recipe to draw attention to your work. If POPL is not selective, then the only way would be to be a student of a famous advisor. In fields such as control theory, top conferences such as CDC have high acceptance rates, and indeed good pedigree is a necessity. In any case, POPL review process should focus on selecting best papers and maintaining high quality, without worrying about other factors. More constructively: conferences such as LICS and CAV use electronic PC meeting. I have been on POPL PC once and PLDI PC once, but I have a lot more experience with LICS and CAV (also as PC Chair for both). The problem with POPL (or LICS/CAV for that matter) initial reviews is not the quality (with some exceptions, most papers' contribution is very clear from the reviews), and also not the selective biases of individuals (which are a given, and also, useful, otherwise no evaluation would be possible), but rather that assignment of letter grades A/B/C/D in a distributed manner. For example, a clear technical advance on a well-studied problem may get a B or a C depending on the reviewer. This can make a huge difference, and thus, the same paper rejected from one conference may get accepted the next time, making the process unpredictable. The goal of the PC meeting is to correct for this bias. But the physical meeting is not conducive to correcting this. For a given paper, the opinion of the person whose interests match most closely with the paper, counts more (but it should not: experts' reviews are useful, not his/her biases on what to do with supposedly incremental, or supposedly theoretical-that-will-never-work, or supposedly practical-but-not-conceptually-deep papers). Also, more vocal people get more influence. Time pressure impacts decisions. In practice, PC members are actively involved only in papers they have been assigned, maintaining the distributed nature of the process. What one says on the spur of moment weighs more than what one writes after careful thought, editing, and sanity checks. Thus, physical PC meeting adds unpredictable noise in the selection process. These are less of a problem in an electronic PC meeting. I think every PC member needs to look at all the papers, and focus on selecting best X submissions based on reviews by applying his/her bias uniformly (and not just to one's own pile). This is easier to do on a longer time scale of electronic PC meeting. Bottomline: not clear why POPL does not switch to electronic PC meeting. More dramatically: I mentioned this to Jens after this year's POPL meeting: abolish the PC (i.e. reduce its role to a "reviewers committee" of an exapnded size). Two or three co-chairs can collect reviews for each paper from those who are real experts on the subject. Then based on the reviews, make a decision applying fair and uniform standards. This is not as bad as it sounds. Jens indeed spent a lot of time browsing through all submissions anyway, and could have easily picked the papers after looking at the reviews. Maybe a single person's bias would be detrimental, but, say 3, would make the process better than it is now (and reduce the cumulative amount of time one would spend on POPL PC duties). best regards --rajeev
