I concur with Brian's concerns - I would strongly suggest that the WG chair(s) take on two responsibilities for this experiment with Gen-ART reviews at WGLC:
(1) Commit to monitoring/mediating the WG mailing list interaction between the WG members and the Gen-ART reviewer. Letting the WG membership loose "no holds barred" on a Gen-ART reviewer who is not an expert in the protocol being reviewed may not work out well, e.g., it could generate list reactions along the lines of "if the Gen-ART reviewer doesn't understand this basic concept, why should we pay serious attention to anything else in the review?" I've recently had a behind the scenes interaction on a technical issue from a Gen-ART review where the involvement of the responsible WG chair made a significant difference. (2) Determine whether the draft is mature enough for a Gen-ART review at WGLC. At a high level, the question to be asked is roughly "if WG LC resulted in no changes to this draft, would you be comfortable submitting it to your AD with a publication request?" If the answer is "no", a Gen-ART review is probably premature. A potentially useful mechanism could be to ask the WG chair to prepare a publication request writeup (Barry's short form should suffice) at the start of WGLC in order to request a Gen-ART review. If the WG chair believes that a writeup is premature, then the Gen-ART review is probably also premature. For example, if a major purpose of a WGLC is to drive open technical issues to resolution, Gen-ART review is probably premature, and a writeup would note the open technical issues that need resolution. Thanks, --David > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of > Brian E Carpenter > Sent: Thursday, May 23, 2013 10:27 PM > To: Joel M. Halpern > Cc: General Area Review Team; The IESG > Subject: Re: [Gen-art] timing of reviews > > On 24/05/2013 11:43, Joel M. Halpern wrote: > > In principle I am happy perform reviews in conjunction with WG last > > call, even though that would probably produce a small increase in loading. > > I think time will tell whether it's small. Getting into the thick of > WG discussion in an area one doesn't know well can be quite a shock > to the system. That's not to say we shouldn't try it, but I would ask > for it to be a time-limited experiment initially. > > It seems important to inform WGs explicitly that outside review > is taking place *and* has to be taken seriously (because if not, > the same reviewer will be back at a later stage with the same > questions). > > Then there is a question: will Gen-ART re-review at the time of > IETF Last Call as well as when the draft reaches the IESG agenda? > > BTW, the concept of early review by Gen-ART is not new, but I think > it was only tried for one or two drafts in the past. > > Brian > > > > > There is however a complication. Many WG email lists are "ember only", > > for good reason. I do not wish to end up on all those lists. Which > > means that the moderator has to let the message through, and then > > probably remember to whitelist the reviewer. Ad folk have to keep > > copying the reviewer explicitly with responses. I think this can be > > done, but we should remember such procedural glitches. > > > > Yours, > > Joel > > > > On 5/23/2013 6:28 PM, Jari Arkko wrote: > >> Thanks again for all your hard work in doing the reviews. They make > >> it possible for me to concentrate on reviewing just those documents > >> where there are problems or I have deeper expertise. And the quality > >> of the protocol specifications coming out of the IETF is obviously > >> very important, particularly for protocols that are gaining > >> significant use. > >> > >> As you may have seen, the IESG and the community has been wondering > >> if there'd be something that we could do about the IETF process, in > >> the sense that there's quite many things happening at the very end of > >> the document's life cycle. This results in some surprises, and often > >> also moves some important decisions out of the working group and to > >> author/shepherd/AD hands. A while ago we met for the IESG retreat and > >> wanted to experiment with three specific changes: > >> > >> - sending work back to the WGs when significant changes are needed, > >> and making the WG the central place of the edits - moving some > >> directorate reviews earlier, without impact reviewer effort too much > >> - inviting some of the shepherds onto tele chats > >> > >> I am writing to you in order to discuss the second item. How big of a > >> change would it be to have Gen-ART reviews be invoked during WGLC, > >> while the documents are still in the working groups? The goal of the > >> reviews would still be the same, e.g., I would be checking that the > >> reviews were positive and/or that the issues brought up have been > >> properly addressed. > >> > >> There are important details to consider, however, and I would like to > >> get your feedback on how you would seem them having an effect, and > >> what would be the best way to organise this, if we decide to go ahead > >> with the change for the Gen-ART. > >> > >> Triggering the review would have to be done by something else than > >> IETF last call announcement. Is the best approach is to have the WG > >> chairs manually request for this? Note that sometimes there are > >> multiple WGLCs. I presume that it would be preferable to have a > >> Gen-ART review be done only once at this stage, as otherwise the work > >> load would increase. The chairs may have some idea of whether they > >> are likely to need another WGLC before they start one. > >> > >> There may be possibly bigger changes and time lag between the first > >> Gen-ART review and the one that checks that the changes are ok. > >> > >> Some specs may not make it through WGLC, and a review at that stage > >> may increase the effort you guys are putting in, by reviewing those > >> specs that will fail. > >> > >> Jari > >> > >> _______________________________________________ Gen-art mailing list > >> [email protected] https://www.ietf.org/mailman/listinfo/gen-art > >> > > _______________________________________________ > > Gen-art mailing list > > [email protected] > > https://www.ietf.org/mailman/listinfo/gen-art > > > _______________________________________________ > Gen-art mailing list > [email protected] > https://www.ietf.org/mailman/listinfo/gen-art _______________________________________________ Gen-art mailing list [email protected] https://www.ietf.org/mailman/listinfo/gen-art
