The comments from David, Brian, Mary and Joel are broadly in line with my thinking on this. One additional point: during the discussion that Jari's original question provoked I suggested that a written summary of the design decisions that had led to the chosen protocol design could be helpful for the 'tail' discussion process. Although it would make some extra work for authors and WG chairs, having this information would make the gen-art (and other) reviewing easier and make interpreting what is going on during WGLC somewhat easier.
One way of doing this would be to make the summary an appendix to the proto write up. Doing this might also serve to bring out any remaining issues that need to be sorted out during WGLC - and thereby provide evidence of the draft maturity as highlighted by David. And, perhaps, lessen the chance (or at least the impact) of attempts to reopen decided issues. All this being said, I would certainly be prepared to give it a go on a trial basis if our secretary can cope with he extra work until the tools could be extended. Regards, Elwyn On Fri, 2013-05-24 at 12:00 -0400, Black, David wrote: > I concur with Brian's concerns - I would strongly suggest that the WG chair(s) > take on two responsibilities for this experiment with Gen-ART reviews at WGLC: > > (1) Commit to monitoring/mediating the WG mailing list interaction between the > WG members and the Gen-ART reviewer. Letting the WG membership loose "no > holds > barred" on a Gen-ART reviewer who is not an expert in the protocol being > reviewed > may not work out well, e.g., it could generate list reactions along the lines > of > "if the Gen-ART reviewer doesn't understand this basic concept, why should we > pay > serious attention to anything else in the review?" I've recently had a behind > the scenes interaction on a technical issue from a Gen-ART review where the > involvement of the responsible WG chair made a significant difference. > > (2) Determine whether the draft is mature enough for a Gen-ART review at WGLC. > At a high level, the question to be asked is roughly "if WG LC resulted in no > changes to this draft, would you be comfortable submitting it to your AD with > a publication request?" If the answer is "no", a Gen-ART review is probably > premature. A potentially useful mechanism could be to ask the WG chair to > prepare a publication request writeup (Barry's short form should suffice) at > the start of WGLC in order to request a Gen-ART review. If the WG chair > believes > that a writeup is premature, then the Gen-ART review is probably also > premature. > For example, if a major purpose of a WGLC is to drive open technical issues > to resolution, Gen-ART review is probably premature, and a writeup would note > the open technical issues that need resolution. > > Thanks, > --David > > > -----Original Message----- > > From: [email protected] [mailto:[email protected]] On Behalf > > Of > > Brian E Carpenter > > Sent: Thursday, May 23, 2013 10:27 PM > > To: Joel M. Halpern > > Cc: General Area Review Team; The IESG > > Subject: Re: [Gen-art] timing of reviews > > > > On 24/05/2013 11:43, Joel M. Halpern wrote: > > > In principle I am happy perform reviews in conjunction with WG last > > > call, even though that would probably produce a small increase in loading. > > > > I think time will tell whether it's small. Getting into the thick of > > WG discussion in an area one doesn't know well can be quite a shock > > to the system. That's not to say we shouldn't try it, but I would ask > > for it to be a time-limited experiment initially. > > > > It seems important to inform WGs explicitly that outside review > > is taking place *and* has to be taken seriously (because if not, > > the same reviewer will be back at a later stage with the same > > questions). > > > > Then there is a question: will Gen-ART re-review at the time of > > IETF Last Call as well as when the draft reaches the IESG agenda? > > > > BTW, the concept of early review by Gen-ART is not new, but I think > > it was only tried for one or two drafts in the past. > > > > Brian > > > > > > > > There is however a complication. Many WG email lists are "ember only", > > > for good reason. I do not wish to end up on all those lists. Which > > > means that the moderator has to let the message through, and then > > > probably remember to whitelist the reviewer. Ad folk have to keep > > > copying the reviewer explicitly with responses. I think this can be > > > done, but we should remember such procedural glitches. > > > > > > Yours, > > > Joel > > > > > > On 5/23/2013 6:28 PM, Jari Arkko wrote: > > >> Thanks again for all your hard work in doing the reviews. They make > > >> it possible for me to concentrate on reviewing just those documents > > >> where there are problems or I have deeper expertise. And the quality > > >> of the protocol specifications coming out of the IETF is obviously > > >> very important, particularly for protocols that are gaining > > >> significant use. > > >> > > >> As you may have seen, the IESG and the community has been wondering > > >> if there'd be something that we could do about the IETF process, in > > >> the sense that there's quite many things happening at the very end of > > >> the document's life cycle. This results in some surprises, and often > > >> also moves some important decisions out of the working group and to > > >> author/shepherd/AD hands. A while ago we met for the IESG retreat and > > >> wanted to experiment with three specific changes: > > >> > > >> - sending work back to the WGs when significant changes are needed, > > >> and making the WG the central place of the edits - moving some > > >> directorate reviews earlier, without impact reviewer effort too much > > >> - inviting some of the shepherds onto tele chats > > >> > > >> I am writing to you in order to discuss the second item. How big of a > > >> change would it be to have Gen-ART reviews be invoked during WGLC, > > >> while the documents are still in the working groups? The goal of the > > >> reviews would still be the same, e.g., I would be checking that the > > >> reviews were positive and/or that the issues brought up have been > > >> properly addressed. > > >> > > >> There are important details to consider, however, and I would like to > > >> get your feedback on how you would seem them having an effect, and > > >> what would be the best way to organise this, if we decide to go ahead > > >> with the change for the Gen-ART. > > >> > > >> Triggering the review would have to be done by something else than > > >> IETF last call announcement. Is the best approach is to have the WG > > >> chairs manually request for this? Note that sometimes there are > > >> multiple WGLCs. I presume that it would be preferable to have a > > >> Gen-ART review be done only once at this stage, as otherwise the work > > >> load would increase. The chairs may have some idea of whether they > > >> are likely to need another WGLC before they start one. > > >> > > >> There may be possibly bigger changes and time lag between the first > > >> Gen-ART review and the one that checks that the changes are ok. > > >> > > >> Some specs may not make it through WGLC, and a review at that stage > > >> may increase the effort you guys are putting in, by reviewing those > > >> specs that will fail. > > >> > > >> Jari > > >> > > >> _______________________________________________ Gen-art mailing list > > >> [email protected] https://www.ietf.org/mailman/listinfo/gen-art > > >> > > > _______________________________________________ > > > Gen-art mailing list > > > [email protected] > > > https://www.ietf.org/mailman/listinfo/gen-art > > > > > _______________________________________________ > > Gen-art mailing list > > [email protected] > > https://www.ietf.org/mailman/listinfo/gen-art > > _______________________________________________ > Gen-art mailing list > [email protected] > https://www.ietf.org/mailman/listinfo/gen-art _______________________________________________ Gen-art mailing list [email protected] https://www.ietf.org/mailman/listinfo/gen-art
