I am sharing Mary's concerns - also some of the ones expressed by other. The principal problem I see with moving the Gen-ART review in parallel with the WGLC rather than IETF LC is related to the level of maturity of documents when they reach WGLC. This is of course variable according to the area and WG, but it is clear that most of the documents are in a less stable status when they reach WGLC than when they reach IETFLC. As a result the Gen-ART reviewers will see their load increase not only as a result of having to review a number of documents that will never exit the WG (I do not know how many these are, can we see statistics?), but also because they will need to follow the full WGLC cycle which may include several revisions, extend over multiple Last Calls, and lead to considerable changes in the documents resulting from other WGLC comments, changes which also need to be reviewed.
I do believe that there is however a solution to this problem, but it does not come for free either. Let the WGLC take place as it happens right now, but move the Expert Reviews (a.k.a. Directorate Reviews) also within the responsibility of the WG, after the WGLC completes. This will make the WG cycle longer by a few weeks, but will allow for the Directorate comments to be processed by the WG, but on a stable version, which the WG considers as having passed the WGLC. Would this work? We may want to hear the wg-chairs opinion, I am concerned about to intensive cross-posting. Regards, Dan > -----Original Message----- > From: [email protected] [mailto:[email protected]] On > Behalf Of Mary Barnes > Sent: Friday, May 24, 2013 4:15 PM > To: Jari Arkko > Cc: General Area Review Team; The IESG > Subject: Re: [Gen-art] timing of reviews > > I will caveat my comments with the fact that I am no longer taking on > new documents for review. However, having been involved in the process > for 10 years now on the gen-art side and having done the reviews for 3 > IETF chairs, and having much experience as a chair and document editor, > I do have some concerns as laid out below. > > Mary. > > On Fri, May 24, 2013 at 1:11 AM, Jari Arkko <[email protected]> > wrote: > > > >> I think time will tell whether it's small. Getting into the thick of > >> WG discussion in an area one doesn't know well can be quite a shock > >> to the system. That's not to say we shouldn't try it, but I would ask > >> for it to be a time-limited experiment initially. > > > > Yes - or scope limited (only some WGs or areas at the beginning). > [MB] Absolutely, it needs to have a limited scope initially. As Brian > said, the concept is not new, it's just that few took advantage of it. > I would prefer that this stage remain optimal and up to the WG chairs > as to whether it's necessary. Without the tool support that is > available for the regular LC and Telechat reviews, I don't think we want > to burden Jean too much until we have a very good idea of the level of > effort and until there is tool support (e.g., chairs enter the docs into > a tool that gets the doc into Jean's incoming queue for assignment). I > also believe this increases the burden on the reviewers significantly > more than the burden when we added the LC reviews. For folks that > haven't been around a while, the original Gen-ART process only did > reviews prior to the telechats. Adding reviews at LC did increase the > workload for reviewers, however, we did see a somewhat > increase in documents ready at Telechat time. [/MB] > > > >> It seems important to inform WGs explicitly that outside review is > >> taking place *and* has to be taken seriously (because if not, the > >> same reviewer will be back at a later stage with the same > >> questions).' > > > > Good point. > [MB] Exactly. [/MB] > > [MB] My biggest concern is that many documents that go through WGLC in > the WGs in which I am active are NOT ready at that stage. I also have > seen in cases of Gen-ART reviews, that documents clearly weren't > reviewed by the shepherd or ADs due to the level of editorial issues and > technical (the p2psip reload document is a good example of this). > There were concerns raised in the WG with regards to its readiness but > it was still progressed. Certainly, I hope this is the exception. > However, my guess is that many documents will change quite a bit after > LC. When I chaired the SIPPING WG, we had multiple stages of review in > the WG where we had assigned reviewers. We had an initial review when > the doc was first accepted as a WG document, then we had a pre-WGLC > review and then the real WGLC. Many of the documents went through a 2nd > last call. If we could get more WGs to be more disciplined about > ensuring that documents receive adequate review prior to WGLC, then this > idea might work. At this point, I see very few WGs that have an > effective process in place. One also has to consider that in some WGs > (many that I participate in and write docs for), there are very often > folks that don't review any version of a document in detail until it > goes through WGLC. So, I am personally quite skeptical that this > proposal adds significant value to the overall process, in particular > based on the level of effort required by both the gen-art reviewers. I > think this is just moving an existing problem around and asking the > wrong set of folks to help solve the problem. I think a more effective > change would be for ADs to expect WGs to ensure the docs are thoroughly > reviewed earlier in the stage by the WG. > > [/MB] > > > >> Then there is a question: will Gen-ART re-review at the time of IETF > >> Last Call as well as when the draft reaches the IESG agenda? > > > > My initial though was that we'd do the same as we do today for drafts > that change, i.e., re-review by the same person. The likelihood for > draft changes is of course bigger when starting from the WGLC. Another > possible work load increase to watch for. I'd be interested in your > thoughts on other approaches to deal with re-reviews though. > > > > Jari > > > > _______________________________________________ > > Gen-art mailing list > > [email protected] > > https://www.ietf.org/mailman/listinfo/gen-art > _______________________________________________ > Gen-art mailing list > [email protected] > https://www.ietf.org/mailman/listinfo/gen-art _______________________________________________ Gen-art mailing list [email protected] https://www.ietf.org/mailman/listinfo/gen-art
