Re: FW: Last Call: draft-farrell-ft-03.txt (A Fast-Track way to RFCwith Running Code) to Experimental RFC
On 01/28/2013 04:27 AM, Joe Touch wrote: About the idea of an experiment: Right. The context being its an RFC 3933 IETF process experiment. On 1/25/2013 5:07 AM, Stephen Farrell wrote: Responses to some points below but I'd really like to ask people to consider a few things here: - what's proposed is an experiment, it'd likely get tried out a few times and won't consume any huge resource anywhere If this is an experiment, then you presumably answers to the following questions: 1- what is your an hypothesis? 2- what you intend to measure? 3- what is your 'control' against which to compare the results? 4- what is your objective metric for success/failure? Well, its not a scientific experiment, and nor does it need to be. Quoting 3933: -A statement of the problem expected to be resolved is desirable but not required (the intent is to keep the firm requirements for such an experiment as lightweight as possible). Similarly, specific experimental or evaluative criteria, although highly desirable, are not required -- for some of the process changes we anticipate, having the IESG reach a conclusion at the end of the sunset period that the community generally believes things to be better (or worse) will be both adequate and sufficient. My take is that even though 3933 says desirable a couple of times there, it's probably best to not go there, at least in this case, but for now probably more generally. The reason I think that is that I reckon the IETF has got itself into a log-jam that almost prevents us from modifying our processes in any way and every additional word provides additional barriers to doing anything, since there'll always be someone for whom those added words raise what seems to them to be a red flag. I do agree that's far from ideal, but so is our current process-change log-jam. (BTW, I already said this a few months ago in response to a similar comment on I think -00 or -01.) Separately, I don't believe this proposed experiment does have any objective criteria on which we could get consensus and that would be practical to measure - I think the outcome is more likely to be subjective, (e.g. seemed to help because of X) and that the follow-up, if any, would require further consideration - my guess is that if the experiment were some form of success then we'd need to start over on documenting the thing(s) that might be worth including in an update to BCP9, but then that'd be based on some experience with running the experiment rather than on your or my speculation as to what might or might not work well or badly. I've heard only one hypothesis - that this reduces time to publication. I disagree that this is a useful hypothesis to test for the following reasons: - time to publication isn't a goal of the IETF IMO, any doc that isn't useful in 5 years ought to not be published here; we don't need to document every sneeze IMO reduced time to publication is definitely *a* goal. I've heard lots of IETF participants complain about how long things take, lots of times. Perhaps you're in the rough on that. (I also note that this experiment doesn't actually aim for any major reduction in time to publication as it says in the draft.) The draft itself also discusses reasons why running code might also lead to better quality specifications. - thorough review ought to be a requirement and this 'experiment' potentially compromises that by reducing the overall time of review I think the liklihood of that doing damage during the experiment is very small. In addition, I might be wrong, but the thoroughness of review doesn't correlate that well with the duration of review would be my guess. While secdir reviews are pretty variable, they are almost all done in a tight timeframe and a good number of them are very good. Same for gen-art and apps-dir reviews. So we do have evidence that good review can be done in a short timeframe. Equally, I've seen quite a few drafts that have reached the IESG after years that still have quite a lot that needs to be fixed and that a thorough review ought have caught. - community resources ought to be considered and this 'experiment' burns group resources due to having a broad group concurrently review a doc that could have been reviewed by smaller groups first Yes, this experiment should bring a few drafts to IETF review earlier. The WG chairs are still supposed to only send a publication request when they're satisfied the draft represents the rough consensus of the WG. However, I'm looking at changing some of the wording on this in response to John's point about having two parallel discussions. I'll post to the list when I've had a chance to write that up in the working version. Given the limited cycles this community has to review docs, I cannot see a benefit to this
Re: FW: Last Call: draft-farrell-ft-03.txt (A Fast-Track way to RFCwith Running Code) to Experimental RFC
On 1/28/2013 3:12 AM, Stephen Farrell wrote: On 01/28/2013 04:27 AM, Joe Touch wrote: ... If this is an experiment, then you presumably answers to the following questions: 1- what is your an hypothesis? 2- what you intend to measure? 3- what is your 'control' against which to compare the results? 4- what is your objective metric for success/failure? Well, its not a scientific experiment, and nor does it need to be. Quoting 3933: -A statement of the problem expected to be resolved is desirable but not required (the intent is to keep the firm requirements for such an experiment as lightweight as possible). Similarly, specific experimental or evaluative criteria, although highly desirable, are not required -- for some of the process changes we anticipate, having the IESG reach a conclusion at the end of the sunset period that the community generally believes things to be better (or worse) will be both adequate and sufficient. My take is that even though 3933 says desirable a couple of times there, it's probably best to not go there, at least in this case, but for now probably more generally. The reason I think that is that I reckon the IETF has got itself into a log-jam that almost prevents us from modifying our processes in any way and every additional word provides additional barriers to doing anything, since there'll always be someone for whom those added words raise what seems to them to be a red flag. Lightweight != vacuous. Perhaps the lack of the discussion of these issues is part of the reason so many previous proposals have not been tried. That isn't a reason to ignore those issues here. But a bigger concern is your reasoning as presented in this response: I've heard only one hypothesis - that this reduces time to publication. I disagree that this is a useful hypothesis to test for the following reasons: - time to publication isn't a goal of the IETF IMO, any doc that isn't useful in 5 years ought to not be published here; we don't need to document every sneeze IMO reduced time to publication is definitely *a* goal. I've heard lots of IETF participants complain about how long things take, lots of times. Perhaps you're in the rough on that. (I also note that this experiment doesn't actually aim for any major reduction in time to publication as it says in the draft.) There are plenty of places where existing process is in a logjam. My experience over the past 15 years is most of the delay happens during the following: - changes in the IETF process where ideas like this throw a wrench into the process, and create confusion I had a few informational docs wait over a year in the queue because a process change was put into effect before the necessary boilerplate was resolved. -- this has a *simple* solution. Grandfather everything that has been submitted to a given queue to changes in that queue's process. - IESG review, during which issues are often raised that were addressed during WG review that are then re-hashed, even though they are not relevant to the area of their appointment Overlapping community review has no impact on either of these. The draft itself also discusses reasons why running code might also lead to better quality specifications. No disagreement there, but better quality doesn't mean the doc wouldn't still need - and substantively benefit from - serial review in increasingly larger communities. - thorough review ought to be a requirement and this 'experiment' potentially compromises that by reducing the overall time of review I think the likelihood of that doing damage during the experiment is very small. Damage = - publishing ideas not sufficiently vetted that later need to be updated/errata'd - wasting the community's time by having a large group review an issue that could have been addressed and corrected within a smaller community Do you seriously think these concerns should outweigh a few weeks of reduced time to publication? In addition, I might be wrong, but the thoroughness of review doesn't correlate that well with the duration of review would be my guess. I've heard many on this list - including myself - point out specific reasons why this should not be believed. If still believe every process can be accelerated, I encourage you to review Brooks the Mythical Man Month. Having this entire community burn cycles on this document speaks for itself. It should have been vetted in a smaller, more invested community first. I'm following the 3933 process. I don't know of any smaller but open venue for discussing rfc 3933 experiments. Perhaps there ought be such, but given how rarely 3933 experiments are
Re: FW: Last Call: draft-farrell-ft-03.txt (A Fast-Track way to RFCwith Running Code) to Experimental RFC
About the idea of an experiment: On 1/25/2013 5:07 AM, Stephen Farrell wrote: Responses to some points below but I'd really like to ask people to consider a few things here: - what's proposed is an experiment, it'd likely get tried out a few times and won't consume any huge resource anywhere If this is an experiment, then you presumably answers to the following questions: 1- what is your an hypothesis? 2- what you intend to measure? 3- what is your 'control' against which to compare the results? 4- what is your objective metric for success/failure? I've heard only one hypothesis - that this reduces time to publication. I disagree that this is a useful hypothesis to test for the following reasons: - time to publication isn't a goal of the IETF IMO, any doc that isn't useful in 5 years ought to not be published here; we don't need to document every sneeze - thorough review ought to be a requirement and this 'experiment' potentially compromises that by reducing the overall time of review - community resources ought to be considered and this 'experiment' burns group resources due to having a broad group concurrently review a doc that could have been reviewed by smaller groups first Given the limited cycles this community has to review docs, I cannot see a benefit to this experiment that is worth the cost. Having this entire community burn cycles on this document speaks for itself. It should have been vetted in a smaller, more invested community first. Calling something an 'experiment' doesn't make it worthwhile to test. Joe
Re: FW: Last Call: draft-farrell-ft-03.txt (A Fast-Track way to RFCwith Running Code) to Experimental RFC
- Original Message - From: Thomas Narten nar...@us.ibm.com To: Joe Touch to...@isi.edu Cc: adr...@olddog.co.uk; ietf@ietf.org Sent: Tuesday, January 22, 2013 9:31 PM FWIW, I share Joe's concerns. And Stephen's responses don't really change my mind. This document seems to have a bit of missing the forest for the trees. In the overall scheme of things, I don't believe the draft will materially help, and is at best a distraction from dealing with meaningful problems. I agree. This proposal asks the ADs and WG Chairs to do more work and I perceive lack of resouces in that area to be the single biggest reason for I-Ds to fail to progress as fast as they might. Putting more load on the critical path can only make things worse, for the work as a whole. Some favoured I-Ds will progress faster but overall, RFCs will take longer to appear. In this vein, I have commented before how progress in the IETF tends to take place in three bursts, in the course of each year, and that if only we had a way of bringing the three pauses to life, then progress would be much improved. Quality would almost certainly also improve, since data would not get lost or forgotten in the pauses. This month, most WG mailing lists seem to be in a mega-pause (I keep checking the archives to see if I have been accidentally unsubscribed!): that is the real problem. Tom Petch The crux of the issue is that any attempt at fast tracking is fundamentally about short-circuiting some aspect of our review processes. We should only do that very carefully. In almost all cases, individual judgement is needed as to when it is appropriate to do that. ADs/WG chairs already have some flexibility here. e.g., a WG can skip WG LC if they think its not needed. And nothing stops an AD from going to WGLC before doing a careful review. But I'd rather that be done because the case at hand justifies such an optimization, not because there is some running code checkmark that allows a special case to be fast tracked.. This document risks substituting process for judgement. I'd rather see more of the latter and less of the former. Some specifics from the doc: In summary this experiment collapses three stages of the publication process to run concurrently: working group last call, AD review, and IETF last call. Additionally, the experiment will only require IMO, it is almost never appropriate to collapse WG LC and IETF LC. In cases where it is appropriate to do so, there presumably isn't a need for a WG LC to start with. And an AD has always been able to start an IETF LC without doing a detailed review. The point, is judgement is needed, and all such cases are best handled on a case-by-case basis. Surely, we don't need a document to justify doing this in those cases where it makes sense... IETF last call. Additionally, the experiment will only require issues raised during these three stages to be addressed if they meet the IESG's Discuss criteria. The former is not formally a process and later: 4. Only comments that would be DISCUSS-worthy according to the IESG Discuss Criteria [DCRIT] need be handled during last call. Other comments can be handled or not, at the authors/editors discretion. The above can easily become a way to dismiss legitimate review comments. No doubt, when a AD or reviewer suggests this needs fixing, the proponents will hold up this document and say you shouldn't do this, per the RFC -- you're violating the spirit of this document, only really really critical stuff needs to get addressed... No thanks. That is not what the IETF is about. If it is really urgent to get a document done, it is far better to take steps to make sure the editors are engaged and responsive. That is more likely the real issue!!! It is understood that the savings in time for the end-to-end delivery of a proposed standard or experimental RFC may be modest, however, the modifications to normal IETF process will also serve as an indication of the importance that the IETF places in running code. Running code is valuable. Always has been, always will be. But we need to resist the temptation of making running code more equal than other criteria or putting it on a pedestal as some sort of holy grail. Running code by itself is just a sound bite. The importance of running code is what it tells us about a protocol specification. Just the mere fact that there is running code doesn't mean there is anything particularly enlightening to learn from an implementation. For example, running code of a router function in software doesn't necessarily say anything as to whether the code can be implemented efficiently using ASICs, which may be the real requirement. 5. The responsible AD for the WG concerned makes the decision as to whether changes are required as a result of review comments, and determines
Re: FW: Last Call: draft-farrell-ft-03.txt (A Fast-Track way to RFCwith Running Code) to Experimental RFC
Folks, I'm very much on the same page as Tom, the normal problem we have is not too much review, it is that we don't have enough. Running code is valuable, but does not normally replace review. /Loa On 2013-01-25 11:02, t.p. wrote: - Original Message - From: Thomas Narten nar...@us.ibm.com To: Joe Touch to...@isi.edu Cc: adr...@olddog.co.uk; ietf@ietf.org Sent: Tuesday, January 22, 2013 9:31 PM FWIW, I share Joe's concerns. And Stephen's responses don't really change my mind. This document seems to have a bit of missing the forest for the trees. In the overall scheme of things, I don't believe the draft will materially help, and is at best a distraction from dealing with meaningful problems. I agree. This proposal asks the ADs and WG Chairs to do more work and I perceive lack of resouces in that area to be the single biggest reason for I-Ds to fail to progress as fast as they might. Putting more load on the critical path can only make things worse, for the work as a whole. Some favoured I-Ds will progress faster but overall, RFCs will take longer to appear. In this vein, I have commented before how progress in the IETF tends to take place in three bursts, in the course of each year, and that if only we had a way of bringing the three pauses to life, then progress would be much improved. Quality would almost certainly also improve, since data would not get lost or forgotten in the pauses. This month, most WG mailing lists seem to be in a mega-pause (I keep checking the archives to see if I have been accidentally unsubscribed!): that is the real problem. Tom Petch The crux of the issue is that any attempt at fast tracking is fundamentally about short-circuiting some aspect of our review processes. We should only do that very carefully. In almost all cases, individual judgement is needed as to when it is appropriate to do that. ADs/WG chairs already have some flexibility here. e.g., a WG can skip WG LC if they think its not needed. And nothing stops an AD from going to WGLC before doing a careful review. But I'd rather that be done because the case at hand justifies such an optimization, not because there is some running code checkmark that allows a special case to be fast tracked.. This document risks substituting process for judgement. I'd rather see more of the latter and less of the former. Some specifics from the doc: In summary this experiment collapses three stages of the publication process to run concurrently: working group last call, AD review, and IETF last call. Additionally, the experiment will only require IMO, it is almost never appropriate to collapse WG LC and IETF LC. In cases where it is appropriate to do so, there presumably isn't a need for a WG LC to start with. And an AD has always been able to start an IETF LC without doing a detailed review. The point, is judgement is needed, and all such cases are best handled on a case-by-case basis. Surely, we don't need a document to justify doing this in those cases where it makes sense... IETF last call. Additionally, the experiment will only require issues raised during these three stages to be addressed if they meet the IESG's Discuss criteria. The former is not formally a process and later: 4. Only comments that would be DISCUSS-worthy according to the IESG Discuss Criteria [DCRIT] need be handled during last call. Other comments can be handled or not, at the authors/editors discretion. The above can easily become a way to dismiss legitimate review comments. No doubt, when a AD or reviewer suggests this needs fixing, the proponents will hold up this document and say you shouldn't do this, per the RFC -- you're violating the spirit of this document, only really really critical stuff needs to get addressed... No thanks. That is not what the IETF is about. If it is really urgent to get a document done, it is far better to take steps to make sure the editors are engaged and responsive. That is more likely the real issue!!! It is understood that the savings in time for the end-to-end delivery of a proposed standard or experimental RFC may be modest, however, the modifications to normal IETF process will also serve as an indication of the importance that the IETF places in running code. Running code is valuable. Always has been, always will be. But we need to resist the temptation of making running code more equal than other criteria or putting it on a pedestal as some sort of holy grail. Running code by itself is just a sound bite. The importance of running code is what it tells us about a protocol specification. Just the mere fact that there is running code doesn't mean there is anything particularly enlightening to learn from an implementation. For example, running code of a router function in software doesn't necessarily say anything as to whether the code can be implemented efficiently using ASICs, which
Re: FW: Last Call: draft-farrell-ft-03.txt (A Fast-Track way to RFCwith Running Code) to Experimental RFC
Responses to some points below but I'd really like to ask people to consider a few things here: - what's proposed is an experiment, it'd likely get tried out a few times and won't consume any huge resource anywhere - its optional, WG chairs that want to try it could, those that don't can just ignore the whole thing - argument that there are other or better process issues to address is misdirected - on a meta-level, if every process proposal, even small experiments like this, generates huge concerns then we're not in a good place and it seems to me that we are in that bad place (that's not an argument to do this experiment, but just an observation) On 01/25/2013 10:54 AM, Loa Andersson wrote: Running code is valuable, but does not normally replace review. This proposal doesn't replace review. Its an optional way to get some of that done in parallel with a few other rule tweaks. On 01/25/2013 10:30 AM, Brian E Carpenter wrote: Speaking as a Gen-ART reviewer, I am indeed worried by this aspect. I feel I would have to spend much longer reviewing a draft if I knew it had not been through WGLC. Its an experiment that would be used for a few drafts, your concern might be justified if this were proposed as a permanent change (it's not) that applied to many drafts (it won't ever IMO, but certainly not during the experiment). On 01/25/2013 10:02 AM, t.p. wrote: This proposal asks the ADs and WG Chairs to do more work It does not. WG chairs *choose* to take this route if they want to. The timers and checking the implementation could increase the workload for the responsible AD at a time not of her choosing and its the timing that's more pertinent but not so bad since ADs are already clearly gullible enough when it comes to additional workload:-) On 01/24/2013 06:34 PM, John C Klensin wrote: As co-author of the process experiment model, I also object to its use when it is not demonstrably necessary, That's a really weird objection. I just re-read 3933 and it says nothing whatsoever like that, in fact it says 'This specification permits and encourages the IESG to adopt and institute process experiments...' and the whole tenor of the RFC is that stuff ought be tried out. John also said: this document moves a number of informal IESG procedures -- including the DISCUSS criteria and even the Shepherd report requirements (now Informational in RFC 4858 plus a number of less formal statements and templates that have sometimes changed rapidly) -- into the status of formal procedural requirements That's just wrong. It includes those as part of an experiment that is optional to use which I don't think qualifies as formal process requirements in any useful sense. And lastly... In my experience in the IETF, we almost never allow a individual submission document that is obviously very controversial to turn into a extended debate on the IETF list with the author trying to refute every comment. Trying to refute every comment is (IMO wildly) inaccurate. This was proposed a couple of months ago. Some people liked the idea of trying it out, others didn't. I'm following up on that as described in 3933. Every proposal to do with process seems this controversial unfortunately, even teeny little ones like this. I'd love if it were otherwise. And if Adrian or the IESG decide we should not go ahead and try this particular experiment, that's just fine and the world will not end. (And nor will it end if this experiment is run.) What's less fine, but I suspect is also the case, is that any proposal for process change would get the same kinds of negative reception from a similarly sized set of folks and with similarly good, bad and irrelevant arguments against whatever change or experiment is proposed. Regards, S.
