Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
Hi. The recent discussion about DISCUSS and DS/IS documents has inspired me to go back and think about the two maturity levels draft again. Sadly, it hasn't changed my mind but has, in some respects, reinforced and strengthened my earlier view that this is not a good idea and is not harmless. To avoid repeating myself and making this note too long, I won't repeat myself but would like much of the content of my notes at https://www.ietf.org/ibin/c5i?mid=6rid=49gid=0k1=933k2=59000tid=131473 https://www.ietf.org/ibin/c5i?mid=6rid=49gid=0k1=933k2=59008tid=131473 and https://www.ietf.org/ibin/c5i?mid=6rid=49gid=0k1=933k2=59018tid=131473 to be considered as part of this response. However, in particular and independent of any of the other issues, there is just no evidence that the two maturity level change will make any difference at all. The problems (and possible solutions) lie elsewhere and, if one examines the evolution of the various drafts of that proposal, it is clear that the community, and probably the authors, understand that it will make no significant difference other than to spare the IESG one extra review in a case that rarely occurs (more rarely in some areas than others). So the question becomes whether this change, which might help (that is as close as anyone can get -- we are still without a compelling argument) is, at worst, harmless. I don't think it is, for at least two reasons: (1) We have ample historical evidence that making a change like this will block consideration of more substantive changes, changes that might actually improve things. People will argue against consideration of such changes on basis of community exhaustion (and will probably be right). Others will argue that we need to see if this experiment works rather than pushing ahead with other changes that might create confusion about where any effects came from or they will speculate that other changes might mess this one up. They might be right too -- certainly if the IETF sets a two levels direction, it would be unwise to immediately propose a one level and lots of annotation and explanation model. (2) Having procedures that we don't use exactly, or that the community doesn't follow, merely means that we haven't updated those procedures (for whatever reason). We live with it and so do ITU-T, ISO, IEEE, and so on with procedures they have altered but not yet captured in formal documentation. But to change procedures from something we don't follow to something that won't work is silly, should be embarrassing, and exposes us to increased vunerability to attacks from others (including other SDOs who want to take over IETF's place in the world). It is worth the risk to change procedures and risk new ones not working if we have good reasons to believe that they will work, or that they will bring significant benefits, or both. But a change to something for which there is no evidence that it will work, and some evidence to believe that it solves the wrong problem, should be a non-starter. If we want to fix the standards track, then let's start by seeing if we have consensus about what is broken (I think we probably do and that the discussion of this draft has had the immensely useful side-effect of producing useful discussion on that topic). Let's try some bold experiments in turning things around, e.g., having the IESG move toward actually following the intent of RFC 2026 wrt acceptance of documents at Proposed (discussed in the recently-expired draft-bradner-restore-proposed), encouraging some areas -- ideally those with the lowest rates of advancement beyond PS-- experiment with different considerations for advancement to DS and IS (discussed in the Discuss criteria ... thread and especially in the third mail message cited above). Or let's ask the question of whether broad-category maturity levels actually serve our needs, and those of the community, well in our present environment of increasingly complex and option-laden standards (from the standpoint of maturity level categories, the only really good standard is one that contains no requirements other than MUST and MUST NOT -- no SHOULD or MAY variations) and move to one maturity level supplemented by more Applicability Statements or commentary and annotation external to the protocol specifications themselves. regards, john ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
--On Thursday, September 01, 2011 08:10 -0700 Dave CROCKER d...@dcrocker.net wrote: ... Folks should remember that this is a system that has been functioning quite well for some decades and I am not aware of any recent emergencies that justify starting over or making major changes. The policy when seeking to change an established, essential, well-running systems is to make as /few/ changes as possible, not as /many/... Ideally, this means making no changes at all, of course. That is, any proposal for a change MUST explain why the change is /essential/. I agree completely. Please apply this logic to draft-housley-two-maturity-levels, of which you are a co-author. john ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
I pretty much agree, although one form of discuss might be reasonable: This document needs to be recycled at Proposed Standard because of the following *observed* interoperability problem: In other words, once we have got this BCP out (soon, please), the IESG should update the DISCUSS criteria specifically to narrow them for the PS-IS transition. I also pretty much agree, and the IESG discuss criteria document (http://www.ietf.org/iesg/statement/discuss-criteria.html) would certainly benefit from a respin to provide guidelines for -bis documents and documents advancing further. However, as with most things I don't think there are hard and fast rules. I can imagine a very old RFC being respin or advancing where I'd want some rework. For instance, a vulnerability that was discovered after the orginal RFC should be described, perhaps even dealt with somehow. Jari ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
On Aug 14, 2011, at 5:49 AM, jari.ar...@piuha.net wrote: I pretty much agree, although one form of discuss might be reasonable: This document needs to be recycled at Proposed Standard because of the following *observed* interoperability problem: In other words, once we have got this BCP out (soon, please), the IESG should update the DISCUSS criteria specifically to narrow them for the PS-IS transition. I also pretty much agree, and the IESG discuss criteria document (http://www.ietf.org/iesg/statement/discuss-criteria.html) would certainly benefit from a respin to provide guidelines for -bis documents and documents advancing further. However, as with most things I don't think there are hard and fast rules. I can imagine a very old RFC being respin or advancing where I'd want some rework. For instance, a vulnerability that was discovered after the orginal RFC should be described, perhaps even dealt with somehow. I agree that such things need to be described, but I don't think this description should be gated on, or wait for, advancement in grade. The errata mechanism can be used to report some kinds of vulnerabilities; separate RFCs for others. Right now we have this expectation that we're going to rewrite the entire RFC just to declare something as DS. That is a lot of work, opens up a huge can of worms, imposes a large barrier to the interop testing that we want to encourage, and sometimes introduces confusion by subtly changing the original specification. Old standards do need maintenance. Our process doesn't really account for that. It has this fiction that once a protocol gets to Full Standard, it's good forever - or that it needs a complete and very tedious rewrite and recycle from Proposed. And the current process doesn't (in practice) document the shifting applicability of standards over time. I'd like to see some sort of changes to the process that recognize the need to periodically review and occasionally maintain old standards, update their applicability without changing the text of the document, and that allows for incremental updating (with change bars etc.) without changing the basic form of the text. Of course the changes do need to be reviewed, and IESG is already over-worked. Maybe we need review groups that are analogous to working groups, and a separate supervisory board for reviews. The review groups and board would be limited as to what they could do with a standard - e.g. clarify text where ambiguity had been identified, document new vulnerabilities (whether security, scaling, whatever), note limitations in applicability. They'd have no power to add new features, or maybe only when the feature was narrowly tailored to fix a documented problem with the older spec. This might even lighten IESG's load a bit because a few existing WGs could be changed to review groups. And hopefully the review supervisory board's load would be lighter than IESG's so it wouldn't be so hard to recruit suitable candidates for it. Keith ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
Keith, However, as with most things I don't think there are hard and fast rules. I agree that such things need to be described, but I don't think this description should be gated on, or wait for, advancement in grade. The errata mechanism can be used to report some kinds of vulnerabilities; separate RFCs for others. Right now we have this expectation that we're going to rewrite the entire RFC just to declare something as DS. I do not think we currently have that expectation. I do realize that we've debated several cases and the need or lack thereof for additional changes. But in general, a document that advanced is not expected to be rewritten. At least not in my opinion. When I review -bis or -ds documents I usually focus on the rfcdiff output between the old RFC and the new draft. I also do acknowledge that we (the working groups and the IESG) have made several errors in this respect, sometimes doing more than should really have been done. But I do not think we have the expectation that all -ds documents should be rewritten. That would be wrong. What I tried to say above is that I dislike hard rules such as: - We should never require a -ds document to say additional things - We should always apply the most recent IETF approved rules (such as BCP 109 on key management) to all documents, including the -ds ones But I do agree with you that whether a separate new RFC, editing this RFC, errata or some other form of update should be decided on a case-by-case basis. Jari ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
On Aug 14, 2011, at 9:24 AM, jari.ar...@piuha.net wrote: What I tried to say above is that I dislike hard rules such as: - We should never require a -ds document to say additional things - We should always apply the most recent IETF approved rules (such as BCP 109 on key management) to all documents, including the -ds ones I'm not sure that I agree with that. I think that when we have an old standard that's widely used, it's important to document newly discovered limitations. But sometimes we live with the deficiencies, and sometimes we do what we can to fix the protocol without breaking compatibility. We shouldn't necessarily insist that everything meet current criteria. More generally, I don't think that advancing standards to DS should invite special scrutiny along those lines. I think we should regularly apply such scrutiny to all IETF standards, regardless of their status.If the standard no longer meets our current criteria for standardization, we should say so. But that doesn't necessarily mean we should reclassify it as Historic or otherwise take away its status. If people are still using the protocol, we can be of more service by maintaining the specification than by abandoning it. We should enumerate the known problems, identify the known fixes or workarounds, identify opportunities for new fixes that haven't been developed yet. But I do agree with you that whether a separate new RFC, editing this RFC, errata or some other form of update should be decided on a case-by-case basis. I agree with that as you've stated it. But I think there should be a presumption that writing a new RFC is done only in extreme cases, and that when this is done, the document has to recycle at Proposed. And I'm okay in principle with editing existing RFCs and reissuing them with a new number, but I wish the new ones would have change bars. (and maybe, when rendered in HTML, other indications of specifically which text is changed). Keith ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
jari.ar...@piuha.net wrote: Keith, However, as with most things I don't think there are hard and fast rules. I agree that such things need to be described, but I don't think this description should be gated on, or wait for, advancement in grade. The errata mechanism can be used to report some kinds of vulnerabilities; separate RFCs for others. Right now we have this expectation that we're going to rewrite the entire RFC just to declare something as DS. I do not think we currently have that expectation. I do realize that we've debated several cases and the need or lack thereof for additional changes. But in general, a document that advanced is not expected to be rewritten. At least not in my opinion. When I review -bis or -ds documents I usually focus on the rfcdiff output between the old RFC and the new draft. I also do acknowledge that we (the working groups and the IESG) have made several errors in this respect, sometimes doing more than should really have been done. But I do not think we have the expectation that all -ds documents should be rewritten. That would be wrong. It is good to read this is recognized, especially for people who wish to remain on the sideline as online participants and depend on engineering common sense an faith among their peers and the IETF/IESG leaders and really don't have the time to be getting involved deeply in the IETF process. I don't think anything is cut and dry, but I will note an increasing concern that in the name of fast tracking, politics and who you know, there have been a perception of rubber stamping BIS work. Overall, not necessarily a bad thing, but serious care should be taken when existing base of protocol users are now asked to change again. When done at the expense of ignoring input as watered down concerns using rough consensus to exclude what is otherwise a less than 50%, but nonetheless significant weight of people, then the real losers is the IETF process and the IETF future. [I personally believe, the Rough Consensus model needs a serious review on how its applied today.] I personally can recognize the need to move documents faster, but if only to serve as a reminder, it should not come at the expense of the long held guidelines the IETF/IESG has followed to make sure errors are not made. If anything, the two-maturity level proposal should increase the engineering expertise and scrutiny requirements for reviewers. -- Hector Santos, CTO http://www.santronics.com http://santronics.blogspot.com ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
I really have to wonder if the entire yes/no-obj/discuss voting model is appropriate for document advancement. For initial approval at proposed, sure, having the ability to discuss the document makes all sorts of sense. But for subsequent steps that virtue is a lot obvious, to me at least. This, IMHO, is the right question: Does yes/no-obj/discuss resolve the right issues when advancing from PS to DS? IMO, IESG members ought to be able to vote no if, after first voting discuss and not having the issue resolved within a well-defined amount of time, they still believe that the document should not progress. That goes for any standards level. (That's not to say that a single no vote should suffice to block progress of a document.) Discuss is not the problem. Discuss is actually a really good idea. The problem is the lack of a no vote, which causes people to interpret discuss as if it were no. We are a bit off topic here, at least for this subject line. The IESG did make some changes to the voting procedures a couple of years ago. The change was to make it clear that a single DISCUSS position could not block a document. That is, the IESG believes in rough consensus too. The current rules are available here: http://www.ietf.org/iesg/voting-procedures.html Now, please return to the Last Call discussion of draft-housley-two-maturity-levels on this subject line. If we want to discuss the IESG ballot process further, please start a new thread. Russ ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
To add one observation to SM's comment and other observations that the scarcity of implementation reports implies that they are somehow difficult... --On Friday, August 05, 2011 02:45 -0700 SM s...@resistor.net wrote: I presume that the IESG will only use the following criteria for advancement: - two independent interoperating implementations with widespread deployment and successful operational experience - no errata against the specification - no unused features in the specification And there won't be any DISCUSSes along the lines of: I don't think the implementation reports are adequate for me to meet the requirements of 2026. It does not clearly identify what software was used or show support of each of the individual options and features. Examples througout the document make use of non-example domains. The implementation report is woefully inadequate to document there are interoperable implementations of all the features from two different code bases. My Discuss was not addressed at all - I believe that the WG ignored the spirit of the implementation report requirement - my Discuss said that we should know that there are multiple implementations that have handled the significant changes in the recycling of this Draft Standard. The group apparently refused to update its implementation report An alternate hypothesis about the low numbers of implementation reports on file is that people try them, get responses like the ones SM lists above, and react with why bother -- this is a waste of time that I can better spend in other ways. And that is the end, not just of the report being commented on, but any others that could have come from that author or participant. That pretty much nails it. We've allowed exactly these sorts of vague and nonspecific discusses to stand, with the result that the pain of getting to draft or full standard is seen as far greater than the gain. The really surprising thing, frankly, is that we have as many draft and full standards as we do. While it has not been raised as a specific argument for this two-maturity-levels proposal, getting rid of formal implementation reports might, for that reason, be useful in getting documents advanced. However, it does not improve the case for two levels instead of three and won't help if IESG members express the same thoughts --not in terms of interoperability reports but in terms of lack of conviction the interoperability had actually been demonstrated in the case of that particular spec. Actually it does in a way. Part of the problem with going to draft is that in addition to it being painful and difficult, when it's over you're still not done. And you have to wait for the final step, and when you get there it's not all that well defined, either. Really, this is Psych 101 stuff - it's been shown over and over that immediate gratification wins out over long term satisfaction. Our three step process sucks most if not all of the immediate gratification out of moving to draft. And this is why I strongly support a move to a two step process. I think the other changes that have been proposed are also important but I'll take what I can get. I note that, when RFC 4846 and what became 5742 were under discussion, a then-IESG member pointed out to me that, despite the restrictions in those documents, if something in a draft really offended him, he could always find a way to state his objections in a way that would conform to the 4846/5742 rules. The same comment presumably applies to presumed proofs of independent interoperable implementations. I really have to wonder if the entire yes/no-obj/discuss voting model is appropriate for document advancement. For initial approval at proposed, sure, having the ability to discuss the document makes all sorts of sense. But for subsequent steps that virtue is a lot obvious, to me at least. Ned ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
On 2011-08-14 06:29, ned+i...@mauve.mrochek.com wrote: ... I really have to wonder if the entire yes/no-obj/discuss voting model is appropriate for document advancement. For initial approval at proposed, sure, having the ability to discuss the document makes all sorts of sense. But for subsequent steps that virtue is a lot obvious, to me at least. s/lot/lot less/ I pretty much agree, although one form of discuss might be reasonable: This document needs to be recycled at Proposed Standard because of the following *observed* interoperability problem: In other words, once we have got this BCP out (soon, please), the IESG should update the DISCUSS criteria specifically to narrow them for the PS-IS transition. Brian ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
ned+i...@mauve.mrochek.com ned+i...@mauve.mrochek.com wrote: I really have to wonder if the entire yes/no-obj/discuss voting model is appropriate for document advancement. For initial approval at proposed, sure, having the ability to discuss the document makes all sorts of sense. But for subsequent steps that virtue is a lot obvious, to me at least. This, IMHO, is the right question: Does yes/no-obj/discuss resolve the right issues when advancing from PS to DS? There is a perception, at least, that any AD can block progress to DS (though in fact the process is more complicated, that's the way it's perceived). I could somewhat-seriously suggest that the IESG shouldn't have to vote at all on advancement to DS -- the right issues are questions of fact which could perfectly well be delegated. (And wearing my Narrative-Scribe hat, I know how confusing it is for IESG members to figure out what's expected of them when the question is advancement of maturity level.) -- John Leslie j...@jlc.net ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
On 2011-08-14 06:29, ned+i...@mauve.mrochek.com wrote: ... I really have to wonder if the entire yes/no-obj/discuss voting model is appropriate for document advancement. For initial approval at proposed, sure, having the ability to discuss the document makes all sorts of sense. But for subsequent steps that virtue is a lot obvious, to me at least. s/lot/lot less/ Yes, sorry. I pretty much agree, although one form of discuss might be reasonable: This document needs to be recycled at Proposed Standard because of the following *observed* interoperability problem: In other words, once we have got this BCP out (soon, please), the IESG should update the DISCUSS criteria specifically to narrow them for the PS-IS transition. +1 Ned ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
On Aug 13, 2011, at 4:43 PM, John Leslie wrote: ned+i...@mauve.mrochek.com ned+i...@mauve.mrochek.com wrote: I really have to wonder if the entire yes/no-obj/discuss voting model is appropriate for document advancement. For initial approval at proposed, sure, having the ability to discuss the document makes all sorts of sense. But for subsequent steps that virtue is a lot obvious, to me at least. This, IMHO, is the right question: Does yes/no-obj/discuss resolve the right issues when advancing from PS to DS? IMO, IESG members ought to be able to vote no if, after first voting discuss and not having the issue resolved within a well-defined amount of time, they still believe that the document should not progress. That goes for any standards level. (That's not to say that a single no vote should suffice to block progress of a document.) Discuss is not the problem. Discuss is actually a really good idea. The problem is the lack of a no vote, which causes people to interpret discuss as if it were no. If an IESG member sees something about a document that is suspect, it's entirely appropriate for him or her to raise questions about it. That's their job. And if the explanation is satisfactory, or the problem is fixed, it's then appropriate for the vote to be changed to no objection or yes. But if the problem isn't fixed, and the problem is substantial, the IESG member ought to be able to - IMO, has an obligation to - vote no. Anything else is dishonest and irresponsible. The history of engineering is full of serious accidents that have been caused by technical reviews being ignored due to political pressure. Anybody can make a mistake, including IESG members. It's okay to make an occasional bad review. It's not okay to lie about that review. Now there's a quite valid question about what kinds of objections IESG should be able to raise at PS vs DS or whatever. And it's arguable that the criteria for PS and DS (or IS) should be different. In particular, a revision of a document for DS is not an invitation to change the protocol, or even to fix deficiencies in the original design of the protocol. I don't even think that advancement to DS should be considered an invitation to rewrite the document. That's not to say that standards-track RFCs should never be revised, but that revising them should be done very conservatively (so as to make as few changes as possible to the text and especially organization of the document) and that this should be treated separately from advancement in grade. Keith ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
+1 d/ -- Dave Crocker bbiw.net via mobile -Original Message- I pretty much agree, although one form of discuss might be reasonable: This document needs to be recycled at Proposed Standard because of the following *observed* interoperability problem: In other words, once we have got this BCP out (soon, please), the IESG should update the DISCUSS criteria specifically to narrow them for the PS-IS transit +1 ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
On Wed, Jul 27, 2011 at 07:02:07PM -0700, The IESG wrote: The IESG has received a request from an individual submitter to consider the following document: - 'Reducing the Standards Track to Two Maturity Levels' draft-housley-two-maturity-levels-08.txt as a BCP I have reviewed this version of the document and also the changes since -07. While i am not sure that there is really a problem to solve, the current version appears reasonable and an improvement over -07. However, it would benefit from a few clarifications before publication. Also, even if it will become part of BCP 9, i consider it a bit undesirable to distribute the core IETF standards process across more than one document. For the short term, that should be OK, though. 1. Introduction This document proposes several changes to the Internet Standards Process defined in RFC 2026 [1]. In recent years, the Internet Engineering Task Force (IETF) has witnessed difficulty in advancing documents through the maturity levels: Proposed Standard, Draft Standard, and finally Standard. These changes are designed to simplify the Standards Process and reduce impediments to standards progression while preserving the most important benefits of the IETF engineering approach. In addition, the requirement for annual review of standards-track documents that have not reached the top of the maturity ladder is removed from the Internet Standards Process. this informal language is a nice read, but consistency with RFC 2026 would suggest to say standards track instead of maturity ladder. 2. Two Maturity Levels This document, once approved, replaces the three-tier maturity ladder defined in RFC 2026 [1] with a two-tier maturity ladder. Specifications become Internet Standards through a set of two maturity levels known as the standards track. These maturity levels are Proposed Standard and Internet Standard. A specification may be, and indeed, is likely to be, revised as it advances from Proposed Standard to Internet Standard. When a revised specification is proposed for advancement to Internet Standard, the IESG shall determine the scope and significance of the changes to the specification, and, if necessary and appropriate, modify the recommended action. Minor revisions and the removal of unused features are expected, but a significant revision may require that the specification accumulate more experience at Proposed Standard before progressing. Suggest to add a reference to section 6.1.1 of RFC 2026 to clarify that recommended action is actually the WG's request for advancement. 2.2. The Second Maturity Level: Internet Standard This maturity level is a merger of Draft Standard and Standard as specified in RFC 2026 [1]. The chosen name avoids confusion between Draft Standard and Internet-Draft. The characterization of an Internet Standard remains as described in RFC 2026 [1], which says: An Internet Standard is characterized by a high degree of technical maturity and by a generally held belief that the specified protocol or service provides significant benefit to the Internet community. The criteria for advancing from Proposed Standard to Internet Standard are confirmed by the IESG in an IETF-wide Last Call of at least four weeks. The request for reclassification is sent to the IESG along with an explanation of how the criteria have been met. The criteria are: (1) There are at least two independent interoperating implementations with widespread deployment and successful operational experience. (2) There are no errata against the specification that would cause a new implementation to fail to interoperate with deployed ones. (3) There are no unused features in the specification that greatly increase implementation complexity. Operational complexity should be of equal concern. (4) If patented or otherwise controlled technology is required for implementation, the implementations demonstrate at least two separate and successful uses of the licensing process. After review and consideration of significant errata, the IESG will perform an IETF-wide Last Call of at least four weeks on the requested reclassification. Is this another four weeks or the same as mentioned above? In case of the latter the first occurence could be removed to maintain chronological ordering. requested reclassification. If there is consensus for reclassification, the RFC will be reclassified without publication of a new RFC. So, how would a modified specification (say, with unused features not passing criterion 3) be advanced? As stated in RFC 2026 [1], in a timely fashion after the expiration of the Last Call period, the IESG shall make its final determination and notify the IETF of its decision via
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
To add one observation to SM's comment and other observations that the scarcity of implementation reports implies that they are somehow difficult... --On Friday, August 05, 2011 02:45 -0700 SM s...@resistor.net wrote: I presume that the IESG will only use the following criteria for advancement: - two independent interoperating implementations with widespread deployment and successful operational experience - no errata against the specification - no unused features in the specification And there won't be any DISCUSSes along the lines of: I don't think the implementation reports are adequate for me to meet the requirements of 2026. It does not clearly identify what software was used or show support of each of the individual options and features. Examples througout the document make use of non-example domains. The implementation report is woefully inadequate to document there are interoperable implementations of all the features from two different code bases. My Discuss was not addressed at all - I believe that the WG ignored the spirit of the implementation report requirement - my Discuss said that we should know that there are multiple implementations that have handled the significant changes in the recycling of this Draft Standard. The group apparently refused to update its implementation report An alternate hypothesis about the low numbers of implementation reports on file is that people try them, get responses like the ones SM lists above, and react with why bother -- this is a waste of time that I can better spend in other ways. And that is the end, not just of the report being commented on, but any others that could have come from that author or participant. While it has not been raised as a specific argument for this two-maturity-levels proposal, getting rid of formal implementation reports might, for that reason, be useful in getting documents advanced. However, it does not improve the case for two levels instead of three and won't help if IESG members express the same thoughts --not in terms of interoperability reports but in terms of lack of conviction the interoperability had actually been demonstrated in the case of that particular spec. I note that, when RFC 4846 and what became 5742 were under discussion, a then-IESG member pointed out to me that, despite the restrictions in those documents, if something in a draft really offended him, he could always find a way to state his objections in a way that would conform to the 4846/5742 rules. The same comment presumably applies to presumed proofs of independent interoperable implementations. john ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
RE: Last Call: draft-housley-two-maturity-levels-08.txt(Reducing the Standards Track to Two Maturity Levels) to BCP
Hi Medel, At 17:57 07-08-2011, GT RAMIREZ, Medel G. wrote: 1)Is there such thing as a good enough Criteria to handle this concern? Glen Zorn asked a question during the last plenary and there was a discussion [1] about criteria on this mailing list. To be fair, I'll quote a message from Warren Kumari [2]: While not all ADs read all drafts, most read a large fraction of them (and read them carefully and thoughtfully enough to catch a number of large issues (and nits) *that were not caught in LC*) -- I think that they deserve recognition for performing a valuable and un-fun function... 2)Or as usual it passes rough consensus process? The process is based on consensus and not rough consensus. Regards, -sm P.S. http://www.ietf.org/mail-archive/web/ietf/current/msg67494.html 1. http://www.ietf.org/mail-archive/web/ietf/current/msg68008.html 2. http://www.ietf.org/mail-archive/web/ietf/current/msg68008.html ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
Hi Hector, At 18:22 07-08-2011, Hector Santos wrote: Of course, when implementation reports are written, one has to watchful for the summarized analytical results that either attempt to add weight to an desired goal or mask the undesired goal and natural result. An implementation report is written to further a desired goal, i.e. advancement of a specification along the Standards Track. This is not an exercise we should have to go through. Engineers must have complete faith in implementation reports. Faith-based engineering and reality are mutually exclusive. :-) Regards, -sm ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
SM wrote: This is not an exercise we should have to go through. Engineers must have complete faith in implementation reports. Faith-based engineering and reality are mutually exclusive. :-) Touche! -- Hector Santos, CTO http://www.santronics.com ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
--On Saturday, August 06, 2011 07:15 -0700 Bob Hinden bob.hin...@gmail.com wrote: If a document no longer has anyone watching it, there's a reasonable concern that it no longer has much constituency. In that case, it's better to treat it as immature rather than mature. In order to have reached Draft Standard, the w.g. must have shown multiple implementations. In at least one case I can think about the protocol is very widely used operationally. The working group is disbanded, only the mailing list remains. The protocol is very mature and stable. It's only the IETF activity that is inactive. Moreover, if the community that produced the Draft Standard came to the IETF for the purpose of working on the particular specification (and, presumably, its predecessors), decided they were finished, and went away, perhaps after seeing it widely deployed, there may be an even stronger argument for moving it to Internet Standard than would be the case if there are still people around and messing with it. You can't measure maturity of a protocol at draft standard by looking at the IETF activity. Right. john ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
RE: Last Call: draft-housley-two-maturity-levels-08.txt(Reducing the Standards Track to Two Maturity Levels) to BCP
Hi SM, Pardon me; 1)Is there such thing as a good enough Criteria to handle this concern? 2)Or as usual it passes rough consensus process? Regards, Medel -Original Message- From: ietf-boun...@ietf.org [mailto:ietf-boun...@ietf.org] On Behalf Of SM Sent: Friday, August 05, 2011 5:45 PM To: Russ Housley Cc: ietf@ietf.org Subject: Re: Last Call: draft-housley-two-maturity-levels-08.txt(Reducing the Standards Track to Two Maturity Levels) to BCP Hi Russ, At 12:28 PM 8/3/2011, Russ Housley wrote: I am well aware of the implementation reports. The premise here is that the protocol specification is good enough there are at least two interoperable implementations and the protocol is deployed widely. The implementation report would become optional. One of the advantages of an implementation report is that it provides a statement about interoperability between two or more known implementations. If there is any dispute about that claim, it can be resolved in a non-controversial way. Determining whether a protocol is widely deployed is not always a clear-cut decision. People are not doing many implementation reports. As you say above, there are only about 75 of them. How many protocols are documented in RFCs? That is a very low percentage in my view. Yes, it's a very low percentage. I don't have the figure for the number of protocols documented. Given the low barrier for such reports, I would have expected to see more reports. After all, if the RFC has been published, the protocol has been widely deployed, it should simply have been a matter of filing the short report. From draft-housley-two-maturity-levels-08: this document measures interoperability through widespread deployment of multiple implementations from different code bases, thus condensing the two separate metrics into one. This change is expected to solve the problem. I am not convinced that the metrics is the problem. So, I see the cost quite differently. Most protocols are published as Proposed Standards, and they are never advanced. I'm seeking a process where implementation and deployment experience actually improves the protocol specifications. Today, that rarely happens, and when it does, the Agreed. I didn't find any incentive to inject implementation and deployment experience into the process. This is an argument for the status quo. We have decades of experience with that not working. That is essentially an argument for a single maturity level; that is how the process is really working today. I am not arguing for a single maturity level (the status quo). I do not agree with the conclusion that the decades of stagnation is due to the three maturity level. This document is not about IESG review time, except for the elimination of the requirement for annual reviews which are not done anyway. If that is what you get from the document, then I have done a very poor job in writing it. That is not the point at all. I don't think that you did a poor job. A three maturity level requires three IESG Evaluations. A two maturity level requires two IESG Evaluations. If more documents moved from Proposed Standard to the next level, it would obviously take more IESG review time. I presume that the IESG will only use the following criteria for advancement: - two independent interoperating implementations with widespread deployment and successful operational experience - no errata against the specification - no unused features in the specification And there won't be any DISCUSSes along the lines of: I don't think the implementation reports are adequate for me to meet the requirements of 2026. It does not clearly identify what software was used or show support of each of the individual options and features. Examples througout the document make use of non-example domains. The implementation report is woefully inadequate to document there are interoperable implementations of all the features from two different code bases. My Discuss was not addressed at all - I believe that the WG ignored the spirit of the implementation report requirement - my Discuss said that we should know that there are multiple implementations that have handled the significant changes in the recycling of this Draft Standard. The group apparently refused to update its implementation report Regards, -sm ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf This e-mail message (including attachments, if any) is intended for the use of the individual or the entity to whom it is addressed and may contain information that is privileged, proprietary, confidential and exempt from disclosure. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this communication is strictly
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
SM wrote: People are not doing many implementation reports. As you say above, there are only about 75 of them. How many protocols are documented in RFCs? That is a very low percentage in my view. Yes, it's a very low percentage. I don't have the figure for the number of protocols documented. Given the low barrier for such reports, I would have expected to see more reports. After all, if the RFC has been published, the protocol has been widely deployed, it should simply have been a matter of filing the short report. Of course, when implementation reports are written, one has to watchful for the summarized analytical results that either attempt to add weight to an desired goal or mask the undesired goal and natural result. In one such report, I was able to show the data collection indicated an undesired result was indeed the natural and dominant implementation mode. The 2nd revision tweaked the semantics enough to hide the higher weight of the data collected for the undesirable dominant mode. When that change was also seen and noted, it was added back in the 3rd revision as a oops exclusion. This is not an exercise we should have to go through. Engineers must have complete faith in implementation reports. -- Hector Santos, CTO http://www.santronics.com http://santronics.blogspot.com ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
Dave, On Fri, Jul 29, 2011 at 9:53 AM, Dave CROCKER d...@dcrocker.net wrote: On 7/29/2011 11:13 AM, Russ Housley wrote: (2) At any time after two years from the approval of this document as a BCP, the IESG may choose to reclassify any Draft Standard document as Proposed Standard. I think this is unfair to the people who have done considerable work to get a document to Draft Standard. I hope that the IESG would only do this after giving a lot of notice to the authors, appropriate working groups, and the IETF community to give them the opportunity to request advancement to Internet Standard. This was added after the discussion that Draft Standards could linger for a very long time. Some people said that would not be a problem, and other people said it would be harmful. I conclude that no one knows, so we should include the powers necessary to resolve the problems if they emerge. If they do not emerge, there is no requirement that the IESG do anything. If a document no longer has anyone watching it, there's a reasonable concern that it no longer has much constituency. In that case, it's better to treat it as immature rather than mature. In order to have reached Draft Standard, the w.g. must have shown multiple implementations. In at least one case I can think about the protocol is very widely used operationally. The working group is disbanded, only the mailing list remains. The protocol is very mature and stable. It's only the IETF activity that is inactive. You can't measure maturity of a protocol at draft standard by looking at the IETF activity. Bob I think this Section of the document needs to provide additional detail on how this should work. I do not think we should add speculation about the potential problems to this document. +1 d/ -- Dave Crocker Brandenburg InternetWorking bbiw.net ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
Hi Russ, At 12:28 PM 8/3/2011, Russ Housley wrote: I am well aware of the implementation reports. The premise here is that the protocol specification is good enough there are at least two interoperable implementations and the protocol is deployed widely. The implementation report would become optional. One of the advantages of an implementation report is that it provides a statement about interoperability between two or more known implementations. If there is any dispute about that claim, it can be resolved in a non-controversial way. Determining whether a protocol is widely deployed is not always a clear-cut decision. People are not doing many implementation reports. As you say above, there are only about 75 of them. How many protocols are documented in RFCs? That is a very low percentage in my view. Yes, it's a very low percentage. I don't have the figure for the number of protocols documented. Given the low barrier for such reports, I would have expected to see more reports. After all, if the RFC has been published, the protocol has been widely deployed, it should simply have been a matter of filing the short report. From draft-housley-two-maturity-levels-08: this document measures interoperability through widespread deployment of multiple implementations from different code bases, thus condensing the two separate metrics into one. This change is expected to solve the problem. I am not convinced that the metrics is the problem. So, I see the cost quite differently. Most protocols are published as Proposed Standards, and they are never advanced. I'm seeking a process where implementation and deployment experience actually improves the protocol specifications. Today, that rarely happens, and when it does, the Agreed. I didn't find any incentive to inject implementation and deployment experience into the process. This is an argument for the status quo. We have decades of experience with that not working. That is essentially an argument for a single maturity level; that is how the process is really working today. I am not arguing for a single maturity level (the status quo). I do not agree with the conclusion that the decades of stagnation is due to the three maturity level. This document is not about IESG review time, except for the elimination of the requirement for annual reviews which are not done anyway. If that is what you get from the document, then I have done a very poor job in writing it. That is not the point at all. I don't think that you did a poor job. A three maturity level requires three IESG Evaluations. A two maturity level requires two IESG Evaluations. If more documents moved from Proposed Standard to the next level, it would obviously take more IESG review time. I presume that the IESG will only use the following criteria for advancement: - two independent interoperating implementations with widespread deployment and successful operational experience - no errata against the specification - no unused features in the specification And there won't be any DISCUSSes along the lines of: I don't think the implementation reports are adequate for me to meet the requirements of 2026. It does not clearly identify what software was used or show support of each of the individual options and features. Examples througout the document make use of non-example domains. The implementation report is woefully inadequate to document there are interoperable implementations of all the features from two different code bases. My Discuss was not addressed at all - I believe that the WG ignored the spirit of the implementation report requirement - my Discuss said that we should know that there are multiple implementations that have handled the significant changes in the recycling of this Draft Standard. The group apparently refused to update its implementation report Regards, -sm ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
Hector, On 2011-08-04 14:35, Hector Santos wrote: Brian E Carpenter asked: Can you be more specific? Are you talking about a) drafts that appear in the WG with very mature text, so complete the WG progress very quickly? b) drafts that are direct submissions to the IESG, and go through IETF Last Call and IESG review without coming near the WG? c) drafts that are Independent Submissions to the RFC Editor, so are only subject to minimal review by the IESG and are not IETF documents at all? And in case a) or b), are you talking about standards track and BCP documents, or about informational or experimental ones? Hi Brian, From what I have experienced, it doesn't matter what the fast track non-WG RFC status is - any status can have a surprising WG goal changing influence. I don't really see that as the main concern until it is a competing or conflictive externally produced RFC, especially if it was originally deem out of scope in a WG charter. IMO, in such a case, it should be part of a WG charter change and a WG vetting process before it is made into an RFC and/or allowed to change the WG goals. I'm still not getting your point. External events of all kinds can affect a WG; for example publication of a much better technique in an academic journal, or publication of a patent that is essential to the chosen solution. Are you saying that the existing review process for direct submission or Independent Submission RFCs fails to detect work that overlaps with WGs? I am not an expert with IETF procedures and I don't wish to call that a Loop Hole but it appears to be one. It seems to me a two-maturity level may inadvertently create more of these WG conflicts without the proper due diligence to watch for these type of situations. How? It only concerns documents that have already gone through the whole process of becoming Proposed Standard; essentially the changes beyond that point in the standards track should always be quite minor. Maybe the experience in the DKIM WG was an exception and not the rule typical of a WG, but if was a new reflection of an IETF change making it more possible to fast track RFCs, the review process needs to take into account the consequences it can have in any related WG. Is that a time problem? A WG Chair/AD issue? Too much work for the AD reviews? Not enough synergism of among all parties? I can't say and quite maybe it was just an exceptional experience and not the norm. But I believe a watchdog for these type of possibilities will help. Which is exactly why we do have a review process for non-WG RFCs. Brian ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
Brian E Carpenter wrote: Are you saying that the existing review process for direct submission or Independent Submission RFCs fails to detect work that overlaps with WGs? At least in one experience, I would not say it was a failure per se but more realistically, for many possible reasons, it simply fell through the cracks. If interested, I can provide offlist summary outline example. Which is exactly why we do have a review process for non-WG RFCs. Brian, no doubt. The IETF Review process and the procedures are in place for anyone to raise concerns and even appeal. So in hindsight, it probably unrealistic to expect any IETF reviewer to catch consequential conflicts and in this case, I should of continued the appeal process as I had contemplated but several reasons I decided not to follow through. Only then can concerns be raised for proper IETF review. Maybe to help alleviate possible concerns with fast-track RFCs and perhaps reduce IETF reviewer time and provide some guidance, a short pre-review questionnaire asking relevant questions designed to highlight any possible conflict as known by the author and something the Reviewers might wish to know about, might be appropriate. Example questions might be: 1) Is this RFC directly or indirectly related to one or more existing IETF Working Groups? If so, please list them below and continue with the additional questions. 2) Is there any concept in your RFC which is deemed out of scope in the related Working Group? If so, which state the out of scope concept(s) in itemized format. 3) Is there any concept in your RFC which competes and/or conflicts with the chartered goals of the related Working Group? 4) Was this RFC vetted by the Working Group? If so, what was the conclusion? Please note: In no way am I stating that these type of non-WG RFC work should not be written, put to a stop or not accepted by the IETF reviewers. But with awareness, there be a reviewer problem solving suggestion to remove conflicts. Maybe it just a matter of word-smithing to make corrections. Again, if you wish, I can give you an example off-list to see why questions like these can help. -- Hector Santos, CTO http://www.santronics.com http://santronics.blogspot.com ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
SM: From Section 2.1: no existing published requirements are relaxed. Are these published requirements BCPs? Yes. From Section 2.2: 'This maturity level is a merger of Draft Standard and Standard as specified in RFC 2026 [1]. The chosen name avoids confusion between Draft Standard and Internet-Draft.' Shouldn't that be Internet Standard instead of Standard? Look at RFC 2026. The three level defined there are Proposed Standard, Draft Standard, and Standard. The draft-housley-two-maturity-levels proposes two levels: Proposed Standard and Internet Standard. The request for reclassification is sent to the IESG along with an explanation of how the criteria have been met. The criteria are: (1) There are at least two independent interoperating implementations with widespread deployment and successful operational experience. (2) There are no errata against the specification that would cause a new implementation to fail to interoperate with deployed ones. (3) There are no unused features in the specification that greatly increase implementation complexity. The document that has been the subject of innumerable messages highlights how difficult it can be to reclassify a RFC. Moreover, it amplified the divide between application folks and operators. The IESG could have used the review clause from RFC 2026 and: (i) requested an implementation report from the people in favor of the proposed standard; and (ii) a statement about deployment to determine whether there are operational issues that have to be addressed. I don't know whether application folks and operators agree that cross-area requires mutual understanding. The creation of interoperable protocol implementations requires clear specifications. Interoperability does not mean that the protocol would not have unintended effects on the network. That is where operational experience comes in. It can serve as valuable input to improve a specification. For what it is worth, there approximately 75 implementation reports have been submitted since 1996. I am well aware of the implementation reports. The premise here is that the protocol specification is good enough there are at least two interoperable implementations and the protocol is deployed widely. The implementation report would become optional. A two-step maturity level folds the two different classes of issues into one. Quoting RFC 5657, which this draft unfortunately does not reference: Moving documents along the standards track can be an important signal to the user and implementor communities, and the process of submitting a standard for advancement can help improve that standard or the quality of implementations that participate. During a discussion on another mailing list, it has been mentioned that such an effort has a cost. Lumping all the issues together can only increase that cost. People are not doing many implementation reports. As you say above, there are only about 75 of them. How many protocols are documented in RFCs? That is a very low percentage in my view. So, I see the cost quite differently. Most protocols are published as Proposed Standards, and they are never advanced. I'm seeking a process where implementation and deployment experience actually improves the protocol specifications. Today, that rarely happens, and when it does, the document recycles at Proposed Standard and the reader does not know whether it happened or not. Strangely, this draft argues for measuring interoperability through widespread deployment of multiple implementations from different code bases. It will be more difficult to remove features once implementations are widely deployed. Keeping the feature fight within the Draft Standard discussion can reduce the level of controversy at the last step. As a side note, it would be less costly if feature evaluation was based on implementations instead of what people would like to keep (only one implementation of a feature). This is an argument for the status quo. We have decades of experience with that not working. That is essentially an argument for a single maturity level; that is how the process is really working today. Once a Proposed Standard is published, it is expected that people will go and write code, if they have not done so yet, and evaluate whether their implementation can interoperate with other implementations. I don't see anything in this draft that encourages that. Of course that is what we want. Even better if the running code is used to make decisions before the protocol become a Proposed Standard. In the RFC 5000 range, there are 7 Internet Standards, 13 Draft Standards and 537 Proposed Standards. One of the arguments for this draft is that it reduces the number of IESG evaluations, and other review cycles, from three to two.
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
I appreciate this exchange here. I have a better idea of the draft and your intention I have a few comments. What I have noticed of late are fast track RFCs are coming out of no where, very fast and sometimes are indirectly related to a WG but not a WG chartered work item, and it may have an unforeseen influence in the WG end results. While not all WGs are the same, this is what I experienced in the DKIM WG. The problem comes when there is little or isolating vetting to this external work. For the most part, once work gets an RFC or even just an published I-D, it also comes with a complex to be labeled as a standard to throw the book at others; you are not following the standard. While they are technically wrong, it happens and it puts others in defensive position. But it also shows why there are derivative work or why an RFC does not get followed 100%. I guess, if anything, if we are going to allow for faster maturity, we probably need some guidelines (if not already in place) in how non-WG RFC productions could influence a current WG. -- Hector Santos, CTO http://www.santronics.com http://santronics.blogspot.com Russ Housley wrote: SM: From Section 2.1: no existing published requirements are relaxed. Are these published requirements BCPs? Yes. From Section 2.2: 'This maturity level is a merger of Draft Standard and Standard as specified in RFC 2026 [1]. The chosen name avoids confusion between Draft Standard and Internet-Draft.' Shouldn't that be Internet Standard instead of Standard? Look at RFC 2026. The three level defined there are Proposed Standard, Draft Standard, and Standard. The draft-housley-two-maturity-levels proposes two levels: Proposed Standard and Internet Standard. The request for reclassification is sent to the IESG along with an explanation of how the criteria have been met. The criteria are: (1) There are at least two independent interoperating implementations with widespread deployment and successful operational experience. (2) There are no errata against the specification that would cause a new implementation to fail to interoperate with deployed ones. (3) There are no unused features in the specification that greatly increase implementation complexity. The document that has been the subject of innumerable messages highlights how difficult it can be to reclassify a RFC. Moreover, it amplified the divide between application folks and operators. The IESG could have used the review clause from RFC 2026 and: (i) requested an implementation report from the people in favor of the proposed standard; and (ii) a statement about deployment to determine whether there are operational issues that have to be addressed. I don't know whether application folks and operators agree that cross-area requires mutual understanding. The creation of interoperable protocol implementations requires clear specifications. Interoperability does not mean that the protocol would not have unintended effects on the network. That is where operational experience comes in. It can serve as valuable input to improve a specification. For what it is worth, there approximately 75 implementation reports have been submitted since 1996. I am well aware of the implementation reports. The premise here is that the protocol specification is good enough there are at least two interoperable implementations and the protocol is deployed widely. The implementation report would become optional. A two-step maturity level folds the two different classes of issues into one. Quoting RFC 5657, which this draft unfortunately does not reference: Moving documents along the standards track can be an important signal to the user and implementor communities, and the process of submitting a standard for advancement can help improve that standard or the quality of implementations that participate. During a discussion on another mailing list, it has been mentioned that such an effort has a cost. Lumping all the issues together can only increase that cost. People are not doing many implementation reports. As you say above, there are only about 75 of them. How many protocols are documented in RFCs? That is a very low percentage in my view. So, I see the cost quite differently. Most protocols are published as Proposed Standards, and they are never advanced. I'm seeking a process where implementation and deployment experience actually improves the protocol specifications. Today, that rarely happens, and when it does, the document recycles at Proposed Standard and the reader does not know whether it happened or not. Strangely, this draft argues for measuring interoperability through widespread deployment of multiple implementations from different code bases. It will be more difficult to remove features once implementations are widely deployed. Keeping the feature fight within the Draft
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
Hector, On 2011-08-04 09:19, Hector Santos wrote: I appreciate this exchange here. I have a better idea of the draft and your intention I have a few comments. What I have noticed of late are fast track RFCs are coming out of no where, very fast and sometimes are indirectly related to a WG but not a WG chartered work item, and it may have an unforeseen influence in the WG end results. While not all WGs are the same, this is what I experienced in the DKIM WG. The problem comes when there is little or isolating vetting to this external work. Can you be more specific? Are you talking about a) drafts that appear in the WG with very mature text, so complete the WG progress very quickly? b) drafts that are direct submissions to the IESG, and go through IETF Last Call and IESG review without coming near the WG? c) drafts that are Independent Submissions to the RFC Editor, so are only subject to minimal review by the IESG and are not IETF documents at all? And in case a) or b), are you talking about standards track and BCP documents, or about informational or experimental ones? For the most part, once work gets an RFC or even just an published I-D, it also comes with a complex to be labeled as a standard to throw the book at others; you are not following the standard. While they are technically wrong, it happens and it puts others in defensive position. But it also shows why there are derivative work or why an RFC does not get followed 100%. I guess, if anything, if we are going to allow for faster maturity, we probably need some guidelines (if not already in place) in how non-WG RFC productions could influence a current WG. I don't understand that without understanding what kind of document you are concerned about. Brian ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
Brian E Carpenter asked: Can you be more specific? Are you talking about a) drafts that appear in the WG with very mature text, so complete the WG progress very quickly? b) drafts that are direct submissions to the IESG, and go through IETF Last Call and IESG review without coming near the WG? c) drafts that are Independent Submissions to the RFC Editor, so are only subject to minimal review by the IESG and are not IETF documents at all? And in case a) or b), are you talking about standards track and BCP documents, or about informational or experimental ones? Hi Brian, From what I have experienced, it doesn't matter what the fast track non-WG RFC status is - any status can have a surprising WG goal changing influence. I don't really see that as the main concern until it is a competing or conflictive externally produced RFC, especially if it was originally deem out of scope in a WG charter. IMO, in such a case, it should be part of a WG charter change and a WG vetting process before it is made into an RFC and/or allowed to change the WG goals. I am not an expert with IETF procedures and I don't wish to call that a Loop Hole but it appears to be one. It seems to me a two-maturity level may inadvertently create more of these WG conflicts without the proper due diligence to watch for these type of situations. Maybe the experience in the DKIM WG was an exception and not the rule typical of a WG, but if was a new reflection of an IETF change making it more possible to fast track RFCs, the review process needs to take into account the consequences it can have in any related WG. Is that a time problem? A WG Chair/AD issue? Too much work for the AD reviews? Not enough synergism of among all parties? I can't say and quite maybe it was just an exceptional experience and not the norm. But I believe a watchdog for these type of possibilities will help. -- Hector Santos, CTO http://www.santronics.com ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
Hi, I generally support this proposal, but have some questions on Section 2.3, Transition to a Standards Track with Two Maturity Levels. I am both an author of several Draft Standards and have chaired working groups that have produced them. Any protocol or service that is currently at the abandoned Draft Standard maturity level will retain that classification, absent explicit actions. Two possible actions are available: (1) A Draft Standard may be reclassified as an Internet Standard as soon as the criteria in Section 2.2 are satisfied. What is the process for this? Is the IESG going to review all Draft Standards. Should authors and/or working groups propose a change of status as defined in the document? Something else? Most draft standards very likely meet most of the requirements listed in the document for Internet Standard. (2) At any time after two years from the approval of this document as a BCP, the IESG may choose to reclassify any Draft Standard document as Proposed Standard. I think this is unfair to the people who have done considerable work to get a document to Draft Standard. I hope that the IESG would only do this after giving a lot of notice to the authors, appropriate working groups, and the IETF community to give them the opportunity to request advancement to Internet Standard. I think this Section of the document needs to provide additional detail on how this should work. Regards, Bob ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
Bob: I generally support this proposal, but have some questions on Section 2.3, Transition to a Standards Track with Two Maturity Levels. I am both an author of several Draft Standards and have chaired working groups that have produced them. Any protocol or service that is currently at the abandoned Draft Standard maturity level will retain that classification, absent explicit actions. Two possible actions are available: (1) A Draft Standard may be reclassified as an Internet Standard as soon as the criteria in Section 2.2 are satisfied. What is the process for this? Is the IESG going to review all Draft Standards. Should authors and/or working groups propose a change of status as defined in the document? Something else? Most draft standards very likely meet most of the requirements listed in the document for Internet Standard. Section 2.2 is pretty clear I think. A request to reclassification must be sent to the IESG. ... The request for reclassification is sent to the IESG along with an explanation of how the criteria have been met. The criteria are: (1) There are at least two independent interoperating implementations with widespread deployment and successful operational experience. (2) There are no errata against the specification that would cause a new implementation to fail to interoperate with deployed ones. (3) There are no unused features in the specification that greatly increase implementation complexity. (4) If patented or otherwise controlled technology is required for implementation, the implementations demonstrate at least two separate and successful uses of the licensing process. (2) At any time after two years from the approval of this document as a BCP, the IESG may choose to reclassify any Draft Standard document as Proposed Standard. I think this is unfair to the people who have done considerable work to get a document to Draft Standard. I hope that the IESG would only do this after giving a lot of notice to the authors, appropriate working groups, and the IETF community to give them the opportunity to request advancement to Internet Standard. This was added after the discussion that Draft Standards could linger for a very long time. Some people said that would not be a problem, and other people said it would be harmful. I conclude that no one knows, so we should include the powers necessary to resolve the problems if they emerge. If they do not emerge, there is no requirement that the IESG do anything. I think this Section of the document needs to provide additional detail on how this should work. I do not think we should add speculation about the potential problems to this document. Russ ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
On 7/29/2011 11:13 AM, Russ Housley wrote: (2) At any time after two years from the approval of this document as a BCP, the IESG may choose to reclassify any Draft Standard document as Proposed Standard. I think this is unfair to the people who have done considerable work to get a document to Draft Standard. I hope that the IESG would only do this after giving a lot of notice to the authors, appropriate working groups, and the IETF community to give them the opportunity to request advancement to Internet Standard. This was added after the discussion that Draft Standards could linger for a very long time. Some people said that would not be a problem, and other people said it would be harmful. I conclude that no one knows, so we should include the powers necessary to resolve the problems if they emerge. If they do not emerge, there is no requirement that the IESG do anything. If a document no longer has anyone watching it, there's a reasonable concern that it no longer has much constituency. In that case, it's better to treat it as immature rather than mature. I think this Section of the document needs to provide additional detail on how this should work. I do not think we should add speculation about the potential problems to this document. +1 d/ -- Dave Crocker Brandenburg InternetWorking bbiw.net ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
At 07:02 PM 7/27/2011, The IESG wrote: The IESG has received a request from an individual submitter to consider the following document: - 'Reducing the Standards Track to Two Maturity Levels' draft-housley-two-maturity-levels-08.txt as a BCP The IESG plans to make a decision in the next few weeks, and solicits final comments on this action. Please send substantive comments to the ietf@ietf.org mailing lists by 2011-08-24. Exceptionally, comments may be From Section 2.1: no existing published requirements are relaxed. Are these published requirements BCPs? From Section 2.2: 'This maturity level is a merger of Draft Standard and Standard as specified in RFC 2026 [1]. The chosen name avoids confusion between Draft Standard and Internet-Draft.' Shouldn't that be Internet Standard instead of Standard? The request for reclassification is sent to the IESG along with an explanation of how the criteria have been met. The criteria are: (1) There are at least two independent interoperating implementations with widespread deployment and successful operational experience. (2) There are no errata against the specification that would cause a new implementation to fail to interoperate with deployed ones. (3) There are no unused features in the specification that greatly increase implementation complexity. The document that has been the subject of innumerable messages highlights how difficult it can be to reclassify a RFC. Moreover, it amplified the divide between application folks and operators. The IESG could have used the review clause from RFC 2026 and: (i) requested an implementation report from the people in favor of the proposed standard; and (ii) a statement about deployment to determine whether there are operational issues that have to be addressed. I don't know whether application folks and operators agree that cross-area requires mutual understanding. The creation of interoperable protocol implementations requires clear specifications. Interoperability does not mean that the protocol would not have unintended effects on the network. That is where operational experience comes in. It can serve as valuable input to improve a specification. For what it is worth, there approximately 75 implementation reports have been submitted since 1996. A two-step maturity level folds the two different classes of issues into one. Quoting RFC 5657, which this draft unfortunately does not reference: Moving documents along the standards track can be an important signal to the user and implementor communities, and the process of submitting a standard for advancement can help improve that standard or the quality of implementations that participate. During a discussion on another mailing list, it has been mentioned that such an effort has a cost. Lumping all the issues together can only increase that cost. Strangely, this draft argues for measuring interoperability through widespread deployment of multiple implementations from different code bases. It will be more difficult to remove features once implementations are widely deployed. Keeping the feature fight within the Draft Standard discussion can reduce the level of controversy at the last step. As a side note, it would be less costly if feature evaluation was based on implementations instead of what people would like to keep (only one implementation of a feature). Once a Proposed Standard is published, it is expected that people will go and write code, if they have not done so yet, and evaluate whether their implementation can interoperate with other implementations. I don't see anything in this draft that encourages that. In the RFC 5000 range, there are 7 Internet Standards, 13 Draft Standards and 537 Proposed Standards. One of the arguments for this draft is that it reduces the number of IESG evaluations, and other review cycles, from three to two. Basically, this draft is to adjust the environment so that the IESG can review everything. It does not reduce the barrier of intended proposed standards to the RFC 2026 level. It does not offer any incentive to advance document along the Standards Track. I unfortunately cannot support this draft. Regards, -sm ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
On 7/28/11 1:05 AM, Mykyta Yevstifeyev wrote: Hello, The new version is obviously shorter, but it omits some points. With eliminating of DS level, RFC 5657 makes no sense more. Wrong. The *title* needs to be adjusted, but mutatis mutandis the general advice is useful. It should be obsoleted and moved to Historic by your document, if IESG decides to eliminate the requirement for interoperability documentation, which I am opposed to (see my LC comments to -06). I see no reason to move RFC 5657 to Historic. Another issue is STD numbers. Mentioning that they are still assigned to ISs in Section 2.2 should be fine. The STD issue is orthogonal. Also, Section 3.3: (2) At any time after two years from the approval of this document as a BCP, the IESG may choose to reclassify any Draft Standard document as Proposed Standard. Won't such action be allowed after 2 years from approval? That's what the text says, no? Peter -- Peter Saint-Andre https://stpeter.im/ ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
28.07.2011 16:52, Peter Saint-Andre wrote: On 7/28/11 1:05 AM, Mykyta Yevstifeyev wrote: Hello, The new version is obviously shorter, but it omits some points. With eliminating of DS level, RFC 5657 makes no sense more. Wrong. The *title* needs to be adjusted, but mutatis mutandis the general advice is useful. The document says: RFC 2026 [1] requires a report that documents interoperability between at least two implementations from different code bases as an interim step (Draft Standard) before a specification can be advanced further to the third and final maturity level (Standard) based on widespread deployment and use. In contrast, this document measures interoperability through widespread deployment of multiple implementations from different code bases, thus condensing the two separate metrics into one. which implies that no requirement for interoperability reports is set. RFC 5735 defines what the implementation report is; so I think retaining RFC 5735 as BCP will create confusion, as it will define procedures for the issue which no longer exists. It should be obsoleted and moved to Historic by your document, if IESG decides to eliminate the requirement for interoperability documentation, which I am opposed to (see my LC comments to -06). I see no reason to move RFC 5657 to Historic. See above. Another issue is STD numbers. Mentioning that they are still assigned to ISs in Section 2.2 should be fine. The STD issue is orthogonal. Also, Section 3.3: (2) At any time after two years from the approval of this document as a BCP, the IESG may choose to reclassify any Draft Standard document as Proposed Standard. Won't such action be allowed after 2 years from approval? That's what the text says, no? Yes, I misunderstood the statement. I thought that before 2 years come, DS-PS may be OK while after these 2 years this won't be OK, while it's vice versa. Mykyta Peter ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Last Call: draft-housley-two-maturity-levels-08.txt (Reducing the Standards Track to Two Maturity Levels) to BCP
Hello, The new version is obviously shorter, but it omits some points. With eliminating of DS level, RFC 5657 makes no sense more. It should be obsoleted and moved to Historic by your document, if IESG decides to eliminate the requirement for interoperability documentation, which I am opposed to (see my LC comments to -06). Another issue is STD numbers. Mentioning that they are still assigned to ISs in Section 2.2 should be fine. Also, Section 3.3: (2) At any time after two years from the approval of this document as a BCP, the IESG may choose to reclassify any Draft Standard document as Proposed Standard. Won't such action be allowed after 2 years from approval? Mykyta Yevstifeyev 28.07.2011 5:02, The IESG wrote: The IESG has received a request from an individual submitter to consider the following document: - 'Reducing the Standards Track to Two Maturity Levels' draft-housley-two-maturity-levels-08.txt as a BCP ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
