Thanks, Joel. Based on your review and inputs the latest version is now on the agenda of the IESG telechat this Thursday.
Dan > -----Original Message----- > From: Joel M. Halpern [mailto:[EMAIL PROTECTED] > Sent: Friday, May 02, 2008 2:21 PM > To: [email protected] > Cc: Romascanu, Dan (Dan) > Subject: [Fwd: Re: [Gen-art] Gen-Art IETF LC review: > draft-ietf-ipfix-testing-04.txt] > > Since this did not make it to the spreadsheeting, I am > forwarding the note again. > the -05 of the ipfix testing document addressed my concerns > and as far as I can tell is now ready for publication as an > Informational RFC. > > Yours, > Joel > > -------- Original Message -------- > Subject: Re: [Gen-art] Gen-Art IETF LC review: > draft-ietf-ipfix-testing-04.txt > Date: Fri, 18 Apr 2008 17:42:06 -0400 > From: Joel M. Halpern <[EMAIL PROTECTED]> > To: Benoit Claise <[EMAIL PROTECTED]> > CC: Romascanu, Dan (Dan) <[EMAIL PROTECTED]>, > [email protected], [EMAIL PROTECTED], > [EMAIL PROTECTED], > [EMAIL PROTECTED] > References: > <[EMAIL PROTECTED]> > <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> > > Thank you. These changes sufficiently address my concerns. > > (I rather like the one case where the text reads "the tester > must check and if necessary improve the software so that > ...". Why do I think that this is something folks have > tripped over before? :-) > > I appreciate your efforts in addressing the issues. > Yours, > Joel M. Halpern > > Benoit Claise wrote: > > Dear Joel, > > > > I posted a new version of the testing draft > > > > A new version of I-D, draft-ietf-ipfix-testing-05.txt > has been successfuly submitted by Benoit Claise and posted to > the IETF repository. > > > > Filename: draft-ietf-ipfix-testing > > Revision: 05 > > Title: Guidelines for IP Flow Information > eXport (IPFIX) Testing > > Creation_date: 2008-04-14 > > WG ID: ipfix > > Number_of_pages: 38 > > > > Abstract: > > This document presents a list of tests for implementers > of IP Flow > > Information Export (IPFIX) compliant Exporting Processes and > > Collecting Processes. This document specifies guidelines for a > > series of tests that can be run on the IPFIX Exporting > Process and > > Collecting Process in order to probe the conformity and > robustness of > > the IPFIX protocol implementations. These tests cover > all important > > functions, in order to gain a level of confidence in the IPFIX > > implementation. Therefore they allow the implementer to perform > > interoperability or plug tests with other IPFIX > Exporting Processes > > and Collecting Processes.Conventions used in this document > > > > We hope to have addressed your concerns successfully. Would > you mind > > checking this latest version. > > Note: we have addressed your point 10), as we thought we > didn't want > > to be that formal in our test description. > > > > Regards, Benoit. > > > > > >> I have been selected as the General Area Review Team (Gen-ART) > >> reviewer for this draft (for background on Gen-ART, please see > >> http://www.alvestrand.no/ietf/gen/art/gen-art-FAQ.html ). > >> > >> Please resolve these comments along with any other Last > Call comments > >> you may receive. > >> > >> Document: Guidelines for IP Flow Information eXport (IPFIX) Testing > >> Reviewer: Joel M. Halpern > >> Review Date: 15-Feb-2008 > >> IETF LC End Date: 26-Feb-2008 > >> IESG Telechat date: N/A > >> > >> Summary: This document needs some additional work before > publication > >> as an informational RFC. > >> I would particularly recommend considering addressing at least the > >> first comment below prior to RFC publication. > >> I would also suggest that the test descriptions need some > >> clarification as described in the technical section below, > >> particularly items 5 and 6. > >> > >> Comments: > >> > >> Conceptual: > >> 1) While the document is being published as an information > RFC, the > >> wording of the abstract and introduction make it seem that this > >> document is actually defining conformance to the IPFIX RFCs. The > >> IETF has generally carefully steered clear of defining > such conformance. > >> So, while publishing a useful test suite is probably a > good idea, I > >> strongly recommend fixing the wording of at least the abstract and > >> introduction to make it quite clear that these are not mandatory > >> tests, and that these tests do not define conformance. > >> Related to this, please do not assert (in section 3) that passing > >> this test suite constitutes conformance to the IPFIX > architecture and > >> protocol. (Among other things,test suite passage proves nothing > >> about architectural conformance.) > >> > >> > >> Technical: > >> 2) In the terminology section, an Observation Point is > defined simply > >> as a place where packets can be observed. An Observation > Domain is a > >> collection of Observation points. Then, in the middle of the > >> definition of an Observation domain it say "In the IPFIX > MEssage it > >> generates..." but up till now none of the things that have been > >> defined generate IPFIX messages. It is possible that the > "it" in the > >> quote is supposed to be the "Metering Process" mentioned > in passing > >> earlier in the definition. But the English grammar does > not lead the > >> reader to such a conclusion. Later in that same > definition, it beings > >> to appear that an Observation Domain (which is a collection of > >> points, not a process or entity) is supposed to generate IPFIX > >> messages, since it is supposed to include a Domain ID in > the messages it generates. > >> This definition for an Observation Domain needs to be reworked, to > >> avoid confusing the Domain with the Measurement Process which is > >> running in / for / on the Domain. > >> > >> 3) The use of capital "MUST" in section 3.1 is almost > certainly wrong. > >> Firstly, what I think that section is saying is that being able to > >> correctly perform the basic tests is a precondition for > being able to > >> perform further test successfully. Thats a precondition, > not a "MUST". > >> Of lesser significance, this document does not provide any > >> description of what it means by "MUST". We are usually > careful about > >> how such language is used in informational RFCs. I think > the meaning > >> would be clearer if the real intent were stated. I > suspect that some > >> readers of this review may find my concern here pedantic. But the > >> continual use of MUST in the document really, really > bothers me. (I > >> hope the next comment helps explain why it bothers me so much.) > >> > >> 4) Then, the test descriptions go on to keep using this language. > >> This is a test suite description document. Simply state > how to run > >> the test. There is no need for "MUST". Section 3 should indicate > >> that the test descriptions describe the preconditions and > steps that > >> the tester goes through. So section 3.1 would begin "The tester > >> creates one Exporting Process and one collection process, > configures > >> the Exporting Process to ..." > >> > >> 5) It is not clear what test steps like ~The tester > ensures that an > >> SCTP association is established.~ (Or worse, the actual text which > >> reads "the test MUST ensure that an SCTP association is > established." > >> are supposed to do. Is this an instruction to the tester to use > >> network management tools or CLI to verify a connection on both > >> devices? Is it an instruction to perform additional configuration? > >> How does the tester "ensure". A test suite should tell a > tester what > >> steps to undertake, and what observations to perform. "Ensure" is > >> not either one of those. > >> 5a) To elaborate on this issue, in the middle of the test > step about > >> ensuring that Data Records are actually exported, we finally get a > >> testable instruction, to whit, use a packet sniffer and check that > >> the packets are coming by. > >> > >> 6) I believe I understand how a tester would create templates, for > >> the template test. But how is the tester to create data sets. > >> Particularly data sets with specific properties, such as > the padding > >> in section 3.2.3 and 3.2.4? The best conclusion I can come to is > >> that this is a collector test, and that it assumes a > packet generator > >> which can generate IPFIX packets. Having such a device in a test > >> setup makes sense. But the test description does not say > "configure > >> a packet generator to generate an IPFIX packet with ..." > (There are > >> other ways to say this, but there needs to be some > description of how > >> testers are expected to create data sets.) > >> 6a) Related to this, I find reading this document rather > odd. I have > >> read many test suites for protocols and implementations of > protocols. > >> They generally focus on a Device (or implementation, or > entity) Under > >> Test, and the framing around that Device. This suite > appears to be > >> trying to test two interacting devices simultaneously. That is > >> extremely difficult, and extremely confusing. It is particularly > >> hard because then the tester doesn't have enough points of > control to > >> perform the tests and observe the results meaningfully. It is > >> possible that this combined suite is right for this > problem. But if > >> so, a lot of explanation of why it is done that way and how the > >> tester is to accomplish his goals is needed. > >> > >> Minor: > >> 7) The abstract is worded as if one could not perform > >> interoperability testing without first running the tests in this > >> document. While having run the tests in this document will > >> presumably increase the chances of a successful interoperability > >> test, they are not an inherent requirement for such testing. > >> > >> 8) I would probably be inclined to lighten up the > Motivation section > >> a bit. Or even remove it. I don't think we need to > explain why test > >> suites are useful. If we really need a motivation > section, then it > >> should explain something about why it is particularly > complex to test > >> IPFIX implementations (if it is) and thus why the IETF feels it is > >> particularly useful to publish a test suite ourselves in this case. > >> > >> 9) The definition of Transport Session is actually the > definition of > >> various kinds of transport sessions, and how they are identified. > >> Could the definition start with an actual definition please. (I.e. > >> the communication over time used to carry X between Y and Z? Or > >> something.) > >> > >> 10) As an editorial matter, most testers I have worked > with strongly > >> prefer if every step in a test is explicitly separate and named / > >> numbered. That way, they can check off each step as it > goes. So the > >> beginning of 3.1.1 would be > >> i) Create One Exporting Process > >> ii) Create One Collection Process > >> iii) Configure the Exporting Process ... > >> > >> 11) It is particularly odd to see a set of Stress/Load tests that > >> simultaneously claim to be measuring conformance and to > not specify > >> the level of Stress / Load. Having a description of how > to perform > >> load tests is useful. But its relationship to the other tests is > >> confusing. (This obviously is helped once we no longer claim that > >> this is a conformance test.) > >> > >> > > > _______________________________________________ > Gen-art mailing list > [email protected] > https://www.ietf.org/mailman/listinfo/gen-art > > _______________________________________________ Gen-art mailing list [email protected] https://www.ietf.org/mailman/listinfo/gen-art
