Joel, > Thank you. These changes sufficiently address my concerns. > > (I rather like the one case where the text reads "the tester must > check and if necessary improve the software so that ...". Why do I > think that this is something folks have tripped over before? :-) Oops. This was not supposed to be there. It was agreed to remove that part of the sentence before posting the draft ;-)
OLD: The tester must check and if necessary improve the software so that the Templates and the associated Data Records are correctly received and decoded by the Collecting Process. NEW: The tester must check that the Templates and the associated Data Records are correctly received and decoded by the Collecting Process. > > I appreciate your efforts in addressing the issues. You're welcome. Regards, Benoit. > Yours, > Joel M. Halpern > > Benoit Claise wrote: >> Dear Joel, >> >> I posted a new version of the testing draft >> >> A new version of I-D, draft-ietf-ipfix-testing-05.txt has been >> successfuly submitted by Benoit Claise and posted to the IETF >> repository. >> >> Filename: draft-ietf-ipfix-testing >> Revision: 05 >> Title: Guidelines for IP Flow Information eXport (IPFIX) >> Testing >> Creation_date: 2008-04-14 >> WG ID: ipfix >> Number_of_pages: 38 >> >> Abstract: >> This document presents a list of tests for implementers of IP Flow >> Information Export (IPFIX) compliant Exporting Processes and >> Collecting Processes. This document specifies guidelines for a >> series of tests that can be run on the IPFIX Exporting Process and >> Collecting Process in order to probe the conformity and >> robustness of >> the IPFIX protocol implementations. These tests cover all important >> functions, in order to gain a level of confidence in the IPFIX >> implementation. Therefore they allow the implementer to perform >> interoperability or plug tests with other IPFIX Exporting Processes >> and Collecting Processes.Conventions used in this document >> >> We hope to have addressed your concerns successfully. Would you mind >> checking this latest version. >> Note: we have addressed your point 10), as we thought we didn't want >> to be that formal in our test description. >> >> Regards, Benoit. >> >> >>> I have been selected as the General Area Review Team (Gen-ART) >>> reviewer for this draft (for background on Gen-ART, please see >>> http://www.alvestrand.no/ietf/gen/art/gen-art-FAQ.html ). >>> >>> Please resolve these comments along with any other Last Call comments >>> you may receive. >>> >>> Document: Guidelines for IP Flow Information eXport (IPFIX) Testing >>> Reviewer: Joel M. Halpern >>> Review Date: 15-Feb-2008 >>> IETF LC End Date: 26-Feb-2008 >>> IESG Telechat date: N/A >>> >>> Summary: This document needs some additional work before publication >>> as an informational RFC. >>> I would particularly recommend considering addressing at least the >>> first comment below prior to RFC publication. >>> I would also suggest that the test descriptions need some >>> clarification as described in the technical section below, >>> particularly items 5 and 6. >>> >>> Comments: >>> >>> Conceptual: >>> 1) While the document is being published as an information RFC, the >>> wording of the abstract and introduction make it seem that this >>> document is actually defining conformance to the IPFIX RFCs. The >>> IETF has generally carefully steered clear of defining such >>> conformance. >>> So, while publishing a useful test suite is probably a good idea, I >>> strongly recommend fixing the wording of at least the abstract and >>> introduction to make it quite clear that these are not mandatory >>> tests, and that these tests do not define conformance. >>> Related to this, please do not assert (in section 3) that passing >>> this test suite constitutes conformance to the IPFIX architecture >>> and protocol. (Among other things,test suite passage proves nothing >>> about architectural conformance.) >>> >>> >>> Technical: >>> 2) In the terminology section, an Observation Point is defined >>> simply as a place where packets can be observed. An Observation >>> Domain is a collection of Observation points. Then, in the middle >>> of the definition of an Observation domain it say "In the IPFIX >>> MEssage it generates..." but up till now none of the things that >>> have been defined generate IPFIX messages. It is possible that the >>> "it" in the quote is supposed to be the "Metering Process" mentioned >>> in passing earlier in the definition. But the English grammar does >>> not lead the reader to such a conclusion. Later in that same >>> definition, it beings to appear that an Observation Domain (which is >>> a collection of points, not a process or entity) is supposed to >>> generate IPFIX messages, since it is supposed to include a Domain ID >>> in the messages it generates. This definition for an Observation >>> Domain needs to be reworked, to avoid confusing the Domain with the >>> Measurement Process which is running in / for / on the Domain. >>> >>> 3) The use of capital "MUST" in section 3.1 is almost certainly >>> wrong. Firstly, what I think that section is saying is that being >>> able to correctly perform the basic tests is a precondition for >>> being able to perform further test successfully. Thats a >>> precondition, not a "MUST". >>> Of lesser significance, this document does not provide any >>> description of what it means by "MUST". We are usually careful >>> about how such language is used in informational RFCs. I think the >>> meaning would be clearer if the real intent were stated. I suspect >>> that some readers of this review may find my concern here pedantic. >>> But the continual use of MUST in the document really, really bothers >>> me. (I hope the next comment helps explain why it bothers me so much.) >>> >>> 4) Then, the test descriptions go on to keep using this language. >>> This is a test suite description document. Simply state how to run >>> the test. There is no need for "MUST". Section 3 should indicate >>> that the test descriptions describe the preconditions and steps that >>> the tester goes through. So section 3.1 would begin "The tester >>> creates one Exporting Process and one collection process, configures >>> the Exporting Process to ..." >>> >>> 5) It is not clear what test steps like ~The tester ensures that an >>> SCTP association is established.~ (Or worse, the actual text which >>> reads "the test MUST ensure that an SCTP association is >>> established." are supposed to do. Is this an instruction to the >>> tester to use network management tools or CLI to verify a connection >>> on both devices? Is it an instruction to perform additional >>> configuration? How does the tester "ensure". A test suite should >>> tell a tester what steps to undertake, and what observations to >>> perform. "Ensure" is not either one of those. >>> 5a) To elaborate on this issue, in the middle of the test step about >>> ensuring that Data Records are actually exported, we finally get a >>> testable instruction, to whit, use a packet sniffer and check that >>> the packets are coming by. >>> >>> 6) I believe I understand how a tester would create templates, for >>> the template test. But how is the tester to create data sets. >>> Particularly data sets with specific properties, such as the padding >>> in section 3.2.3 and 3.2.4? The best conclusion I can come to is >>> that this is a collector test, and that it assumes a packet >>> generator which can generate IPFIX packets. Having such a device in >>> a test setup makes sense. But the test description does not say >>> "configure a packet generator to generate an IPFIX packet with ..." >>> (There are other ways to say this, but there needs to be some >>> description of how testers are expected to create data sets.) >>> 6a) Related to this, I find reading this document rather odd. I >>> have read many test suites for protocols and implementations of >>> protocols. They generally focus on a Device (or implementation, or >>> entity) Under Test, and the framing around that Device. This suite >>> appears to be trying to test two interacting devices >>> simultaneously. That is extremely difficult, and extremely >>> confusing. It is particularly hard because then the tester doesn't >>> have enough points of control to perform the tests and observe the >>> results meaningfully. It is possible that this combined suite is >>> right for this problem. But if so, a lot of explanation of why it >>> is done that way and how the tester is to accomplish his goals is >>> needed. >>> >>> Minor: >>> 7) The abstract is worded as if one could not perform >>> interoperability testing without first running the tests in this >>> document. While having run the tests in this document will >>> presumably increase the chances of a successful interoperability >>> test, they are not an inherent requirement for such testing. >>> >>> 8) I would probably be inclined to lighten up the Motivation section >>> a bit. Or even remove it. I don't think we need to explain why >>> test suites are useful. If we really need a motivation section, >>> then it should explain something about why it is particularly >>> complex to test IPFIX implementations (if it is) and thus why the >>> IETF feels it is particularly useful to publish a test suite >>> ourselves in this case. >>> >>> 9) The definition of Transport Session is actually the definition of >>> various kinds of transport sessions, and how they are identified. >>> Could the definition start with an actual definition please. (I.e. >>> the communication over time used to carry X between Y and Z? Or >>> something.) >>> >>> 10) As an editorial matter, most testers I have worked with strongly >>> prefer if every step in a test is explicitly separate and named / >>> numbered. That way, they can check off each step as it goes. So >>> the beginning of 3.1.1 would be >>> i) Create One Exporting Process >>> ii) Create One Collection Process >>> iii) Configure the Exporting Process ... >>> >>> 11) It is particularly odd to see a set of Stress/Load tests that >>> simultaneously claim to be measuring conformance and to not specify >>> the level of Stress / Load. Having a description of how to perform >>> load tests is useful. But its relationship to the other tests is >>> confusing. (This obviously is helped once we no longer claim that >>> this is a conformance test.) >>> >>> >> _______________________________________________ Gen-art mailing list [email protected] https://www.ietf.org/mailman/listinfo/gen-art
