At 11:09 PM -0800 1/26/02, Kyle Lussier wrote: >I seem to be getting two conflicting viewpoints: > > #1 Vendors can only be trusted to be interoperable on their own, > and can not be forced to conform. > > #2 Vendors absolutely can't be trusted to be interoperable, > without conformance testing.
Missing is "#3 Vendors can't be trusted to be interoperable without interoperability testing." VPNC performs conformance testing for IPsec, and there are plenty of examples of our members who conform but do not interoperate without a lot of knob twiddling and an occasional bug fix. The long experience with IPsec interoperability events (which VPNC does not hold) has shown that A, B, and C might conform, and A and B can interoperate fine, but A and C cannot interoperate. This is usually due to administrative interfaces either not having the right knobs, the defaults for C being valid for conformance but not for interoperability, or weird magic. Is interoperability testing needed for end users? Possibly, but it won't happen until someone comes up with a good business model for the testing agency. When anyone comes up with one, I'd love to hear it. VPNC was originally formed to do good interop testing for the IPsec industry, but when we figured out what that would cost all of the members, there was no longer any interest. The basic problem: either each of the 35 members is responsible for running and debugging the test with the other 34 members, or they are willing to pay someone to run and debug the 1225 (35^2) tests for them. In the former case, the best statement of why that was not attractive was "if I have a staff person who has that much skill with our product and the at least 100 hours it will take, I have much more important work for them". In the latter case, there was immediate history of another interop testing agency who both charged a large amount of money to do the tests and a fair amount of vendor staff time to do debugging in order to do about one fifth the number of tests. Thus, VPNC is left doing conformance testing with verifiable results, which is admittedly not nearly as valuable to end users. (See <http://www.vpnc.org/conformance.html>, particularly near the end, for more details.) VPNC also does some small-to-medium sized interop demos, but these are not formal interop tests with formal results. Of course, this is not to say that formal interoperability testing is impossible. There are examples of where it happens today in the Internet industry. But there are probably one or two orders of magnitude of examples of where it does not happen. Informal interoperabilty events, where there are no results published but lots of good interaction between vendors, have helped the industry a great deal but are largely invisible to end users (and still don't produce the level of interoperability that people in this discussion say is "required"). Does the IETF or ISOC want to get into either conformance or interoperability testing? It is fairly safe to say this not going to happen without a business model to pay for the short-term and long-term costs. So far, no business model has appeared in the discussion. --Paul Hoffman, Director --VPN Consortium
