Wendy, Good proposal! Two comments:
1. I liked that the interop is for a setting that is close to real-life deployment. In particular, my understanding of the proposal is that the interop should leave as much unspecified as possible. A first reaction then is how to validate the correctness. For example, the response of resource-id is not known ahead of time. I assume that then the interop participants will need a second channel (e.g., human in the loop) to verify that a client c1 gets the correct response from a server s1. Is this the intended setting? 2. A second comment is that it will be interesting if the interop design will also become a starting point of ALTO compliance tests. We partition the tests into MUST and optional so that a test case is a MUST case if and only if (iff) it is a MUST in RFC7285. In the current proposal below, I see some mix (e.g., some services in the interop is not a MUST in RFC7285). Make sense? Richard On Fri, May 29, 2015 at 10:24 AM, Wendy Roome <[email protected]> wrote: > Folks, > > Rather than simply update the last test, I suggest a new approach, > outlined below. > > **** General **** > > The interop test defines two network maps: the default network map and an > alternate network map. We specify the PIDs and CIDRs for each map. For > each network map, the interop test defines the numerical-mode values for > the routingcost and hopcount cost-metrics. The interop test does not > define ordinal values for those metrics. Instead, each ALTO server may > assign ordinal values however it wants, as long as the ordinal values > preserve the numerical values. > > The interop test also defines a custom endpoint property, say > "priv:ietf-type", and specifies the values. > > > **** Approach & Philosophy **** > > Rather than specifying every detail, as in the previous interop test, this > iteration only specifies what is absolutely necessary. That is, the PIDs > in the network maps, the costs between PIDs, and the enpdoint property > values. Everything else -- resource ids, uris, cost-type names, etc. -- is > up to the server. Thus each server is completely characterized by two > parameters: > > * The URI of its IRD. > * The resource id of its alternate map (if provided). > > When a client is paired with a server, the client is given those two > parameters. The client is expected to determine everything else it needs > (e.g., the URIs of the default network map and the numerical routingcost > cost map) from the IRD. > > Client-writers are encouraged to script their client so that it validates > the server's responses. That is, rather than just printing a cost map, the > client is encouraged to verify that it has the expected values. The > interop test will specify the maps in enough detail that clients will know > exactly what the server should return for any request. > > Server-writers are encouraged to configure their server (especially the > IRD) so as to stress-test clients. That is, do things which are legal, but > unusual. For example, instead of having all resources in one IRD, the > server might have several linked (cascaded) IRDs. Or instead of offering > one filtered cost map, which handles all four combinations of cost metrics > & modes, plus constraints, the server might offer four separate filtered > cost map resources: one which handles numerical mode routingcost & > hopcount, but does not accept constraints, a second that handles the same > cost types, but which does accept constraints, and two more for ordinal > mode costs. This would verify that a client is able to select the correct > resource for a given request. > > > **** Server Requirements **** > > Each server MUST offer the following resources: > * A full network map for the default network. > * Four full cost maps for the default network map: the routingcost & > hopcount metrics, in both numerical & ordinal modes. > * An endpoint property service for the pid property for the default > network map. > > Each server MAY offer the following optional services: > * A filtered network map for the default network. > * A filtered cost map for the default network, optionally allowing cost > constraints. > * An endpoint cost service, covering the 4 cost metric/mode combinations, > optionally allowing cost constraints. The values must be derived from the > default network & cost maps. > * The filtered cost map & endpoint cost services may support constraints. > * An alternate network map plus the 4 associated full cost maps. > * An endpoint property service for the alternate network's pid property. > This may be the same EPS resource as for the default network, or a > different resource. > * A filtered network map for the alternate network. > * A filtered cost map for the alternate network map. > > If offered, the filtered cost map & endpoint cost resources MUST cover all > four metric/mode combinations. > > If offered, the costs for the endpoint cost service MUST must be derived > from the default network map. > > > **** Client Requirements **** > > When a client is paired with a server, the client is given the URI of the > server's IRD, and the resource id of its alternate map (if the server & > client support alternate maps). The client must be able to extract all > other information from the IRD -- full map resources, their URIs, etc. > > Each client MUST: > > * Fetch the IRD. > * Locate & fetch the default network map and its four associated full cost > maps. > * Locate the EPS for the default network map's pid resource, and determine > the pid for a set of endpoints specified by the interop test. > > Note that clients are not required to fetch the alternate network map (if > present). But every client must be able to parse an IRD with an alternate > network map, and distinguish the default network map, and its associate > resources, from the alternate map. > > Optional client requirements: > * Use the filtered network map for the default network map to fetch PIDs > specified by the interop test. > * Use the filtered cost map for the default network map to fetch the costs > for several combinations of source pids, destination pids, metrics/modes, > and optional constraints, as specified by the interop test. > * Use the endpoint cost service to fetch the costs for several > combinations of source addresses, destination addresses, metrics/modes, > and optional constraints, as specified by the interop test. > * Locate & fetch the alternate map and its four full cost maps. > * Use the filtered network map for the alternate network map to fetch PIDs > specified by the interop test. > * Use the filtered cost map for the alternate network map to fetch the > costs for several combinations of source pids, destination pids, > metrics/modes, and optional constraints, as specified by the interop test. > * Fetch the custom endpoint property for a set of endpoints defined by the > interop test. > > > **** Error Testing **** > > We will develop "torture test" scripts of bad client requests, and feed > them to each server to make sure the server gives reasonable error > responses. Participants are encouraged to contribute bad client requests > up to the time of the test. We will not specify the error tests in > advance, but will publish them after the interop is complete. > > Note that the emphasis in on verifying that servers survive client errors. > It does not seem important to verify that clients survive server errors. > If you disagree, speak now or forever hold your peace. > > - Wendy Roome > > > _______________________________________________ > alto mailing list > [email protected] > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_alto&d=AwICAg&c=-dg2m7zWuuDZ0MUcV7Sdqw&r=4G36iiEVb2m_v-0RnP2gx9KZJjYQgfvrOCE3789JGIA&m=kXIGnWFHVmEKkrU-3jzE_hUpglOe7Bj7H4Ys164dZBE&s=eXl0C-Yj1ZVYLA6lRFfpWyP1zKA5InHCoayBzITypGQ&e=
_______________________________________________ alto mailing list [email protected] https://www.ietf.org/mailman/listinfo/alto
