On Sun, 10 Jul 2011 22:34:59 +0200, Aryeh Gregor
<[email protected]> wrote:
On Sun, Jul 10, 2011 at 3:59 PM, Charles McCathieNevile
<[email protected]> wrote:
Privacy and security restrictions leap to mind. There are things that
really are "should" requirements because there are valid use cases for
not applying them, and no reason to break those cases by making the
requirement a "must". In the normal case where they are applied you
want to be able to test.
Were you thinking of specific examples? I can't think of any offhand.
The most recent example is the Contacts API from the DAP group, but this
is a pretty common pattern. Geolocation went through a long discussion on
the topic.
But the difference between "should" and "must" is already two sets of
conformance profiles (or whatever you want to call it), where one
applies always and the other applies unless there's a reason not to
do the thing that is assumed to be normal.
The difference is that if you have "must" requirements that are
specific to a single conformance class, you can write a test suite and
expect every implementation in that class to pass it.
Yep.
For "should" requirements, you're saying it's okay to violate it, so
test suites don't make a lot of sense.
Not quite. I'm saying that there are cases where violoating the
requirement is reasonable, so test results shouldn't determine simple
conformance.
On the other hand, where these are things that in *most* cases we want
interoperability, it makes sense to have test suites so people who don't
violate the requirement can check that what they are doing is consistent
with what others do.
Essentially the question shouldn't revolve around the tests themselves -
tests can be useful even when they're not determining "pure conformance".
What matters is how you use the results.
Performance tests are another class of tests that can be very useful,
allowing people to understand what sort of things work well and what sort
of things might cause real-world problems, without normally giving any
information about actual conformance.
cheers
--
Charles McCathieNevile Opera Software, Standards Group
je parle français -- hablo español -- jeg lærer norsk
http://my.opera.com/chaals Try Opera: http://www.opera.com
