RFC2119 'Key words for use in RFCs to Indicate Requirement Levels'
defines the keyword 'SHOULD' as:
"This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course."
Generally, I think we can agree that anything less than MUST or MUST NOT
requirements in a spec are pretty useless when it comes to conformance
testing. We try to write specs to these keywords but other keywords tend
to creep in to most specifications.
We currently define tests in test suites for SHOULD requirements. A
problem occurs when those tests are used to gauge the overall compliance
of an implementation to the full test suite. An implementation could
theoretically be 100% compliant without needing to pass non-MUST and
non-MUST NOT tests.
Perhaps we should introduce 'bonus' points for SHOULD/SHOULD NOT/MAY and
RECOMMENDED tests and not have them contribute to overall compliance
output, thereby allowing implementations to claim 100% compliance to
MUST/MUST NOT tests. An implementation can then optionally collect any
available, optional bonus points as available from requirements marked
up with other keywords.
Wondering if there is any set W3C thinking on this or a way of including
SHOULD tests in test suites but clearly indicating that they are,
basically, optional and do not count towards the overall compliance
score? I couldn't find anything in [1].
- Rich
[1] http://www.w3.org/TR/test-methodology/
