On Jul 4, 2011, at 11:47 , Rich Tibbett wrote: > Wondering if there is any set W3C thinking on this or a way of including > SHOULD tests in test suites but clearly indicating that they are, basically, > optional and do not count towards the overall compliance score? I couldn't > find anything in [1].
I think that the best way of addressing SHOULDs (apart from avoiding them in the first place) is to test them as if they were MUSTs, but put the results for those in a separate table. In that separate table, for each failure ask the implementer why they fail and document that. This gives a) the possibility of 100% conformance to the MUSTS, which is needed to advance the spec; 2) useful feedback about the spec, notably concerning unrealistic SHOULD or potentially something that could be either removed or upgraded to a MUST; and 3) a potential incentive of sorts for some implementers who are driven more by the ability to use a test report in marketing material than they are to make actual users happy to up their game a little bit. -- Robin Berjon - http://berjon.com/ - @robinberjon
