On 7/14/06, Mario Ivankovits <[EMAIL PROTECTED]> wrote:
I can tell you from years of painful experience, supporting some JSP tag libraries that rendered complex output, that the golden file approach can be really fragile and I'd never do it again :-). The problem we had is two fold:
* Some changes that are innocuous in their effect on the runtime
(such as changing the order of attributes generated in an element)
will still break the golden file. False positive error reports are never
a productivity enhancer :-).
* If you deliberately change the output of a component, the tendency
of the developer is to just re-record the entire golden file, and forget
to examine whether some other bug was introduced (such as omitting
a child element or something). We found ourselves introducing
new errors when this occurred, which kind of defeats the purpose.
Shale's test framework includes support for exactly this scenario (using HtmlUnit). Many of the "system integration" tests in Shale's use cases example use this to evaluate the actual output ... very convenient, because HtmlUnit gives you back a DOM of the output, and you can test your assertions as needed.
That being said, the behavior of a JSF component and renderer combination is complex enough that you probably want both unit tests on the individual methods and integration tests on the overall output.
Deliberately releasing components that don't work with the RI does not seem like something that will increase the market acceptance of MyFaces components. Instead, this would create (or increase) a perception that MyFaces developers are not interested in compatibility. Also, given the fact that the RI has a 1.2 version available and MyFaces doesn't yet, it seems likely to give people a reason to consider switching away.
The best approach is to make the build system so powerful that running your entire test suite against either the MyFaces or RI implementations is a single command line parameter. Wendy's already done that for Shale (so we have no excuses for the framework not supporting both :-), and I'm sure it can be done for the component libraries too.
CraigHi!
>> I noticed that there are only some tests for components in Tomahawk.
>> Wouldn't it be great to have a test-case as a requirement for
>> promoting a component ?
> Sure, if we can decide what reasonable test cases are.
I would like to see more test-cases too, but not by using mock tests. I
think for component testing its way too much work and error prone after
refactoring.
I was told (thanks to matze ;-) ) that trinidad uses so called
gold-files - it compares the resulting html with an "known-to-be-good"
(=gold) html file,
I can tell you from years of painful experience, supporting some JSP tag libraries that rendered complex output, that the golden file approach can be really fragile and I'd never do it again :-). The problem we had is two fold:
* Some changes that are innocuous in their effect on the runtime
(such as changing the order of attributes generated in an element)
will still break the golden file. False positive error reports are never
a productivity enhancer :-).
* If you deliberately change the output of a component, the tendency
of the developer is to just re-record the entire golden file, and forget
to examine whether some other bug was introduced (such as omitting
a child element or something). We found ourselves introducing
new errors when this occurred, which kind of defeats the purpose.
though, what I really would like to see for component
testing is something like htmlunit (http://htmlunit.sourceforge.net/)
which allows some basic _javascript_ testing too.
Shale's test framework includes support for exactly this scenario (using HtmlUnit). Many of the "system integration" tests in Shale's use cases example use this to evaluate the actual output ... very convenient, because HtmlUnit gives you back a DOM of the output, and you can test your assertions as needed.
That being said, the behavior of a JSF component and renderer combination is complex enough that you probably want both unit tests on the individual methods and integration tests on the overall output.
>> And of course, Mike is right in *testing* with RI and example.
>
> Note that I don't think that the component necessarily has to work
> with the RI to be promoted. But the documentation for the component
> should state whether it's expected to work with the RI or not, which
> would be determined by the testing.
This is my point too, we should document if a tag breaks RI
compatibility. As far as I know, only the form and command* stuff will
break RI compatibility and - as long as there is no spec for this, its
hard to make it "any JSF" compatible - we can act as RI and so we can be
RI compatible, but as long as there is no spec for the parameter passing
stuff we cant make it "any JSF" compatible.
However, whats the case why we didnt change our classes to be at least
RI compatible?
Thomas and Martin wanted to look at it, but didnt came up with a
solution - is there something fundamental which prevents it, or is it
simply a matter of time?
Deliberately releasing components that don't work with the RI does not seem like something that will increase the market acceptance of MyFaces components. Instead, this would create (or increase) a perception that MyFaces developers are not interested in compatibility. Also, given the fact that the RI has a 1.2 version available and MyFaces doesn't yet, it seems likely to give people a reason to consider switching away.
The best approach is to make the build system so powerful that running your entire test suite against either the MyFaces or RI implementations is a single command line parameter. Wendy's already done that for Shale (so we have no excuses for the framework not supporting both :-), and I'm sure it can be done for the component libraries too.
Ciao,
Mario
