I have opened Tuscany-1241 to provide the changes my team has made to the CTS. As is mentioned in the Jira, the patch is not entirely complete in and of itself and further may not reflect some of the changes that have gone in recently. However, I'm aware that changes are being made to the CTS and wanted to provide the patch quickly to minimize the complexity of the merge.
As per the above discussion, here is my two cents: Packaging: <kg>We should be able to apply the parameterization feature of junit without it being so prominent in the source code structure.</kg> I agree and this seems to be a consensus. The Use of Paramaterized Tests: <kg>As far as I can see our usage of the junit parameterized testing function is aimed at ensuring consistency between operations performed on graphs when the metadata has been produced a) from an xsd and b) by using the SDO API to create it dynamically. I propose to rehouse these under test.sdo21.consistency.</kg> <rjm>However, I don't think that this should be packaged under a consistency package - for me that has the same problems as being organized under paramatized where you do not get a feel for complete API coverage.</rjm> <fb>Verifying this by repeating every possible test on the exact same DataObjects, just created differently, sounds a little inefficient. What I think we need instead is to have some set of consistency tests that confirm that the types created in various ways are in fact the same.</fb> <kg>However, it's not clear to me whether we can now confirm the types are as they should be through the metadata API alone, or whether there are still requirements on an implementation to preserve elements of metadata that can only be detected by the behaviour seen through the data API. However, if it's purely some facets of XSD behaviour that we needed to test empirically then that wouldn't require a parameterized approach.</kg> My group has been running our version of the CTS and have located several instances of errors specific to one DataObject definiton/creation mechanism. Granted, these are generally specific to the static case. Given this experience, I agree with Kelvin that the running the parameterized tests on the DataObject API adds value. Creating tests that attempt to foresee such instances by working through the metadata API alone would seem to be another instance of "testing the testers" that Kelvin alluded to earlier. Miscellaneous: <fb>I also wonder why ParameterizedTestUtil has it's own equals code (instead of just using EqualityHelper). </fb> Kelvin addressed this, and he correctly guessed my intent when adding the equal code to ParamaterizedTestUtil. <fb>I noticed that the TestHelper is unnecessarily complicated. Instead of having all kinds of getXXXHelper() methods, it should just have one getHelperContext() method - that's the only method that is implementation dependant.</fb> This will be addressed in the resubmission of the patch for Tuscany-1241. It had been addressed in our version of the CTS, but I see that in the first version of the patch provided that is not reflected.
