George
> Regards,
> Oliver
>
> [1]
>
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/[EMAIL PROTECTED]
>
>
>>
>> Thanks for reading this far.
>>
>> Best regards,
>> George
>>
>>
>>
>> George Harley wrote:
>>> Hi,
>>>
>>> Just seen Tim's note on test support classes and it really
caught my
>>> attention as I have been mulling over this issue for a little while
>>> now. I think that it is a good time for us to return to the
topic of
>>> class library test layouts.
>>>
>>> The current proposal [1] sets out to segment our different types of
>>> test by placing them in different file locations. After looking at
>>> the recent changes to the LUNI module tests (where the layout
>>> guidelines were applied) I have a real concern that there are
>>> serious problems with this approach. We have started down a
track of
>>> just continually growing the number of test source folders as new
>>> categories of test are identified and IMHO that is going to bring
>>> complexity and maintenance issues with these tests.
>>>
>>> Consider the dimensions of tests that we have ...
>>>
>>> API
>>> Harmony-specific
>>> Platform-specific
>>> Run on classpath
>>> Run on bootclasspath
>>> Behaves different between Harmony and RI
>>> Stress
>>> ...and so on...
>>>
>>>
>>> If you weigh up all of the different possible permutations and then
>>> consider that the above list is highly likely to be extended as
>>> things progress it is obvious that we are eventually heading for
>>> large amounts of related test code scattered or possibly duplicated
>>> across numerous "hard wired" source directories. How
maintainable is
>>> that going to be ?
>>>
>>> If we want to run different tests in different configurations then
>>> IMHO we need to be thinking a whole lot smarter. We need to be
>>> thinking about keeping tests for specific areas of functionality
>>> together (thus easing maintenance); we need something quick and
>>> simple to re-configure if necessary (pushing whole directories of
>>> files around the place does not seem a particularly lightweight
>>> approach); and something that is not going to potentially mess up
>>> contributed patches when the file they patch is found to have been
>>> recently pushed from source folder A to B.
>>>
>>> To connect into another recent thread, there have been some posts
>>> lately about handling some test methods that fail on Harmony and
>>> have meant that entire test case classes have been excluded from
our
>>> test runs. I have also been noticing some API test methods that
pass
>>> fine on Harmony but fail when run against the RI. Are the different
>>> behaviours down to errors in the Harmony implementation ? An error
>>> in the RI implementation ? A bug in the RI Javadoc ? Only after
some
>>> investigation has been carried out do we know for sure. That takes
>>> time. What do we do with the test methods in the meantime ? Do we
>>> push them round the file system into yet another new source
folder ?
>>> IMHO we need a testing strategy that enables such "problem" methods
>>> to be tracked easily without disruption to the rest of the other
tests.
>>>
>>> A couple of weeks ago I mentioned that the TestNG framework [2]
>>> seemed like a reasonably good way of allowing us to both group
>>> together different kinds of tests and permit the exclusion of
>>> individual tests/groups of tests [3]. I would like to strongly
>>> propose that we consider using TestNG as a means of providing the
>>> different test configurations required by Harmony. Using a
>>> combination of annotations and XML to capture the kinds of
>>> sophisticated test configurations that people need, and that allows
>>> us to specify down to the individual method, has got to be more
>>> scalable and flexible than where we are headed now.
>>>
>>> Thanks for reading this far.
>>>
>>> Best regards,
>>> George
>>>
>>>
>>> [1]
>>>
http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html
>>>
>>> [2] http://testng.org
>>> [3]
>>>
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/[EMAIL PROTECTED]
>>>
>>>
>>>
>>
>>
>>
---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail:
[EMAIL PROTECTED]
>>
>>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
