subject:"Re\: \[classlib\] Testing conventions \- a proposal"




George Harley wrote:

Hi,

If annotations were to be used to help us categorise tests in order to 
simplify the definition of test configurations - what's included and 
excluded etc - then a core set of annotations would need to be agreed 
by the project. Consider the possibilities that the TestNG @Test 
annotation offers us in this respect.


First, if a test method was identified as being broken and needed to 
be excluded from all test runs while awaiting investigation then it 
would be a simple matter of setting its enabled field like this:


   @Test(enabled=false)
   public void myTest() {
   ...
   }

Temporarily disabling a test method in this way means that it can be 
left in its original class and we do not have to refer to it in any 
suite configuration (e.g. in the suite xml file).


If a test method was identified as being broken on a specific platform 
then we could make use of the groups field of the @Test type by 
making the method a member of a group that identifies its predicament. 
Something like this:


   @Test(groups={state.broken.win.IA32})
   public void myOtherTest() {
   ...
   }

The configuration for running tests on Windows would then specifically 
exclude any test method (or class) that was a member of that group.


Making a test method or type a member of a well-known group 
(well-known in the sense that the name and meaning has been agreed 
within the project) is essentially adding some descriptive attributes 
to the test. Like adjectives (the groups) and nouns (the tests) in the 
English language. To take another example, if there was a test class 
that contained methods only intended to be run on Windows and that 
were all specific to Harmony (i.e. not API tests) then  one could 
envisage the following kind of annotation:



@Test(groups={type.impl, os.win.IA32})
public class MyTestClass {

   public void testOne() {
   ...
   }

   public void testTwo() {
   ...
   }

   @Test(enabled=false)
   public void brokenTest() {
   ...
   }
}

Here the annotation on MyTestClass applies to all of its test methods.

So what are the well-known TestNG groups that we could define for use 
inside Harmony ? Here are some of my initial thoughts:



* type.impl  --  tests that are specific to Harmony

* state.broken.platform id  --  tests bust on a specific platform

* state.broken  --  tests broken on every platform but we want to 
decide whether or not to run from our suite configuration


* os.platform id  --  tests that are to be run only on the specified 
platform (a test could be member of more than one of these)



What does everyone else think ? Does such a scheme sound reasonable ?

Just one question: What's the default test annotation? I mean the 
successful api tests which will be run on every platform. Thanks a lot.


Best regards,
Richard

Thanks for reading this far.

Best regards,
George



George Harley wrote:

Hi,

Just seen Tim's note on test support classes and it really caught my 
attention as I have been mulling over this issue for a little while 
now. I think that it is a good time for us to return to the topic of 
class library test layouts.


The current proposal [1] sets out to segment our different types of 
test by placing them in different file locations. After looking at 
the recent changes to the LUNI module tests (where the layout 
guidelines were applied) I have a real concern that there are serious 
problems with this approach. We have started down a track of just 
continually growing the number of test source folders as new 
categories of test are identified and IMHO that is going to bring 
complexity and maintenance issues with these tests.


Consider the dimensions of tests that we have ...

API
Harmony-specific
Platform-specific
Run on classpath
Run on bootclasspath
Behaves different between Harmony and RI
Stress
...and so on...


If you weigh up all of the different possible permutations and then 
consider that the above list is highly likely to be extended as 
things progress it is obvious that we are eventually heading for 
large amounts of related test code scattered or possibly duplicated 
across numerous hard wired source directories. How maintainable is 
that going to be ?


If we want to run different tests in different configurations then 
IMHO we need to be thinking a whole lot smarter. We need to be 
thinking about keeping tests for specific areas of functionality 
together (thus easing maintenance); we need something quick and 
simple to re-configure if necessary (pushing whole directories of 
files around the place does not seem a particularly lightweight 
approach); and something that is not going to potentially mess up 
contributed patches when the file they patch is found to have been 
recently pushed from source folder A to B.


To connect into another recent thread, there have been some posts 
lately about handling some test methods that fail on Harmony and have 
meant that entire test case classes have been

Re: [classlib] Testing conventions - a proposal

According to TestNG Ant Task [1], it seems that the TestNG Ant task 
does not support to fork a new JVM, that is, we must launch ant using 
Harmony itself. Any comments? Thanks a lot.


[1]http://testng.org/doc/ant.html

Best regards,
Richard

George Harley wrote:

Andrew Zhang wrote:

On 7/18/06, George Harley [EMAIL PROTECTED] wrote:


Oliver Deakin wrote:
 George Harley wrote:
 SNIP!

 Here the annotation on MyTestClass applies to all of its test 
methods.


 So what are the well-known TestNG groups that we could define for 
use

 inside Harmony ? Here are some of my initial thoughts:


 * type.impl  --  tests that are specific to Harmony

 So tests are implicitly API unless specified otherwise?

 I'm slightly confused by your definition of impl tests as tests that
are
 specific to Harmony. Does this mean that impl tests are only
 those that test classes in org.apache.harmony packages?
 I thought that impl was our way of saying tests that need to go on
 the bootclasspath.

 I think I just need a little clarification...


Hi Oliver,

I was using the definition of implementation-specific tests that we
currently have on the Harmony testing conventions web page. That is,
implementation-specific tests are those that are dependent on some
aspect of the Harmony implementation and would therefore not pass when
run against the RI or other conforming implementations. It's orthogonal
to the classpath/bootclasspath issue.


 * state.broken.platform id  --  tests bust on a specific platform

 * state.broken  --  tests broken on every platform but we want to
 decide whether or not to run from our suite configuration

 * os.platform id  --  tests that are to be run only on the
 specified platform (a test could be member of more than one of 
these)


 And the defaults for these are an unbroken state and runs on any
 platform.
 That makes sense...

 Will the platform ids be organised in a similar way to the 
platform ids

 we've discussed before for organisation of native code [1]?


The actual string used to identify a particular platform can be 
whatever

we want it to be, just so long as we are consistent. So, yes, the ids
mentioned in the referenced email would seem a good starting point. Do
we need to include a 32-bit/64-bit identifier ?


 So all tests are, by default, in an all-platforms (or shared) group.
 If a test fails on all Windows platforms, it is marked with
 state.broken.windows.
 If a test fails on Windows but only on, say, amd hardware,
 it is marked state.broken.windows.amd.


Yes. Agreed.


 Then when you come to run tests on your windows amd machine,
 you want to include all tests in the all-platform (shared) group,
 os.windows and os.windows.amd, and exclude all tests in
 the state.broken, state.broken.windows and state.broken.windows.amd
 groups.

 Does this tally with what you were thinking?


Yes, that is the idea.




 What does everyone else think ? Does such a scheme sound 
reasonable ?


 I think so - it seems to cover our current requirements. Thanks for
 coming up with this!


Thanks, but I don't see it as final yet really. It would be great to
prove the worth of this by doing a trial on one of the existing 
modules,

ideally something that contains tests that are platform-specific.



Hello George, how about doing a trial on NIO module?

So far as I know, there are several platform dependent tests in NIO 
module.

:)

The assert statements are commented out in these tests, with FIXME 
mark.


Furthurmore, I also find some platform dependent behaviours of 
FileChannel.
If TestNG is applied on NIO, I will supplement new tests for 
FileChannel and

fix the bug of source code.

What's your opnion? Any suggestions/comments?

Thanks!



Hi Andrew,

That sounds like a very good idea. If there is agreement in the 
project that 5.0 annotations are the way to go (as opposed to the 
pre-5.0 Javadoc comment support offered by TestNG) then to the best of 
my knowledge all that is stopping us from doing this trial is the lack 
of a 5.0 VM to run the Harmony tests on. Hopefully that will be 
addressed soon. When it is I would be happy to get stuck into this trial.


Best regards,
George



Best regards,

George


 Regards,
 Oliver

 [1]

http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/[EMAIL PROTECTED] 





 Thanks for reading this far.

 Best regards,
 George



 George Harley wrote:
 Hi,

 Just seen Tim's note on test support classes and it really 
caught my

 attention as I have been mulling over this issue for a little while
 now. I think that it is a good time for us to return to the 
topic of

 class library test layouts.

 The current proposal [1] sets out to segment our different types of
 test by placing them in different file locations. After looking at
 the recent changes to the LUNI module tests (where the layout
 guidelines were applied) I have a real concern that there are
 serious problems with this approach. We have started down a 
track of

 just continually

Re: [classlib] Testing conventions - a proposal

2006-07-19 Thread Alexei Zakharov


Hmm, do we have problems with launching ant? I thought we have
problems with launching TestNG. Just checked - running tests for beans
on j9+fresh classlib works fine. I.e.
ant -Dbuild.module=beans
-Dbuild.compiler=org.eclipse.jdt.core.JDTCompilerAdapter test

2006/7/19, Richard Liang [EMAIL PROTECTED]:

According to TestNG Ant Task [1], it seems that the TestNG Ant task
does not support to fork a new JVM, that is, we must launch ant using
Harmony itself. Any comments? Thanks a lot.

[1]http://testng.org/doc/ant.html

Best regards,
Richard

George Harley wrote:
 Andrew Zhang wrote:
 On 7/18/06, George Harley [EMAIL PROTECTED] wrote:

 Oliver Deakin wrote:
  George Harley wrote:
  SNIP!
 
  Here the annotation on MyTestClass applies to all of its test
 methods.
 
  So what are the well-known TestNG groups that we could define for
 use
  inside Harmony ? Here are some of my initial thoughts:
 
 
  * type.impl  --  tests that are specific to Harmony
 
  So tests are implicitly API unless specified otherwise?
 
  I'm slightly confused by your definition of impl tests as tests that
 are
  specific to Harmony. Does this mean that impl tests are only
  those that test classes in org.apache.harmony packages?
  I thought that impl was our way of saying tests that need to go on
  the bootclasspath.
 
  I think I just need a little clarification...
 

 Hi Oliver,

 I was using the definition of implementation-specific tests that we
 currently have on the Harmony testing conventions web page. That is,
 implementation-specific tests are those that are dependent on some
 aspect of the Harmony implementation and would therefore not pass when
 run against the RI or other conforming implementations. It's orthogonal
 to the classpath/bootclasspath issue.


  * state.broken.platform id  --  tests bust on a specific platform
 
  * state.broken  --  tests broken on every platform but we want to
  decide whether or not to run from our suite configuration
 
  * os.platform id  --  tests that are to be run only on the
  specified platform (a test could be member of more than one of
 these)
 
  And the defaults for these are an unbroken state and runs on any
  platform.
  That makes sense...
 
  Will the platform ids be organised in a similar way to the
 platform ids
  we've discussed before for organisation of native code [1]?
 

 The actual string used to identify a particular platform can be
 whatever
 we want it to be, just so long as we are consistent. So, yes, the ids
 mentioned in the referenced email would seem a good starting point. Do
 we need to include a 32-bit/64-bit identifier ?


  So all tests are, by default, in an all-platforms (or shared) group.
  If a test fails on all Windows platforms, it is marked with
  state.broken.windows.
  If a test fails on Windows but only on, say, amd hardware,
  it is marked state.broken.windows.amd.
 

 Yes. Agreed.


  Then when you come to run tests on your windows amd machine,
  you want to include all tests in the all-platform (shared) group,
  os.windows and os.windows.amd, and exclude all tests in
  the state.broken, state.broken.windows and state.broken.windows.amd
  groups.
 
  Does this tally with what you were thinking?
 

 Yes, that is the idea.


 
 
  What does everyone else think ? Does such a scheme sound
 reasonable ?
 
  I think so - it seems to cover our current requirements. Thanks for
  coming up with this!
 

 Thanks, but I don't see it as final yet really. It would be great to
 prove the worth of this by doing a trial on one of the existing
 modules,
 ideally something that contains tests that are platform-specific.


 Hello George, how about doing a trial on NIO module?

 So far as I know, there are several platform dependent tests in NIO
 module.
 :)

 The assert statements are commented out in these tests, with FIXME
 mark.

 Furthurmore, I also find some platform dependent behaviours of
 FileChannel.
 If TestNG is applied on NIO, I will supplement new tests for
 FileChannel and
 fix the bug of source code.

 What's your opnion? Any suggestions/comments?

 Thanks!


 Hi Andrew,

 That sounds like a very good idea. If there is agreement in the
 project that 5.0 annotations are the way to go (as opposed to the
 pre-5.0 Javadoc comment support offered by TestNG) then to the best of
 my knowledge all that is stopping us from doing this trial is the lack
 of a 5.0 VM to run the Harmony tests on. Hopefully that will be
 addressed soon. When it is I would be happy to get stuck into this trial.

 Best regards,
 George


 Best regards,
 George


  Regards,
  Oliver
 
  [1]
 
 
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/[EMAIL 
PROTECTED]

 
 
 
  Thanks for reading this far.
 
  Best regards,
  George
 
 
 
  George Harley wrote:
  Hi,
 
  Just seen Tim's note on test support classes and it really
 caught my
  attention as I have been mulling over this issue for a little while
  now. I think that it is a good time for us to return to

Re: [classlib] Testing conventions - a proposal


Just thinking about using TestNG to execute Harmony test cases.  :-)

Look at our build.xml (e.g., modules/luni/build.xml), you will see 
something like:

..
junit fork=yes
  forkmode=once
  printsummary=withOutAndErr
  errorproperty=test.errors
  failureproperty=test.failures
  showoutput=on
  dir=${hy.luni.bin.test}
  jvm=${test.jre.home}/bin/java

   jvmarg value=-showversion /
..

My question is the TestNG Ant task testng does not support attributes 
fork and jvm, how to run our test under Harmony VM? Thanks a lot.


Best regards,
Richard

Alexei Zakharov wrote:

Hmm, do we have problems with launching ant? I thought we have
problems with launching TestNG. Just checked - running tests for beans
on j9+fresh classlib works fine. I.e.
ant -Dbuild.module=beans
-Dbuild.compiler=org.eclipse.jdt.core.JDTCompilerAdapter test

2006/7/19, Richard Liang [EMAIL PROTECTED]:

According to TestNG Ant Task [1], it seems that the TestNG Ant task
does not support to fork a new JVM, that is, we must launch ant using
Harmony itself. Any comments? Thanks a lot.

[1]http://testng.org/doc/ant.html

Best regards,
Richard

George Harley wrote:
 Andrew Zhang wrote:
 On 7/18/06, George Harley [EMAIL PROTECTED] wrote:

 Oliver Deakin wrote:
  George Harley wrote:
  SNIP!
 
  Here the annotation on MyTestClass applies to all of its test
 methods.
 
  So what are the well-known TestNG groups that we could define for
 use
  inside Harmony ? Here are some of my initial thoughts:
 
 
  * type.impl  --  tests that are specific to Harmony
 
  So tests are implicitly API unless specified otherwise?
 
  I'm slightly confused by your definition of impl tests as 
tests that

 are
  specific to Harmony. Does this mean that impl tests are only
  those that test classes in org.apache.harmony packages?
  I thought that impl was our way of saying tests that need to 
go on

  the bootclasspath.
 
  I think I just need a little clarification...
 

 Hi Oliver,

 I was using the definition of implementation-specific tests that we
 currently have on the Harmony testing conventions web page. That is,
 implementation-specific tests are those that are dependent on some
 aspect of the Harmony implementation and would therefore not pass 
when
 run against the RI or other conforming implementations. It's 
orthogonal

 to the classpath/bootclasspath issue.


  * state.broken.platform id  --  tests bust on a specific 
platform

 
  * state.broken  --  tests broken on every platform but we want to
  decide whether or not to run from our suite configuration
 
  * os.platform id  --  tests that are to be run only on the
  specified platform (a test could be member of more than one of
 these)
 
  And the defaults for these are an unbroken state and runs on any
  platform.
  That makes sense...
 
  Will the platform ids be organised in a similar way to the
 platform ids
  we've discussed before for organisation of native code [1]?
 

 The actual string used to identify a particular platform can be
 whatever
 we want it to be, just so long as we are consistent. So, yes, the 
ids
 mentioned in the referenced email would seem a good starting 
point. Do

 we need to include a 32-bit/64-bit identifier ?


  So all tests are, by default, in an all-platforms (or shared) 
group.

  If a test fails on all Windows platforms, it is marked with
  state.broken.windows.
  If a test fails on Windows but only on, say, amd hardware,
  it is marked state.broken.windows.amd.
 

 Yes. Agreed.


  Then when you come to run tests on your windows amd machine,
  you want to include all tests in the all-platform (shared) group,
  os.windows and os.windows.amd, and exclude all tests in
  the state.broken, state.broken.windows and 
state.broken.windows.amd

  groups.
 
  Does this tally with what you were thinking?
 

 Yes, that is the idea.


 
 
  What does everyone else think ? Does such a scheme sound
 reasonable ?
 
  I think so - it seems to cover our current requirements. Thanks 
for

  coming up with this!
 

 Thanks, but I don't see it as final yet really. It would be great to
 prove the worth of this by doing a trial on one of the existing
 modules,
 ideally something that contains tests that are platform-specific.


 Hello George, how about doing a trial on NIO module?

 So far as I know, there are several platform dependent tests in NIO
 module.
 :)

 The assert statements are commented out in these tests, with FIXME
 mark.

 Furthurmore, I also find some platform dependent behaviours of
 FileChannel.
 If TestNG is applied on NIO, I will supplement new tests for
 FileChannel and
 fix the bug of source code.

 What's your opnion? Any suggestions/comments?

 Thanks!


 Hi Andrew,

 That sounds like a very good idea. If there is agreement in the
 project that 5.0 annotations are the way to go (as opposed to the
 pre-5.0 Javadoc comment

Re: [classlib] Testing conventions - a proposal


Richard Liang wrote:



George Harley wrote:

Hi,

If annotations were to be used to help us categorise tests in order 
to simplify the definition of test configurations - what's included 
and excluded etc - then a core set of annotations would need to be 
agreed by the project. Consider the possibilities that the TestNG 
@Test annotation offers us in this respect.


First, if a test method was identified as being broken and needed to 
be excluded from all test runs while awaiting investigation then it 
would be a simple matter of setting its enabled field like this:


   @Test(enabled=false)
   public void myTest() {
   ...
   }

Temporarily disabling a test method in this way means that it can be 
left in its original class and we do not have to refer to it in any 
suite configuration (e.g. in the suite xml file).


If a test method was identified as being broken on a specific 
platform then we could make use of the groups field of the @Test 
type by making the method a member of a group that identifies its 
predicament. Something like this:


   @Test(groups={state.broken.win.IA32})
   public void myOtherTest() {
   ...
   }

The configuration for running tests on Windows would then 
specifically exclude any test method (or class) that was a member of 
that group.


Making a test method or type a member of a well-known group 
(well-known in the sense that the name and meaning has been agreed 
within the project) is essentially adding some descriptive attributes 
to the test. Like adjectives (the groups) and nouns (the tests) in 
the English language. To take another example, if there was a test 
class that contained methods only intended to be run on Windows and 
that were all specific to Harmony (i.e. not API tests) then  one 
could envisage the following kind of annotation:



@Test(groups={type.impl, os.win.IA32})
public class MyTestClass {

   public void testOne() {
   ...
   }

   public void testTwo() {
   ...
   }

   @Test(enabled=false)
   public void brokenTest() {
   ...
   }
}

Here the annotation on MyTestClass applies to all of its test methods.

So what are the well-known TestNG groups that we could define for use 
inside Harmony ? Here are some of my initial thoughts:



* type.impl  --  tests that are specific to Harmony

* state.broken.platform id  --  tests bust on a specific platform

* state.broken  --  tests broken on every platform but we want to 
decide whether or not to run from our suite configuration


* os.platform id  --  tests that are to be run only on the 
specified platform (a test could be member of more than one of these)



What does everyone else think ? Does such a scheme sound reasonable ?

Just one question: What's the default test annotation? I mean the 
successful api tests which will be run on every platform. Thanks a lot.


Best regards,
Richard


Hi Richard,

I think that just the basic @Test annotation on its own will suffice. 
Any better suggestions are welcome.


Best regards,
George




Thanks for reading this far.

Best regards,
George



George Harley wrote:

Hi,

Just seen Tim's note on test support classes and it really caught my 
attention as I have been mulling over this issue for a little while 
now. I think that it is a good time for us to return to the topic of 
class library test layouts.


The current proposal [1] sets out to segment our different types of 
test by placing them in different file locations. After looking at 
the recent changes to the LUNI module tests (where the layout 
guidelines were applied) I have a real concern that there are 
serious problems with this approach. We have started down a track of 
just continually growing the number of test source folders as new 
categories of test are identified and IMHO that is going to bring 
complexity and maintenance issues with these tests.


Consider the dimensions of tests that we have ...

API
Harmony-specific
Platform-specific
Run on classpath
Run on bootclasspath
Behaves different between Harmony and RI
Stress
...and so on...


If you weigh up all of the different possible permutations and then 
consider that the above list is highly likely to be extended as 
things progress it is obvious that we are eventually heading for 
large amounts of related test code scattered or possibly duplicated 
across numerous hard wired source directories. How maintainable is 
that going to be ?


If we want to run different tests in different configurations then 
IMHO we need to be thinking a whole lot smarter. We need to be 
thinking about keeping tests for specific areas of functionality 
together (thus easing maintenance); we need something quick and 
simple to re-configure if necessary (pushing whole directories of 
files around the place does not seem a particularly lightweight 
approach); and something that is not going to potentially mess up 
contributed patches when the file they patch is found to have been 
recently pushed from source folder A to B.


To connect into

Re: [classlib] Testing conventions - a proposal

Hi Richard,

Actually the Ant task always runs the tests in a forked VM. At present,
however, the task does not support specifying the forked VM (i.e. there
is no equivalent to the JUnit Ant task's jvm attribute). This matter
has already been raised with the TestNG folks who seem happy to
introduce this.

In the meantime we could run the tests using the Ant java task.

Best regards,
George

Richard Liang wrote:
According to TestNG Ant Task [1], it seems that the TestNG Ant task
does not support to fork a new JVM, that is, we must launch ant using
Harmony itself. Any comments? Thanks a lot.

[1]http://testng.org/doc/ant.html

Best regards,
Richard

George Harley wrote:

Andrew Zhang wrote:

On 7/18/06, George Harley [EMAIL PROTECTED] wrote:

Oliver Deakin wrote:
George Harley wrote:
SNIP!

Here the annotation on MyTestClass applies to all of its test
methods.

So what are the well-known TestNG groups that we could define
for use

inside Harmony ? Here are some of my initial thoughts:

* type.impl -- tests that are specific to Harmony

So tests are implicitly API unless specified otherwise?

I'm slightly confused by your definition of impl tests as tests
that

are
specific to Harmony. Does this mean that impl tests are only
those that test classes in org.apache.harmony packages?
I thought that impl was our way of saying tests that need to go on
the bootclasspath.

I think I just need a little clarification...

Hi Oliver,

I was using the definition of implementation-specific tests that we
currently have on the Harmony testing conventions web page. That is,
implementation-specific tests are those that are dependent on some
aspect of the Harmony implementation and would therefore not pass when
run against the RI or other conforming implementations. It's
orthogonal

to the classpath/bootclasspath issue.

* state.broken.platform id -- tests bust on a specific platform

* state.broken -- tests broken on every platform but we want to
decide whether or not to run from our suite configuration

* os.platform id -- tests that are to be run only on the
specified platform (a test could be member of more than one of
these)

And the defaults for these are an unbroken state and runs on any
platform.
That makes sense...

Will the platform ids be organised in a similar way to the
platform ids

we've discussed before for organisation of native code [1]?

The actual string used to identify a particular platform can be
whatever

we want it to be, just so long as we are consistent. So, yes, the ids
mentioned in the referenced email would seem a good starting point. Do
we need to include a 32-bit/64-bit identifier ?

So all tests are, by default, in an all-platforms (or shared) group.
If a test fails on all Windows platforms, it is marked with
state.broken.windows.
If a test fails on Windows but only on, say, amd hardware,
it is marked state.broken.windows.amd.

Yes. Agreed.

Then when you come to run tests on your windows amd machine,
you want to include all tests in the all-platform (shared) group,
os.windows and os.windows.amd, and exclude all tests in
the state.broken, state.broken.windows and state.broken.windows.amd
groups.

Does this tally with what you were thinking?

Yes, that is the idea.

What does everyone else think ? Does such a scheme sound
reasonable ?

I think so - it seems to cover our current requirements. Thanks for
coming up with this!

Thanks, but I don't see it as final yet really. It would be great to
prove the worth of this by doing a trial on one of the existing
modules,

ideally something that contains tests that are platform-specific.

Hello George, how about doing a trial on NIO module?

So far as I know, there are several platform dependent tests in NIO
module.

The assert statements are commented out in these tests, with FIXME
mark.

Furthurmore, I also find some platform dependent behaviours of
FileChannel.
If TestNG is applied on NIO, I will supplement new tests for
FileChannel and

fix the bug of source code.

What's your opnion? Any suggestions/comments?

Thanks!

Hi Andrew,

That sounds like a very good idea. If there is agreement in the
project that 5.0 annotations are the way to go (as opposed to the
pre-5.0 Javadoc comment support offered by TestNG) then to the best
of my knowledge all that is stopping us from doing this trial is the
lack of a 5.0 VM to run the Harmony tests on. Hopefully that will be
addressed soon. When it is I would be happy to get stuck into this
trial.

Best regards,
George

Best regards,

George

Regards,
Oliver

[1]

http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/[EMAIL PROTECTED]

Thanks for reading this far.

Best regards,
George

George Harley wrote:
Hi,

Just seen Tim's note on test support classes and it really
caught my
attention as I have been mulling over this issue for a little
while
now. I think that it is

Re: [classlib] Testing conventions - a proposal

2006-07-19 Thread Alexei Zakharov


Probably my previous message was not clear enough.
Why can't we just invoke everything including ant on top of Harmony
for now? At least I was able to build and run test-14 examples from
TestNG 4.7 distribution solely on top of j9 + our classlib today.

C:\Java\testng-4.7\test-14set JAVA_HOME=c:\Java\harmony\enhanced\classlib\trunk
\deploy\jdk\jre

C:\Java\testng-4.7\test-14ant -Dbuild.compiler=org.eclipse.jdt.core.JDTCompiler
Adapter run
Buildfile: build.xml

prepare:

compile:
[echo]  -- Compiling JDK 1.4 tests --

run:
[echo]  -- Running JDK 1.4 tests   --
[echo]  -- testng-4.7-jdk14.jar  --

[testng-14] ===
[testng-14] TestNG JDK 1.4
[testng-14] Total tests run: 179, Failures: 10, Skips: 0
[testng-14] ===
...

Exactly the same results as with Sun JDK 1.4.
Note: you may need to hatch the build.xml a little bit to achieve this.

Thanks,

2006/7/19, George Harley [EMAIL PROTECTED]:

Hi Richard,

Actually the Ant task always runs the tests in a forked VM. At present,
however, the task does not support specifying the forked VM (i.e. there
is no equivalent to the JUnit Ant task's jvm attribute). This matter
has already been raised with the TestNG folks who seem happy to
introduce this.

In the meantime we could run the tests using the Ant java task.


Best regards,
George



Richard Liang wrote:
 According to TestNG Ant Task [1], it seems that the TestNG Ant task
 does not support to fork a new JVM, that is, we must launch ant using
 Harmony itself. Any comments? Thanks a lot.

 [1]http://testng.org/doc/ant.html

 Best regards,
 Richard

 George Harley wrote:
 Andrew Zhang wrote:
 On 7/18/06, George Harley [EMAIL PROTECTED] wrote:

 Oliver Deakin wrote:
  George Harley wrote:
  SNIP!
 
  Here the annotation on MyTestClass applies to all of its test
 methods.
 
  So what are the well-known TestNG groups that we could define
 for use
  inside Harmony ? Here are some of my initial thoughts:
 
 
  * type.impl  --  tests that are specific to Harmony
 
  So tests are implicitly API unless specified otherwise?
 
  I'm slightly confused by your definition of impl tests as tests
 that
 are
  specific to Harmony. Does this mean that impl tests are only
  those that test classes in org.apache.harmony packages?
  I thought that impl was our way of saying tests that need to go on
  the bootclasspath.
 
  I think I just need a little clarification...
 

 Hi Oliver,

 I was using the definition of implementation-specific tests that we
 currently have on the Harmony testing conventions web page. That is,
 implementation-specific tests are those that are dependent on some
 aspect of the Harmony implementation and would therefore not pass when
 run against the RI or other conforming implementations. It's
 orthogonal
 to the classpath/bootclasspath issue.


  * state.broken.platform id  --  tests bust on a specific platform
 
  * state.broken  --  tests broken on every platform but we want to
  decide whether or not to run from our suite configuration
 
  * os.platform id  --  tests that are to be run only on the
  specified platform (a test could be member of more than one of
 these)
 
  And the defaults for these are an unbroken state and runs on any
  platform.
  That makes sense...
 
  Will the platform ids be organised in a similar way to the
 platform ids
  we've discussed before for organisation of native code [1]?
 

 The actual string used to identify a particular platform can be
 whatever
 we want it to be, just so long as we are consistent. So, yes, the ids
 mentioned in the referenced email would seem a good starting point. Do
 we need to include a 32-bit/64-bit identifier ?


  So all tests are, by default, in an all-platforms (or shared) group.
  If a test fails on all Windows platforms, it is marked with
  state.broken.windows.
  If a test fails on Windows but only on, say, amd hardware,
  it is marked state.broken.windows.amd.
 

 Yes. Agreed.


  Then when you come to run tests on your windows amd machine,
  you want to include all tests in the all-platform (shared) group,
  os.windows and os.windows.amd, and exclude all tests in
  the state.broken, state.broken.windows and state.broken.windows.amd
  groups.
 
  Does this tally with what you were thinking?
 

 Yes, that is the idea.


 
 
  What does everyone else think ? Does such a scheme sound
 reasonable ?
 
  I think so - it seems to cover our current requirements. Thanks for
  coming up with this!
 

 Thanks, but I don't see it as final yet really. It would be great to
 prove the worth of this by doing a trial on one of the existing
 modules,
 ideally something that contains tests that are platform-specific.


 Hello George, how about doing a trial on NIO module?

 So far as I know, there are several platform dependent tests in NIO
 module.
 :)

 The assert

Re: [classlib] Testing conventions - a proposal


Hi Alexei,

It's encouraging to hear that (Ant + TestNG + sample tests) all worked 
fine together on Harmony. In answer to your question I suppose that the 
ability to fork the tests in a separate VM means that we do not run the 
risk of possible bugs in Harmony affecting the test harness and 
therefore the outcome of the tests.


Best regards,
George


Alexei Zakharov wrote:

Probably my previous message was not clear enough.
Why can't we just invoke everything including ant on top of Harmony
for now? At least I was able to build and run test-14 examples from
TestNG 4.7 distribution solely on top of j9 + our classlib today.

C:\Java\testng-4.7\test-14set 
JAVA_HOME=c:\Java\harmony\enhanced\classlib\trunk

\deploy\jdk\jre

C:\Java\testng-4.7\test-14ant 
-Dbuild.compiler=org.eclipse.jdt.core.JDTCompiler

Adapter run
Buildfile: build.xml

prepare:

compile:
[echo]  -- Compiling JDK 1.4 tests --

run:
[echo]  -- Running JDK 1.4 tests   --
[echo]  -- testng-4.7-jdk14.jar  --

[testng-14] ===
[testng-14] TestNG JDK 1.4
[testng-14] Total tests run: 179, Failures: 10, Skips: 0
[testng-14] ===
...

Exactly the same results as with Sun JDK 1.4.
Note: you may need to hatch the build.xml a little bit to achieve this.

Thanks,

2006/7/19, George Harley [EMAIL PROTECTED]:

Hi Richard,

Actually the Ant task always runs the tests in a forked VM. At present,
however, the task does not support specifying the forked VM (i.e. there
is no equivalent to the JUnit Ant task's jvm attribute). This matter
has already been raised with the TestNG folks who seem happy to
introduce this.

In the meantime we could run the tests using the Ant java task.


Best regards,
George



Richard Liang wrote:
 According to TestNG Ant Task [1], it seems that the TestNG Ant task
 does not support to fork a new JVM, that is, we must launch ant using
 Harmony itself. Any comments? Thanks a lot.

 [1]http://testng.org/doc/ant.html

 Best regards,
 Richard

 George Harley wrote:
 Andrew Zhang wrote:
 On 7/18/06, George Harley [EMAIL PROTECTED] wrote:

 Oliver Deakin wrote:
  George Harley wrote:
  SNIP!
 
  Here the annotation on MyTestClass applies to all of its test
 methods.
 
  So what are the well-known TestNG groups that we could define
 for use
  inside Harmony ? Here are some of my initial thoughts:
 
 
  * type.impl  --  tests that are specific to Harmony
 
  So tests are implicitly API unless specified otherwise?
 
  I'm slightly confused by your definition of impl tests as tests
 that
 are
  specific to Harmony. Does this mean that impl tests are only
  those that test classes in org.apache.harmony packages?
  I thought that impl was our way of saying tests that need to 
go on

  the bootclasspath.
 
  I think I just need a little clarification...
 

 Hi Oliver,

 I was using the definition of implementation-specific tests that we
 currently have on the Harmony testing conventions web page. That 
is,

 implementation-specific tests are those that are dependent on some
 aspect of the Harmony implementation and would therefore not 
pass when

 run against the RI or other conforming implementations. It's
 orthogonal
 to the classpath/bootclasspath issue.


  * state.broken.platform id  --  tests bust on a specific 
platform

 
  * state.broken  --  tests broken on every platform but we 
want to

  decide whether or not to run from our suite configuration
 
  * os.platform id  --  tests that are to be run only on the
  specified platform (a test could be member of more than one of
 these)
 
  And the defaults for these are an unbroken state and runs on any
  platform.
  That makes sense...
 
  Will the platform ids be organised in a similar way to the
 platform ids
  we've discussed before for organisation of native code [1]?
 

 The actual string used to identify a particular platform can be
 whatever
 we want it to be, just so long as we are consistent. So, yes, 
the ids
 mentioned in the referenced email would seem a good starting 
point. Do

 we need to include a 32-bit/64-bit identifier ?


  So all tests are, by default, in an all-platforms (or shared) 
group.

  If a test fails on all Windows platforms, it is marked with
  state.broken.windows.
  If a test fails on Windows but only on, say, amd hardware,
  it is marked state.broken.windows.amd.
 

 Yes. Agreed.


  Then when you come to run tests on your windows amd machine,
  you want to include all tests in the all-platform (shared) group,
  os.windows and os.windows.amd, and exclude all tests in
  the state.broken, state.broken.windows and 
state.broken.windows.amd

  groups.
 
  Does this tally with what you were thinking?
 

 Yes, that is the idea.


 
 
  What does everyone else think ? Does such a scheme sound
 reasonable ?
 
  I think so - it seems to cover our current requirements.

Re: [classlib] Testing conventions - a proposal

2006-07-19 Thread Alexei Zakharov


Hi George,

Agree, we may experience problems in case of VM hang or crash. I
suggest this only as a temporary solution. BTW, the fact that TestNG
ant task still doesn't have such attributes looks like a sign for me -
TestNG can be still immature in some aspects. Still comparing TestNG
and JUnit.

Regards,

2006/7/19, George Harley [EMAIL PROTECTED]:

Hi Alexei,

It's encouraging to hear that (Ant + TestNG + sample tests) all worked
fine together on Harmony. In answer to your question I suppose that the
ability to fork the tests in a separate VM means that we do not run the
risk of possible bugs in Harmony affecting the test harness and
therefore the outcome of the tests.

Best regards,
George


Alexei Zakharov wrote:
 Probably my previous message was not clear enough.
 Why can't we just invoke everything including ant on top of Harmony
 for now? At least I was able to build and run test-14 examples from
 TestNG 4.7 distribution solely on top of j9 + our classlib today.

 C:\Java\testng-4.7\test-14set
 JAVA_HOME=c:\Java\harmony\enhanced\classlib\trunk
 \deploy\jdk\jre

 C:\Java\testng-4.7\test-14ant
 -Dbuild.compiler=org.eclipse.jdt.core.JDTCompiler
 Adapter run
 Buildfile: build.xml

 prepare:

 compile:
 [echo]  -- Compiling JDK 1.4 tests --

 run:
 [echo]  -- Running JDK 1.4 tests   --
 [echo]  -- testng-4.7-jdk14.jar  --

 [testng-14] ===
 [testng-14] TestNG JDK 1.4
 [testng-14] Total tests run: 179, Failures: 10, Skips: 0
 [testng-14] ===
 ...

 Exactly the same results as with Sun JDK 1.4.
 Note: you may need to hatch the build.xml a little bit to achieve this.

 Thanks,

 2006/7/19, George Harley [EMAIL PROTECTED]:
 Hi Richard,

 Actually the Ant task always runs the tests in a forked VM. At present,
 however, the task does not support specifying the forked VM (i.e. there
 is no equivalent to the JUnit Ant task's jvm attribute). This matter
 has already been raised with the TestNG folks who seem happy to
 introduce this.

 In the meantime we could run the tests using the Ant java task.


 Best regards,
 George



 Richard Liang wrote:
  According to TestNG Ant Task [1], it seems that the TestNG Ant task
  does not support to fork a new JVM, that is, we must launch ant using
  Harmony itself. Any comments? Thanks a lot.
 
  [1]http://testng.org/doc/ant.html
 
  Best regards,
  Richard
 
  George Harley wrote:
  Andrew Zhang wrote:
  On 7/18/06, George Harley [EMAIL PROTECTED] wrote:
 
  Oliver Deakin wrote:
   George Harley wrote:
   SNIP!
  
   Here the annotation on MyTestClass applies to all of its test
  methods.
  
   So what are the well-known TestNG groups that we could define
  for use
   inside Harmony ? Here are some of my initial thoughts:
  
  
   * type.impl  --  tests that are specific to Harmony
  
   So tests are implicitly API unless specified otherwise?
  
   I'm slightly confused by your definition of impl tests as tests
  that
  are
   specific to Harmony. Does this mean that impl tests are only
   those that test classes in org.apache.harmony packages?
   I thought that impl was our way of saying tests that need to
 go on
   the bootclasspath.
  
   I think I just need a little clarification...
  
 
  Hi Oliver,
 
  I was using the definition of implementation-specific tests that we
  currently have on the Harmony testing conventions web page. That
 is,
  implementation-specific tests are those that are dependent on some
  aspect of the Harmony implementation and would therefore not
 pass when
  run against the RI or other conforming implementations. It's
  orthogonal
  to the classpath/bootclasspath issue.
 
 
   * state.broken.platform id  --  tests bust on a specific
 platform
  
   * state.broken  --  tests broken on every platform but we
 want to
   decide whether or not to run from our suite configuration
  
   * os.platform id  --  tests that are to be run only on the
   specified platform (a test could be member of more than one of
  these)
  
   And the defaults for these are an unbroken state and runs on any
   platform.
   That makes sense...
  
   Will the platform ids be organised in a similar way to the
  platform ids
   we've discussed before for organisation of native code [1]?
  
 
  The actual string used to identify a particular platform can be
  whatever
  we want it to be, just so long as we are consistent. So, yes,
 the ids
  mentioned in the referenced email would seem a good starting
 point. Do
  we need to include a 32-bit/64-bit identifier ?
 
 
   So all tests are, by default, in an all-platforms (or shared)
 group.
   If a test fails on all Windows platforms, it is marked with
   state.broken.windows.
   If a test fails on Windows but only on, say, amd hardware,
   it is marked state.broken.windows.amd.
  
 
  Yes. Agreed.
 
 
   Then when you come to run

Re: [classlib] Testing conventions - a proposal


Hi Alexei,

I just downloaded the latest working build of TestNG 5.0 [1] and support 
for the jvm attribute is in there. This is not the official release 
build.


Best regards,
George

[1] http://testng.org/testng-5.0.zip


Alexei Zakharov wrote:

Hi George,

Agree, we may experience problems in case of VM hang or crash. I
suggest this only as a temporary solution. BTW, the fact that TestNG
ant task still doesn't have such attributes looks like a sign for me -
TestNG can be still immature in some aspects. Still comparing TestNG
and JUnit.

Regards,

2006/7/19, George Harley [EMAIL PROTECTED]:

Hi Alexei,

It's encouraging to hear that (Ant + TestNG + sample tests) all worked
fine together on Harmony. In answer to your question I suppose that the
ability to fork the tests in a separate VM means that we do not run the
risk of possible bugs in Harmony affecting the test harness and
therefore the outcome of the tests.

Best regards,
George


Alexei Zakharov wrote:
 Probably my previous message was not clear enough.
 Why can't we just invoke everything including ant on top of Harmony
 for now? At least I was able to build and run test-14 examples from
 TestNG 4.7 distribution solely on top of j9 + our classlib today.

 C:\Java\testng-4.7\test-14set
 JAVA_HOME=c:\Java\harmony\enhanced\classlib\trunk
 \deploy\jdk\jre

 C:\Java\testng-4.7\test-14ant
 -Dbuild.compiler=org.eclipse.jdt.core.JDTCompiler
 Adapter run
 Buildfile: build.xml

 prepare:

 compile:
 [echo]  -- Compiling JDK 1.4 
tests --


 run:
 [echo]  -- Running JDK 1.4 
tests   --
 [echo]  -- 
testng-4.7-jdk14.jar  --


 [testng-14] ===
 [testng-14] TestNG JDK 1.4
 [testng-14] Total tests run: 179, Failures: 10, Skips: 0
 [testng-14] ===
 ...

 Exactly the same results as with Sun JDK 1.4.
 Note: you may need to hatch the build.xml a little bit to achieve 
this.


 Thanks,

 2006/7/19, George Harley [EMAIL PROTECTED]:
 Hi Richard,

 Actually the Ant task always runs the tests in a forked VM. At 
present,
 however, the task does not support specifying the forked VM (i.e. 
there
 is no equivalent to the JUnit Ant task's jvm attribute). This 
matter

 has already been raised with the TestNG folks who seem happy to
 introduce this.

 In the meantime we could run the tests using the Ant java task.


 Best regards,
 George



 Richard Liang wrote:
  According to TestNG Ant Task [1], it seems that the TestNG Ant 
task
  does not support to fork a new JVM, that is, we must launch ant 
using

  Harmony itself. Any comments? Thanks a lot.
 
  [1]http://testng.org/doc/ant.html
 
  Best regards,
  Richard
 
  George Harley wrote:
  Andrew Zhang wrote:
  On 7/18/06, George Harley [EMAIL PROTECTED] wrote:
 
  Oliver Deakin wrote:
   George Harley wrote:
   SNIP!
  
   Here the annotation on MyTestClass applies to all of its test
  methods.
  
   So what are the well-known TestNG groups that we could define
  for use
   inside Harmony ? Here are some of my initial thoughts:
  
  
   * type.impl  --  tests that are specific to Harmony
  
   So tests are implicitly API unless specified otherwise?
  
   I'm slightly confused by your definition of impl tests as 
tests

  that
  are
   specific to Harmony. Does this mean that impl tests are only
   those that test classes in org.apache.harmony packages?
   I thought that impl was our way of saying tests that need to
 go on
   the bootclasspath.
  
   I think I just need a little clarification...
  
 
  Hi Oliver,
 
  I was using the definition of implementation-specific tests 
that we

  currently have on the Harmony testing conventions web page. That
 is,
  implementation-specific tests are those that are dependent on 
some

  aspect of the Harmony implementation and would therefore not
 pass when
  run against the RI or other conforming implementations. It's
  orthogonal
  to the classpath/bootclasspath issue.
 
 
   * state.broken.platform id  --  tests bust on a specific
 platform
  
   * state.broken  --  tests broken on every platform but we
 want to
   decide whether or not to run from our suite configuration
  
   * os.platform id  --  tests that are to be run only on the
   specified platform (a test could be member of more than 
one of

  these)
  
   And the defaults for these are an unbroken state and runs 
on any

   platform.
   That makes sense...
  
   Will the platform ids be organised in a similar way to the
  platform ids
   we've discussed before for organisation of native code [1]?
  
 
  The actual string used to identify a particular platform can be
  whatever
  we want it to be, just so long as we are consistent. So, yes,
 the ids
  mentioned in the referenced email would seem a good starting
 point. Do
  we need to include a 32-bit/64-bit identifier ?
 
 
   So all tests are, by default, in an

Re: [classlib] Testing conventions - a proposal




George Harley wrote:

Hi Alexei,

It's encouraging to hear that (Ant + TestNG + sample tests) all worked 
fine together on Harmony. In answer to your question I suppose that 
the ability to fork the tests in a separate VM means that we do not 
run the risk of possible bugs in Harmony affecting the test harness 
and therefore the outcome of the tests.


Do you think it's reasonable to launch ant using Harmony, and then 
TestNG will fork another Harmony VM to run our tests? Thanks a lot.


Best regards,
Richard.


Best regards,
George


Alexei Zakharov wrote:

Probably my previous message was not clear enough.
Why can't we just invoke everything including ant on top of Harmony
for now? At least I was able to build and run test-14 examples from
TestNG 4.7 distribution solely on top of j9 + our classlib today.

C:\Java\testng-4.7\test-14set 
JAVA_HOME=c:\Java\harmony\enhanced\classlib\trunk

\deploy\jdk\jre

C:\Java\testng-4.7\test-14ant 
-Dbuild.compiler=org.eclipse.jdt.core.JDTCompiler

Adapter run
Buildfile: build.xml

prepare:

compile:
[echo]  -- Compiling JDK 1.4 
tests --


run:
[echo]  -- Running JDK 1.4 
tests   --

[echo]  -- testng-4.7-jdk14.jar  --

[testng-14] ===
[testng-14] TestNG JDK 1.4
[testng-14] Total tests run: 179, Failures: 10, Skips: 0
[testng-14] ===
...

Exactly the same results as with Sun JDK 1.4.
Note: you may need to hatch the build.xml a little bit to achieve this.

Thanks,

2006/7/19, George Harley [EMAIL PROTECTED]:

Hi Richard,

Actually the Ant task always runs the tests in a forked VM. At present,
however, the task does not support specifying the forked VM (i.e. there
is no equivalent to the JUnit Ant task's jvm attribute). This matter
has already been raised with the TestNG folks who seem happy to
introduce this.

In the meantime we could run the tests using the Ant java task.


Best regards,
George



Richard Liang wrote:
 According to TestNG Ant Task [1], it seems that the TestNG Ant task
 does not support to fork a new JVM, that is, we must launch ant using
 Harmony itself. Any comments? Thanks a lot.

 [1]http://testng.org/doc/ant.html

 Best regards,
 Richard

 George Harley wrote:
 Andrew Zhang wrote:
 On 7/18/06, George Harley [EMAIL PROTECTED] wrote:

 Oliver Deakin wrote:
  George Harley wrote:
  SNIP!
 
  Here the annotation on MyTestClass applies to all of its test
 methods.
 
  So what are the well-known TestNG groups that we could define
 for use
  inside Harmony ? Here are some of my initial thoughts:
 
 
  * type.impl  --  tests that are specific to Harmony
 
  So tests are implicitly API unless specified otherwise?
 
  I'm slightly confused by your definition of impl tests as tests
 that
 are
  specific to Harmony. Does this mean that impl tests are only
  those that test classes in org.apache.harmony packages?
  I thought that impl was our way of saying tests that need to 
go on

  the bootclasspath.
 
  I think I just need a little clarification...
 

 Hi Oliver,

 I was using the definition of implementation-specific tests 
that we
 currently have on the Harmony testing conventions web page. 
That is,

 implementation-specific tests are those that are dependent on some
 aspect of the Harmony implementation and would therefore not 
pass when

 run against the RI or other conforming implementations. It's
 orthogonal
 to the classpath/bootclasspath issue.


  * state.broken.platform id  --  tests bust on a specific 
platform

 
  * state.broken  --  tests broken on every platform but we 
want to

  decide whether or not to run from our suite configuration
 
  * os.platform id  --  tests that are to be run only on the
  specified platform (a test could be member of more than one of
 these)
 
  And the defaults for these are an unbroken state and runs on any
  platform.
  That makes sense...
 
  Will the platform ids be organised in a similar way to the
 platform ids
  we've discussed before for organisation of native code [1]?
 

 The actual string used to identify a particular platform can be
 whatever
 we want it to be, just so long as we are consistent. So, yes, 
the ids
 mentioned in the referenced email would seem a good starting 
point. Do

 we need to include a 32-bit/64-bit identifier ?


  So all tests are, by default, in an all-platforms (or shared) 
group.

  If a test fails on all Windows platforms, it is marked with
  state.broken.windows.
  If a test fails on Windows but only on, say, amd hardware,
  it is marked state.broken.windows.amd.
 

 Yes. Agreed.


  Then when you come to run tests on your windows amd machine,
  you want to include all tests in the all-platform (shared) 
group,

  os.windows and os.windows.amd, and exclude all tests in
  the state.broken, state.broken.windows and 
state.broken.windows.amd

  groups.
 
  Does this tally with

Re: [classlib] Testing conventions - a proposal




George Harley wrote:

Richard Liang wrote:



George Harley wrote:

Hi,

If annotations were to be used to help us categorise tests in order 
to simplify the definition of test configurations - what's included 
and excluded etc - then a core set of annotations would need to be 
agreed by the project. Consider the possibilities that the TestNG 
@Test annotation offers us in this respect.


First, if a test method was identified as being broken and needed to 
be excluded from all test runs while awaiting investigation then it 
would be a simple matter of setting its enabled field like this:


   @Test(enabled=false)
   public void myTest() {
   ...
   }

Temporarily disabling a test method in this way means that it can be 
left in its original class and we do not have to refer to it in any 
suite configuration (e.g. in the suite xml file).


If a test method was identified as being broken on a specific 
platform then we could make use of the groups field of the @Test 
type by making the method a member of a group that identifies its 
predicament. Something like this:


   @Test(groups={state.broken.win.IA32})
   public void myOtherTest() {
   ...
   }

The configuration for running tests on Windows would then 
specifically exclude any test method (or class) that was a member of 
that group.


Making a test method or type a member of a well-known group 
(well-known in the sense that the name and meaning has been agreed 
within the project) is essentially adding some descriptive 
attributes to the test. Like adjectives (the groups) and nouns (the 
tests) in the English language. To take another example, if there 
was a test class that contained methods only intended to be run on 
Windows and that were all specific to Harmony (i.e. not API tests) 
then  one could envisage the following kind of annotation:



@Test(groups={type.impl, os.win.IA32})
public class MyTestClass {

   public void testOne() {
   ...
   }

   public void testTwo() {
   ...
   }

   @Test(enabled=false)
   public void brokenTest() {
   ...
   }
}

Here the annotation on MyTestClass applies to all of its test methods.

So what are the well-known TestNG groups that we could define for 
use inside Harmony ? Here are some of my initial thoughts:



* type.impl  --  tests that are specific to Harmony

* state.broken.platform id  --  tests bust on a specific platform

* state.broken  --  tests broken on every platform but we want to 
decide whether or not to run from our suite configuration


* os.platform id  --  tests that are to be run only on the 
specified platform (a test could be member of more than one of these)



What does everyone else think ? Does such a scheme sound reasonable ?

Just one question: What's the default test annotation? I mean the 
successful api tests which will be run on every platform. Thanks a lot.


Best regards,
Richard


Hi Richard,

I think that just the basic @Test annotation on its own will suffice. 
Any better suggestions are welcome.



Just thinking about how to filter out the target test groups :-)

I tried to use the following groups to define the win.IA32 API tests, 
but it seems that the tests with the default annotation @Test cannot be 
selected. Do I miss anything? Thanks a lot.


   groups
   run
   include name=.*  /
   include name=os.win.IA32  /
   exclude name=type.impl /
   exclude name=state.broken /
   exclude name=state.broken.win.IA32 /
   exclude name=os.linux.IA32 /
   /run
   /groups

The groups I defined:
@Test
@Test(groups={os.win.IA32})
@Test(groups={os.win.IA32, state.broken.win.IA32})
@Test(groups={type.impl})
@Test(groups={state.broken})
@Test(groups={os.linux.IA32})
@Test(groups={state.broken.linux.IA32})

Best regards,
Richard.

Best regards,
George




Thanks for reading this far.

Best regards,
George



George Harley wrote:

Hi,

Just seen Tim's note on test support classes and it really caught 
my attention as I have been mulling over this issue for a little 
while now. I think that it is a good time for us to return to the 
topic of class library test layouts.


The current proposal [1] sets out to segment our different types of 
test by placing them in different file locations. After looking at 
the recent changes to the LUNI module tests (where the layout 
guidelines were applied) I have a real concern that there are 
serious problems with this approach. We have started down a track 
of just continually growing the number of test source folders as 
new categories of test are identified and IMHO that is going to 
bring complexity and maintenance issues with these tests.


Consider the dimensions of tests that we have ...

API
Harmony-specific
Platform-specific
Run on classpath
Run on bootclasspath
Behaves different between Harmony and RI
Stress
...and so on...


If you weigh up all of the different possible permutations and then 
consider that the above

Re: [classlib] Testing conventions - a proposal

2006-07-18 Thread Oliver Deakin


George Harley wrote:

SNIP!

Here the annotation on MyTestClass applies to all of its test methods.

So what are the well-known TestNG groups that we could define for use 
inside Harmony ? Here are some of my initial thoughts:



* type.impl  --  tests that are specific to Harmony


So tests are implicitly API unless specified otherwise?

I'm slightly confused by your definition of impl tests as tests that are
specific to Harmony. Does this mean that impl tests are only
those that test classes in org.apache.harmony packages?
I thought that impl was our way of saying tests that need to go on
the bootclasspath.

I think I just need a little clarification...


* state.broken.platform id  --  tests bust on a specific platform

* state.broken  --  tests broken on every platform but we want to 
decide whether or not to run from our suite configuration


* os.platform id  --  tests that are to be run only on the specified 
platform (a test could be member of more than one of these)


And the defaults for these are an unbroken state and runs on any platform.
That makes sense...

Will the platform ids be organised in a similar way to the platform ids
we've discussed before for organisation of native code [1]?

So all tests are, by default, in an all-platforms (or shared) group.
If a test fails on all Windows platforms, it is marked with
state.broken.windows.
If a test fails on Windows but only on, say, amd hardware,
it is marked state.broken.windows.amd.

Then when you come to run tests on your windows amd machine,
you want to include all tests in the all-platform (shared) group,
os.windows and os.windows.amd, and exclude all tests in
the state.broken, state.broken.windows and state.broken.windows.amd
groups.

Does this tally with what you were thinking?




What does everyone else think ? Does such a scheme sound reasonable ?


I think so - it seems to cover our current requirements. Thanks for
coming up with this!

Regards,
Oliver

[1] 
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/[EMAIL PROTECTED]




Thanks for reading this far.

Best regards,
George



George Harley wrote:

Hi,

Just seen Tim's note on test support classes and it really caught my 
attention as I have been mulling over this issue for a little while 
now. I think that it is a good time for us to return to the topic of 
class library test layouts.


The current proposal [1] sets out to segment our different types of 
test by placing them in different file locations. After looking at 
the recent changes to the LUNI module tests (where the layout 
guidelines were applied) I have a real concern that there are serious 
problems with this approach. We have started down a track of just 
continually growing the number of test source folders as new 
categories of test are identified and IMHO that is going to bring 
complexity and maintenance issues with these tests.


Consider the dimensions of tests that we have ...

API
Harmony-specific
Platform-specific
Run on classpath
Run on bootclasspath
Behaves different between Harmony and RI
Stress
...and so on...


If you weigh up all of the different possible permutations and then 
consider that the above list is highly likely to be extended as 
things progress it is obvious that we are eventually heading for 
large amounts of related test code scattered or possibly duplicated 
across numerous hard wired source directories. How maintainable is 
that going to be ?


If we want to run different tests in different configurations then 
IMHO we need to be thinking a whole lot smarter. We need to be 
thinking about keeping tests for specific areas of functionality 
together (thus easing maintenance); we need something quick and 
simple to re-configure if necessary (pushing whole directories of 
files around the place does not seem a particularly lightweight 
approach); and something that is not going to potentially mess up 
contributed patches when the file they patch is found to have been 
recently pushed from source folder A to B.


To connect into another recent thread, there have been some posts 
lately about handling some test methods that fail on Harmony and have 
meant that entire test case classes have been excluded from our test 
runs. I have also been noticing some API test methods that pass fine 
on Harmony but fail when run against the RI. Are the different 
behaviours down to errors in the Harmony implementation ? An error in 
the RI implementation ? A bug in the RI Javadoc ? Only after some 
investigation has been carried out do we know for sure. That takes 
time. What do we do with the test methods in the meantime ? Do we 
push them round the file system into yet another new source folder ? 
IMHO we need a testing strategy that enables such problem methods 
to be tracked easily without disruption to the rest of the other tests.


A couple of weeks ago I mentioned that the TestNG framework [2] 
seemed like a reasonably good way of allowing us to both group 
together

Re: [classlib] Testing conventions - a proposal


Oliver Deakin wrote:

George Harley wrote:

SNIP!

Here the annotation on MyTestClass applies to all of its test methods.

So what are the well-known TestNG groups that we could define for use 
inside Harmony ? Here are some of my initial thoughts:



* type.impl  --  tests that are specific to Harmony


So tests are implicitly API unless specified otherwise?

I'm slightly confused by your definition of impl tests as tests that are
specific to Harmony. Does this mean that impl tests are only
those that test classes in org.apache.harmony packages?
I thought that impl was our way of saying tests that need to go on
the bootclasspath.

I think I just need a little clarification...



Hi Oliver,

I was using the definition of implementation-specific tests that we 
currently have on the Harmony testing conventions web page. That is, 
implementation-specific tests are those that are dependent on some 
aspect of the Harmony implementation and would therefore not pass when 
run against the RI or other conforming implementations. It's orthogonal 
to the classpath/bootclasspath issue.




* state.broken.platform id  --  tests bust on a specific platform

* state.broken  --  tests broken on every platform but we want to 
decide whether or not to run from our suite configuration


* os.platform id  --  tests that are to be run only on the 
specified platform (a test could be member of more than one of these)


And the defaults for these are an unbroken state and runs on any 
platform.

That makes sense...

Will the platform ids be organised in a similar way to the platform ids
we've discussed before for organisation of native code [1]?



The actual string used to identify a particular platform can be whatever 
we want it to be, just so long as we are consistent. So, yes, the ids 
mentioned in the referenced email would seem a good starting point. Do 
we need to include a 32-bit/64-bit identifier ?




So all tests are, by default, in an all-platforms (or shared) group.
If a test fails on all Windows platforms, it is marked with
state.broken.windows.
If a test fails on Windows but only on, say, amd hardware,
it is marked state.broken.windows.amd.



Yes. Agreed.



Then when you come to run tests on your windows amd machine,
you want to include all tests in the all-platform (shared) group,
os.windows and os.windows.amd, and exclude all tests in
the state.broken, state.broken.windows and state.broken.windows.amd
groups.

Does this tally with what you were thinking?



Yes, that is the idea.





What does everyone else think ? Does such a scheme sound reasonable ?


I think so - it seems to cover our current requirements. Thanks for
coming up with this!



Thanks, but I don't see it as final yet really. It would be great to 
prove the worth of this by doing a trial on one of the existing modules, 
ideally something that contains tests that are platform-specific.


Best regards,
George



Regards,
Oliver

[1] 
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/[EMAIL PROTECTED] 





Thanks for reading this far.

Best regards,
George



George Harley wrote:

Hi,

Just seen Tim's note on test support classes and it really caught my 
attention as I have been mulling over this issue for a little while 
now. I think that it is a good time for us to return to the topic of 
class library test layouts.


The current proposal [1] sets out to segment our different types of 
test by placing them in different file locations. After looking at 
the recent changes to the LUNI module tests (where the layout 
guidelines were applied) I have a real concern that there are 
serious problems with this approach. We have started down a track of 
just continually growing the number of test source folders as new 
categories of test are identified and IMHO that is going to bring 
complexity and maintenance issues with these tests.


Consider the dimensions of tests that we have ...

API
Harmony-specific
Platform-specific
Run on classpath
Run on bootclasspath
Behaves different between Harmony and RI
Stress
...and so on...


If you weigh up all of the different possible permutations and then 
consider that the above list is highly likely to be extended as 
things progress it is obvious that we are eventually heading for 
large amounts of related test code scattered or possibly duplicated 
across numerous hard wired source directories. How maintainable is 
that going to be ?


If we want to run different tests in different configurations then 
IMHO we need to be thinking a whole lot smarter. We need to be 
thinking about keeping tests for specific areas of functionality 
together (thus easing maintenance); we need something quick and 
simple to re-configure if necessary (pushing whole directories of 
files around the place does not seem a particularly lightweight 
approach); and something that is not going to potentially mess up 
contributed patches when the file they patch is found to have been 
recently pushed from

Re: [classlib] Testing conventions - a proposal

2006-07-18 Thread Oliver Deakin


George Harley wrote:

Oliver Deakin wrote:

George Harley wrote:

SNIP!

Here the annotation on MyTestClass applies to all of its test methods.

So what are the well-known TestNG groups that we could define for 
use inside Harmony ? Here are some of my initial thoughts:



* type.impl  --  tests that are specific to Harmony


So tests are implicitly API unless specified otherwise?

I'm slightly confused by your definition of impl tests as tests that 
are

specific to Harmony. Does this mean that impl tests are only
those that test classes in org.apache.harmony packages?
I thought that impl was our way of saying tests that need to go on
the bootclasspath.

I think I just need a little clarification...



Hi Oliver,

I was using the definition of implementation-specific tests that we 
currently have on the Harmony testing conventions web page. That is, 
implementation-specific tests are those that are dependent on some 
aspect of the Harmony implementation and would therefore not pass when 
run against the RI or other conforming implementations. It's 
orthogonal to the classpath/bootclasspath issue.


OK, that's what I imagined you meant. IMHO using api and impl
in this way makes the most sense (since, as you say, they do not
really relate to the classpath/bootclasspath issue).

So do we also need a pair of groups for classpath/bootclasspath
tests? I'm assuming that this is how we would handle this distinction,
rather than organising them into separate directories in the file system.





* state.broken.platform id  --  tests bust on a specific platform

* state.broken  --  tests broken on every platform but we want to 
decide whether or not to run from our suite configuration


* os.platform id  --  tests that are to be run only on the 
specified platform (a test could be member of more than one of these)


And the defaults for these are an unbroken state and runs on any 
platform.

That makes sense...

Will the platform ids be organised in a similar way to the platform ids
we've discussed before for organisation of native code [1]?



The actual string used to identify a particular platform can be 
whatever we want it to be, just so long as we are consistent. So, yes, 
the ids mentioned in the referenced email would seem a good starting 
point. Do we need to include a 32-bit/64-bit identifier ?


I cannot immediately think of any obvious 32/64-bit specific tests that we
might require in the future (although Id be interested to know if anyone
can think of any!). However, if the need did arise, then I would
suggest that this is incorporated as another tag on the end of the
group name e.g. os.linux.ppc.32.






So all tests are, by default, in an all-platforms (or shared) group.
If a test fails on all Windows platforms, it is marked with
state.broken.windows.
If a test fails on Windows but only on, say, amd hardware,
it is marked state.broken.windows.amd.



Yes. Agreed.



Then when you come to run tests on your windows amd machine,
you want to include all tests in the all-platform (shared) group,
os.windows and os.windows.amd, and exclude all tests in
the state.broken, state.broken.windows and state.broken.windows.amd
groups.

Does this tally with what you were thinking?



Yes, that is the idea.





What does everyone else think ? Does such a scheme sound reasonable ?


I think so - it seems to cover our current requirements. Thanks for
coming up with this!



Thanks, but I don't see it as final yet really. It would be great to 
prove the worth of this by doing a trial on one of the existing 
modules, ideally something that contains tests that are 
platform-specific.


Thanks for volunteering... ;)

...but seriously, do any of our modules currently contain platform 
specific tests?

Have you attempted a TestNG trial on any of the modules (with or
without platform specific tests) and, if so, was it 
simpler/harder/better/worse

than our current setup?

Regards,
Oliver



Best regards,
George



Regards,
Oliver

[1] 
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/[EMAIL PROTECTED] 





Thanks for reading this far.

Best regards,
George



George Harley wrote:

Hi,

Just seen Tim's note on test support classes and it really caught 
my attention as I have been mulling over this issue for a little 
while now. I think that it is a good time for us to return to the 
topic of class library test layouts.


The current proposal [1] sets out to segment our different types of 
test by placing them in different file locations. After looking at 
the recent changes to the LUNI module tests (where the layout 
guidelines were applied) I have a real concern that there are 
serious problems with this approach. We have started down a track 
of just continually growing the number of test source folders as 
new categories of test are identified and IMHO that is going to 
bring complexity and maintenance issues with these tests.


Consider the dimensions of tests that we have ...

API

Re: [classlib] Testing conventions - a proposal

2006-07-18 Thread Andrew Zhang


On 7/18/06, George Harley [EMAIL PROTECTED] wrote:


Oliver Deakin wrote:
 George Harley wrote:
 SNIP!

 Here the annotation on MyTestClass applies to all of its test methods.

 So what are the well-known TestNG groups that we could define for use
 inside Harmony ? Here are some of my initial thoughts:


 * type.impl  --  tests that are specific to Harmony

 So tests are implicitly API unless specified otherwise?

 I'm slightly confused by your definition of impl tests as tests that
are
 specific to Harmony. Does this mean that impl tests are only
 those that test classes in org.apache.harmony packages?
 I thought that impl was our way of saying tests that need to go on
 the bootclasspath.

 I think I just need a little clarification...


Hi Oliver,

I was using the definition of implementation-specific tests that we
currently have on the Harmony testing conventions web page. That is,
implementation-specific tests are those that are dependent on some
aspect of the Harmony implementation and would therefore not pass when
run against the RI or other conforming implementations. It's orthogonal
to the classpath/bootclasspath issue.


 * state.broken.platform id  --  tests bust on a specific platform

 * state.broken  --  tests broken on every platform but we want to
 decide whether or not to run from our suite configuration

 * os.platform id  --  tests that are to be run only on the
 specified platform (a test could be member of more than one of these)

 And the defaults for these are an unbroken state and runs on any
 platform.
 That makes sense...

 Will the platform ids be organised in a similar way to the platform ids
 we've discussed before for organisation of native code [1]?


The actual string used to identify a particular platform can be whatever
we want it to be, just so long as we are consistent. So, yes, the ids
mentioned in the referenced email would seem a good starting point. Do
we need to include a 32-bit/64-bit identifier ?


 So all tests are, by default, in an all-platforms (or shared) group.
 If a test fails on all Windows platforms, it is marked with
 state.broken.windows.
 If a test fails on Windows but only on, say, amd hardware,
 it is marked state.broken.windows.amd.


Yes. Agreed.


 Then when you come to run tests on your windows amd machine,
 you want to include all tests in the all-platform (shared) group,
 os.windows and os.windows.amd, and exclude all tests in
 the state.broken, state.broken.windows and state.broken.windows.amd
 groups.

 Does this tally with what you were thinking?


Yes, that is the idea.




 What does everyone else think ? Does such a scheme sound reasonable ?

 I think so - it seems to cover our current requirements. Thanks for
 coming up with this!


Thanks, but I don't see it as final yet really. It would be great to
prove the worth of this by doing a trial on one of the existing modules,
ideally something that contains tests that are platform-specific.



Hello George, how about doing a trial on NIO module?

So far as I know, there are several platform dependent tests in NIO module.
:)

The assert statements are commented out in these tests, with FIXME mark.

Furthurmore, I also find some platform dependent behaviours of FileChannel.
If TestNG is applied on NIO, I will supplement new tests for FileChannel and
fix the bug of source code.

What's your opnion? Any suggestions/comments?

Thanks!

Best regards,

George


 Regards,
 Oliver

 [1]

http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/[EMAIL
 PROTECTED]



 Thanks for reading this far.

 Best regards,
 George



 George Harley wrote:
 Hi,

 Just seen Tim's note on test support classes and it really caught my
 attention as I have been mulling over this issue for a little while
 now. I think that it is a good time for us to return to the topic of
 class library test layouts.

 The current proposal [1] sets out to segment our different types of
 test by placing them in different file locations. After looking at
 the recent changes to the LUNI module tests (where the layout
 guidelines were applied) I have a real concern that there are
 serious problems with this approach. We have started down a track of
 just continually growing the number of test source folders as new
 categories of test are identified and IMHO that is going to bring
 complexity and maintenance issues with these tests.

 Consider the dimensions of tests that we have ...

 API
 Harmony-specific
 Platform-specific
 Run on classpath
 Run on bootclasspath
 Behaves different between Harmony and RI
 Stress
 ...and so on...


 If you weigh up all of the different possible permutations and then
 consider that the above list is highly likely to be extended as
 things progress it is obvious that we are eventually heading for
 large amounts of related test code scattered or possibly duplicated
 across numerous hard wired source directories. How maintainable is
 that going to be ?

 If we want to run different tests in

Re: [classlib] Testing conventions - a proposal

2006-07-18 Thread Alexei Zakharov


Hi,

George wrote:

 Thanks, but I don't see it as final yet really. It would be great to
 prove the worth of this by doing a trial on one of the existing modules,
 ideally something that contains tests that are platform-specific.


I volunteer to do this trial for beans module. I'm not sure that beans
contains any platform-specific tests, but I know for sure it has a lot
failed tests - so we can try TestNG with the real workload. I also
like to do the same job with JUnit 4.0 and compare the results -
exactly what is simpler/harder/better etc. In the real.

If Andrew does the same job for nio we will have two separate
experiences that help us to move further in choosing the right testing
framework.
Any thoughts, objections?

Thanks,

2006/7/18, Andrew Zhang [EMAIL PROTECTED]:

On 7/18/06, George Harley [EMAIL PROTECTED] wrote:

 Oliver Deakin wrote:
  George Harley wrote:
  SNIP!
 
  Here the annotation on MyTestClass applies to all of its test methods.
 
  So what are the well-known TestNG groups that we could define for use
  inside Harmony ? Here are some of my initial thoughts:
 
 
  * type.impl  --  tests that are specific to Harmony
 
  So tests are implicitly API unless specified otherwise?
 
  I'm slightly confused by your definition of impl tests as tests that
 are
  specific to Harmony. Does this mean that impl tests are only
  those that test classes in org.apache.harmony packages?
  I thought that impl was our way of saying tests that need to go on
  the bootclasspath.
 
  I think I just need a little clarification...
 

 Hi Oliver,

 I was using the definition of implementation-specific tests that we
 currently have on the Harmony testing conventions web page. That is,
 implementation-specific tests are those that are dependent on some
 aspect of the Harmony implementation and would therefore not pass when
 run against the RI or other conforming implementations. It's orthogonal
 to the classpath/bootclasspath issue.


  * state.broken.platform id  --  tests bust on a specific platform
 
  * state.broken  --  tests broken on every platform but we want to
  decide whether or not to run from our suite configuration
 
  * os.platform id  --  tests that are to be run only on the
  specified platform (a test could be member of more than one of these)
 
  And the defaults for these are an unbroken state and runs on any
  platform.
  That makes sense...
 
  Will the platform ids be organised in a similar way to the platform ids
  we've discussed before for organisation of native code [1]?
 

 The actual string used to identify a particular platform can be whatever
 we want it to be, just so long as we are consistent. So, yes, the ids
 mentioned in the referenced email would seem a good starting point. Do
 we need to include a 32-bit/64-bit identifier ?


  So all tests are, by default, in an all-platforms (or shared) group.
  If a test fails on all Windows platforms, it is marked with
  state.broken.windows.
  If a test fails on Windows but only on, say, amd hardware,
  it is marked state.broken.windows.amd.
 

 Yes. Agreed.


  Then when you come to run tests on your windows amd machine,
  you want to include all tests in the all-platform (shared) group,
  os.windows and os.windows.amd, and exclude all tests in
  the state.broken, state.broken.windows and state.broken.windows.amd
  groups.
 
  Does this tally with what you were thinking?
 

 Yes, that is the idea.


 
 
  What does everyone else think ? Does such a scheme sound reasonable ?
 
  I think so - it seems to cover our current requirements. Thanks for
  coming up with this!
 

 Thanks, but I don't see it as final yet really. It would be great to
 prove the worth of this by doing a trial on one of the existing modules,
 ideally something that contains tests that are platform-specific.


Hello George, how about doing a trial on NIO module?

So far as I know, there are several platform dependent tests in NIO module.
:)

The assert statements are commented out in these tests, with FIXME mark.

Furthurmore, I also find some platform dependent behaviours of FileChannel.
If TestNG is applied on NIO, I will supplement new tests for FileChannel and
fix the bug of source code.

What's your opnion? Any suggestions/comments?

Thanks!

Best regards,
 George


  Regards,
  Oliver
 
  [1]
 
 
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/[EMAIL 
PROTECTED]
 
 
 
  Thanks for reading this far.
 
  Best regards,
  George
 
 
 
  George Harley wrote:
  Hi,
 
  Just seen Tim's note on test support classes and it really caught my
  attention as I have been mulling over this issue for a little while
  now. I think that it is a good time for us to return to the topic of
  class library test layouts.
 
  The current proposal [1] sets out to segment our different types of
  test by placing them in different file locations. After looking at
  the recent changes to the LUNI module tests (where the layout
  guidelines were applied) I

Re: [classlib] Testing conventions - a proposal

2006-07-18 Thread Andrew Zhang


On 7/18/06, Alexei Zakharov [EMAIL PROTECTED] wrote:


Hi,

George wrote:
  Thanks, but I don't see it as final yet really. It would be great to
  prove the worth of this by doing a trial on one of the existing
modules,
  ideally something that contains tests that are platform-specific.

I volunteer to do this trial for beans module. I'm not sure that beans
contains any platform-specific tests, but I know for sure it has a lot
failed tests - so we can try TestNG with the real workload. I also
like to do the same job with JUnit 4.0 and compare the results -
exactly what is simpler/harder/better etc. In the real.



Alexei, great! :)

If Andrew does the same job for nio we will have two separate

experiences that help us to move further in choosing the right testing
framework.



So shall we move next step now? That is to say, integrate TestNG, define the
annotation(George has given the first version:) ) .

If no one objects, I volunteer to have a try on nio module. :)
Thanks!

Any thoughts, objections?


Thanks,

2006/7/18, Andrew Zhang [EMAIL PROTECTED]:
 On 7/18/06, George Harley [EMAIL PROTECTED] wrote:
 
  Oliver Deakin wrote:
   George Harley wrote:
   SNIP!
  
   Here the annotation on MyTestClass applies to all of its test
methods.
  
   So what are the well-known TestNG groups that we could define for
use
   inside Harmony ? Here are some of my initial thoughts:
  
  
   * type.impl  --  tests that are specific to Harmony
  
   So tests are implicitly API unless specified otherwise?
  
   I'm slightly confused by your definition of impl tests as tests
that
  are
   specific to Harmony. Does this mean that impl tests are only
   those that test classes in org.apache.harmony packages?
   I thought that impl was our way of saying tests that need to go on
   the bootclasspath.
  
   I think I just need a little clarification...
  
 
  Hi Oliver,
 
  I was using the definition of implementation-specific tests that we
  currently have on the Harmony testing conventions web page. That is,
  implementation-specific tests are those that are dependent on some
  aspect of the Harmony implementation and would therefore not pass when
  run against the RI or other conforming implementations. It's
orthogonal
  to the classpath/bootclasspath issue.
 
 
   * state.broken.platform id  --  tests bust on a specific platform
  
   * state.broken  --  tests broken on every platform but we want to
   decide whether or not to run from our suite configuration
  
   * os.platform id  --  tests that are to be run only on the
   specified platform (a test could be member of more than one of
these)
  
   And the defaults for these are an unbroken state and runs on any
   platform.
   That makes sense...
  
   Will the platform ids be organised in a similar way to the platform
ids
   we've discussed before for organisation of native code [1]?
  
 
  The actual string used to identify a particular platform can be
whatever
  we want it to be, just so long as we are consistent. So, yes, the ids
  mentioned in the referenced email would seem a good starting point. Do
  we need to include a 32-bit/64-bit identifier ?
 
 
   So all tests are, by default, in an all-platforms (or shared) group.
   If a test fails on all Windows platforms, it is marked with
   state.broken.windows.
   If a test fails on Windows but only on, say, amd hardware,
   it is marked state.broken.windows.amd.
  
 
  Yes. Agreed.
 
 
   Then when you come to run tests on your windows amd machine,
   you want to include all tests in the all-platform (shared) group,
   os.windows and os.windows.amd, and exclude all tests in
   the state.broken, state.broken.windows and state.broken.windows.amd
   groups.
  
   Does this tally with what you were thinking?
  
 
  Yes, that is the idea.
 
 
  
  
   What does everyone else think ? Does such a scheme sound reasonable
?
  
   I think so - it seems to cover our current requirements. Thanks for
   coming up with this!
  
 
  Thanks, but I don't see it as final yet really. It would be great to
  prove the worth of this by doing a trial on one of the existing
modules,
  ideally something that contains tests that are platform-specific.


 Hello George, how about doing a trial on NIO module?

 So far as I know, there are several platform dependent tests in NIO
module.
 :)

 The assert statements are commented out in these tests, with FIXME
mark.

 Furthurmore, I also find some platform dependent behaviours of
FileChannel.
 If TestNG is applied on NIO, I will supplement new tests for FileChannel
and
 fix the bug of source code.

 What's your opnion? Any suggestions/comments?

 Thanks!

 Best regards,
  George
 
 
   Regards,
   Oliver
  
   [1]
  
 
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/[EMAIL
 PROTECTED]
  
  
  
   Thanks for reading this far.
  
   Best regards,
   George
  
  
  
   George Harley wrote:
   Hi,
  
   Just seen Tim's note on test support classes and it really caught
my

Re: [classlib] Testing conventions - a proposal


Oliver Deakin wrote:

George Harley wrote:

Oliver Deakin wrote:

George Harley wrote:

SNIP!

Here the annotation on MyTestClass applies to all of its test methods.

So what are the well-known TestNG groups that we could define for 
use inside Harmony ? Here are some of my initial thoughts:



* type.impl  --  tests that are specific to Harmony


So tests are implicitly API unless specified otherwise?

I'm slightly confused by your definition of impl tests as tests 
that are

specific to Harmony. Does this mean that impl tests are only
those that test classes in org.apache.harmony packages?
I thought that impl was our way of saying tests that need to go on
the bootclasspath.

I think I just need a little clarification...



Hi Oliver,

I was using the definition of implementation-specific tests that we 
currently have on the Harmony testing conventions web page. That is, 
implementation-specific tests are those that are dependent on some 
aspect of the Harmony implementation and would therefore not pass 
when run against the RI or other conforming implementations. It's 
orthogonal to the classpath/bootclasspath issue.


OK, that's what I imagined you meant. IMHO using api and impl
in this way makes the most sense (since, as you say, they do not
really relate to the classpath/bootclasspath issue).

So do we also need a pair of groups for classpath/bootclasspath
tests? I'm assuming that this is how we would handle this distinction,
rather than organising them into separate directories in the file system.



Hi Oliver,


I guess that would be possible but, given that the intended classloader 
of a test is probably not going to vary much, I would have no objection 
to that distinction being made in separate directories in the file 
system. In other words, I see the benefits of a suite-driven system 
(TestNG or whatever) as being more applicable to those attributes of 
tests that we identify as being more susceptible to change. Examples 
like currently only works on platform X or broken on platform Y or 
only works when run against Harmony seem like good candidates, 
especially when the time comes to grow Harmony on other platforms. Right 
now, I don't see the runtime classloader fitting into that category. 
Maybe I'm wrong.







* state.broken.platform id  --  tests bust on a specific platform

* state.broken  --  tests broken on every platform but we want to 
decide whether or not to run from our suite configuration


* os.platform id  --  tests that are to be run only on the 
specified platform (a test could be member of more than one of these)


And the defaults for these are an unbroken state and runs on any 
platform.

That makes sense...

Will the platform ids be organised in a similar way to the platform ids
we've discussed before for organisation of native code [1]?



The actual string used to identify a particular platform can be 
whatever we want it to be, just so long as we are consistent. So, 
yes, the ids mentioned in the referenced email would seem a good 
starting point. Do we need to include a 32-bit/64-bit identifier ?


I cannot immediately think of any obvious 32/64-bit specific tests 
that we

might require in the future (although Id be interested to know if anyone
can think of any!). However, if the need did arise, then I would
suggest that this is incorporated as another tag on the end of the
group name e.g. os.linux.ppc.32.




Right.





So all tests are, by default, in an all-platforms (or shared) group.
If a test fails on all Windows platforms, it is marked with
state.broken.windows.
If a test fails on Windows but only on, say, amd hardware,
it is marked state.broken.windows.amd.



Yes. Agreed.



Then when you come to run tests on your windows amd machine,
you want to include all tests in the all-platform (shared) group,
os.windows and os.windows.amd, and exclude all tests in
the state.broken, state.broken.windows and state.broken.windows.amd
groups.

Does this tally with what you were thinking?



Yes, that is the idea.





What does everyone else think ? Does such a scheme sound reasonable ?


I think so - it seems to cover our current requirements. Thanks for
coming up with this!



Thanks, but I don't see it as final yet really. It would be great to 
prove the worth of this by doing a trial on one of the existing 
modules, ideally something that contains tests that are 
platform-specific.


Thanks for volunteering... ;)

...but seriously, do any of our modules currently contain platform 
specific tests?

Have you attempted a TestNG trial on any of the modules (with or
without platform specific tests) and, if so, was it 
simpler/harder/better/worse

than our current setup?

Regards,
Oliver



Yes, I believe that there are platform specific tests out there. NIO and 
auth are two examples that spring to mind. My own experiments with 
TestNG have so far not been with our modules but in separate projects. 
Running the tests didn't seem all that different to what we currently

Re: [classlib] Testing conventions - a proposal


Andrew Zhang wrote:

On 7/18/06, George Harley [EMAIL PROTECTED] wrote:


Oliver Deakin wrote:
 George Harley wrote:
 SNIP!

 Here the annotation on MyTestClass applies to all of its test 
methods.


 So what are the well-known TestNG groups that we could define for use
 inside Harmony ? Here are some of my initial thoughts:


 * type.impl  --  tests that are specific to Harmony

 So tests are implicitly API unless specified otherwise?

 I'm slightly confused by your definition of impl tests as tests that
are
 specific to Harmony. Does this mean that impl tests are only
 those that test classes in org.apache.harmony packages?
 I thought that impl was our way of saying tests that need to go on
 the bootclasspath.

 I think I just need a little clarification...


Hi Oliver,

I was using the definition of implementation-specific tests that we
currently have on the Harmony testing conventions web page. That is,
implementation-specific tests are those that are dependent on some
aspect of the Harmony implementation and would therefore not pass when
run against the RI or other conforming implementations. It's orthogonal
to the classpath/bootclasspath issue.


 * state.broken.platform id  --  tests bust on a specific platform

 * state.broken  --  tests broken on every platform but we want to
 decide whether or not to run from our suite configuration

 * os.platform id  --  tests that are to be run only on the
 specified platform (a test could be member of more than one of these)

 And the defaults for these are an unbroken state and runs on any
 platform.
 That makes sense...

 Will the platform ids be organised in a similar way to the platform 
ids

 we've discussed before for organisation of native code [1]?


The actual string used to identify a particular platform can be whatever
we want it to be, just so long as we are consistent. So, yes, the ids
mentioned in the referenced email would seem a good starting point. Do
we need to include a 32-bit/64-bit identifier ?


 So all tests are, by default, in an all-platforms (or shared) group.
 If a test fails on all Windows platforms, it is marked with
 state.broken.windows.
 If a test fails on Windows but only on, say, amd hardware,
 it is marked state.broken.windows.amd.


Yes. Agreed.


 Then when you come to run tests on your windows amd machine,
 you want to include all tests in the all-platform (shared) group,
 os.windows and os.windows.amd, and exclude all tests in
 the state.broken, state.broken.windows and state.broken.windows.amd
 groups.

 Does this tally with what you were thinking?


Yes, that is the idea.




 What does everyone else think ? Does such a scheme sound reasonable ?

 I think so - it seems to cover our current requirements. Thanks for
 coming up with this!


Thanks, but I don't see it as final yet really. It would be great to
prove the worth of this by doing a trial on one of the existing modules,
ideally something that contains tests that are platform-specific.



Hello George, how about doing a trial on NIO module?

So far as I know, there are several platform dependent tests in NIO 
module.

:)

The assert statements are commented out in these tests, with FIXME 
mark.


Furthurmore, I also find some platform dependent behaviours of 
FileChannel.
If TestNG is applied on NIO, I will supplement new tests for 
FileChannel and

fix the bug of source code.

What's your opnion? Any suggestions/comments?

Thanks!



Hi Andrew,

That sounds like a very good idea. If there is agreement in the project 
that 5.0 annotations are the way to go (as opposed to the pre-5.0 
Javadoc comment support offered by TestNG) then to the best of my 
knowledge all that is stopping us from doing this trial is the lack of a 
5.0 VM to run the Harmony tests on. Hopefully that will be addressed 
soon. When it is I would be happy to get stuck into this trial.


Best regards,
George



Best regards,

George


 Regards,
 Oliver

 [1]

http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/[EMAIL PROTECTED] 





 Thanks for reading this far.

 Best regards,
 George



 George Harley wrote:
 Hi,

 Just seen Tim's note on test support classes and it really caught my
 attention as I have been mulling over this issue for a little while
 now. I think that it is a good time for us to return to the topic of
 class library test layouts.

 The current proposal [1] sets out to segment our different types of
 test by placing them in different file locations. After looking at
 the recent changes to the LUNI module tests (where the layout
 guidelines were applied) I have a real concern that there are
 serious problems with this approach. We have started down a track of
 just continually growing the number of test source folders as new
 categories of test are identified and IMHO that is going to bring
 complexity and maintenance issues with these tests.

 Consider the dimensions of tests that we have ...

 API
 Harmony-specific
 Platform-specific
 Run on

Re: [classlib] Testing conventions - a proposal

2006-07-18 Thread Oliver Deakin


George Harley wrote:

Oliver Deakin wrote:

George Harley wrote:

Oliver Deakin wrote:

George Harley wrote:

SNIP!

Here the annotation on MyTestClass applies to all of its test 
methods.


So what are the well-known TestNG groups that we could define for 
use inside Harmony ? Here are some of my initial thoughts:



* type.impl  --  tests that are specific to Harmony


So tests are implicitly API unless specified otherwise?

I'm slightly confused by your definition of impl tests as tests 
that are

specific to Harmony. Does this mean that impl tests are only
those that test classes in org.apache.harmony packages?
I thought that impl was our way of saying tests that need to go on
the bootclasspath.

I think I just need a little clarification...



Hi Oliver,

I was using the definition of implementation-specific tests that we 
currently have on the Harmony testing conventions web page. That is, 
implementation-specific tests are those that are dependent on some 
aspect of the Harmony implementation and would therefore not pass 
when run against the RI or other conforming implementations. It's 
orthogonal to the classpath/bootclasspath issue.


OK, that's what I imagined you meant. IMHO using api and impl
in this way makes the most sense (since, as you say, they do not
really relate to the classpath/bootclasspath issue).

So do we also need a pair of groups for classpath/bootclasspath
tests? I'm assuming that this is how we would handle this distinction,
rather than organising them into separate directories in the file 
system.




Hi Oliver,


I guess that would be possible but, given that the intended 
classloader of a test is probably not going to vary much, I would have 
no objection to that distinction being made in separate directories in 
the file system. In other words, I see the benefits of a suite-driven 
system (TestNG or whatever) as being more applicable to those 
attributes of tests that we identify as being more susceptible to 
change. Examples like currently only works on platform X or broken 
on platform Y or only works when run against Harmony seem like good 
candidates, especially when the time comes to grow Harmony on other 
platforms. Right now, I don't see the runtime classloader fitting into 
that category. Maybe I'm wrong.




Right - the only thing that prompted me to ask about it was the 
possibility of
uniting tests for a particular class into a single test class. With a 
separate
directory structure for classpath and bootclasspath tests there are 
often two

test classes for each class-under-test - making this distinction with
groups allows us to keep all tests for a class in a single file.

However, I don't feel strongly about this - as long as the distinction 
is clear

and easy to maintain then I'm happy. Also, I seem to remember a
discussion about splitting the tests up into separate classpath and
bootclasspath directories so that IDEs could compile them into
separate bin directories, and run them with the right config when not
using the Ant scripts [1]. This seems like a decent reason to go
with separate directories.

Regards,
Oliver

[1] 
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/[EMAIL PROTECTED]








* state.broken.platform id  --  tests bust on a specific platform

* state.broken  --  tests broken on every platform but we want to 
decide whether or not to run from our suite configuration


* os.platform id  --  tests that are to be run only on the 
specified platform (a test could be member of more than one of these)


And the defaults for these are an unbroken state and runs on any 
platform.

That makes sense...

Will the platform ids be organised in a similar way to the platform 
ids

we've discussed before for organisation of native code [1]?



The actual string used to identify a particular platform can be 
whatever we want it to be, just so long as we are consistent. So, 
yes, the ids mentioned in the referenced email would seem a good 
starting point. Do we need to include a 32-bit/64-bit identifier ?


I cannot immediately think of any obvious 32/64-bit specific tests 
that we

might require in the future (although Id be interested to know if anyone
can think of any!). However, if the need did arise, then I would
suggest that this is incorporated as another tag on the end of the
group name e.g. os.linux.ppc.32.




Right.





So all tests are, by default, in an all-platforms (or shared) group.
If a test fails on all Windows platforms, it is marked with
state.broken.windows.
If a test fails on Windows but only on, say, amd hardware,
it is marked state.broken.windows.amd.



Yes. Agreed.



Then when you come to run tests on your windows amd machine,
you want to include all tests in the all-platform (shared) group,
os.windows and os.windows.amd, and exclude all tests in
the state.broken, state.broken.windows and state.broken.windows.amd
groups.

Does this tally with what you were thinking?



Yes, that is the idea.





What

Re: [classlib] Testing conventions - a proposal


Andrew Zhang wrote:

On 7/18/06, Alexei Zakharov [EMAIL PROTECTED] wrote:


Hi,

George wrote:
  Thanks, but I don't see it as final yet really. It would be great to
  prove the worth of this by doing a trial on one of the existing
modules,
  ideally something that contains tests that are platform-specific.

I volunteer to do this trial for beans module. I'm not sure that beans
contains any platform-specific tests, but I know for sure it has a lot
failed tests - so we can try TestNG with the real workload. I also
like to do the same job with JUnit 4.0 and compare the results -
exactly what is simpler/harder/better etc. In the real.



Alexei, great! :)

If Andrew does the same job for nio we will have two separate

experiences that help us to move further in choosing the right testing
framework.



So shall we move next step now? That is to say, integrate TestNG, 
define the

annotation(George has given the first version:) ) .

If no one objects, I volunteer to have a try on nio module. :)
Thanks!

Any thoughts, objections?



Hi Andrew,

I thought that Oliver had volunteered me to do it :-)

It would be terrific if you were happy to proceed with this trial on 
NIO. Please note that if you intend to use the TestNG annotations 
approach then you will need to wait for a 5.0 VM for Harmony.


Best regards,
George




Thanks,

2006/7/18, Andrew Zhang [EMAIL PROTECTED]:
 On 7/18/06, George Harley [EMAIL PROTECTED] wrote:
 
  Oliver Deakin wrote:
   George Harley wrote:
   SNIP!
  
   Here the annotation on MyTestClass applies to all of its test
methods.
  
   So what are the well-known TestNG groups that we could define for
use
   inside Harmony ? Here are some of my initial thoughts:
  
  
   * type.impl  --  tests that are specific to Harmony
  
   So tests are implicitly API unless specified otherwise?
  
   I'm slightly confused by your definition of impl tests as tests
that
  are
   specific to Harmony. Does this mean that impl tests are only
   those that test classes in org.apache.harmony packages?
   I thought that impl was our way of saying tests that need to 
go on

   the bootclasspath.
  
   I think I just need a little clarification...
  
 
  Hi Oliver,
 
  I was using the definition of implementation-specific tests that we
  currently have on the Harmony testing conventions web page. That is,
  implementation-specific tests are those that are dependent on some
  aspect of the Harmony implementation and would therefore not pass 
when

  run against the RI or other conforming implementations. It's
orthogonal
  to the classpath/bootclasspath issue.
 
 
   * state.broken.platform id  --  tests bust on a specific 
platform

  
   * state.broken  --  tests broken on every platform but we want to
   decide whether or not to run from our suite configuration
  
   * os.platform id  --  tests that are to be run only on the
   specified platform (a test could be member of more than one of
these)
  
   And the defaults for these are an unbroken state and runs on any
   platform.
   That makes sense...
  
   Will the platform ids be organised in a similar way to the 
platform

ids
   we've discussed before for organisation of native code [1]?
  
 
  The actual string used to identify a particular platform can be
whatever
  we want it to be, just so long as we are consistent. So, yes, the 
ids
  mentioned in the referenced email would seem a good starting 
point. Do

  we need to include a 32-bit/64-bit identifier ?
 
 
   So all tests are, by default, in an all-platforms (or shared) 
group.

   If a test fails on all Windows platforms, it is marked with
   state.broken.windows.
   If a test fails on Windows but only on, say, amd hardware,
   it is marked state.broken.windows.amd.
  
 
  Yes. Agreed.
 
 
   Then when you come to run tests on your windows amd machine,
   you want to include all tests in the all-platform (shared) group,
   os.windows and os.windows.amd, and exclude all tests in
   the state.broken, state.broken.windows and 
state.broken.windows.amd

   groups.
  
   Does this tally with what you were thinking?
  
 
  Yes, that is the idea.
 
 
  
  
   What does everyone else think ? Does such a scheme sound 
reasonable

?
  
   I think so - it seems to cover our current requirements. Thanks 
for

   coming up with this!
  
 
  Thanks, but I don't see it as final yet really. It would be great to
  prove the worth of this by doing a trial on one of the existing
modules,
  ideally something that contains tests that are platform-specific.


 Hello George, how about doing a trial on NIO module?

 So far as I know, there are several platform dependent tests in NIO
module.
 :)

 The assert statements are commented out in these tests, with FIXME
mark.

 Furthurmore, I also find some platform dependent behaviours of
FileChannel.
 If TestNG is applied on NIO, I will supplement new tests for 
FileChannel

and
 fix the bug of source code.

 What's your opnion? Any suggestions/comments?

 Thanks!

 Best

Re: [classlib] Testing conventions - a proposal