Re: [classlib] millions of rmi tests
Alexei Zakharov skrev den 31-05-2006 11:50: Hi, Just like to pay your attention on JUnit best practices concerning this topic (in case if you are talking about UNIT tests): http://junit.sourceforge.net/doc/faq/faq.htm#best_2 Especially paragraphs two and three. If I understand this correctly this is about testing RMI where the whole point is that get/set are not too simple to break anymore :) Glad that care is taken to getting this absolutely right. -- Thorbjørn smime.p7s Description: S/MIME Cryptographic Signature
Re: [classlib] millions of rmi tests
Mikhail Loenko wrote: 2006/5/31, Geir Magnusson Jr <[EMAIL PROTECTED]>: Are there really 5000+ tests against *one* CTOR? [SNIP] I did not review all the 5000+ tests manually. I'm actually very happy to hear that. geir - Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [classlib] millions of rmi tests
2006/5/31, Geir Magnusson Jr <[EMAIL PROTECTED]>: Mikhail Loenko wrote: > 2006/5/30, Geir Magnusson Jr <[EMAIL PROTECTED]>: >> >> >> Mikhail Loenko wrote: >> > It seems that instead of those million test cases we need just a few >> > that would verify that getXXX() methods return what was passed into >> > constructor >> > plus possibly some tests that pass 'suspicious' parameters like null. >> > >> > Thoughts? >> >> I don't agree that would be enough - having tests that assualt the CTORs >> w/ combinations of crap to see what happens isn't a bad thing. Not a >> fun thing to write, but certainly not something I vote on throwing away >> if someone wrote it already. > > They were not written, they were generated. And I suspect that if we change > some options to test generator then it can generate as many tests as you > have > free space on your hard drive. Do we need them all in the test suite? > > My question is given that so far we had ~7500 tests against all the API do > we need 5000+ tests against a single trivial constructor? Are there really 5000+ tests against *one* CTOR? Against two. 800+ tests call the constructor only. All other tests that I've seen call constructor plus some getter (I think they are also against the constructors). Possibly there are tests that do something else, I did not review all the 5000+ tests manually. Thanks, Mikhail > >> Why do you see a problem? > > The problem is test execution time. Then pitch what you think isn't needed.Any other thoughts? > >> I think that it's misleading to trust only that getters return what's >> passed in... > > Sorry, I did not catch I mean that you had suggested that we'd be ok by testing to see if getters (getXxxx()) returned what was passed into a CTOR correctly, and I thought that wasn't a safe assumption. geir - Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [classlib] millions of rmi tests
Mikhail Loenko wrote: 2006/5/30, Geir Magnusson Jr <[EMAIL PROTECTED]>: Mikhail Loenko wrote: > It seems that instead of those million test cases we need just a few > that would verify that getXXX() methods return what was passed into > constructor > plus possibly some tests that pass 'suspicious' parameters like null. > > Thoughts? I don't agree that would be enough - having tests that assualt the CTORs w/ combinations of crap to see what happens isn't a bad thing. Not a fun thing to write, but certainly not something I vote on throwing away if someone wrote it already. They were not written, they were generated. And I suspect that if we change some options to test generator then it can generate as many tests as you have free space on your hard drive. Do we need them all in the test suite? My question is given that so far we had ~7500 tests against all the API do we need 5000+ tests against a single trivial constructor? Are there really 5000+ tests against *one* CTOR? Why do you see a problem? The problem is test execution time. Then pitch what you think isn't needed.Any other thoughts? I think that it's misleading to trust only that getters return what's passed in... Sorry, I did not catch I mean that you had suggested that we'd be ok by testing to see if getters (getXxxx()) returned what was passed into a CTOR correctly, and I thought that wasn't a safe assumption. geir - Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [classlib] millions of rmi tests
Hi,
Just like to pay your attention on JUnit best practices concerning
this topic (in case if you are talking about UNIT tests):
http://junit.sourceforge.net/doc/faq/faq.htm#best_2
Especially paragraphs two and three.
2006/5/31, Mikhail Loenko <[EMAIL PROTECTED]>:
Hi Daniel,
I think what we need is tests that verify that our implementation works
according to our design.
As I understand the design for that constructor is as follows: "store all
parameters in private fields and don't do any special for special parameters".
That means that we need just two test methods:
- test that passes 'normal' parameters to the constructor and verifies
private fields
by getXX methods (or with access to internal stuff)
- test that passes 'suspicious' values for all the parameters and verifies that
no exception thrown and private fields are set as expected.
The broad testing you've mentioned does make much sense on design stage:
e.g. if the spec is not clear enough you might want to do a black-box
testing of RI.
For example, for an integer parameter of a method you might want to go
through all the integer values and explore how it works on RI.
But once you explored it and found that e.g. it works this way for all positive
values and that way for all negative values and zero, you do an
appropriate design
and include a small number of unit tests to the test suite (like passing
0, 1, -1, 10, 10, minvalue, maxvalue, couple more). You don't have to
include all the tests that you used for design.
Please let me know what you think
Thanks,
Mikhail
2006/5/30, Daniel Gandara <[EMAIL PROTECTED]>:
> Mikhail,
>Let me explain the reasons why we developed this amount of tests;
> We decided to develop implementation independant tests cases which
> will show if a given implementation of the package behaves as it should
> (RI); we also decided to implement them at the same time the package
> was being coded, so no implementation specific detail was known by
> the time the tests were developed.
> In order to do this we developed:
> - Unit tests: for them we took different approachs, which were:
>- broad tests: where we test each public method with a set of
> possible inputs
>- specific tests: using 'suspicious' parameters like null
>unit tests were developed using a mix of automatic code generation
>and regular coding.
> - Integration tests: http tunneling and distributed integration tests
>which provide a more realistic and "smart" test for the package.
>
> All this tests allowed us to check the correct functionality of our
> package.
>
> That said, I will agree with you that at this moment some of this tests
> (mostly basic JUnit tests) might not be very usefull, and I would add
> that more and complex integration tests should be written to check
> the correct functionality of rmi package.
> At this moment we are running all tests (both itc and intel) against
> contributed rmi packages (both itc and intel); I'll send the report as
> soon as I have it.
>
> Thanks,
>
> Daniel
> note: latest tests source code and doc can be found at
> http://issues.apache.org/jira/browse/HARMONY-473
> btw: there were not millions but a few thousands tests :)
>
> - Original Message -
> From: "Mikhail Loenko" <[EMAIL PROTECTED]>
> To:
> Sent: Monday, May 29, 2006 10:01 AM
> Subject: [classlib] millions of rmi tests
>
>
> I've tried to integrate rmi2 tests to rmi module, and found some odd
> things.
>
> Let's take a look for example at TestActivationGroupDesc.java
>
> it has 5158 test methods, most of which are very similar.
> For example it has 855 tests that invoke constructor with various parameters
> and check that new did not return null and no exception was thrown:
>
> Compare
>
> public void
>
testActivationGroupDescStringStringMarshalledObjectPropertiesCommandEnvironment006()
> {
>
>try{
>Properties p= new Properties();
>assertNotNull(msgNotNull, new ActivationGroupDesc(null , null ,
>new MarshalledObject(new Double(23.4)) , new Properties()
> ,
>new ActivationGroupDesc.CommandEnvironment("Hola la",
>new String[0])));
>} catch (Throwable e) {
>fail(msgNoException+e);
>}
> }
>
> and
>
> public void
>
testActivationGroupDescStringStringMarshalledObjectPropertiesCommandEnvironment007()
> {
>try{
>Properties p= new Properties();
>assertNotNull(msgNotNull, new ActivationGroupDesc(null , null ,
>new MarshalledObject(new Double(23.4)) , new Properties()
> ,
>new ActivationGroupDesc.CommandEnvironment("", null)));
>} catch (Throwable e) {
>fail(msgNoException+e);
>}
> }
>
>
> This is how the constructor under test looks like:
> public ActivationGroupDesc(String className, String codebase,
>MarshalledObject data, Properties props,
>ActivationGroupDesc.CommandEnvironment env) {
>
>th
Re: [classlib] millions of rmi tests
Hi Daniel,
I think what we need is tests that verify that our implementation works
according to our design.
As I understand the design for that constructor is as follows: "store all
parameters in private fields and don't do any special for special parameters".
That means that we need just two test methods:
- test that passes 'normal' parameters to the constructor and verifies
private fields
by getXX methods (or with access to internal stuff)
- test that passes 'suspicious' values for all the parameters and verifies that
no exception thrown and private fields are set as expected.
The broad testing you've mentioned does make much sense on design stage:
e.g. if the spec is not clear enough you might want to do a black-box
testing of RI.
For example, for an integer parameter of a method you might want to go
through all the integer values and explore how it works on RI.
But once you explored it and found that e.g. it works this way for all positive
values and that way for all negative values and zero, you do an
appropriate design
and include a small number of unit tests to the test suite (like passing
0, 1, -1, 10, 10, minvalue, maxvalue, couple more). You don't have to
include all the tests that you used for design.
Please let me know what you think
Thanks,
Mikhail
2006/5/30, Daniel Gandara <[EMAIL PROTECTED]>:
Mikhail,
Let me explain the reasons why we developed this amount of tests;
We decided to develop implementation independant tests cases which
will show if a given implementation of the package behaves as it should
(RI); we also decided to implement them at the same time the package
was being coded, so no implementation specific detail was known by
the time the tests were developed.
In order to do this we developed:
- Unit tests: for them we took different approachs, which were:
- broad tests: where we test each public method with a set of
possible inputs
- specific tests: using 'suspicious' parameters like null
unit tests were developed using a mix of automatic code generation
and regular coding.
- Integration tests: http tunneling and distributed integration tests
which provide a more realistic and "smart" test for the package.
All this tests allowed us to check the correct functionality of our
package.
That said, I will agree with you that at this moment some of this tests
(mostly basic JUnit tests) might not be very usefull, and I would add
that more and complex integration tests should be written to check
the correct functionality of rmi package.
At this moment we are running all tests (both itc and intel) against
contributed rmi packages (both itc and intel); I'll send the report as
soon as I have it.
Thanks,
Daniel
note: latest tests source code and doc can be found at
http://issues.apache.org/jira/browse/HARMONY-473
btw: there were not millions but a few thousands tests :)
- Original Message -
From: "Mikhail Loenko" <[EMAIL PROTECTED]>
To:
Sent: Monday, May 29, 2006 10:01 AM
Subject: [classlib] millions of rmi tests
I've tried to integrate rmi2 tests to rmi module, and found some odd
things.
Let's take a look for example at TestActivationGroupDesc.java
it has 5158 test methods, most of which are very similar.
For example it has 855 tests that invoke constructor with various parameters
and check that new did not return null and no exception was thrown:
Compare
public void
testActivationGroupDescStringStringMarshalledObjectPropertiesCommandEnvironment006()
{
try{
Properties p= new Properties();
assertNotNull(msgNotNull, new ActivationGroupDesc(null , null ,
new MarshalledObject(new Double(23.4)) , new Properties()
,
new ActivationGroupDesc.CommandEnvironment("Hola la",
new String[0])));
} catch (Throwable e) {
fail(msgNoException+e);
}
}
and
public void
testActivationGroupDescStringStringMarshalledObjectPropertiesCommandEnvironment007()
{
try{
Properties p= new Properties();
assertNotNull(msgNotNull, new ActivationGroupDesc(null , null ,
new MarshalledObject(new Double(23.4)) , new Properties()
,
new ActivationGroupDesc.CommandEnvironment("", null)));
} catch (Throwable e) {
fail(msgNoException+e);
}
}
This is how the constructor under test looks like:
public ActivationGroupDesc(String className, String codebase,
MarshalledObject data, Properties props,
ActivationGroupDesc.CommandEnvironment env) {
this.className = className;
this.location = codebase;
this.data = data;
this.props = props;
this.env = env;
}
It seems that instead of those million test cases we need just a few
that would verify that getXXX() methods return what was passed into
constructor
plus possibly some tests that pass 'suspicious' parameters like null.
Thoughts?
Thanks,
Mikhail
Re: [classlib] millions of rmi tests
2006/5/30, Geir Magnusson Jr <[EMAIL PROTECTED]>:
Mikhail Loenko wrote:
> I've tried to integrate rmi2 tests to rmi module, and found some odd
> things.
>
> Let's take a look for example at TestActivationGroupDesc.java
>
> it has 5158 test methods, most of which are very similar.
> For example it has 855 tests that invoke constructor with various
> parameters
> and check that new did not return null and no exception was thrown:
Can 'new' actually return null? My read of the JLS says that it won't.
it should be a bug in VM to return null. So comparing to null is not necessary
in the API tests.
>
> Compare
>
> public void
>
testActivationGroupDescStringStringMarshalledObjectPropertiesCommandEnvironment006()
>
> {
>
>try{
>Properties p= new Properties();
>assertNotNull(msgNotNull, new ActivationGroupDesc(null , null ,
>new MarshalledObject(new Double(23.4)) , new
> Properties() ,
>new ActivationGroupDesc.CommandEnvironment("Hola la",
>new String[0])));
>} catch (Throwable e) {
>fail(msgNoException+e);
>}
> }
>
> and
>
> public void
>
testActivationGroupDescStringStringMarshalledObjectPropertiesCommandEnvironment007()
>
> {
>try{
>Properties p= new Properties();
>assertNotNull(msgNotNull, new ActivationGroupDesc(null , null ,
>new MarshalledObject(new Double(23.4)) , new
> Properties() ,
>new ActivationGroupDesc.CommandEnvironment("", null)));
>} catch (Throwable e) {
>fail(msgNoException+e);
>}
> }
>
>
> This is how the constructor under test looks like:
> public ActivationGroupDesc(String className, String codebase,
>MarshalledObject data, Properties props,
>ActivationGroupDesc.CommandEnvironment env) {
>
>this.className = className;
>this.location = codebase;
>this.data = data;
>this.props = props;
>this.env = env;
> }
>
> It seems that instead of those million test cases we need just a few
> that would verify that getXXX() methods return what was passed into
> constructor
> plus possibly some tests that pass 'suspicious' parameters like null.
>
> Thoughts?
I don't agree that would be enough - having tests that assualt the CTORs
w/ combinations of crap to see what happens isn't a bad thing. Not a
fun thing to write, but certainly not something I vote on throwing away
if someone wrote it already.
They were not written, they were generated. And I suspect that if we change
some options to test generator then it can generate as many tests as you have
free space on your hard drive. Do we need them all in the test suite?
My question is given that so far we had ~7500 tests against all the API do
we need 5000+ tests against a single trivial constructor?
Why do you see a problem?
The problem is test execution time.
I think that it's misleading to trust only that getters return what's
passed in...
Sorry, I did not catch
Thanks,
Mikhail
geir
-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: [classlib] millions of rmi tests
Mikhail,
Let me explain the reasons why we developed this amount of tests;
We decided to develop implementation independant tests cases which
will show if a given implementation of the package behaves as it should
(RI); we also decided to implement them at the same time the package
was being coded, so no implementation specific detail was known by
the time the tests were developed.
In order to do this we developed:
- Unit tests: for them we took different approachs, which were:
- broad tests: where we test each public method with a set of
possible inputs
- specific tests: using 'suspicious' parameters like null
unit tests were developed using a mix of automatic code generation
and regular coding.
- Integration tests: http tunneling and distributed integration tests
which provide a more realistic and "smart" test for the package.
All this tests allowed us to check the correct functionality of our
package.
That said, I will agree with you that at this moment some of this tests
(mostly basic JUnit tests) might not be very usefull, and I would add
that more and complex integration tests should be written to check
the correct functionality of rmi package.
At this moment we are running all tests (both itc and intel) against
contributed rmi packages (both itc and intel); I'll send the report as
soon as I have it.
Thanks,
Daniel
note: latest tests source code and doc can be found at
http://issues.apache.org/jira/browse/HARMONY-473
btw: there were not millions but a few thousands tests :)
- Original Message -
From: "Mikhail Loenko" <[EMAIL PROTECTED]>
To:
Sent: Monday, May 29, 2006 10:01 AM
Subject: [classlib] millions of rmi tests
I've tried to integrate rmi2 tests to rmi module, and found some odd
things.
Let's take a look for example at TestActivationGroupDesc.java
it has 5158 test methods, most of which are very similar.
For example it has 855 tests that invoke constructor with various parameters
and check that new did not return null and no exception was thrown:
Compare
public void
testActivationGroupDescStringStringMarshalledObjectPropertiesCommandEnvironment006()
{
try{
Properties p= new Properties();
assertNotNull(msgNotNull, new ActivationGroupDesc(null , null ,
new MarshalledObject(new Double(23.4)) , new Properties()
,
new ActivationGroupDesc.CommandEnvironment("Hola la",
new String[0])));
} catch (Throwable e) {
fail(msgNoException+e);
}
}
and
public void
testActivationGroupDescStringStringMarshalledObjectPropertiesCommandEnvironment007()
{
try{
Properties p= new Properties();
assertNotNull(msgNotNull, new ActivationGroupDesc(null , null ,
new MarshalledObject(new Double(23.4)) , new Properties()
,
new ActivationGroupDesc.CommandEnvironment("", null)));
} catch (Throwable e) {
fail(msgNoException+e);
}
}
This is how the constructor under test looks like:
public ActivationGroupDesc(String className, String codebase,
MarshalledObject data, Properties props,
ActivationGroupDesc.CommandEnvironment env) {
this.className = className;
this.location = codebase;
this.data = data;
this.props = props;
this.env = env;
}
It seems that instead of those million test cases we need just a few
that would verify that getXXX() methods return what was passed into
constructor
plus possibly some tests that pass 'suspicious' parameters like null.
Thoughts?
Thanks,
Mikhail
-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: [classlib] millions of rmi tests
Mikhail Loenko wrote:
I've tried to integrate rmi2 tests to rmi module, and found some odd
things.
Let's take a look for example at TestActivationGroupDesc.java
it has 5158 test methods, most of which are very similar.
For example it has 855 tests that invoke constructor with various
parameters
and check that new did not return null and no exception was thrown:
Can 'new' actually return null? My read of the JLS says that it won't.
Compare
public void
testActivationGroupDescStringStringMarshalledObjectPropertiesCommandEnvironment006()
{
try{
Properties p= new Properties();
assertNotNull(msgNotNull, new ActivationGroupDesc(null , null ,
new MarshalledObject(new Double(23.4)) , new
Properties() ,
new ActivationGroupDesc.CommandEnvironment("Hola la",
new String[0])));
} catch (Throwable e) {
fail(msgNoException+e);
}
}
and
public void
testActivationGroupDescStringStringMarshalledObjectPropertiesCommandEnvironment007()
{
try{
Properties p= new Properties();
assertNotNull(msgNotNull, new ActivationGroupDesc(null , null ,
new MarshalledObject(new Double(23.4)) , new
Properties() ,
new ActivationGroupDesc.CommandEnvironment("", null)));
} catch (Throwable e) {
fail(msgNoException+e);
}
}
This is how the constructor under test looks like:
public ActivationGroupDesc(String className, String codebase,
MarshalledObject data, Properties props,
ActivationGroupDesc.CommandEnvironment env) {
this.className = className;
this.location = codebase;
this.data = data;
this.props = props;
this.env = env;
}
It seems that instead of those million test cases we need just a few
that would verify that getXXX() methods return what was passed into
constructor
plus possibly some tests that pass 'suspicious' parameters like null.
Thoughts?
I don't agree that would be enough - having tests that assualt the CTORs
w/ combinations of crap to see what happens isn't a bad thing. Not a
fun thing to write, but certainly not something I vote on throwing away
if someone wrote it already.
Why do you see a problem?
I think that it's misleading to trust only that getters return what's
passed in...
geir
-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: [classlib] millions of rmi tests
I would agree with the inference that this seems overly verbose and complex.
I think your suggestion seems appropriate in this case. We don't need to
test ever possible combination of parameters, especially when the method
isn't doing anything.
-Nathan
> -Original Message-
> From: Mikhail Loenko [mailto:[EMAIL PROTECTED]
> Sent: Monday, May 29, 2006 8:02 AM
> To: harmony-dev@incubator.apache.org
> Subject: [classlib] millions of rmi tests
>
> I've tried to integrate rmi2 tests to rmi module, and found some odd
> things.
>
> Let's take a look for example at TestActivationGroupDesc.java
>
> it has 5158 test methods, most of which are very similar.
> For example it has 855 tests that invoke constructor with various
> parameters
> and check that new did not return null and no exception was thrown:
>
> Compare
>
> public void
> testActivationGroupDescStringStringMarshalledObjectPropertiesCommandEnviro
> nment006()
> {
>
> try{
> Properties p= new Properties();
> assertNotNull(msgNotNull, new ActivationGroupDesc(null , null ,
> new MarshalledObject(new Double(23.4)) , new
> Properties() ,
> new ActivationGroupDesc.CommandEnvironment("Hola la",
> new String[0])));
> } catch (Throwable e) {
> fail(msgNoException+e);
> }
> }
>
> and
>
> public void
> testActivationGroupDescStringStringMarshalledObjectPropertiesCommandEnviro
> nment007()
> {
> try{
> Properties p= new Properties();
> assertNotNull(msgNotNull, new ActivationGroupDesc(null , null ,
> new MarshalledObject(new Double(23.4)) , new
> Properties() ,
> new ActivationGroupDesc.CommandEnvironment("", null)));
> } catch (Throwable e) {
> fail(msgNoException+e);
> }
> }
>
>
> This is how the constructor under test looks like:
> public ActivationGroupDesc(String className, String codebase,
> MarshalledObject data, Properties props,
> ActivationGroupDesc.CommandEnvironment env) {
>
> this.className = className;
> this.location = codebase;
> this.data = data;
> this.props = props;
> this.env = env;
> }
>
> It seems that instead of those million test cases we need just a few
> that would verify that getXXX() methods return what was passed into
> constructor
> plus possibly some tests that pass 'suspicious' parameters like null.
>
> Thoughts?
>
> Thanks,
> Mikhail
>
> -
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
