Hi everybody,
Apologies for the cross posting - but the implementation of the TCK
is relevant to both camps.
One of the things we need to do before making a release available is to
produce a TCK. By "TCK" I mean
a test harness that will:
1. Setup a JVM into a known configuration and generate a dump.
2. Open a dump with the Kato API and run unit tests against it.
Whether or not these are just functional tests or technology compliance
is something I will just gloss over for now
as just a classification issue.
As we all know, there is a problem with optionality in the API. There is
information that will be lost when using
certain dump types.
Steve has suggested two modes for running the TCK.
Legacy
========
This mode runs through the API and confirms that the basic API behaves
as it is intended to do so.
It checks that the rules of the API are adhered to, but doesn't expect
any method will return any
particular piece of information.
For example:
List<ImageProcess> ImageAddressSpace.getProcesses();
processes would be checked to ensure that it is not null. If there were
contents
they would be queried, and their methods executed.
For example:
String ImageProcess.getID();
Would be expected to return a String or throw a DataUnavailable or
CorruptDataException exception.
The actual contents of the string wouldn't be checked - just so long as
it isn't null.
(incidentally, actually checking the value in this example for
correctness is difficult).
1.0
=====
This is a test for exact compliance with the API. All of the information
that can be retrieved from the dump
must be retrieved and it must all be correct.
With some caveats.
Like with ImageProcess.getID(), there is some information that is poorly
specified or difficult to know in
advance.
The poorly specified aspects of the API are:
Platform specific items.
For example:
Implementation specific items:
toString() methods - what do they print?
Things that are difficult to know in advance:
Native stack frames - ordering and naming.
JavaThreads - there will be threads that are implementation specific.
There problems aren't intractable. A certain amount of flexibility will
be necessary to accommodate variations.
We should expect there to be more than what we expect, so we should just
look for what we expect to be there,
and ignore the rest.
The poorly specified aspects of the API should just be more precisely
specified. Although it means that the TCK will
have to be aware of the different platforms and their behaviour.
I'm not a great fan of the "Legacy" mode. While it may have a basic use for
testing basic behaviour, I would like for there to be stronger statement about what we
expect the API to do.
For example, if we call JavaRuntime.getThreads(), I would expect a List<JavaThread> to be returned. However, if it wasn't empty, I would want the threads we put into the testcase to be there and identifiable by their (correct) names.
i.e. if there is something to test we should test it.
I call this mode "Permissive" (alternatives: lenient, acquiescent, lax,
liberal, tolerant)
There are, of course, problems. But I would expect that is this mode wasn't
possible or advisable, it would be improbable that a program could be writtent
to call the API.
The end result is that if you get information out of the API, it must be
correct. Where you don't, it is ignored.
Thoughts?
Thoughts?
Regards,
Stuart
--
Stuart Monteith
http://blog.stoo.me.uk/
