Hi Jens,
The dependencies are straightforward: OpenCMIS Client API and Ant for
the Ant Task. That's it. It's pretty lightweight.
JUnit is designed for a different kind of tests. A test fails if a check
fails. That's the right behavior for a unit test but not for a TCK.
If, let's say, the allowable actions of an object don't match our
expectations, it shouldn't stop us from checking the properties, the ACL
and the version history. We want that recorded but, as long as it is not
a fatal error, it should not interrupt the test flow.
Another detail that the framework provides is a logical stack trace.
Let's say we test document creation. We fetch the document right after
it has been created in order to check its properties, etc. If a property
value check fails the report could look like this:
FAILED: document creation
FAILED: check of newly created document '0000-1111-2222-3333'
FAILED: property check 'my:prop1'
INFO: excepted "2" / actual "3"
FAILED: property check 'my:prop2'
INFO: excepted "abc" / actual "xyz"
FAILED: property check 'my:prop3'
INFO: excepted "true" / actual "false"
Junit would just return the 'excepted "2" / actual "3"' part and a stack
trace. The context, the document id and the property id would be lost.
And it would stop right after the first mismatch.
- Florian
On 27/10/2010 07:32, Jens Hübel wrote:
Hi all,
I agree that this would be useful and I think the opencmis project would be a
good place to maintain and publish this code base.
Could you tell us a bit more about the dependencies of the test framework? At
least at first glance I wonder if we really need more than junit and perhaps a
thin layer on top of it.
-----Original Message-----
From: Florian Müller [mailto:[email protected]]
Sent: Dienstag, 26. Oktober 2010 16:35
To: [email protected]
Subject: CMIS TCK Framework
Hi all,
We (Alfresco) are implementing a TCK like CMIS test framework that is based on
OpenCMIS. If the Apache Chemistry community is interested in this we would
contribute it to OpenCMIS.
Here is what is does:
- A TCK test tests one specific aspect, for example the creation of a folder or
a query feature. A test consists of a series of checks, comparable to JUnit
asserts. In contrast to JUnit, a check can return one of these states:
* SKIPPED - The check has been skipped because the binding or the repository
doesn't support it
* OK - The check has passed.
* WARNING - The check hasn't passed. It is not a spec violation but you want
to have a look at it. For example, an object has an empty cmis:name property or
a type definition has no display name.
* FAILURE - The check hasn't passed. It is some kind of a spec violation.
Failures can be fatal or not. If a failure is fatal, the test stops here. If it
is not fatal, the test moves on.
* UNEXPECTED_EXCEPTION - The framework caught an exception that hasn't been
handled by the test.
- Tests are organized in groups. The TCK consists of a collection of groups.
Each group and each test can be individually enabled and disabled.
- The framework can generate reports. Currently, we have simple text, XML and
HTML report generators. The reports are not very pretty yet and need a bit of
work.
- There are multiple ways to start the TCK. At the moment, we have a simple
console runner and an ANT task. But it should be pretty easy to run it from a
servlet or embed into a Swing application or something like that.
In a first step we want to port all the tests in the OpenCMIS client binding
package to the framework and add the AtomPub tests that Dave developed a while
ago. After that we will add more and more tests until we think we have covered
most of the spec.
Any opinions?
Cheers,
Florian
