This is an automated email from the ASF dual-hosted git repository.
veithen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ws-axiom.git
The following commit(s) were added to refs/heads/master by this push:
new 5be18c92f Add matrix-testsuite docs and remove design document
5be18c92f is described below
commit 5be18c92f814640ea92082957d300043a5eb90b0
Author: Andreas Veithen-Knowles <[email protected]>
AuthorDate: Sun Mar 1 07:30:45 2026 +0000
Add matrix-testsuite docs and remove design document
Add README.md with an overview of the new MatrixTestSuite framework
and migration.md with step-by-step instructions for migrating from
MatrixTestSuiteBuilder. Delete the design document that preceded the
implementation.
---
docs/design/README.md | 1 -
docs/design/test-suite-pattern.md | 612 ----------------------------------
testing/matrix-testsuite/README.md | 199 +++++++++++
testing/matrix-testsuite/migration.md | 290 ++++++++++++++++
4 files changed, 489 insertions(+), 613 deletions(-)
diff --git a/docs/design/README.md b/docs/design/README.md
index dedcaefe7..4308e0f3c 100644
--- a/docs/design/README.md
+++ b/docs/design/README.md
@@ -23,4 +23,3 @@ Design documents
| Title/link | Status |
| ---------- | ------ |
| [OSGi integration and separation between API and
implementation](osgi-integration.md) | Implemented |
-| [Reusable test suites and parameterization](test-suite-pattern.md) | In
review |
diff --git a/docs/design/test-suite-pattern.md
b/docs/design/test-suite-pattern.md
deleted file mode 100644
index f239b1778..000000000
--- a/docs/design/test-suite-pattern.md
+++ /dev/null
@@ -1,612 +0,0 @@
-<!--
- ~ Licensed to the Apache Software Foundation (ASF) under one
- ~ or more contributor license agreements. See the NOTICE file
- ~ distributed with this work for additional information
- ~ regarding copyright ownership. The ASF licenses this file
- ~ to you under the Apache License, Version 2.0 (the
- ~ "License"); you may not use this file except in compliance
- ~ with the License. You may obtain a copy of the License at
- ~
- ~ http://www.apache.org/licenses/LICENSE-2.0
- ~
- ~ Unless required by applicable law or agreed to in writing,
- ~ software distributed under the License is distributed on an
- ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- ~ KIND, either express or implied. See the License for the
- ~ specific language governing permissions and limitations
- ~ under the License.
- -->
-
-Reusable test suites and parameterization
-=========================================
-
-## Introduction
-
-The Axiom project provides reusable API test suites that can be applied to
different
-implementations of the same API. For example, `saaj-testsuite` defines tests
for the
-SAAJ API that can be executed against any `SAAJMetaFactory`.
-
-Test suites also execute tests across multiple dimensions. For instance, SAAJ
tests
-run against both SOAP 1.1 and SOAP 1.2.
-
-This document examines the current pattern used to implement these test suites
and
-evaluates whether JUnit 5's `@TestFactory` mechanism offers a better approach.
-
-## Current pattern: MatrixTestSuiteBuilder (JUnit 3)
-
-### Infrastructure classes
-
-The current pattern is built on the following custom infrastructure in the
`testutils`
-module:
-
-* **`MatrixTestCase`** — extends `junit.framework.TestCase`. Each test case
is a
- separate class that overrides `runTest()`. Test parameters (e.g. SOAP
version) are
- added via `addTestParameter(name, value)`, which appends `[name=value]` to
the test
- name for display purposes.
-
-* **`MatrixTestSuiteBuilder`** — builds a `junit.framework.TestSuite` by
collecting
- `MatrixTestCase` instances via `addTest()` calls in the abstract
`addTests()` method.
- Supports exclusions using LDAP-style filters on test parameters.
-
-### How it works in saaj-testsuite
-
-The saaj-testsuite uses this pattern as follows:
-
-1. Each test case is a separate class extending `SAAJTestCase` (which extends
- `MatrixTestCase`). For example, `TestAddChildElementReification`,
- `TestExamineMustUnderstandHeaderElements`, etc.
-
-2. `SAAJTestSuiteBuilder` extends `MatrixTestSuiteBuilder`. Its `addTests()`
method
- calls a private `addTests(SOAPSpec)` helper for both `SOAPSpec.SOAP11` and
- `SOAPSpec.SOAP12`, instantiating each test case class with the SAAJ
implementation
- and the SOAP spec.
-
-3. `SAAJTestCase` provides convenience methods `newMessageFactory()` and
- `newSOAPFactory()` that create the appropriate factory for the current
SOAP version.
-
-4. Consumers create a JUnit 3 runner class with a `static suite()` method:
-
- ```java
- public class SAAJRITest extends TestCase {
- public static TestSuite suite() throws Exception {
- return new SAAJTestSuiteBuilder(new SAAJMetaFactoryImpl()).build();
- }
- }
- ```
-
-### File inventory for saaj-testsuite
-
-For 6 test cases × 2 SOAP versions = 12 test instances, the main files are:
-
-| File | Role |
-|------|------|
-| `SAAJTestCase.java` | Abstract base class for all SAAJ tests |
-| `SAAJTestSuiteBuilder.java` | Suite builder; registers all tests × SOAP
versions |
-| `SAAJImplementation.java` | Wraps `SAAJMetaFactory` with reflective access |
-| `TestAddChildElementReification.java` | Test case class |
-| `TestAddChildElementLocalName.java` | Test case class |
-| `TestAddChildElementLocalNamePrefixAndURI.java` | Test case class |
-| `TestSetParentElement.java` | Test case class |
-| `TestGetOwnerDocument.java` | Test case class |
-| `TestExamineMustUnderstandHeaderElements.java` | Test case class |
-| `SAAJRITest.java` | JUnit 3 runner for the reference implementation |
-
-## Alternative: JUnit 5 @TestFactory + DynamicTest
-
-JUnit 5 provides a built-in mechanism for dynamic test generation that directly
-addresses the same use case.
-
-### Key JUnit 5 features
-
-* **`@TestFactory`** — a method that returns a `Stream<DynamicNode>` (or
`Collection`,
- `Iterable`, etc.). Each `DynamicNode` becomes a test in the test tree.
-
-* **`DynamicContainer`** — groups `DynamicNode` instances under a named
container,
- enabling hierarchical test organization (e.g. grouping by SOAP version).
-
-* **`DynamicTest`** — a named test with an `Executable` body. Replaces the
need for
- a separate class per test case.
-
-* **`@ParameterizedTest`** + `@MethodSource` — an alternative for simpler
- parameterization where each test method receives parameters directly.
-
-### What saaj-testsuite would look like
-
-The reusable test suite module would define an abstract class:
-
-```java
-public abstract class SAAJTests {
- private final SAAJImplementation impl;
-
- protected SAAJTests(SAAJMetaFactory metaFactory) {
- this.impl = new SAAJImplementation(metaFactory);
- }
-
- @TestFactory
- Stream<DynamicContainer> saajTests() {
- return Multiton.getInstances(SOAPSpec.class).stream().map(spec ->
- DynamicContainer.dynamicContainer(spec.getName(), Stream.of(
- testAddChildElementReification(spec),
- testExamineMustUnderstandHeaderElements(spec),
- testAddChildElementLocalName(spec),
- testAddChildElementLocalNamePrefixAndURI(spec),
- testSetParentElement(spec),
- testGetOwnerDocument(spec)
- ))
- );
- }
-
- private DynamicTest testAddChildElementReification(SOAPSpec spec) {
- return DynamicTest.dynamicTest("addChildElementReification", () -> {
- MessageFactory mf = spec.getAdapter(FactorySelector.class)
- .newMessageFactory(impl);
- SOAPBody body = mf.createMessage().getSOAPBody();
- SOAPElement child = body.addChildElement(
- (SOAPElement)
body.getOwnerDocument().createElementNS("urn:test", "p:test"));
- assertThat(child).isInstanceOf(SOAPBodyElement.class);
- });
- }
-
- // ... other test methods ...
-}
-```
-
-Consumers would subclass per implementation:
-
-```java
-class SAAJRITests extends SAAJTests {
- SAAJRITests() {
- super(new SAAJMetaFactoryImpl());
- }
-}
-```
-
-### Comparison
-
-| Concern | MatrixTestSuiteBuilder (JUnit 3) | JUnit 5 @TestFactory |
-|---------|----------------------------------|----------------------|
-| Framework version | JUnit 3 | JUnit 5 (Jupiter) |
-| Test registration | Explicit `addTest()` in builder | Return
`Stream<DynamicNode>` |
-| One class per test case | Required | Not required — tests are methods
returning `DynamicTest` |
-| Boilerplate for saaj-testsuite | 10 files | 2–3 files |
-| Test tree in IDE | Flat list with `[spec=SOAP11]` in name | Nested: SOAP11 >
testName, SOAP12 > testName |
-| Exclusion mechanism | LDAP filter on parameter dictionary | LDAP filter on
`MatrixTestNode` tree (see below) |
-| Reusability across implementations | Subclass `TestCase` + pass factory to
builder | Subclass base test class + pass factory to constructor |
-| Custom infrastructure needed | `MatrixTestSuiteBuilder`, `MatrixTestCase` |
None (built into JUnit 5) |
-
-## Considerations for migration
-
-### saaj-testsuite
-
-For the saaj-testsuite, migrating to JUnit 5 `@TestFactory` would:
-
-* Collapse 6 test case classes into methods within a single class.
-* Remove the need for `SAAJTestSuiteBuilder` entirely.
-* Replace the `SAAJTestCase` base class with a simpler abstract class.
-* Reduce the file count from 10 to approximately 3 (`SAAJImplementation`,
`SAAJTests`,
- `SAAJRITests`).
-
-The `SAAJImplementation` class (which uses reflection to access protected
methods on
-`SAAJMetaFactory`) would be retained as-is.
-
-### Replacement for MatrixTestSuiteBuilder: MatrixTestNode tree with Guice
-
-Since `DynamicContainer` and `DynamicTest` are `final` in JUnit 5, they cannot
be
-subclassed to attach test parameters for LDAP-style filtering. Instead, a
parallel
-class hierarchy acts as a factory for `DynamicNode` instances while carrying
the
-parameters needed for exclusion filtering.
-
-Tests continue to be structured as one test case per class extending
-`junit.framework.TestCase`, with each class overriding `runTest()`. Rather than
-receiving test parameters via constructor arguments, test cases declare their
-dependencies using `@Inject` annotations and are instantiated by Google Guice.
-`MatrixTestContainer` builds a Guice injector hierarchy — one child injector
per
-dimension value — so that by the time a leaf `MatrixTest` is reached, the
-accumulated injector can satisfy all `@Inject` dependencies for the test case
class.
-
-#### Class hierarchy
-
-```java
-/**
- * Base class mirroring {@link DynamicNode}. Represents a node in the test tree
- * that can be filtered before conversion to JUnit 5's dynamic test API.
- *
- * <p>The {@code parentInjector} parameter threads through the tree: each
- * fan-out node ({@link DimensionFanOutNode}, {@link ParameterFanOutNode})
creates
- * child injectors from it, and each {@link MatrixTest} uses it to instantiate
- * the test class.
- */
-public abstract class MatrixTestNode {
- abstract Stream<DynamicNode> toDynamicNodes(Injector parentInjector,
- Dictionary<String, String> inheritedParameters,
- MatrixTestFilters excludes);
-}
-```
-
-```java
-/**
- * Abstract base class for fan-out nodes that iterate over a list of values,
- * creating one {@link DynamicContainer} per value. For each value, a child
- * Guice injector is created that binds the value type to the specific
instance.
- *
- * <p>Subclasses define how test parameters (used for display names and LDAP
- * filter matching) are extracted from each value:
- * <ul>
- * <li>{@link DimensionFanOutNode} — for types that implement {@link
Dimension},
- * using {@link Dimension#addTestParameters}.
- * <li>{@link ParameterFanOutNode} — for arbitrary types, using a
caller-supplied
- * parameter name and {@link Function}.
- * </ul>
- *
- * @param <T> the value type
- */
-public abstract class AbstractFanOutNode<T> extends MatrixTestNode {
- private final Class<T> type;
- private final List<T> values;
- private final List<MatrixTestNode> children = new ArrayList<>();
-
- protected AbstractFanOutNode(Class<T> type, List<T> values) {
- this.type = type;
- this.values = values;
- }
-
- public void addChild(MatrixTestNode child) {
- children.add(child);
- }
-
- /**
- * Extracts test parameters from the given value. The returned map entries
- * are used for the display name and for LDAP filter matching.
- */
- protected abstract Map<String, String> extractParameters(T value);
-
- @Override
- Stream<DynamicNode> toDynamicNodes(Injector parentInjector,
- Dictionary<String, String> inheritedParameters,
- MatrixTestFilters excludes) {
- return values.stream().map(value -> {
- Injector childInjector = parentInjector.createChildInjector(new
AbstractModule() {
- @Override
- protected void configure() {
- bind(type).toInstance(value);
- }
- });
-
- Map<String, String> parameters = extractParameters(value);
- Hashtable<String, String> params = new Hashtable<>();
- for (Enumeration<String> e = inheritedParameters.keys();
e.hasMoreElements(); ) {
- String key = e.nextElement();
- params.put(key, inheritedParameters.get(key));
- }
- parameters.forEach(params::put);
- String displayName = parameters.entrySet().stream()
- .map(e -> e.getKey() + "=" + e.getValue())
- .collect(Collectors.joining(", "));
- return DynamicContainer.dynamicContainer(displayName,
- children.stream()
- .flatMap(child ->
child.toDynamicNodes(childInjector, params, excludes)));
- });
- }
-}
-```
-
-```java
-/**
- * Fan-out node for types that implement {@link Dimension}. Parameters are
- * extracted via {@link Dimension#addTestParameters}.
- *
- * <p>For types that do <em>not</em> implement {@code Dimension}, use
- * {@link ParameterFanOutNode} instead.
- *
- * @param <D> the dimension type
- */
-public class DimensionFanOutNode<D extends Dimension> extends
AbstractFanOutNode<D> {
- public DimensionFanOutNode(Class<D> dimensionType, List<D> dimensions) {
- super(dimensionType, dimensions);
- }
-
- @Override
- protected Map<String, String> extractParameters(D dimension) {
- Map<String, String> parameters = new LinkedHashMap<>();
- dimension.addTestParameters(new TestParameterTarget() {
- @Override
- public void addTestParameter(String name, String value) {
- parameters.put(name, value);
- }
-
- @Override
- public void addTestParameter(String name, boolean value) {
- addTestParameter(name, String.valueOf(value));
- }
-
- @Override
- public void addTestParameter(String name, int value) {
- addTestParameter(name, String.valueOf(value));
- }
- });
- return parameters;
- }
-}
-```
-
-```java
-/**
- * Fan-out node for arbitrary value types that do not implement {@link
Dimension}.
- * The caller supplies a parameter name and a {@link Function} that maps each
- * value to its parameter value (used for display names and LDAP filter
matching).
- *
- * <p>For example, {@code SOAPSpec} does not implement {@code Dimension}, so
- * it is handled with:
- *
- * <pre>
- * new ParameterFanOutNode<>(SOAPSpec.class,
- * Multiton.getInstances(SOAPSpec.class),
- * "spec", SOAPSpec::getName)
- * </pre>
- *
- * @param <T> the value type
- */
-public class ParameterFanOutNode<T> extends AbstractFanOutNode<T> {
- private final String parameterName;
- private final Function<T, String> parameterValueFunction;
-
- public ParameterFanOutNode(Class<T> type, List<T> values,
- String parameterName, Function<T, String> parameterValueFunction) {
- super(type, values);
- this.parameterName = parameterName;
- this.parameterValueFunction = parameterValueFunction;
- }
-
- @Override
- protected Map<String, String> extractParameters(T value) {
- return Map.of(parameterName, parameterValueFunction.apply(value));
- }
-}
-```
-
-```java
-/**
- * Mirrors {@link DynamicTest}. A leaf node that instantiates a
- * {@link junit.framework.TestCase} subclass via Guice and executes it.
- *
- * <p>The test class must have an injectable constructor (either a no-arg
- * constructor or one annotated with {@code @Inject}). Field injection is
- * also supported. The injector received from the ancestor
- * {@code MatrixTestContainer} chain will have bindings for all dimension
- * types, plus any implementation-level bindings from the root injector
- * (e.g. {@code SAAJImplementation}).
- *
- * <p>Once the instance is created, it is executed via {@link
TestCase#runBare()},
- * which invokes the full {@code setUp()} → {@code runTest()} → {@code
tearDown()}
- * lifecycle.
- */
-public class MatrixTest extends MatrixTestNode {
- private final Class<? extends TestCase> testClass;
-
- public MatrixTest(Class<? extends TestCase> testClass) {
- this.testClass = testClass;
- }
-
- @Override
- Stream<DynamicNode> toDynamicNodes(Injector injector,
- Dictionary<String, String> inheritedParameters,
- MatrixTestFilters excludes) {
- if (excludes.test(testClass, inheritedParameters)) {
- return Stream.empty(); // Excluded
- }
- return Stream.of(DynamicTest.dynamicTest(testClass.getSimpleName(), ()
-> {
- TestCase testInstance = injector.getInstance(testClass);
- testInstance.setName(testClass.getSimpleName());
- testInstance.runBare();
- }));
- }
-}
-```
-
-```java
-/**
- * Root of a test suite. Owns the Guice root injector and the tree of
- * {@link MatrixTestNode} instances. Provides a {@link
#toDynamicNodes(MatrixTestFilters)}
- * method that converts the tree to JUnit 5 dynamic nodes, applying the
- * supplied exclusion filters.
- *
- * <p>Exclusion filters are <em>not</em> owned by the suite itself because
- * they are specific to each consumer (implementation under test), whereas
- * the suite structure and bindings are defined by the test suite author.
- */
-public class MatrixTestSuite {
- private final Injector rootInjector;
- private final List<MatrixTestNode> children = new ArrayList<>();
-
- public MatrixTestSuite(Module... modules) {
- this.rootInjector = Guice.createInjector(modules);
- }
-
- public void addChild(MatrixTestNode child) {
- children.add(child);
- }
-
- public Stream<DynamicNode> toDynamicNodes(MatrixTestFilters excludes) {
- return children.stream()
- .flatMap(child -> child.toDynamicNodes(
- rootInjector, new Hashtable<>(), excludes));
- }
-}
-```
-
-#### Guice injector hierarchy
-
-The injector hierarchy mirrors the fan-out node nesting. The root injector is
created
-by the consumer and binds implementation-level objects. Each
`DimensionFanOutNode` or
-`ParameterFanOutNode` level creates one child injector per value, binding the
value type.
-By the time a leaf `MatrixTest` is reached, the injector can satisfy all
`@Inject`
-dependencies.
-
-```
-Root Injector
- binds: SAAJImplementation → instance
- │
- ├─ Child Injector (SOAPSpec → SOAP11)
- │ │
- │ ├─ MatrixTest →
injector.getInstance(TestAddChildElementReification.class)
- │ │ → testInstance.runBare()
- │ └─ MatrixTest → injector.getInstance(TestGetOwnerDocument.class)
- │ → testInstance.runBare()
- │
- └─ Child Injector (SOAPSpec → SOAP12)
- │
- ├─ MatrixTest →
injector.getInstance(TestAddChildElementReification.class)
- │ → testInstance.runBare()
- └─ MatrixTest → injector.getInstance(TestGetOwnerDocument.class)
- → testInstance.runBare()
-```
-
-#### What test case classes look like
-
-Test case classes continue to extend `junit.framework.TestCase` (or a
domain-specific
-subclass) and override `runTest()`. The key difference is that dependencies
are injected
-by Guice rather than passed via constructor arguments.
-
-**Before (constructor parameters):**
-
-```java
-public class TestAddChildElementReification extends SAAJTestCase {
- public TestAddChildElementReification(SAAJImplementation
saajImplementation, SOAPSpec spec) {
- super(saajImplementation, spec);
- }
-
- @Override
- protected void runTest() throws Throwable {
- SOAPBody body = newMessageFactory().createMessage().getSOAPBody();
- SOAPElement child = body.addChildElement(
- (SOAPElement)
body.getOwnerDocument().createElementNS("urn:test", "p:test"));
- assertThat(child).isInstanceOf(SOAPBodyElement.class);
- }
-}
-```
-
-**After (Guice injection):**
-
-```java
-public class TestAddChildElementReification extends SAAJTestCase {
- @Override
- protected void runTest() throws Throwable {
- SOAPBody body = newMessageFactory().createMessage().getSOAPBody();
- SOAPElement child = body.addChildElement(
- (SOAPElement)
body.getOwnerDocument().createElementNS("urn:test", "p:test"));
- assertThat(child).isInstanceOf(SOAPBodyElement.class);
- }
-}
-```
-
-The intermediate base class `SAAJTestCase` uses `@Inject` for its dependencies:
-
-```java
-public abstract class SAAJTestCase extends TestCase {
- @Inject protected SAAJImplementation saajImplementation;
- @Inject protected SOAPSpec spec;
-
- protected final MessageFactory newMessageFactory() throws SOAPException {
- return
spec.getAdapter(FactorySelector.class).newMessageFactory(saajImplementation);
- }
-
- protected final SOAPFactory newSOAPFactory() throws SOAPException {
- return
spec.getAdapter(FactorySelector.class).newSOAPFactory(saajImplementation);
- }
-}
-```
-
-Note that the existing `Dimension.addTestParameters()` mechanism is **not**
used by test
-case classes at all. Parameters are extracted only by `DimensionFanOutNode`
(via
-`Dimension.addTestParameters()`) or `ParameterFanOutNode` (via the supplied
function) for
-display names and filter matching. Test cases interact with values purely as
typed
-objects obtained through injection.
-
-#### How filtering works
-
-Each fan-out node level holds a list of values. When `toDynamicNodes()` is
called,
-each node produces one `DynamicContainer` per value, and parameters accumulate
from
-the root down. For types that implement `Dimension`, use
`DimensionFanOutNode`; for
-arbitrary types (like `SOAPSpec`), use `ParameterFanOutNode`:
-
-```
-ParameterFanOutNode(SOAPSpec.class, [SOAPSpec.SOAP11, SOAPSpec.SOAP12],
"spec", SOAPSpec::getName)
- MatrixTest(TestAddChildElementReification.class)
- MatrixTest(TestGetOwnerDocument.class)
-
-→ spec=soap11 [child injector binds SOAPSpec]
- → TestAddChildElementReification (filtered against {spec=soap11})
- → TestGetOwnerDocument (filtered against {spec=soap11})
- spec=soap12
- → TestAddChildElementReification (filtered against {spec=soap12})
- → TestGetOwnerDocument (filtered against {spec=soap12})
-```
-
-The **test suite author** (in `saaj-testsuite`) defines the suite structure —
which
-test classes to include, which dimensions to iterate over, and what Guice
bindings
-are needed — accepting only the implementation-specific factory as a parameter:
-
-```java
-public class SAAJTestSuite {
- public static MatrixTestSuite create(SAAJMetaFactory metaFactory) {
- SAAJImplementation impl = new SAAJImplementation(metaFactory);
- MatrixTestSuite suite = new MatrixTestSuite(new AbstractModule() {
- @Override
- protected void configure() {
- bind(SAAJImplementation.class).toInstance(impl);
- }
- });
-
- ParameterFanOutNode<SOAPSpec> specs = new ParameterFanOutNode<>(
- SOAPSpec.class, Multiton.getInstances(SOAPSpec.class),
- "spec", SOAPSpec::getName);
- specs.addChild(new MatrixTest(TestAddChildElementReification.class));
- specs.addChild(new
MatrixTest(TestExamineMustUnderstandHeaderElements.class));
- specs.addChild(new MatrixTest(TestAddChildElementLocalName.class));
- specs.addChild(new
MatrixTest(TestAddChildElementLocalNamePrefixAndURI.class));
- specs.addChild(new MatrixTest(TestSetParentElement.class));
- specs.addChild(new MatrixTest(TestGetOwnerDocument.class));
- suite.addChild(specs);
-
- return suite;
- }
-}
-```
-
-The **consumer** (in the implementation module) supplies the concrete factory
and
-any implementation-specific exclusion filters:
-
-```java
-class SAAJRITests {
- @TestFactory
- Stream<DynamicNode> saajTests() {
- MatrixTestSuite suite = SAAJTestSuite.create(new
SAAJMetaFactoryImpl());
- MatrixTestFilters excludes = MatrixTestFilters.builder()
- .add(TestGetOwnerDocument.class, "(spec=soap12)")
- .build();
- return suite.toDynamicNodes(excludes);
- }
-}
-```
-
-#### Benefits over MatrixTestSuiteBuilder
-
-* Produces a hierarchical test tree in the IDE (grouped by dimension)
instead of a
- flat list with parameter suffixes in the test name.
-* Parameters are distributed across the tree (one `Dimension` per container
level,
- possibly contributing multiple parameters) rather than accumulated on each
leaf
- test case, making the structure explicit.
-* Uses standard JUnit 5 `DynamicNode` for execution while keeping the
filtering
- infrastructure in the intermediate `MatrixTestNode` layer.
-* The LDAP-style filter mechanism is preserved unchanged.
-* **Guice injection decouples test cases from the tree structure.** Test
cases declare
- what they need (`@Inject SOAPSpec spec`) without knowing how or where in
the tree
- hierarchy that binding is provided. Adding a new dimension to the tree
does not
- require changing test case constructors.
-* **No boilerplate parameter passing in builders.** The current pattern
requires each
- `addTest()` call to manually pass all dimension values to the test
constructor.
- With Guice, `MatrixTest` only needs the test class; the injector supplies
- everything.
-* **Test case base classes become simpler.** `SAAJTestCase` no longer needs
- constructor parameters or chains of `super(...)` calls — it simply declares
- `@Inject` fields.
diff --git a/testing/matrix-testsuite/README.md
b/testing/matrix-testsuite/README.md
new file mode 100644
index 000000000..c8d6599d6
--- /dev/null
+++ b/testing/matrix-testsuite/README.md
@@ -0,0 +1,199 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+# Matrix Test Suite Framework
+
+## Overview
+
+The matrix-testsuite module provides infrastructure for building reusable,
+parameterized test suites that can be applied to different implementations of
the
+same API. It combines JUnit 5's `@TestFactory` / `DynamicNode` mechanism with
+Google Guice dependency injection to produce hierarchical, filterable test
trees.
+
+## Key concepts
+
+### Test tree
+
+A test suite is a tree of `MatrixTestNode` instances. Interior nodes are
+**fan-out nodes** that iterate over a list of dimension values, creating one
+`DynamicContainer` per value. Leaf nodes are **`MatrixTest`** instances that
+instantiate and run a `junit.framework.TestCase` subclass.
+
+### Guice injector hierarchy
+
+Each fan-out level creates a child Guice injector that binds its value type to
the
+current value. By the time a leaf `MatrixTest` is reached, the accumulated
injector
+can satisfy all `@Inject` dependencies declared by the test case class.
+
+```
+Root Injector
+ binds: implementation-level objects
+ │
+ ├─ Child Injector (DimensionA → value1)
+ │ ├─ MatrixTest → injector.getInstance(SomeTestCase.class) → runBare()
+ │ └─ MatrixTest → injector.getInstance(AnotherTestCase.class) → runBare()
+ │
+ └─ Child Injector (DimensionA → value2)
+ ├─ MatrixTest → injector.getInstance(SomeTestCase.class) → runBare()
+ └─ MatrixTest → injector.getInstance(AnotherTestCase.class) → runBare()
+```
+
+### Filtering
+
+Parameters accumulate from the root down through the tree. At each leaf, the
+accumulated parameter dictionary is checked against `MatrixTestFilters` — an
+immutable set of LDAP-style filter expressions optionally scoped to a test
class.
+Excluded tests produce an empty `Stream<DynamicNode>` and do not appear in the
+test tree.
+
+## Classes
+
+### `MatrixTestNode`
+
+Abstract base class for all nodes in the test tree. Defines a single method:
+
+```java
+abstract Stream<DynamicNode> toDynamicNodes(
+ Injector parentInjector,
+ Dictionary<String, String> inheritedParameters,
+ MatrixTestFilters excludes);
+```
+
+### `AbstractFanOutNode<T>`
+
+Abstract fan-out node that iterates over a list of values of type `T`. For each
+value, it:
+
+1. Creates a child Guice injector binding `T` to the value.
+2. Extracts test parameters (via the abstract `extractParameters` method).
+3. Produces a `DynamicContainer` containing the results of recursing into its
+ child nodes.
+
+Subclasses:
+
+- **`DimensionFanOutNode<D extends Dimension>`** — for types that implement the
+ `Dimension` interface. Parameters are extracted via
+ `Dimension.addTestParameters()`.
+
+- **`ParameterFanOutNode<T>`** — for arbitrary types. The caller supplies a
+ parameter name and a `Function<T, String>` to extract the parameter value.
+
+### `MatrixTest`
+
+Leaf node. Instantiates a `junit.framework.TestCase` subclass via Guice and
+executes it through `TestCase.runBare()` (which runs the full `setUp()` →
+`runTest()` → `tearDown()` lifecycle). The test is skipped if matched by the
+exclusion filters.
+
+### `MatrixTestSuite`
+
+Root of a test suite. Owns the Guice root injector (created from
caller-supplied
+modules) and a list of top-level `MatrixTestNode` children. Provides:
+
+```java
+public Stream<DynamicNode> toDynamicNodes(MatrixTestFilters excludes)
+```
+
+### `MatrixTestFilters`
+
+Immutable set of exclusion filters. Each filter entry optionally constrains by
+test class and/or an LDAP filter expression on the parameter dictionary (using
+OSGi's `FrameworkUtil.createFilter()`). Built via
`MatrixTestFilters.builder()`.
+
+## Writing a test case
+
+Test cases extend `junit.framework.TestCase` (or a domain-specific subclass)
and
+override `runTest()`. Dependencies are declared with `@Inject` — either on
fields
+or via constructor. The test case does **not** receive parameters through its
+constructor and does **not** call `addTestParameter()`.
+
+```java
+public abstract class MyTestCase extends TestCase {
+ @Inject protected SomeImplementation impl;
+ @Inject protected SomeDimension dimension;
+
+ // convenience methods using impl and dimension ...
+}
+```
+
+```java
+public class TestSomeBehavior extends MyTestCase {
+ @Override
+ protected void runTest() throws Throwable {
+ // test logic using inherited injected fields
+ }
+}
+```
+
+## Defining a test suite
+
+The test suite author creates a factory method that builds a `MatrixTestSuite`,
+adds fan-out nodes for each dimension, and registers test classes as
`MatrixTest`
+leaf nodes:
+
+```java
+public class MyTestSuite {
+ public static MatrixTestSuite create(SomeFactory factory) {
+ SomeImplementation impl = new SomeImplementation(factory);
+ MatrixTestSuite suite = new MatrixTestSuite(new AbstractModule() {
+ @Override
+ protected void configure() {
+ bind(SomeImplementation.class).toInstance(impl);
+ }
+ });
+
+ ParameterFanOutNode<SomeDimension> dimensions = new
ParameterFanOutNode<>(
+ SomeDimension.class,
+ Multiton.getInstances(SomeDimension.class),
+ "dimension",
+ SomeDimension::getName);
+ dimensions.addChild(new MatrixTest(TestSomeBehavior.class));
+ dimensions.addChild(new MatrixTest(TestOtherBehavior.class));
+ suite.addChild(dimensions);
+
+ return suite;
+ }
+}
+```
+
+## Consuming a test suite
+
+Consumers create a JUnit 5 test class with a `@TestFactory` method:
+
+```java
+class MyImplTest {
+ @TestFactory
+ Stream<DynamicNode> tests() {
+ MatrixTestSuite suite = MyTestSuite.create(new MyFactoryImpl());
+ MatrixTestFilters excludes = MatrixTestFilters.builder()
+ .add(TestSomeBehavior.class, "(dimension=problematicValue)")
+ .build();
+ return suite.toDynamicNodes(excludes);
+ }
+}
+```
+
+## Legacy classes
+
+The following classes from the old JUnit 3 based framework still exist in this
+package but are deprecated and will be removed once all test suites have been
+migrated:
+
+- `MatrixTestCase`
+- `MatrixTestSuiteBuilder`
diff --git a/testing/matrix-testsuite/migration.md
b/testing/matrix-testsuite/migration.md
new file mode 100644
index 000000000..a04d10c96
--- /dev/null
+++ b/testing/matrix-testsuite/migration.md
@@ -0,0 +1,290 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+# Migration guide: MatrixTestSuiteBuilder → MatrixTestSuite
+
+This document describes how to migrate a test suite from the old
+`MatrixTestSuiteBuilder` / `MatrixTestCase` pattern (JUnit 3) to the new
+`MatrixTestSuite` / `MatrixTestNode` pattern (JUnit 5 + Guice).
+
+For a completed example of this migration, see the `saaj-testsuite` module.
+
+## Prerequisites
+
+The module being migrated must depend on:
+
+```xml
+<dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter</artifactId>
+</dependency>
+<dependency>
+ <groupId>com.google.inject</groupId>
+ <artifactId>guice</artifactId>
+</dependency>
+```
+
+These are already declared in the root POM's `<dependencyManagement>`.
+
+## Step-by-step migration
+
+### 1. Update the test case base class
+
+The domain-specific base class (e.g. `SAAJTestCase`) currently:
+
+- Extends `MatrixTestCase`
+- Accepts all dimension values and implementation objects via constructor
+ parameters
+- Calls `addTestParameter(name, value)` in the constructor
+
+**Change it to:**
+
+- Extend `junit.framework.TestCase` directly
+- Declare dependencies as `@Inject` fields (using `com.google.inject.Inject`)
+- Remove the constructor entirely (or make it no-arg)
+- Remove all `addTestParameter()` calls
+
+**Before:**
+
+```java
+public abstract class SAAJTestCase extends MatrixTestCase {
+ protected final SAAJImplementation saajImplementation;
+ protected final SOAPSpec spec;
+
+ public SAAJTestCase(SAAJImplementation saajImplementation, SOAPSpec spec) {
+ this.saajImplementation = saajImplementation;
+ this.spec = spec;
+ addTestParameter("spec", spec.getName());
+ }
+
+ protected final MessageFactory newMessageFactory() throws SOAPException {
+ return
spec.getAdapter(FactorySelector.class).newMessageFactory(saajImplementation);
+ }
+}
+```
+
+**After:**
+
+```java
+public abstract class SAAJTestCase extends TestCase {
+ @Inject protected SAAJImplementation saajImplementation;
+ @Inject protected SOAPSpec spec;
+
+ protected final MessageFactory newMessageFactory() throws SOAPException {
+ return
spec.getAdapter(FactorySelector.class).newMessageFactory(saajImplementation);
+ }
+}
+```
+
+### 2. Update each test case class
+
+Each test case class currently accepts constructor parameters and forwards them
+to the base class.
+
+**Remove the constructor.** The `runTest()` method stays unchanged. Any imports
+of the implementation class and dimension types that were only used in the
+constructor can be removed.
+
+**Before:**
+
+```java
+public class TestAddChildElementReification extends SAAJTestCase {
+ public TestAddChildElementReification(SAAJImplementation
saajImplementation, SOAPSpec spec) {
+ super(saajImplementation, spec);
+ }
+
+ @Override
+ protected void runTest() throws Throwable {
+ // ... test logic unchanged ...
+ }
+}
+```
+
+**After:**
+
+```java
+public class TestAddChildElementReification extends SAAJTestCase {
+ @Override
+ protected void runTest() throws Throwable {
+ // ... test logic unchanged ...
+ }
+}
+```
+
+### 3. Replace the suite builder with a suite factory
+
+The old `*TestSuiteBuilder` class extends `MatrixTestSuiteBuilder` and
overrides
+`addTests()` to register test instances for each dimension combination.
+
+**Replace it** with a class that has a static factory method returning a
+`MatrixTestSuite`. The factory method:
+
+1. Creates a `MatrixTestSuite` with a Guice module that binds
+ implementation-level objects.
+2. Creates fan-out nodes for each dimension.
+3. Adds `MatrixTest` leaf nodes for each test case class.
+
+Use `ParameterFanOutNode` for types that don't implement `Dimension`
(supplying a
+parameter name and a function to extract the display value). Use
+`DimensionFanOutNode` for types that implement `Dimension`.
+
+**Before:**
+
+```java
+public class SAAJTestSuiteBuilder extends MatrixTestSuiteBuilder {
+ private final SAAJImplementation saajImplementation;
+
+ public SAAJTestSuiteBuilder(SAAJMetaFactory metaFactory) {
+ saajImplementation = new SAAJImplementation(metaFactory);
+ }
+
+ @Override
+ protected void addTests() {
+ addTests(SOAPSpec.SOAP11);
+ addTests(SOAPSpec.SOAP12);
+ }
+
+ private void addTests(SOAPSpec spec) {
+ addTest(new TestAddChildElementReification(saajImplementation, spec));
+ addTest(new TestGetOwnerDocument(saajImplementation, spec));
+ // ...
+ }
+}
+```
+
+**After:**
+
+```java
+public class SAAJTestSuite {
+ public static MatrixTestSuite create(SAAJMetaFactory metaFactory) {
+ SAAJImplementation impl = new SAAJImplementation(metaFactory);
+ MatrixTestSuite suite = new MatrixTestSuite(new AbstractModule() {
+ @Override
+ protected void configure() {
+ bind(SAAJImplementation.class).toInstance(impl);
+ }
+ });
+
+ ParameterFanOutNode<SOAPSpec> specs = new ParameterFanOutNode<>(
+ SOAPSpec.class,
+ Multiton.getInstances(SOAPSpec.class),
+ "spec",
+ SOAPSpec::getName);
+ specs.addChild(new MatrixTest(TestAddChildElementReification.class));
+ specs.addChild(new MatrixTest(TestGetOwnerDocument.class));
+ // ...
+ suite.addChild(specs);
+
+ return suite;
+ }
+}
+```
+
+Key differences:
+
+- Test classes are registered **once** as `MatrixTest` instances under the
+ appropriate fan-out node, rather than once per dimension combination.
+- Dimension values are listed via `Multiton.getInstances()` (or an explicit
list)
+ in the fan-out node, not iterated manually.
+- No constructor arguments are passed to test classes.
+
+### 4. Replace the consumer test class
+
+The old consumer class uses JUnit 3's `static suite()` method.
+
+**Replace it** with a JUnit 5 class that has a `@TestFactory` method returning
+`Stream<DynamicNode>`.
+
+**Before:**
+
+```java
+public class SAAJRITest extends TestCase {
+ public static TestSuite suite() throws Exception {
+ return new SAAJTestSuiteBuilder(new SAAJMetaFactoryImpl()).build();
+ }
+}
+```
+
+**After:**
+
+```java
+public class SAAJRITest {
+ @TestFactory
+ public Stream<DynamicNode> saajTests() {
+ return SAAJTestSuite.create(new SAAJMetaFactoryImpl())
+ .toDynamicNodes(MatrixTestFilters.builder().build());
+ }
+}
+```
+
+### 5. Migrate exclusions
+
+If the old consumer called `exclude()` on the builder, convert those calls to
+`MatrixTestFilters.builder().add(...)` entries.
+
+**Before:**
+
+```java
+SAAJTestSuiteBuilder builder = new SAAJTestSuiteBuilder(factory);
+builder.exclude(TestGetOwnerDocument.class, "(spec=soap12)");
+builder.exclude(TestSomething.class);
+builder.exclude("(parser=StAX)");
+return builder.build();
+```
+
+**After:**
+
+```java
+MatrixTestFilters excludes = MatrixTestFilters.builder()
+ .add(TestGetOwnerDocument.class, "(spec=soap12)")
+ .add(TestSomething.class)
+ .add("(parser=StAX)")
+ .build();
+return MyTestSuite.create(factory).toDynamicNodes(excludes);
+```
+
+The filter syntax and semantics are identical.
+
+### 6. Update dependencies in pom.xml
+
+Add `junit-jupiter` and `guice` to the module's `<dependencies>`. If the module
+uses `Multiton.getInstances()`, also add a dependency on the `multiton` module.
+
+Remove any dependency on `junit:junit` if no code in the module still uses
JUnit 3
+or 4 APIs directly. (Note: test case classes still extend
+`junit.framework.TestCase`, which comes from `junit:junit` transitively through
+`matrix-testsuite`.)
+
+### 7. Delete the old builder class
+
+The old `*TestSuiteBuilder` class can be deleted once the new `*TestSuite`
factory
+is in place and all consumers have been updated.
+
+## Checklist
+
+- [ ] Base test case class: extends `TestCase`, uses `@Inject` fields, no
+ constructor
+- [ ] All test case classes: constructor removed, `runTest()` unchanged
+- [ ] Suite factory class: creates `MatrixTestSuite` with Guice module, builds
+ fan-out tree with `MatrixTest` leaves
+- [ ] Consumer test class: uses `@TestFactory` returning `Stream<DynamicNode>`
+- [ ] Exclusions: converted to `MatrixTestFilters.builder()` calls
+- [ ] `pom.xml`: `junit-jupiter`, `guice`, and (if needed) `multiton` added
+- [ ] Old builder class deleted
+- [ ] Tests pass: `mvn clean test -pl <module> -am`
