Hi all, On December 13, I created an issue on GitHub, pls pay attention to that! Here is the link: https://github.com/apache/maven-surefire/issues/3231 This was an old plan from milestone versions 3.0.0-Mx and it is still valid.
We may really break the sustainability of this project if we would not really have our common agreements. If you are supporting this project long, you will understand the common denominator of the issues in the following features: 1. Parallel tests 2. Flaky tests 3. Re-run tests 4. Multiple test assertions So, many people may say "a report of parallel xyz does not work", or "report with flaky tests does not work", or "re-run is broken" is caused by this, and every time you fix one, you break another feature. Why is that? 1. the Provider does not fire all information needed for a deterministic behavior of reporters, 2. the reporters have a guess implementation at a low level and it is completely stateful blindly anticipating a sequential order of events. Yes, the implementation uses Strings and flaky tests are determined by ordering the events somehow which may not always be a correct guess. Instead of guessing the status of the test at current time to be stateless, you are guessing the status upon the previous sequence of test status which makes you stateful. You should not do this! The test Provider should indicate that this is flaky or rerun, but it does not make it properly. In 2023 I implemented a fix or proposal. This has good abstractions instead of too low level guess code: https://github.com/Tibor17/maven-surefire/commit/8b7d32d4f872e48aaa6f32db2245debfc016ff95#diff-c06584a1810a66dff65d33c70b0c59e7dba80edbe998c9cc67c492bcbe561517 We are targeting these two classes: 1. TestSetRunListener 2. StatelessReportEventListener We have encapsulated test operations within one aggregation object: 1. ReportData We are able to identify the particular test run using UniqueID instead of a combination of Class + Method. We have introduced an abstraction of test operation: 1. TestOperation<T extends ReportEntry> and we introduced concrete operations so that we would no more guess the operation because the Provider informed us about the operation and we don't have to be stateful: 1. TestSucceededOperation 2. TestStartingOperation 3. TestSetStartingOperation 4. TestSetCompletedOperation 5. TestFailedOperation 6. TestErrorOperation 7. TestSkippedOperation 8. TestAssumptionFailureOperation 9. TestExecutionSkippedByUserOperation Cheers Tibor
