Hello!
I recently wrote a script to run FiveAM tests for one of my
libraries on many different implementations on your own machine:
https://gitlab.common-lisp.net/uri-template/uri-template2/blob/master/run-tests.lisp
It would be really nice if I did not have to copy-paste that script
to my other libraries, and instead could contribute a generalized
version to Roswell (on which the script is based) and have it work
for any project using any test library.
What I would like to be able to do, for any system:
(handler-case (asdf:test-system "system")
(asdf:test-op-test-failure (condition)
(princ condition uiop:*stderr*)
(uiop:quit 1)))
The attached patch adds an ASDF:TEST-OP-TEST-FAILURE condition that
test libraries can inherit from. I also added the necessary
functionality to FiveAM:
https://github.com/sionescu/fiveam/pull/58
It should be easy to add similar functionality to other testing
libraries. This will make test automation trivial, with few, if any,
changes required to systems that use ASDF and the testing libraries.
One thing I would like to discuss is which condition
ASDF:TEST-OP-TEST-FAILURE should inherit from. It definitely should
not be ERROR - there is no error in test-op, or in running the tests;
test failures are a regular and expected occurrence of running tests.
Also problematic is widespread abuse of HANDLER-CASE to catch any
ERROR; I am afraid if signaled from popular test libraries it would
break someone's test running code somewhere. WARNING seems like a
nice parent condition, but maybe having ASDF:TEST-OP-TEST-FAILURE
inherit from CONDITION is a better idea. Thoughts?
Vladimir
>From 40c733bdd0401daa48466b807c5cf64cb14002be Mon Sep 17 00:00:00 2001
From: Vladimir Sedach <v...@oneofus.la>
Date: Sat, 14 Sep 2019 19:26:02 -0700
Subject: [PATCH] Add a TEST-OP-TEST-FAILURE condition for test libraries to
sub-class
---
doc/asdf.texinfo | 13 ++++++++++---
interface.lisp | 1 +
lisp-action.lisp | 15 ++++++++++++++-
3 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/doc/asdf.texinfo b/doc/asdf.texinfo
index d663b5cc..04ccdb9a 100644
--- a/doc/asdf.texinfo
+++ b/doc/asdf.texinfo
@@ -2448,9 +2448,16 @@ in a way that is compatible with all of the various test libraries
and test techniques in use in the community, and
given the fact that ASDF operations do not return a value indicating
success or failure.
-For those willing to go to the effort, we suggest defining conditions to
-signal when a @code{test-op} fails, and storing in those conditions
-information that describes which tests fail.
+
+We suggest signaling a sub-class of the @code{test-op-test-failure}
+condition when a @code{test-op} fails.
+The test library you are using may already be doing this.
+Sub-classes of the @code{test-op-test-failure} condition should have a
+report function that prints the total number of tests run, the number
+of failed tests, the identities of the failed tests, and enough
+information about the failures to facilitate debugging.
+You should also signal a subclass of this condition when
+@code{test-op} did not result in any tests being run.
People typically define a separate test @emph{system} to hold the tests.
Doing this avoids unnecessarily adding a test framework as a dependency
diff --git a/interface.lisp b/interface.lisp
index 411ab547..644741e7 100644
--- a/interface.lisp
+++ b/interface.lisp
@@ -99,6 +99,7 @@
#:asdf-version
#:compile-condition #:compile-file-error #:compile-warned-error #:compile-failed-error
#:compile-warned-warning #:compile-failed-warning
+ #:test-op-test-failure
#:error-name
#:error-pathname
#:load-system-definition-error
diff --git a/lisp-action.lisp b/lisp-action.lisp
index 8bdeb3bf..c17193c9 100644
--- a/lisp-action.lisp
+++ b/lisp-action.lisp
@@ -12,7 +12,8 @@
#:load-op #:prepare-op #:compile-op #:test-op #:load-source-op #:prepare-source-op
#:call-with-around-compile-hook
#:perform-lisp-compilation #:perform-lisp-load-fasl #:perform-lisp-load-source
- #:lisp-compilation-output-files))
+ #:lisp-compilation-output-files
+ #:test-op-test-failure))
(in-package :asdf/lisp-action)
@@ -242,6 +243,18 @@ an OPERATION and a COMPONENT."
;;;; test-op
(with-upgradability ()
+ (define-condition test-op-test-failure (warning) ()
+ (:documentation
+ "Parent condition to be sub-classed and signaled by test libraries
+to indicate test failure.
+
+Test libraries should also signal a subclass of this condition when
+test-op did not result in any tests being run.
+
+Sub-classes of this condition should have a report function that
+prints the total number of tests run, the number of failed tests, the
+identities of the failed tests, and enough information about the
+failures to facilitate debugging."))
(defmethod perform ((o test-op) (c component))
nil)
(defmethod operation-done-p ((o test-op) (c system))
--
2.11.0
