Hi Ralf,
Thanks for your comments.
I attached an updated patch which addresses your comments, except the
conditional definition of variables in atconfig.
On Saturday 01 August 2009 14:26:11 Ralf Wildenhues wrote:
> the idea seems ok to me. Please test distcheck before committing.
I checked it passes.
> AT_CHECK has measures to complain if it is not between AT_SETUP and
> AT_CLEANUP. Can you please check that AT_CHECK_EUNIT complains, too
> (it should, because it uses AT_CHECK)? Thanks.
AT_CHECK_EUNIT is defined using _AT_DEFINE_SETUP, like AT_CHECK.
I checked that it fails if not between AT_SETUP and AT_CLEANUP.
> Would users want to write tests that are expected to fail/exit with a
> nonzero status (from one of the involved tools)?
EUnit is a library, which doesn't exit the process in any case, and unit tests
executed by EUnit are not expected to exit the process either. For EUnit,
either a test passes or it fails. There is no other information returned
(except for information output on stdout). So AT_CHECK_EUNIT generates and
execute a "wrapper" Erlang module that calls the EUnit library with the given
test spec, and returns an appropriate exit status depending on the result of
the test:
- 77 if EUnit is not found;
- 1 if the test fails;
- 0 if the test passes.
There's no need and no way to distinguish other cases.
> > +If the testsuite is run in verbose mode, with option @option{--verbose},
> > +EUnit is also run in verbose mode to output more details about
> > +individual unittests.
>
> s/unittests/unit tests/ ? (more instances below)
OK, done.
> > --- a/lib/autoconf/autotest.m4
> > +++ b/lib/autoconf/autotest.m4
> > @@ -84,6 +84,14 @@ at_top_builddir=\$at_top_build_prefix
> > AUTOTEST_PATH='m4_default([$2], [$1])'
> >
> > SHELL=\${CONFIG_SHELL-'$SHELL'}
> > +
> > +# Required to run EUnit unittests.
> > +ERL='$ERL'
> > +ERLC='$ERLC'
> > +ERLCFLAGS='$ERLCFLAGS'
> > ATEOF
> > -])
> > +],
> > +[ERL="$ERL"
> > +ERLC="$ERLC"
> > +ERLCFLAGS="$ERLCFLAGS"])
>
> Can we make this code addition conditional to test suites in which
> AT_CHECK_EUNIT is used? (Diversions ought to help here.)
> We don't want unnecessary code expansions.
I don't see how to do that easily, since AT_CONFIG_TESTDIR doesn't parse the
test suite at all. How can I detect, in configure, the use of a macro in a
testsuite? We would have to do something like AT_INIT_AUTOMAKE, i.e. have the
testsuite generation also produce aclocal macros like Automake?
It should be better to make the definition of those variables conditional to
the use of AC_ERLANG_PATH_ERL and AC_ERLANG_PATH_ERLC in configure.ac, as it
should be the normal way of defining those variables?
> > +_AT_DEFINE_SETUP([AT_CHECK_EUNIT],
> > +[AT_CHECK([test -f "$ERL" && test -f "$ERLC" || exit 77])
>
> AT_SKIP_IF has lower overhead.
OK, replaced with:
AT_SKIP_IF([test ! -f "$ERL" || test ! -f "$ERLC"])
> > +AT_CHECK(["$ERLC" $ERLCFLAGS -b beam $1.erl])
>
> Does that never produce any output? Wow.
If compilation is fine, no output is produced. But it outputs something in
case of warnings or errors, which in this case indicates that the test-spec
has an invalid syntax, which should fail the test. I think that this is a
legitimate use of AT_CHECK?
> > +AT_CHECK([/bin/sh ./compile])
>
> $CONFIG_SHELL, please.
OK, done.
Thanks,
--
Romain Lenglet
From 2e53c3ef15c01b21df7012c965f4be96d89842da Mon Sep 17 00:00:00 2001
From: Romain Lenglet <[email protected]>
Date: Wed, 29 Jul 2009 21:25:21 +0900
Subject: [PATCH] Add AT_CHECK_EUNIT macro to run Erlang EUnit unittests, with its autotests and documentation.
---
ChangeLog | 13 ++++++
NEWS | 3 +
doc/autoconf.texi | 52 +++++++++++++++++++++++++
lib/autoconf/autotest.m4 | 10 ++++-
lib/autotest/Makefile.am | 2 +-
lib/autotest/autotest.m4 | 1 +
lib/autotest/erlang.m4 | 95 ++++++++++++++++++++++++++++++++++++++++++++++
tests/autotest.at | 74 +++++++++++++++++++++++++++++++++++
8 files changed, 248 insertions(+), 2 deletions(-)
create mode 100644 lib/autotest/erlang.m4
diff --git a/ChangeLog b/ChangeLog
index c307701..f3d1d7f 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,16 @@
+2009-08-01 Romain Lenglet <[email protected]>
+
+ * lib/autotest/erlang.m4 (AT_CHECK_EUNIT): New file defining new
+ AT_CHECK_EUNIT macro.
+ * lib/autotest/Makefile.am (dist_autotestlib_DATA): Add erlang.m4.
+ * lib/autoconf/autotest.m4 (AC_CONFIG_TESTDIR): Add definitions of
+ variables used by AT_CHECK_EUNIT macro: ERL, ERLC, ERLCFLAGS.
+ * tests/autotest.at (Erlang Eunit unit tests): Add test for macro
+ AT_CHECK_EUNIT.
+ * doc/autoconf.texi (Writing Testsuites): Document macro
+ AT_CHECK_EUNIT.
+ * NEWS: Mention macro AT_CHECK_EUNIT.
+
2009-07-30 Paolo Bonzini <[email protected]>
Joel E. Denny <[email protected]>
diff --git a/NEWS b/NEWS
index 7e8a522..50f4e0e 100644
--- a/NEWS
+++ b/NEWS
@@ -9,6 +9,9 @@ GNU Autoconf NEWS - User visible changes.
These macros are present only for backwards compatibility purposes.
+** The following documented autotest macros are new:
+ AT_CHECK_EUNIT
+
* Major changes in Autoconf 2.64 (2009-07-26) [stable]
Released by Eric Blake, based on git versions 2.63b.*.
diff --git a/doc/autoconf.texi b/doc/autoconf.texi
index c920d73..c087444 100644
--- a/doc/autoconf.texi
+++ b/doc/autoconf.texi
@@ -22527,6 +22527,8 @@ of debugging scripts has the purpose of easing the chase for bugs.
@item all the files created with @code{AT_DATA}
+...@item all the Erlang source code files created with @code{AT_CHECK_EUNIT}
+
@item a log of the run, named @file{testsuite.log}
@end itemize
@@ -22901,6 +22903,56 @@ parameter, and the standard error contents are compared with
@end table
@end defmac
+...@defmac AT_CHECK_EUNIT (@var{module}, @var{test-spec}, @ovar{erlflags}, @
+ @ovar{run-if-fail}, @ovar{run-if-pass})
+...@atindex{check_eunit}
+Initialize and execute an Erlang module named @var{module} that performs
+tests following the @var{test-spec} EUnit test specification.
+...@var{test-spec} must be a valid EUnit test specification, as defined in
+the @uref{http://@/erlang.org/@/doc/@/apps/@/eunit/@/index.html, EUnit
+Reference Manual}. @var{erlflags} are optional command-line options
+passed to the Erlang interpreter to execute the test Erlang module.
+Typically, @var{erlflags} defines at least the paths to directories
+containing the compiled Erlang modules under test, as @samp{-pa path1
+path2 ...}.
+
+For example, the unit tests associated with Erlang module @samp{testme},
+which compiled code is in subdirectory @file{src}, can be performed
+with:
+
+...@example
+AT_CHECK_EUNIT([testme_testsuite], [...@{module, tes...@}],
+ [-pa "$...@{abs_top_builddir@}/src"])
+...@end example
+
+This macro must be invoked in between @code{AT_SETUP} and @code{AT_CLEANUP}.
+
+Variables @code{ERL}, @code{ERLC}, and (optionally) @code{ERLCFLAGS}
+must be defined as the path of the Erlang interpreter, the path of the
+Erlang compiler, and the command-line flags to pass to the compiler,
+respectively. Otherwise, the test group is skipped. Those variables
+should be configured in @file{configure.ac}, typically using the
+...@command{ac_erlang_path_erl} and @command{AC_ERLANG_PATH_ERLC} macros,
+and the configured values of those variables are automatically defined
+in the testsuite.
+
+If the EUnit library cannot be found, i.e. if module @code{eunit} cannot
+be loaded, the test group is skipped. Otherwise, if @var{test-spec} is
+an invalid EUnit test specification, the test group fails. Otherwise,
+if the EUnit test passes, shell commands @var{run-if-pass} are executed
+or, if the EUnit test fails, shell commands @var{run-if-fail} are
+executed and the test group fails.
+
+Only the generated test Erlang module is automatically compiled and
+executed. If @var{test-spec} involves testing other Erlang modules,
+e.g. module @samp{testme} in the example above, those modules must be
+already compiled.
+
+If the testsuite is run in verbose mode, with option @option{--verbose},
+EUnit is also run in verbose mode to output more details about
+individual unit tests.
+...@end defmac
+
@node testsuite Invocation
@section Running @command{testsuite} Scripts
diff --git a/lib/autoconf/autotest.m4 b/lib/autoconf/autotest.m4
index 72b4b7f..a962880 100644
--- a/lib/autoconf/autotest.m4
+++ b/lib/autoconf/autotest.m4
@@ -84,6 +84,14 @@ at_top_builddir=\$at_top_build_prefix
AUTOTEST_PATH='m4_default([$2], [$1])'
SHELL=\${CONFIG_SHELL-'$SHELL'}
+
+# Required to run EUnit unit tests.
+ERL='$ERL'
+ERLC='$ERLC'
+ERLCFLAGS='$ERLCFLAGS'
ATEOF
-])
+],
+[ERL="$ERL"
+ERLC="$ERLC"
+ERLCFLAGS="$ERLCFLAGS"])
])# AC_CONFIG_TESTDIR
diff --git a/lib/autotest/Makefile.am b/lib/autotest/Makefile.am
index d310e46..4335549 100644
--- a/lib/autotest/Makefile.am
+++ b/lib/autotest/Makefile.am
@@ -16,7 +16,7 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
autotestlibdir = $(pkgdatadir)/autotest
-dist_autotestlib_DATA = autotest.m4 general.m4
+dist_autotestlib_DATA = autotest.m4 general.m4 erlang.m4
nodist_autotestlib_DATA = autotest.m4f
CLEANFILES = $(nodist_autotestlib_DATA)
diff --git a/lib/autotest/autotest.m4 b/lib/autotest/autotest.m4
index dead4c0..f9b8e51 100644
--- a/lib/autotest/autotest.m4
+++ b/lib/autotest/autotest.m4
@@ -45,3 +45,4 @@
# to the GPL from your modified version.
m4_include([autotest/general.m4])
+m4_include([autotest/erlang.m4])
diff --git a/lib/autotest/erlang.m4 b/lib/autotest/erlang.m4
new file mode 100644
index 0000000..b74c37c
--- /dev/null
+++ b/lib/autotest/erlang.m4
@@ -0,0 +1,95 @@
+# This file is part of Autoconf. -*- Autoconf -*-
+# M4 macros used in running Erlang EUnit unit tests in test suites.
+m4_define([_AT_COPYRIGHT_YEARS],
+[Copyright (C) 2009 Free Software Foundation, Inc.])
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+# 02110-1301, USA.
+
+# As a special exception, the Free Software Foundation gives unlimited
+# permission to copy, distribute and modify the configure scripts that
+# are the output of Autoconf. You need not follow the terms of the GNU
+# General Public License when using or distributing such scripts, even
+# though portions of the text of Autoconf appear in them. The GNU
+# General Public License (GPL) does govern all other use of the material
+# that constitutes the Autoconf program.
+#
+# Certain portions of the Autoconf source text are designed to be copied
+# (in certain cases, depending on the input) into the output of
+# Autoconf. We call these the "data" portions. The rest of the Autoconf
+# source text consists of comments plus executable code that decides which
+# of the data portions to output in any given case. We call these
+# comments and executable code the "non-data" portions. Autoconf never
+# copies any of the non-data portions into its output.
+#
+# This special exception to the GPL applies to versions of Autoconf
+# released by the Free Software Foundation. When you make and
+# distribute a modified version of Autoconf, you may extend this special
+# exception to the GPL to apply to your modified version as well, *unless*
+# your modified version has the potential to copy into its output some
+# of the text that was the non-data portion of the version that you started
+# with. (In other words, unless your change moves or copies text from
+# the non-data portions to the data portions.) If your modification has
+# such potential, you must delete any notice of this special exception
+# to the GPL from your modified version.
+#
+# Written by Romain Lenglet
+
+
+## ------------------------ ##
+## Erlang EUnit unit tests. ##
+## ------------------------ ##
+
+# AT_CHECK_EUNIT(MODULE, SPEC, [ERLFLAGS], [RUN-IF-FAIL], [RUN-IF-PASS])
+# ----------------------------------------------------------------------
+# Check that the EUnit test specification SPEC passes. The ERLFLAGS
+# optional flags are passed to the Erlang interpreter command line to
+# execute the test. The test is executed from an automatically
+# generated Erlang module named MODULE. Each call to this macro should
+# have a distinct MODULE name within each test group, to ease
+# debugging.
+# An Erlang/OTP version which contains the eunit library must be
+# installed, in order to execute this macro in a test suite. The ERL,
+# ERLC, and ERLCFLAGS variables must be defined in configure.ac,
+# typically by using the AC_ERLANG_PATH_ERL and AC_ERLANG_PATH_ERLC
+# Autoconf macros.
+_AT_DEFINE_SETUP([AT_CHECK_EUNIT],
+[AT_SKIP_IF([test ! -f "$ERL" || test ! -f "$ERLC"])
+## A wrapper to EUnit, to exit the Erlang VM with the right exit code:
+AT_DATA([$1.erl],
+[[-module($1).
+-export([test/0, test/1]).
+test() -> test([]).
+test(Options) ->
+ TestSpec = $2,
+ ReturnValue = case code:load_file(eunit) of
+ {module, _} -> case eunit:test(TestSpec, Options) of
+ ok -> 0; %% test passes
+ _ -> 1 %% test fails
+ end;
+ _ -> 77 %% EUnit not found, test skipped
+ end,
+ init:stop(ReturnValue).
+]])
+AT_CHECK(["$ERLC" $ERLCFLAGS -b beam $1.erl])
+## Make EUnit verbose when testsuite is verbose:
+if test -z "$at_verbose"; then
+ at_eunit_options="verbose"
+else
+ at_eunit_options=""
+fi
+AT_CHECK(["$ERL" $3 -s $1 test $at_eunit_options -noshell], [0], [ignore], [],
+ [$4], [$5])
+])
diff --git a/tests/autotest.at b/tests/autotest.at
index 7836439..13e2e5d 100644
--- a/tests/autotest.at
+++ b/tests/autotest.at
@@ -1429,3 +1429,77 @@ m4_include([sub/two spaces.at])
AT_CHECK([$CONFIG_SHELL ./suite], [0], [stdout])
AT_CHECK([grep 'two spaces' suite.log], [1])
AT_CLEANUP
+
+
+## ------------------------- ##
+## Erlang EUnit unit tests. ##
+## ------------------------- ##
+
+AT_SETUP([Erlang Eunit unit tests])
+AT_KEYWORDS([Erlang])
+
+mkdir pkg
+mkdir pkg/s pkg/t
+AT_DATA([pkg/configure.ac], [[AC_INIT
+AC_ERLANG_PATH_ERL([not found])
+AC_ERLANG_PATH_ERLC([not found])
+if test "$ERL" = "not found" || test "$ERLC" = "not found"; then exit 77; fi
+AC_ERLANG_CHECK_LIB([eunit], [], [exit 77])
+AC_CONFIG_TESTDIR([t])
+AC_CONFIG_FILES([s/compile])
+AC_OUTPUT
+]])
+cp "$abs_top_srcdir/build-aux/install-sh" pkg
+
+# Erlang module to test:
+AT_DATA([pkg/s/testme.erl],
+[[-module(testme).
+-export([foo/1]).
+foo(1) -> one;
+foo(2) -> two;
+foo(_) -> other.
+]])
+# Corresponding Eunit unit test module:
+AT_DATA([pkg/s/testme_tests.erl],
+[[-module(testme_tests).
+-include_lib("eunit/include/eunit.hrl").
+foo_one_test() -> ?assertEqual(one, testme:foo(1)).
+foo_two_test() -> ?assertEqual(two, testme:foo(2)).
+foo_other_test() -> ?assertEqual(other, testme:foo(42)).
+]])
+# Compilation script:
+AT_DATA([pkg/s/compile.in],
+[["@ERLC@" -b beam testme.erl testme_tests.erl
+]])
+
+cd pkg
+AT_CHECK_AUTOCONF
+cd ..
+
+
+AT_CHECK_AT_PREP([suite],
+[[AT_INIT([suite to check EUnit integration])
+AT_SETUP([my only test])
+AT_CHECK_EUNIT([my_testsuite], [{module, testme}],
+ [-pa "${abs_top_builddir}/s"])
+AT_CLEANUP
+]], [], [], [], [pkg/t])
+
+cd pkg
+AT_CHECK_CONFIGURE
+cd ..
+
+AT_CHECK([grep '^ERL='\''.*'\' pkg/t/atconfig], [], [ignore])
+AT_CHECK([grep '^ERLC='\''.*'\' pkg/t/atconfig], [], [ignore])
+AT_CHECK([grep '^ERLCFLAGS='\''.*'\' pkg/t/atconfig], [], [ignore])
+
+cd pkg/s
+chmod +x "./compile"
+AT_CHECK([$CONFIG_SHELL ./compile])
+cd ../..
+
+cd pkg/t
+AT_CHECK([$CONFIG_SHELL ./suite], [0], [ignore])
+cd ../..
+
+AT_CLEANUP
--
1.6.3.1
