Martin,
Thank you for your thoughtful reply. Your email ended up in my spam
folder so I did not see it until now.
I am new to autopkgtest, and feel like I got things working with CI
testing in a convoluted way, that there is probably a cleaner
solution. I looked through everything I could find on Debian, Ubuntu,
and your blog page to sort out an ideal way of using autopkgtest, and
do not think I am there yet...I think there still is some
clarification that could be added to the autopkgtest tutorial.
I DO want to have test scripts in the source tree that are written in
a general enough way to run on more than Debian and even more than
GNU/Linux, rather than writing custom test scripts for debian/tests if
possible. That was part of my motivation. My "make installcheck"
target works on many systems, but with Debian I had to avoid using it.
My "test/test-all" script works when invoked by "make installcheck",
when invoked in the source tree directly, and (now) when invoked from
the Debian CI test infrastructure.
This, for example, is my debian/tests/control file for my utfcheck package:[1]
Test-Command: cd test && progloc=`which utfcheck` &&
utfcheck_bindir=`dirname $progloc` ./test-all
I did that because it was only clear that I could find my program in
$PATH, not that I could count on it being available in /usr/bin as per
the FHS. I thought it might be possible that CI testing could use an
intermediate location in a way I did not know.
My test scripts then invoke "${utfcheck_bindir}/utfcheck"; see for example [2].
If I could just change the control file to be
Test-Command: make installcheck
That certainly would be simpler and more straightforward, but the test
environment would at least need $prefix to be set to override the
/usr/local default. As a not-quite-ideal alternative,
Test-Command: cd test && utfcheck_bindir=/usr/bin ./test-all
would at least be simpler; I just did not know from the autopkgtest
tutorial if it was possible.
That utfcheck package has a Debian bug filed against it. I want to
fix the bug, but I also am hoping to improve some other things related
to Debian packaging when I do. CI testing is one such thing.
On Sun, Aug 19, 2018 at 1:52 PM Martin Pitt <mp...@debian.org> wrote:
>
> Hello Paul,
>
> Paul Hardy [2018-08-19 7:44 -0700]:
>> ...
> > By defining a few environment variables (like setting bindir to
> > /usr/bin), it will be more straightforward for people running
> > autopkgtest with Autotools to use a default "make installcheck"
> > target.
>
> Maybe, but I still don't believe this actually makes sense:
>
> * Defining random variables with generic names like $bindir might also
> influence other tests or packages, which makes them work differently than
> they would in a normal OS install.
My thinking was to confine such environment variables to the GNU
standard directory variables; see [3] in the automake manual.
Note that the default value for $prefix with automake is usually
/usr/local. With a Debian build, $prefix is set to /usr for final
installation of course. It was at least $prefix that I was hoping to
have defined in the environment; everything else could fall into place
from that.
> * autopkgtest is meant to be neutral to the test frame work that you use, as
> there are literally hundreds of them. There is no way it could maintain
> support for these.
And regretfully I do not know enough about that to understand the
implications, so I had to ask.
> * This is too magic. Test commands should be very explicit and allow the
> developer to reproduce them step by step in a shell.
>
> * For even `make` to work, you need at least a configured tree, and thus all
> the build dependencies, and most likely even need to build the package.
> autopkgtests should generally avoid that.
>
> * I don't believe that setting static values for these variables even works
> for all automake projects. If they would, why are they even variables?
Most automake projects might not even implement an "installcheck"
target; it is not mandatory (neither is even a "check" target). For
those that do, they would likely be working from the $prefix tree to
locate installed files. I would rather not change the
prefix=/usr/local default, because I would like test scripts and such
work as expected on non-Debian systems also.
> > If we can expect that autopkgtest uses package executables in
> > /usr/bin, package data files in /usr/share/<package>, likewise with
> > other FHS directories, could you update the tutorial page to mention
> > that? Right now there is just an implication on that page from the
> > sample script that the binary executable will be in a directory in
> > $PATH.
>
> Right, as Debian policy mandates that user-executable programs should be in
> $PATH. So it's generally preferrable to call programs in $PATH without an
> explicit path, to make sure they are actually in the correct place.
> But note that there's also e. g. /sbin, or you might want to test a
> Python module, or even the validity of a documentation package.
Okay, so our tests can assume that installed package data files, for
example, are in /usr/share/package. That is something I wanted to
determine for another package I have not yet uploaded, because of
course data files will not be in the $PATH variable. Can you mention
that in the tutorial?
As someone else who filed a bug mentioned, these things are probably
obvious to the autopkgtest team, but they are ambiguities for a
someone trying to use autopkgtest for the first time.
I think adding just a little more about the runtime environment
someone can expect would make it easier for people to get their CI
scripts working on the first attempt.
Thanks,
Paul Hardy
[1]https://sources.debian.org/src/utfcheck/1.1-2/debian/tests/control/
[2]https://sources.debian.org/src/utfcheck/1.1-2/test/test-ascii/
[3]https://www.gnu.org/software/automake/manual/html_node/Standard-Directory-Variables.html
