Bug#907846: autopkgtest: Please consider additional section for tutorial web page
A general comment: although this TUTORIAL file originally was just meant to reflect one talk from 2015, I recommend letting it evolve to be _the_ tutorial for using autopkgtest in the Debian CI environment, because this is one link that search engines find when someone searches for "autopkgtest tutorial", and it has "tutorial" in its name. There are autopkgtest information files all over the place, and it isn't immediately clear which should be an "official" Debian reference tutorial. Ideally this TUTORIAL file could evolve into _the_ tutorial, rather than having someone go through every link they can find trying to piece things together (which is what I did). On Tue, Sep 4, 2018 at 11:59 AM Paul Gevers wrote: > ... > You're looking at this file: > https://salsa.debian.org/ci-team/debci/blob/master/docs/TUTORIAL.md > > It would be awesome if you could propose changes to that file instead of > the generated html file. However, I already have some comments. I did not know that the HTML document I changed had its origin in another format. I _could_ modify the md file, but I do not work in md usually and I would be taking a chance of something being lost in translation. If nobody who is familiar with the Debian md format has time I can do that, but then please compare any formatting with the HTML file that I submitted; I do know HTML well and what I did there is what I intended. HTML is my native language. :-) > > On 03-09-18 06:42, Paul Hardy wrote: > > Please consider adding "The test environment" section (or something > > like it) that I added to the autopkgtest tutorial page. It gives an > > introductory description of the general environment in which a CI test > > script runs in a way that I have not found elsewhere. > > Although I appreciate what you try to do here, it rather feels weird, as > you are just describing a very standard Debian setup. I really think > this is not the place to describe how Debian works and where packages > install their files. Policy already does that. So your first paragraph > could be "The test environment is a minimalistic standard Debian > installation." That is good, but I would still make it clear that the installation is not in some sub-tree (for example, /var/tmp/ci/sid/...). I would explicitly mention as an example that data files for "mypackage" are in "/usr/share/mypackage". That should get the point across. > > Your second sentence basically repeats stuff that is in the autopkgtest > spec. Maybe we should more clearly link that? > https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst I think that AUTOPKGTEST_TMP, etc., is so important that it merits mentioning directly in the document. Not everyone will click a link; put the information in this TUTORIAL document where it cannot be missed. > > $PATH will just contain the default Debian $PATH, again, I think this is > described in Debian policy. The location of the scripts is described in > the spec again. Yes, but I didn't know that the files were installed with "/" as the absolute top-level directory when I began. I recommend conveying that somehow explicitly. Because I only knew that installed binaries could be located using $PATH from the "trivial example" and the bug report that initiated that example, this is what the debian/tests/control file did for utfcheck-1.1-2: Test-Command: cd test && progloc=`which utfcheck` && utfcheck_bindir=`dirname $progloc` ./test-all Now that I know, for example, that my utfcheck program is installed as per the FHS (and not in some sub-tree for example under /var/tmp/ci/sid/usr/bin using $DESTDIR), the debian/tests/control file for utfcheck-1.2-1 (my most recent upload) has simplified the above to Test-Command: cd test && utfcheck_bindir=/usr/bin ./test-all Does this example make it clear what I did not understand from the autopkgtest and CI documentation? So I recommend adding enough to make it clear that absolute file locations are per the FHS and Debian Policy. You can say that in an abbreviated way compared to what I wrote in the HTML file I sent, but not so abbreviated that it might require the kind of experimentation that I did with utfcheck-1.1-2 and -1.2-1 (and I'm still not done; I want to simplify it further by just being able to say "make installcheck", incorporating what I learned into my Makefiles). When I get that far, I intend to write up what I did to get everything to work between GNU Autotools, autopkgtest in the Debian CI environment, and test scripts in the source tree that work on many systems (not just with autopkgtest). > ... > > I copied the two paragraphs that describe AUTOPKGTEST_TMP and > > AUTOPKGTEST_ARCHIVE from > > > > https://people.debian.org/~mpitt/autopkgtest/README.package-tests.html > > Hmm, we live on salsa nowadays, please follow the link above. And > instead of repeating stuff, can't we make smart text around links? I recommend directly adding mention of those two
Bug#907846: autopkgtest: Please consider additional section for tutorial web page
On Wed, Sep 5, 2018 at 6:46 AM Antonio Terceiro wrote:
>
> On Tue, Sep 04, 2018 at 08:59:12PM +0200, Paul Gevers wrote:
> > Control: reassign -1 debci
> >
> > Dear Paul
> >
> > You're looking at this file:
> > https://salsa.debian.org/ci-team/debci/blob/master/docs/TUTORIAL.md
> >
> > It would be awesome if you could propose changes to that file instead of
> > the generated html file. However, I already have some comments.
>
> Moreover: as is said in the beginning of that tutorial, it is a
> transcription of a talk I gave at Debconf15. What is not said, however,
> is that transcription is incomplete: it stops at example ("tip") 1, and
> the talk goes up to tip 9. Finishing that transcription would be a nice
> contribuition, if anyone out there is listening (hint, hint).
>
> > On 03-09-18 06:42, Paul Hardy wrote:
> > > Please consider adding "The test environment" section (or something
> > > like it) that I added to the autopkgtest tutorial page...
> >
> > Although I appreciate what you try to do here, it rather feels weird, as
> > you are just describing a very standard Debian setup. I really think
> > this is not the place to describe how Debian works and where packages
> > install their files. Policy already does that. So your first paragraph
> > could be "The test environment is a minimalistic standard Debian
> > installation."
> >
> > Your second sentence basically repeats stuff that is in the autopkgtest
> > spec. Maybe we should more clearly link that?
> > https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst
> >
> > $PATH will just contain the default Debian $PATH, again, I think this is
> > described in Debian policy. The location of the scripts is described in
> > the spec again.
>
> I think one point that maybe still didn't get across here is that
> autopkgtest is intended to test packages in their *installed* form. So,
> it should be obvious where stuff is: programs are in $PATH, libraries
> are in the expected locations, services are supposed to be running, etc.
It was obvious to me from the documentation that the CI environment
tested installed packages. What was not obvious, because I do not
know all of the possible virtual environment setups, is that the
packages are not installed in some sub-tree, for example using
$DESTDIR. It seemed that one CI system possibly could have different
sub-trees for sid, testing, etc. Now, from what Martin Pitt said, it
sounds like $DESTDIR will always be null. That is why I did not know
that ordinary executables could just be referenced in /usr/bin.
>
> i.e. the environment in which tests run is a standard Debian system,
> with nothing special, and with -- and only with -- the stuff you said
> needed to be installed for the tests.
>
> [...]
> > Thanks for trying to improve our documentation.
>
> Yes, thanks a lot.
>
> I realize that CI/autopkgtest is a somewhat daunting subject as there
> are several moving parts, and it can be confusing in the beginning. At
> some point it would be very useful to be able to "follow along" people
> who are learning it, capture all the issues that come up and try to
> produce a guide that could make it easier for people doing the same at
> the future.
>
> In fact I think that would make for a cool project for our outreach
> programs.
My life would have been easier if all I attempted was to write tests
that only worked on Debian. I am trying to expand the envelope with
tests that should work on any system where Autotools works. I have
test scripts that work when run from my package "test" directory, and
also with Autotools "check", "distcheck", and "installcheck" targets
from the top-level source directory, except I am not yet at the point
where my test scripts will work with "make installcheck" in the CI
environment on Debian. That is the last piece of the puzzle I need to
solve. At that point, my scripts could be used as examples of how to
write such multi-system test scripts that play nice with Debian if
there is interest in that.
I will do more on this during the weekend but just wanted to give this
initial response.
Thank you for what documentation does exist--there is a lot of it. It
was just the beginning description of what environment someone can
expect with the CI environment that wasn't completely clear to me.
Paul Hardy
Bug#907846: autopkgtest: Please consider additional section for tutorial web page
On Tue, Sep 04, 2018 at 08:59:12PM +0200, Paul Gevers wrote:
> Control: reassign -1 debci
>
> Dear Paul
>
> You're looking at this file:
> https://salsa.debian.org/ci-team/debci/blob/master/docs/TUTORIAL.md
>
> It would be awesome if you could propose changes to that file instead of
> the generated html file. However, I already have some comments.
Moreover: as is said in the beginning of that tutorial, it is a
transcription of a talk I gave at Debconf15. What is not said, however,
is that transcription is incomplete: it stops at example ("tip") 1, and
the talk goes up to tip 9. Finishing that transcription would be a nice
contribuition, if anyone out there is listening (hint, hint).
> On 03-09-18 06:42, Paul Hardy wrote:
> > Please consider adding "The test environment" section (or something
> > like it) that I added to the autopkgtest tutorial page. It gives an
> > introductory description of the general environment in which a CI test
> > script runs in a way that I have not found elsewhere.
>
> Although I appreciate what you try to do here, it rather feels weird, as
> you are just describing a very standard Debian setup. I really think
> this is not the place to describe how Debian works and where packages
> install their files. Policy already does that. So your first paragraph
> could be "The test environment is a minimalistic standard Debian
> installation."
>
> Your second sentence basically repeats stuff that is in the autopkgtest
> spec. Maybe we should more clearly link that?
> https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst
>
> $PATH will just contain the default Debian $PATH, again, I think this is
> described in Debian policy. The location of the scripts is described in
> the spec again.
I think one point that maybe still didn't get across here is that
autopkgtest is intended to test packages in their *installed* form. So,
it should be obvious where stuff is: programs are in $PATH, libraries
are in the expected locations, services are supposed to be running, etc.
i.e. the environment in which tests run is a standard Debian system,
with nothing special, and with -- and only with -- the stuff you said
needed to be installed for the tests.
[...]
> Thanks for trying to improve our documentation.
Yes, thanks a lot.
I realize that CI/autopkgtest is a somewhat daunting subject as there
are several moving parts, and it can be confusing in the beginning. At
some point it would be very useful to be able to "follow along" people
who are learning it, capture all the issues that come up and try to
produce a guide that could make it easier for people doing the same at
the future.
In fact I think that would make for a cool project for our outreach
programs.
signature.asc
Description: PGP signature
Bug#907846: autopkgtest: Please consider additional section for tutorial web page
Control: reassign -1 debci Dear Paul You're looking at this file: https://salsa.debian.org/ci-team/debci/blob/master/docs/TUTORIAL.md It would be awesome if you could propose changes to that file instead of the generated html file. However, I already have some comments. On 03-09-18 06:42, Paul Hardy wrote: > Please consider adding "The test environment" section (or something > like it) that I added to the autopkgtest tutorial page. It gives an > introductory description of the general environment in which a CI test > script runs in a way that I have not found elsewhere. Although I appreciate what you try to do here, it rather feels weird, as you are just describing a very standard Debian setup. I really think this is not the place to describe how Debian works and where packages install their files. Policy already does that. So your first paragraph could be "The test environment is a minimalistic standard Debian installation." Your second sentence basically repeats stuff that is in the autopkgtest spec. Maybe we should more clearly link that? https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst $PATH will just contain the default Debian $PATH, again, I think this is described in Debian policy. The location of the scripts is described in the spec again. > The file is > small, so I am attaching the whole gzipped file to avoid possible > issues with diff on the very long line that is the table of contents > list at the start of the file. The original web page is at > > https://ci.debian.net/doc/file.TUTORIAL.html As mentioned above, this is a generated file, please have a look at the source linked above. > I copied the two paragraphs that describe AUTOPKGTEST_TMP and > AUTOPKGTEST_ARCHIVE from > > https://people.debian.org/~mpitt/autopkgtest/README.package-tests.html Hmm, we live on salsa nowadays, please follow the link above. And instead of repeating stuff, can't we make smart text around links? > I do not know if what I wrote is accurate, because I do not understand > every virtual environment where autopkgtest can run. Resolving my > uncertainty will hopefully let me write optimal CI test scripts in the > future. What I have written, though, is background information that I > did not see anywhere. It would have saved me time when I started > trying to use CI testing; hopefully it will save others time. The README you referred to is an old one. I propose you read the current one and propose improvements to that file instead of the one you are aiming at now (although all improvements to that one are very welcome, don't let my response put you off). > I am submitting this as a new bug rather than modifying bug #906617, > because I would like to still have at least prefix=/usr and DESTDIR="" > defined if that sounds reasonable. I will add a note to that effect > on that bug later. If they are defined like that, I can provide a > further patch to add mention of them to this tutorial web page. Thanks for trying to improve our documentation. Paul signature.asc Description: OpenPGP digital signature
Bug#907846: autopkgtest: Please consider additional section for tutorial web page
Package: autopkgtest Version: 5.5 Severity: wishlist Please consider adding "The test environment" section (or something like it) that I added to the autopkgtest tutorial page. It gives an introductory description of the general environment in which a CI test script runs in a way that I have not found elsewhere. The file is small, so I am attaching the whole gzipped file to avoid possible issues with diff on the very long line that is the table of contents list at the start of the file. The original web page is at https://ci.debian.net/doc/file.TUTORIAL.html I copied the two paragraphs that describe AUTOPKGTEST_TMP and AUTOPKGTEST_ARCHIVE from https://people.debian.org/~mpitt/autopkgtest/README.package-tests.html I do not know if what I wrote is accurate, because I do not understand every virtual environment where autopkgtest can run. Resolving my uncertainty will hopefully let me write optimal CI test scripts in the future. What I have written, though, is background information that I did not see anywhere. It would have saved me time when I started trying to use CI testing; hopefully it will save others time. I am submitting this as a new bug rather than modifying bug #906617, because I would like to still have at least prefix=/usr and DESTDIR="" defined if that sounds reasonable. I will add a note to that effect on that bug later. If they are defined like that, I can provide a further patch to add mention of them to this tutorial web page. Thanks, Paul Hardy autopkgtest-tutorial-new.html.gz.sig Description: PGP signature autopkgtest-tutorial-new.html.gz Description: application/gzip
