Re: Package testing with Salsa CI for new packages
On Monday, 21 August 2023 15:01:26 CEST Carles Pina i Estany wrote:
>
> If you want, you can simplify more (it's not exactly the same, so it
> might or might not help). There is a way on GitLab to point to the
> latest build of a job. For example, you have the following URL for one
> of the git repos:
>
> https://salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/artifacts/raw/
aptly
>
> You could use instead (to avoid the pipeline number):
> https://salsa.debian.org/moin-team/emeraldtree/-/jobs/artifacts/debian/
master/raw/aptly?job=aptly
That is a good idea, since I always intend the latest job to be successful.
:-)
> Which is a redirect to the latest pipeline. Currently:
>
> $ curl -s -I
> "https://salsa.debian.org/moin-team/emeraldtree/-/jobs/artifacts/debian/mas
> ter/raw/aptly?job=aptly" | grep -E -i "^(http|location)" HTTP/2 302
> location:
> https://salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/artifacts/raw/
aptly
>
>
> Follows this format:
> BRANCH=debian/master
> DIRECTORY=aptly
> JOB_NAME=aptly
> https://salsa.debian.org/moin-team/emeraldtree/-/jobs/artifacts/${BRANCH}/ra
> w/${DIRECTORY}?job=${JOB_NAME}
Thanks for demonstrating this. I imagine that it is documented somewhere in
the GitLab manuals, too. One note of caution, though. It seems that using
these redirected URLs is not necessarily possible with apt, and so it might be
advisible to obtain the redirected URL as shown in the output above.
In my script I now get curl (and not wget any more) to obtain the final URL to
provide to the apt configuration, and I also use this job-specific URL to
obtain each GPG key. I suppose I could change the apt configuration to allow
redirects instead.
> Just a side note: be careful about expiring artifacts. In some projects
> (settings dependant) only the latest artifact is kept and older ones
> might be expired (deleted) after some time. I don't think that this is
> the case of the moin-team/emeraldtree after a quick check... but I'm
> unsure where this is properly checked on GitLab.
"CI/CD Settings", "Artifacts", "Keep artifacts from most recent successful
jobs" is set, so I imagine that as long as the latest job is successful,
everything will still work. Indeed, it is desirable to rely on the latest job
under such conditions: one would not want a newer dependency build to break
the Moin package because an older set of artefacts has been deleted.
Thanks once again for the feedback and advice!
Paul
Re: Package testing with Salsa CI for new packages
Hi,
On 20 Aug 2023 at 23:16:57, Paul Boddie wrote:
> On Sunday, 20 August 2023 14:06:37 CEST Carles Pina i Estany wrote:
[...]
Thanks for sharing your path here, and I'm happy that you are reaching
your destination :-)
> autopkgtest:
> extends: .test-autopkgtest
> variables:
> SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=debian/salsa/add-
> repositories.sh'
>
> piuparts:
> extends: .test-piuparts
> variables:
> SALSA_CI_PIUPARTS_PRE_INSTALL_SCRIPT: 'debian/salsa/add-repositories.sh'
>
> You can see that by defining the variables to customise the tools, I am able
> to work with the existing job definitions. This simplifies the CI description
> file considerably:
>
> https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa-ci.yml
>
> The script is pretty straightforward, too:
>
> https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa/add-repositories.sh
If you want, you can simplify more (it's not exactly the same, so it
might or might not help). There is a way on GitLab to point to the
latest build of a job. For example, you have the following URL for one
of the git repos:
https://salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/artifacts/raw/aptly
You could use instead (to avoid the pipeline number):
https://salsa.debian.org/moin-team/emeraldtree/-/jobs/artifacts/debian/master/raw/aptly?job=aptly
Which is a redirect to the latest pipeline. Currently:
$ curl -s -I
"https://salsa.debian.org/moin-team/emeraldtree/-/jobs/artifacts/debian/master/raw/aptly?job=aptly;
| grep -E -i "^(http|location)"
HTTP/2 302
location:
https://salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/artifacts/raw/aptly
Follows this format:
BRANCH=debian/master
DIRECTORY=aptly
JOB_NAME=aptly
https://salsa.debian.org/moin-team/emeraldtree/-/jobs/artifacts/${BRANCH}/raw/${DIRECTORY}?job=${JOB_NAME}
Just a side note: be careful about expiring artifacts. In some projects
(settings dependant) only the latest artifact is kept and older ones
might be expired (deleted) after some time. I don't think that this is
the case of the moin-team/emeraldtree after a quick check... but I'm
unsure where this is properly checked on GitLab.
[...]
Cheers,
--
Carles Pina i Estany
https://carles.pina.cat || Wiktionary translations: https://kamus.pina.cat
Re: Package testing with Salsa CI for new packages
On Sunday, 20 August 2023 14:06:37 CEST Carles Pina i Estany wrote: > > Interesting approach, but if you could use the variables and ship the > shell script it might reduce some duplication between jobs (if possible, > I haven't look into too much detail in your case) and more importantly > you might be able to use the standard "autopkgtest" or "piuparts" > (instead of redefining them). I have continued along such a path. > Just last week I created a new autopkgtest (I have it running on > bookworm and bullseye). It's done this way: > -- > autopkgtest-bullseye: > extends: .test-autopkgtest > variables: > RELEASE: "bullseye-backports" > SALSA_CI_AUTOPKGTEST_ARGS: > '--setup-commands=ci/pin-django-from-backports.sh > --skip-test=debusine-doc-linkcheck' -- > > Or, for more context: > https://salsa.debian.org/freexian-team/debusine/-/blob/add-bullseye-support/ > .gitlab-ci.yml#L84 It ends up having "autopkgtest" and > "autopkgtest-bullseye". But using the variable SALSA_CI_DISABLE_AUTOPKGTEST > I could disable the one without my extends. Previously, I had merely redefined the autopkgtest and piuparts jobs, replacing them with ones that use the customised code: autopkgtest: extends: .test-autopkgtest-revised piuparts: extends: .test-piuparts-revised I had assumed that such redefinition was possible. When people decide to introduce a new language, as GitLab appear to have done, they should realise that they need to match the considerable levels of documentation that existing languages may have. Fortunately, my assumptions were correct. Following your lead, I have since found that the following works: autopkgtest: extends: .test-autopkgtest variables: SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=debian/salsa/add- repositories.sh' piuparts: extends: .test-piuparts variables: SALSA_CI_PIUPARTS_PRE_INSTALL_SCRIPT: 'debian/salsa/add-repositories.sh' You can see that by defining the variables to customise the tools, I am able to work with the existing job definitions. This simplifies the CI description file considerably: https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa-ci.yml The script is pretty straightforward, too: https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa/add-repositories.sh It is so mundane that I could imagine it being replaced by standard functionality very easily. [Job numbers] > If it's in the same pipeline: you don't need the numbers to get the > artifacts (the .deb files from build). The .deb files are produced by the pipelines associated with other package repositories. Thus, Aptly is used to make the .deb files available to the Moin package's pipeline. I imagine that I could build the dependencies in the same pipeline, however. Apparently, GitLab's non-free editions support "multi- project" pipelines, but I suspect that Aptly has been introduced to get around the lack of such functionality in the free edition. > For the aptly generated artifact: I will investigate this tomorrow (to > fetch it and try to use) (I want to replace some code that I did with > the aptly repo, if possible). Good luck with that task! :-) As a result of this discussion and a lot of effort, I have managed to get autopkgtest and piuparts to see and to use the package dependencies needed by the Moin package. This allowed me to tackle the test suite that had caused me to look at all of this in the beginning. And now, it seems that I can get all the reasonable tests to pass: https://salsa.debian.org/moin-team/moin/-/pipelines/567871 Once again, many thanks for all the guidance! Paul
Re: Package testing with Salsa CI for new packages
Hi, On 20 Aug 2023 at 00:49:20, Paul Boddie wrote: > On Saturday, 19 August 2023 20:32:59 CEST Carles Pina i Estany wrote: > > > > Quick answer for now... > > And a very quick reply from me... :-) > > [...] > > > I don't know if a .asc files are allowed. Actually, from my > > .bash_history: > > --- > > wget -O- https://www.virtualbox.org/download/oracle_vbox_2016.asc | sudo gpg > > --dearmor --yes --output /usr/share/keyrings/oracle-virtualbox-2016.gpg --- > > > > (I was following instructions, I didn't try leaving the .asc file there) > > They are allowed in recent versions of apt, as confirmed by the man page. > > [...] Cool! I'll try next time (I cannot see it on my man apt-get, apt or sources.list on bookworm but I might need a newer apt) > > You could try using: > > --- > > variables: > > SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=ci/setup-the-repo.sh' > > --- > > (and write and ship the ci/setup-the-repo.sh in the repo, do whatever > > you need there) > > This could be very useful. Also available (and also added last year for a similar case): - variables: SALSA_CI_PIUPARTS_PRE_INSTALL_SCRIPT: 'ci/pin-django-from-backports.sh' - (and SALSA_CI_PIUPARTS_POST_INSTALL_SCRIPT) > > Or, probably even better but less flexible: > > --- > > variables: > > SALSA_CI_AUTOPKGTEST_ARGS: '--add-apt-source="deb http://MIRROR SUITE > > COMPONENT' --- > > (I found add-apt-source via "man autopkgtest" in TEST BED SETUP OPTIONS. > > I haven't used add-apt-source, I don't know what happens with the gpg > > keys... but you could use [trusted=yes] :-| ) > > > > For the MIRROR you have salsa variables that might help if you need to > > specify the pipeline. > > So could this. > > > > So, what I now need to do is to find out how I can make the new packages > > > available to autopkgtest specifically. > > > > We will get there! (one way or another) > > Well, I have engineered something very inelegant, revealed in full here: > > https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa-ci.yml I, more or less, see what you did! Interesting approach, but if you could use the variables and ship the shell script it might reduce some duplication between jobs (if possible, I haven't look into too much detail in your case) and more importantly you might be able to use the standard "autopkgtest" or "piuparts" (instead of redefining them). Just last week I created a new autopkgtest (I have it running on bookworm and bullseye). It's done this way: -- autopkgtest-bullseye: extends: .test-autopkgtest variables: RELEASE: "bullseye-backports" SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=ci/pin-django-from-backports.sh --skip-test=debusine-doc-linkcheck' -- Or, for more context: https://salsa.debian.org/freexian-team/debusine/-/blob/add-bullseye-support/.gitlab-ci.yml#L84 It ends up having "autopkgtest" and "autopkgtest-bullseye". But using the variable SALSA_CI_DISABLE_AUTOPKGTEST I could disable the one without my extends. [...] > > hopefully SALSA_CI_AUTOPKGTEST_ARGS will help you. I added this in > > salsa-ci/pipeline last year because I was trying to do a similar thing > > to what you are doing (I had to pin a package from backports). > > > > (I'm just a user of salsa-ci/pipeline, not a member of the team neither > > I can speak for them!) > > If this can simplify what I've done, which is really quite horrible, then I > will adopt it instead. The way that the artefacts of the dependencies are > bound up in specifically numbered jobs is also particularly unfortunate. Of If it's in the same pipeline: you don't need the numbers to get the artifacts (the .deb files from build). For the aptly generated artifact: I will investigate this tomorrow (to fetch it and try to use) (I want to replace some code that I did with the aptly repo, if possible). Cheers, -- Carles Pina i Estany https://carles.pina.cat || Wiktionary translations: https://kamus.pina.cat
Re: Package testing with Salsa CI for new packages
On Saturday, 19 August 2023 20:32:59 CEST Carles Pina i Estany wrote: > > Quick answer for now... And a very quick reply from me... :-) [...] > I don't know if a .asc files are allowed. Actually, from my > .bash_history: > --- > wget -O- https://www.virtualbox.org/download/oracle_vbox_2016.asc | sudo gpg > --dearmor --yes --output /usr/share/keyrings/oracle-virtualbox-2016.gpg --- > > (I was following instructions, I didn't try leaving the .asc file there) They are allowed in recent versions of apt, as confirmed by the man page. [...] > You could try using: > --- > variables: > SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=ci/setup-the-repo.sh' > --- > (and write and ship the ci/setup-the-repo.sh in the repo, do whatever > you need there) This could be very useful. > Or, probably even better but less flexible: > --- > variables: > SALSA_CI_AUTOPKGTEST_ARGS: '--add-apt-source="deb http://MIRROR SUITE > COMPONENT' --- > (I found add-apt-source via "man autopkgtest" in TEST BED SETUP OPTIONS. > I haven't used add-apt-source, I don't know what happens with the gpg > keys... but you could use [trusted=yes] :-| ) > > For the MIRROR you have salsa variables that might help if you need to > specify the pipeline. So could this. > > So, what I now need to do is to find out how I can make the new packages > > available to autopkgtest specifically. > > We will get there! (one way or another) Well, I have engineered something very inelegant, revealed in full here: https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa-ci.yml It defines a package list file and uses wget to obtain each repository's public key. Fortunately, wget works at the "script" level of the jobs involved, whereas it didn't work at what must have been the global "before_script" level (suggested in the documentation) and needed to be installed first. To make autopkgtest and piuparts find the packages, the repository details are then copied into the appropriate location in each case. However, since such customisation is not readily incorporated into the existing job descriptions, I have had to duplicate the existing job description code and insert additional statements in the script sections. As a consequence of these changes, the extra packages are found and installed within the environments concerned. This is enough to make the piuparts job succeed, but autopkgtest still fails. Fortunately, that may be due only to problems with the actual tests: I think the upstream maintainers may have made some errors. [...] > hopefully SALSA_CI_AUTOPKGTEST_ARGS will help you. I added this in > salsa-ci/pipeline last year because I was trying to do a similar thing > to what you are doing (I had to pin a package from backports). > > (I'm just a user of salsa-ci/pipeline, not a member of the team neither > I can speak for them!) If this can simplify what I've done, which is really quite horrible, then I will adopt it instead. The way that the artefacts of the dependencies are bound up in specifically numbered jobs is also particularly unfortunate. Of course, if the packages enter the Debian archives, all of this extra effort will be made largely redundant, anyway, even if coordinated testing of new package versions would still benefit from such effort. [...] > I don't know if an FAQ, conventions, best practises or something might > help... I think that the documentation could be reworked, but that is largely true for a lot of the materials available for Debian. And since some of that documentation resides on the Debian Wiki, that is another reason why I am trying to move this particular packaging exercise forwards. Thanks once again! Paul
Re: Package testing with Salsa CI for new packages
Hi, Quick answer for now... On 19 Aug 2023 at 19:04:09, Paul Boddie wrote: > On Friday, 18 August 2023 19:54:55 CEST Carles Pina i Estany wrote: > > > > Ha! I wasn't aware of the aptly option > > (https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built-> > > apt-repository and SALSA_CI_DISABLE_APTLY=0). > > I think that I might have re-invented the wheel in a tiny part of > > Debusine CI/CD. > > It is certainly a way of propagating packages to those that might need > them. However, the instructions indicating how a package might access > these dependencies appear to be deficient. I haven't used aptly on salsa but on Monday I want to give it a try, to see if I can replace my solution... > It does not appear to be sufficient to merely specify the dependency > package repositories and mark them as trusted. Doing that will just > cause the repositories to be regarded as ignored and the GPG keys > signalled as unrecognised. > > So, the GPG keys need to be obtained. This is a hassle because something like > wget is needed to do that, and then apt has to be persuaded not to fail in an > opaque way. So the modified recipe is something like this: > > before_script: > - apt-get update > - NON_INTERACTIVE=1 apt-get install -y wget > - echo "deb https://salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/ > artifacts/raw/aptly unstable main" | tee /etc/apt/sources.list.d/pkga.list > ... > - wget -q -O /etc/apt/trusted.gpg.d/emeraldtree.asc https:// > salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/artifacts/raw/aptly/ > public-key.asc I'm ensure that this is correct. I have: - deb [arch=amd64 signed-by=/usr/share/keyrings/oracle-virtualbox-2016.gpg] https://download.virtualbox.org/virtualbox/debian bookworm contrib - So I thought that the "signed-by" was needed; and that the files that I have there are binary files (not .asc) of this type: carles@pinux:~$ file /etc/apt/trusted.gpg.d/skype.gpg /etc/apt/trusted.gpg.d/skype.gpg: OpenPGP Public Key Version 4, Created Wed Jun 22 09:36:35 2016, RSA (Encrypt or Sign, 2048 bits); User ID; Signature; OpenPGP Certificate carles@pinux:~$ I don't know if a .asc files are allowed. Actually, from my .bash_history: --- wget -O- https://www.virtualbox.org/download/oracle_vbox_2016.asc | sudo gpg --dearmor --yes --output /usr/share/keyrings/oracle-virtualbox-2016.gpg --- (I was following instructions, I didn't try leaving the .asc file there) > ... > - apt-get update > > This seems to make the various jobs happy, but the one that I was most > concerned with remains unhappy! I don't actually need the dependencies > for anything other than autopkgtest, but that job employs its own > environment where the above recipe has no effect. You could try using: --- variables: SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=ci/setup-the-repo.sh' --- (and write and ship the ci/setup-the-repo.sh in the repo, do whatever you need there) Or, probably even better but less flexible: --- variables: SALSA_CI_AUTOPKGTEST_ARGS: '--add-apt-source="deb http://MIRROR SUITE COMPONENT' --- (I found add-apt-source via "man autopkgtest" in TEST BED SETUP OPTIONS. I haven't used add-apt-source, I don't know what happens with the gpg keys... but you could use [trusted=yes] :-| ) For the MIRROR you have salsa variables that might help if you need to specify the pipeline. > So, what I now need to do is to find out how I can make the new packages > available to autopkgtest specifically. We will get there! (one way or another) > > I will point out at some things that might safe some time to you > > (great) or not (ignore! :-) ): > > > > debusine's .gitlab-ci.yml: > > https://salsa.debian.org/freexian-team/debusine/-/blob/devel/.gitlab-ci.yml > > This is definitely useful for examples of the CI definition syntax and how > hidden jobs can be used to provide additional script fragments. > > [...] > > > The job autopkgtest, via debci command, runs autopkgtest: > > https://salsa.debian.org/freexian-team/debusine/-/jobs/4574458#L65 > > The difficult thing appears to be the configuration of this testing > environment. One approach might involve overriding the existing job > definitions to incorporate the injection of new packages. hopefully SALSA_CI_AUTOPKGTEST_ARGS will help you. I added this in salsa-ci/pipeline last year because I was trying to do a similar thing to what you are doing (I had to pin a package from backports). (I'm just a user of salsa-ci/pipeline, not a member of the team neither I can speak for them!) > [...] > > > It's possible for sure. Other people in this list might come up with a > > different idea. I don't have almost any experience with Debian > > packaging, but I have some experience on the salsa CI. So I might be > > giving you solutions that might be sub-optimal! :-) > > I think that you are giving me some good ideas at least. I do feel that I am > encountering problems that must have been
Re: Package testing with Salsa CI for new packages
On Friday, 18 August 2023 19:54:55 CEST Carles Pina i Estany wrote: > > Ha! I wasn't aware of the aptly option > (https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built-> > apt-repository and SALSA_CI_DISABLE_APTLY=0). > I think that I might have re-invented the wheel in a tiny part of > Debusine CI/CD. It is certainly a way of propagating packages to those that might need them. However, the instructions indicating how a package might access these dependencies appear to be deficient. It does not appear to be sufficient to merely specify the dependency package repositories and mark them as trusted. Doing that will just cause the repositories to be regarded as ignored and the GPG keys signalled as unrecognised. So, the GPG keys need to be obtained. This is a hassle because something like wget is needed to do that, and then apt has to be persuaded not to fail in an opaque way. So the modified recipe is something like this: before_script: - apt-get update - NON_INTERACTIVE=1 apt-get install -y wget - echo "deb https://salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/ artifacts/raw/aptly unstable main" | tee /etc/apt/sources.list.d/pkga.list ... - wget -q -O /etc/apt/trusted.gpg.d/emeraldtree.asc https:// salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/artifacts/raw/aptly/ public-key.asc ... - apt-get update This seems to make the various jobs happy, but the one that I was most concerned with remains unhappy! I don't actually need the dependencies for anything other than autopkgtest, but that job employs its own environment where the above recipe has no effect. So, what I now need to do is to find out how I can make the new packages available to autopkgtest specifically. > I will point out at some things that might safe some time to you > (great) or not (ignore! :-) ): > > debusine's .gitlab-ci.yml: > https://salsa.debian.org/freexian-team/debusine/-/blob/devel/.gitlab-ci.yml This is definitely useful for examples of the CI definition syntax and how hidden jobs can be used to provide additional script fragments. [...] > The job autopkgtest, via debci command, runs autopkgtest: > https://salsa.debian.org/freexian-team/debusine/-/jobs/4574458#L65 The difficult thing appears to be the configuration of this testing environment. One approach might involve overriding the existing job definitions to incorporate the injection of new packages. [...] > It's possible for sure. Other people in this list might come up with a > different idea. I don't have almost any experience with Debian > packaging, but I have some experience on the salsa CI. So I might be > giving you solutions that might be sub-optimal! :-) I think that you are giving me some good ideas at least. I do feel that I am encountering problems that must have been encountered many times before, but I could imagine that people do not use these tools in this way or have different expectations of what they are able to accomplish. As for Debian's tooling, I consider it rather chaotic. When I previously did packaging, things like pbuilder were still in use, being slowly pushed aside by sbuild and probably other things that are now forgotten or obsolete. There always seems to be new things coming along, new mixtures of technologies, and the potential for more confusion and uncertainty. Paul
Re: Package testing with Salsa CI for new packages
Hi, On 18 Aug 2023 at 19:03:48, Paul Boddie wrote: > On Friday, 18 August 2023 16:12:19 CEST Carles Pina i Estany wrote: > > > > For the jobs it is happening via > > https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built- > apt-repository > > Reviewing this documentation is actually more helpful than I thought it would > be. I had noticed the "aptly" task in the YAML files, and I had started to > wonder if that meant that some kind of repository publishing was occurring > somewhere. Ha! I wasn't aware of the aptly option (https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built-apt-repository and SALSA_CI_DISABLE_APTLY=0). I think that I might have re-invented the wheel in a tiny part of Debusine CI/CD. I will point out at some things that might safe some time to you (great) or not (ignore! :-) ): debusine's .gitlab-ci.yml: https://salsa.debian.org/freexian-team/debusine/-/blob/devel/.gitlab-ci.yml Latest CI build: https://salsa.debian.org/freexian-team/debusine/-/pipelines/566796 In the "build" job: https://salsa.debian.org/freexian-team/debusine/-/jobs/4574451 You can see what's doing and it's using salsa-ci/pipeline's defaults in the build. All comes from https://salsa.debian.org/salsa-ci-team/pipeline/ , specifically from: https://salsa.debian.org/salsa-ci-team/pipeline/-/blob/master/salsa-ci.yml#L201 (I sometimes grep salsa-ci-team/pipeline for commands that I see in debusine's CI output to understand the flow...) In the autopkgtest job it sets things up and invokes autopkgtest: https://salsa.debian.org/freexian-team/debusine/-/jobs/4574458 The *.deb from build are made available via "artifacts" (and that autopkgtest needs the artifact of build). The job autopkgtest, via debci command, runs autopkgtest: https://salsa.debian.org/freexian-team/debusine/-/jobs/4574458#L65 And, to be honest, I didn't look into these details for long time. From my point of view I just write debian/tests/control with the tests (commands) to run: https://salsa.debian.org/freexian-team/debusine/-/blob/devel/debian/tests/control And that things run in a container/chroot (depends), etc. You can invoke autopkgtest locally as well... > > > In the Salsa CI environment, I would need to have the built packages > > > (found in the artefacts for each package's build job) copied somewhere > > > that can then be found by the Moin package's pipeline jobs and the > > > scripts creating a special repository of new packages. > > > > Archiving artifacts should happen automatically on the "build" step of > > Salsa CI (salsa-ci/pipeline). If I understand correctly what you > > wanted... > > I think you have a good understanding of what I am trying to achieve. If I > can > get the new package dependencies (emeraldtree, feedgen, and so on) to yield > installable packages when built that can then be referenced by Salsa CI as it > runs the build jobs for Moin, I have a chance of running the test suite. yep! > I'm currently persuading the CI system to run the "aptly" task and to publish > package repositories. I will then augment a customised YAML file for the Moin I might look more about this on Monday (I will write it down now) because I am using "dpkg-scanpackages" and I might be able to avoid it via SALSA_CI_DISABLE_APTLY: 0 I see the salsa-CI like a fancy environment where to run scripts, generate files, pass files between scripts, refactor scripts (in YAML in my .gitlab-ci.yml or to upstream salsa-ci/pipeline), scheduled jobs, etc. so almost anything that you can think of you can do it. Either with artifacts to keep and retrieve files between "jobs" or via GitHub Pages. If you need help creating a new job (a new bubble in the pipeline) let me know (even if you want off-list, if this is getting off-topic of debian-python). Or look at the https://salsa.debian.org/freexian-team/debusine/-/blob/devel/.gitlab-ci.yml where were created the stage "upstream-tests" with things like "code-linting". So You could even create dependencies of your task before the tests are executed and pass the .deb files via artifacts (or even repositories). > package with references to these repositories and see if the test > suite can be invoked somewhat more successfully as a consequence. It's possible for sure. Other people in this list might come up with a different idea. I don't have almost any experience with Debian packaging, but I have some experience on the salsa CI. So I might be giving you solutions that might be sub-optimal! :-) Cheers, -- Carles Pina i Estany https://carles.pina.cat || Wiktionary translations: https://kamus.pina.cat
Re: Package testing with Salsa CI for new packages
On Fri, Aug 18, 2023 at 07:19:18PM +0200, Paul Boddie wrote: > > > Here, it would seem that the most prudent approach is to use the Salsa CI > > > service instead of trying to get the test suite to run during the package > > > build process. > > > > You should do both if possible, assuming that by "Salsa CI service" you > > mean autopkgtests which you can, and IMO should, also run locally. > > I'm not really clear on what autopkgtest really is, other than a tool that > uses some kind of test suite description that may reside in debian/test. I'm > also not completely clear on what is supposed to invoke it, other than either > the Salsa CI mechanism or dh_auto_test. The maintainer is supposed to invoke it before uploading the package, and the Debian infra invokes it on all packages that are uploaded. Notably, dh_auto_test is unrelated. > In the Debian Wiki documentation... > > https://wiki.debian.org/Python/LibraryStyleGuide > > ...it mentions a field in debian/control: > > Testsuite: autopkgtest-pkg-python > > My impression is that this calls autodep8 to generate some kind of test suite Yes. Though note that it generates a trivial test that only checks a top-level import (just like the wiki page says). > description which is then invoked by dh_auto_test. It's not invoked by dh_auto_test. autopkgtests are not a part of the package build process and they operate on built binary packages. > It doesn't help that there > is an alternative to this that resembles it but behaves differently: > > Testsuite: autopkgtest-pkg-pybuild It just generates a different (better) test. > > > I have also found it difficult to persuade the tests to run successfully > > > during the build process. A few of these attempt to invoke the moin > > > program, but this cannot be located since it is not installed in the > > > build environment. > > > > This should also be fine, unless it's completely impossible to run it > > without installing into /. > > The moin program is made available in setup.py using an entry point. Maybe if > there were some kind of script instead, it would work. AFAIK there should be ways to work with this. Ideally, of course, the upstream test suite should support this, but I understand that not all upstreams use common practices. > > > However, one conclusion is that testing a system, as some of the > > > test cases appear to do, and as opposed to testing library functionality, > > > is not necessarily appropriate when directed by something like > > > dh_auto_test. > > > > If there are tests that can't be run at build time you can skip those. You > > can even ask the upstream to provide tool arguments to simplify that. > > I may well discuss such matters with them. One challenge that is relevant in > this situation is that upstream have been working in their own virtualenv (or > venv, or whatever it is now) world for a few years, using plenty of > dependencies that were not packaged in Debian. Which is by itself not a problem from the technical side, as you could just package those (I understand that this requires effort and may have other problems, but by itself it's normal).
Re: Package testing with Salsa CI for new packages
On Friday, 18 August 2023 18:21:00 CEST Andrey Rakhmatullin wrote: > On Thu, Aug 17, 2023 at 05:10:08PM +0200, Paul Boddie wrote: > > Here, it would seem that the most prudent approach is to use the Salsa CI > > service instead of trying to get the test suite to run during the package > > build process. > > You should do both if possible, assuming that by "Salsa CI service" you > mean autopkgtests which you can, and IMO should, also run locally. I'm not really clear on what autopkgtest really is, other than a tool that uses some kind of test suite description that may reside in debian/test. I'm also not completely clear on what is supposed to invoke it, other than either the Salsa CI mechanism or dh_auto_test. In the Debian Wiki documentation... https://wiki.debian.org/Python/LibraryStyleGuide ...it mentions a field in debian/control: Testsuite: autopkgtest-pkg-python My impression is that this calls autodep8 to generate some kind of test suite description which is then invoked by dh_auto_test. It doesn't help that there is an alternative to this that resembles it but behaves differently: Testsuite: autopkgtest-pkg-pybuild What I have previously managed to do with the Salsa CI mechanism is to run tests specified in debian/test, disabling these tests at build time using the following in debian/rules: export PYBUILD_DISABLE=test For another package which has limited dependencies, I do this because the tests take a long time to run, and I even need to increase the timeout for the Salsa CI jobs. For this package, the tests don't take too long to run (those that do run), but some tests fail for reasons previously described, and this is due to some configuration issue that is unpleasant to debug. Success with testing using Salsa CI steers me away from using the Testsuite field in the normal build environment because the resulting test suite invocation failures suggest that some customisation is needed or that the environment is inappropriate in some way (even if I specify numerous build dependencies). > > One motivation for doing so involves not having to specify > > build dependencies for packages only needed for test suite execution, > > which itself requires the invocation of gbp buildpackage with > > --extra-package arguments since some packages are completely new to > > Debian. > > This is fine. Not to mention that the same problem exists for > autopkgtests, as you say below. > > > I have also found it difficult to persuade the tests to run successfully > > during the build process. A few of these attempt to invoke the moin > > program, but this cannot be located since it is not installed in the > > build environment. > > This should also be fine, unless it's completely impossible to run it > without installing into /. The moin program is made available in setup.py using an entry point. Maybe if there were some kind of script instead, it would work. > > However, one conclusion is that testing a system, as some of the > > test cases appear to do, and as opposed to testing library functionality, > > is not necessarily appropriate when directed by something like > > dh_auto_test. > > If there are tests that can't be run at build time you can skip those. You > can even ask the upstream to provide tool arguments to simplify that. I may well discuss such matters with them. One challenge that is relevant in this situation is that upstream have been working in their own virtualenv (or venv, or whatever it is now) world for a few years, using plenty of dependencies that were not packaged in Debian. That hasn't been entirely conducive to adoption of Moin 2 which has in turn isolated its developers further. Fortunately, the gulf between that world and Debian can be bridged, and some of the more tedious packaging seems to have been done for many of the dependencies involved. It just needs someone to shepherd this effort to an acceptable conclusion, which is of concern to Debian itself since Moin runs the Debian Wiki. Thanks for the encouragement and advice. Paul
Re: Package testing with Salsa CI for new packages
On Friday, 18 August 2023 16:12:19 CEST Carles Pina i Estany wrote: > > For the jobs it is happening via > https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built- apt-repository Reviewing this documentation is actually more helpful than I thought it would be. I had noticed the "aptly" task in the YAML files, and I had started to wonder if that meant that some kind of repository publishing was occurring somewhere. > > In the Salsa CI environment, I would need to have the built packages > > (found in the artefacts for each package's build job) copied somewhere > > that can then be found by the Moin package's pipeline jobs and the > > scripts creating a special repository of new packages. > > Archiving artifacts should happen automatically on the "build" step of > Salsa CI (salsa-ci/pipeline). If I understand correctly what you > wanted... I think you have a good understanding of what I am trying to achieve. If I can get the new package dependencies (emeraldtree, feedgen, and so on) to yield installable packages when built that can then be referenced by Salsa CI as it runs the build jobs for Moin, I have a chance of running the test suite. I'm currently persuading the CI system to run the "aptly" task and to publish package repositories. I will then augment a customised YAML file for the Moin package with references to these repositories and see if the test suite can be invoked somewhat more successfully as a consequence. Thanks once again for the guidance! Paul
Re: Package testing with Salsa CI for new packages
On Thu, Aug 17, 2023 at 05:10:08PM +0200, Paul Boddie wrote: > Here, it would seem that the most prudent approach is to use the Salsa CI > service instead of trying to get the test suite to run during the package > build process. You should do both if possible, assuming that by "Salsa CI service" you mean autopkgtests which you can, and IMO should, also run locally. > One motivation for doing so involves not having to specify > build dependencies for packages only needed for test suite execution, which > itself requires the invocation of gbp buildpackage with --extra-package > arguments since some packages are completely new to Debian. This is fine. Not to mention that the same problem exists for autopkgtests, as you say below. > I have also found it difficult to persuade the tests to run successfully > during the build process. A few of these attempt to invoke the moin program, > but this cannot be located since it is not installed in the build > environment. This should also be fine, unless it's completely impossible to run it without installing into /. > However, one conclusion is that testing a system, as some of the > test cases appear to do, and as opposed to testing library functionality, is > not necessarily appropriate when directed by something like dh_auto_test. If there are tests that can't be run at build time you can skip those. You can even ask the upstream to provide tool arguments to simplify that.
Re: Package testing with Salsa CI for new packages
Hi, On 18 Aug 2023 at 13:16:15, Paul Boddie wrote: > On Friday, 18 August 2023 09:51:29 CEST Carles Pina i Estany wrote: > > > > I'm not a Debian developer but I have some experience on Salsa CI, so I > > thought that I might be able to help... but then I was confused by a > > specific part of the message: > > > > On 17 Aug 2023 at 17:10:08, Paul Boddie wrote: > > > > [...] > > > > > For another package I have been working on, the Salsa CI facility has > > > proven to be usable, configured using files in debian/test, particularly > > > as it allows test-related dependencies to be specified independently. > > > However, this other package has no dependencies that are currently > > > unpackaged in Debian. Meanwhile, the testing of this new Moin package > > > depends on brand new packages somehow being made available. > > > > If this dependencies are available on the "build" step: could they be > > made available on the autopkgtest? I didn't quite understand why this is > > not possible. I've found the autopkgtest quite flexible (since the tests > > are scripts that could prepare some environment) > > The package has dependencies on installation but these dependencies > are not strictly necessary when building. However, if I wanted to run I have the same situation on this side. Two solutions for two places. Debusine uses https://salsa.debian.org/salsa-ci-team/pipeline approach (extending it) > the test suite when building, I would indeed need to pull in these > dependencies as build dependencies so that the software being tested > can run without import errors. In our repo I have two solutions (or hacks, or workarounds) for this situation. AUTOPKGTEST TIME (after build) == In the autopkgtest job (from salsa-ci/pipeline), autopkgtest runs a shell script (below integration-tests-generic). Some of them set things up: -- Tests: integration-tests-generic Depends: debusine-client, debusine-server, debusine-worker, postgresql, redis-server, sudo, nginx Restrictions: allow-stderr, needs-root -- It could pull things from the internet (Pypi, git...) if it also has, in Restrictions: "needs-internet" Alternatively (and perhaps better, to keep things separated?), you could use salsa CI variable: SALSA_CI_AUTOPKGTEST_ARGS with this format: SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=ci/pin-django-from-backports.sh Instead of pinning you could install packages. I don't know if it needs "needs-internet" in Restrictions. Note that the debusine-* packages are made available automatically thanks to Salsa CI pipeline and that tests-autopkgtest has: -- needs: - job: build artifacts: true -- So I don't need to do any repository. The build job just creates the *.deb packages (in a specific directory?), they are saved as artifacts and made available to "autopkgtest" job. UNIT-TESTS (before build) = We have a job before build doing (I'm simplifying): - unit-tests: stage: upstream-tests dependencies: [] script: - NON_INTERACTIVE=1 bin/quick-setup.sh install_packages - make coverage # to run the tests with coverage - In "bin/quick-setup.sh" it installs the required packages (some from Debian, some from Pypi if need to be) In here we execute tests before packaging them (because we are upstream as well). > I have to add that the other package I refer to has a test suite that takes a > long time to run, so that is another reason why I chose Salsa CI for that > package instead of letting autopkgtest do its work: Note that when I say "autopkgtest" above I always think of autopkgtest ran by Salsa CI. > One can imagine having a common storage area holding these newly Until now I see "the whole internet" as a "common storage area" (I know, not ideal! could be closer to the process, more robust, faster, etc.) > introduced packages that the CI scripts could access in preference to > the usual archives. In fact, this would be something that might also > affect existing packages. Consider the situation where fixes to a > dependency are required to fix functionality in a particular package. > One would have to wait for the fixed dependency to become integrated > into the unstable archive before the principal package's tests would > start to work again. Semi-offtopic: MMmmm... Debusine is the project that I mentioned before where I implemented different things in .gitlab-ci.yml. What you mentioned here is a thing that debusine, in the future (not impelmented yet) might help. But other people can talk better about this than me... and it's not available right now. > > I also created a repo and hosted it on a Salsa CI page for internal > > testing but this is a bit of a workaround. This is in a new job but just > > download the artifacts (via a Salsa CI dependency) and run > > dpkg-scanpackages and copy the files to the right place). > > This sounds like something related to what might be required. In
Re: Package testing with Salsa CI for new packages
On Friday, 18 August 2023 09:51:29 CEST Carles Pina i Estany wrote: > > I'm not a Debian developer but I have some experience on Salsa CI, so I > thought that I might be able to help... but then I was confused by a > specific part of the message: > > On 17 Aug 2023 at 17:10:08, Paul Boddie wrote: > > [...] > > > For another package I have been working on, the Salsa CI facility has > > proven to be usable, configured using files in debian/test, particularly > > as it allows test-related dependencies to be specified independently. > > However, this other package has no dependencies that are currently > > unpackaged in Debian. Meanwhile, the testing of this new Moin package > > depends on brand new packages somehow being made available. > > If this dependencies are available on the "build" step: could they be > made available on the autopkgtest? I didn't quite understand why this is > not possible. I've found the autopkgtest quite flexible (since the tests > are scripts that could prepare some environment) The package has dependencies on installation but these dependencies are not strictly necessary when building. However, if I wanted to run the test suite when building, I would indeed need to pull in these dependencies as build dependencies so that the software being tested can run without import errors. Since some of these dependencies are new packages, I had been specifying them using the --extra-package option of gbp buildpackage in order to satisfy such build dependencies, but these were really "test dependencies". I have since decided that I would rather distinguish between build and test dependencies, as previously noted. I have to add that the other package I refer to has a test suite that takes a long time to run, so that is another reason why I chose Salsa CI for that package instead of letting autopkgtest do its work: https://salsa.debian.org/pboddie/shedskin When I get the courage to do so, I will probably submit an intention-to- package for that package. [...] > For me, the packages on the build job are made available via a Salsa > artifact automatically (and easy to download as a .zip of the *.deb). I > think that this happens on all the "build" jobs on Salsa CI. They can be > downloaded as a .zip file > (e.g. https://salsa.debian.org/freexian-team/debusine/-/jobs/4564890, > right hand side "Job artifacts" and click "Download"). Would that be > enough? I think the problem is that when running the test suite, the Moin package would need to obtain the test dependencies specified in debian/test/control. These include various new packages that cannot be obtained from the usual Debian archives, and so some kind of equivalent to gbp's --extra-package would be needed in the Salsa CI environment. One can imagine having a common storage area holding these newly introduced packages that the CI scripts could access in preference to the usual archives. In fact, this would be something that might also affect existing packages. Consider the situation where fixes to a dependency are required to fix functionality in a particular package. One would have to wait for the fixed dependency to become integrated into the unstable archive before the principal package's tests would start to work again. > I also created a repo and hosted it on a Salsa CI page for internal > testing but this is a bit of a workaround. This is in a new job but just > download the artifacts (via a Salsa CI dependency) and run > dpkg-scanpackages and copy the files to the right place). This sounds like something related to what might be required. In effect, you seem to be doing what I am doing when I actually install my built packages in a chroot. I run apt-ftparchive (which runs dpkg-scanpackages) to generate Packages, Sources and Release files that tells apt about the new packages when added as another package source. In the Salsa CI environment, I would need to have the built packages (found in the artefacts for each package's build job) copied somewhere that can then be found by the Moin package's pipeline jobs and the scripts creating a special repository of new packages. Thanks for replying to my enquiry! Regards, Paul
Re: Package testing with Salsa CI for new packages
Hi, I'm not a Debian developer but I have some experience on Salsa CI, so I thought that I might be able to help... but then I was confused by a specific part of the message: On 17 Aug 2023 at 17:10:08, Paul Boddie wrote: [...] > For another package I have been working on, the Salsa CI facility has proven > to be usable, configured using files in debian/test, particularly as it > allows > test-related dependencies to be specified independently. However, this other > package has no dependencies that are currently unpackaged in Debian. > Meanwhile, the testing of this new Moin package depends on brand new packages > somehow being made available. If this dependencies are available on the "build" step: could they be made available on the autopkgtest? I didn't quite understand why this is not possible. I've found the autopkgtest quite flexible (since the tests are scripts that could prepare some environment) > So, I would appreciate guidance on how I might enable testing of this new > Moin > package, along with its new dependencies, in the Salsa environment or any > other appropriate environment that would satisfy Debian packaging needs and > policies. It would also be quite helpful if built packages might be published > somewhere for people to test, but this is a separate consideration even if > such packages would obviously need to be generated as part of the testing > regime. For me, the packages on the build job are made available via a Salsa artifact automatically (and easy to download as a .zip of the *.deb). I think that this happens on all the "build" jobs on Salsa CI. They can be downloaded as a .zip file (e.g. https://salsa.debian.org/freexian-team/debusine/-/jobs/4564890, right hand side "Job artifacts" and click "Download"). Would that be enough? I also created a repo and hosted it on a Salsa CI page for internal testing but this is a bit of a workaround. This is in a new job but just download the artifacts (via a Salsa CI dependency) and run dpkg-scanpackages and copy the files to the right place). Cheers, -- Carles Pina i Estany https://carles.pina.cat || Wiktionary translations: https://kamus.pina.cat
