Re: [Debian-salsa-ci] Publishing multiple packages with aptly in Salsa CI
On Friday, 15 September 2023 16:33:25 CEST Philip Hands wrote: > > For another angle, see: > > https://salsa.debian.org/philh/user-setup/-/pipelines/576662 > > In which I have a `harvest-repos` job that grabs artifacts from `build` > jobs in other pipelines, and an `aptly-plus` job that's got an added > `needs: harvest-repos` that can combine the artifacts from its build and > harvest-repos jobs and lump them all together. > > In the resulting aptly repo you can see that it includes both the > local package (user-setup) under the 'u' directory, and 'grub-installer' > under the 'g' directory: > > https://salsa.debian.org/philh/user-setup/-/jobs/4671054/artifacts/browse/ aptly/pool/main/ Yes, that was the desired effect: have packages corresponding to several source packages being made available in the pool and thus the created package repository. > That's done with: > > https://salsa.debian.org/installer-team/branch2repo/ > > Quite a lot of that is already part of the standard salsa-CI pipeline, > and my aim is that branch2repo will pretty-much disapear, with its > components being optional bits of the standard pipeline, and maybe a few > variable settings. Indeed. As noted before, my modifications were effectively (1) to keep the source repository around to get access to my scripts, (2) to be able to add package repositories to the apt configuration, (3) to download packages so that aptly can process them. (1) is just preferable to writing scripts in the horrible YAML syntax and then doing fragment inclusion, which I found can lead to opaque error messages. Being able to obtain a set of scripts would be quite helpful generally as the environment can vary substantially from one kind of job to another. (2) is something that could be part of the standard job definitions, particularly since other jobs like autopkgtest need to know about additional repositories in certain cases, brand new packages being one of them. (3) is just a small enhancement for this specific scenario. I thought that someone must have done this before, even if the standard pipeline didn't support it. I imagine that some coordination would be desirable to prevent fragmentation and people like me introducing our own ways of addressing this particular need. Paul
Re: [Debian-salsa-ci] Publishing multiple packages with aptly in Salsa CI
Hi, Someone wrote: >> However, one other thing I wanted to achieve was to take the complete set of >> new packages and to publish them in a single package repository. This would >> allow people to install and test the built packages in a more convenient >> fashion than asking them to hunt down each built package from job artefacts >> or >> to build the packages themselves. >> >> Obviously, the aptly job in the standard Debian CI pipeline publishes a >> single >> package (or maybe a collection of packages built from a single source >> package), but I wanted to aggregate all packages published by a collection >> of >> aptly repositories. Fortunately, it seems that this is possible by >> augmenting >> the existing aptly job definition as shown in the following file: >> >> https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa-ci.yml For another angle, see: https://salsa.debian.org/philh/user-setup/-/pipelines/576662 In which I have a `harvest-repos` job that grabs artifacts from `build` jobs in other pipelines, and an `aptly-plus` job that's got an added `needs: harvest-repos` that can combine the artifacts from its build and harvest-repos jobs and lump them all together. In the resulting aptly repo you can see that it includes both the local package (user-setup) under the 'u' directory, and 'grub-installer' under the 'g' directory: https://salsa.debian.org/philh/user-setup/-/jobs/4671054/artifacts/browse/aptly/pool/main/ That's done with: https://salsa.debian.org/installer-team/branch2repo/ Quite a lot of that is already part of the standard salsa-CI pipeline, and my aim is that branch2repo will pretty-much disapear, with its components being optional bits of the standard pipeline, and maybe a few variable settings. Cheers, Phil. -- Philip Hands -- https://hands.com/~phil signature.asc Description: PGP signature
Re: Publishing multiple packages with aptly in Salsa CI
Hi, > Hello, > > A few weeks ago, I asked about techniques for making new packages available > to > other new packages so that the autopkgtest job could be run successfully in a > pipeline in the Salsa CI environment. Eventually, this was made to work by > taking advantage of the aptly job defined in the standard Debian CI pipeline. > > To recap, the aptly job publishes the package built during the execution of a > pipeline by creating a dedicated apt-compatible package repository just for > that package. Such repositories can then be made available to various jobs in > the pipeline of another package, allowing the successful installation of that > package and its dependencies, including new packages, and the successful > execution of jobs like autopkgtest. > > However, one other thing I wanted to achieve was to take the complete set of > new packages and to publish them in a single package repository. This would > allow people to install and test the built packages in a more convenient > fashion than asking them to hunt down each built package from job artefacts > or > to build the packages themselves. > > Obviously, the aptly job in the standard Debian CI pipeline publishes a > single > package (or maybe a collection of packages built from a single source > package), but I wanted to aggregate all packages published by a collection of > aptly repositories. Fortunately, it seems that this is possible by augmenting > the existing aptly job definition as shown in the following file: > > https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa-ci.yml [snip] Please, considering discussing about this on the Salsa CI mailing list: debian-salsa...@alioth-lists.debian.net (taking the liberty to CC:) > Ignoring the "ls" command which was there to troubleshoot this rather opaque > environment, one script adds repository definitions to the apt configuration, > whereas the other performs the appropriate "apt source" and "apt download" > commands, making the source and binary package files available for aptly to > process. Consequently, aptly will include all the dependency packages in the > final package repository. > > (Since the scripts reside in my package's debian directory, I also had to > request the availability of the Git repository for the job. Generally, I have > found it challenging to have these job definitions make effective use of > scripts due to environmental inconsistencies.) With my Salsa CI maintainer's hat, I am happy to receive MRs ;-) > > I imagine that publishing packages like this is not particularly desirable, > at > least if done widely, but I hope it shows that it can be done relatively > easily, at least if all the right incantations have been discovered. It depends. Right now, I am looking forward to ease testing reverse dependencies, which is somehow related. As far as Salsa CI also makes it possible to control the number of jobs, and to don't abuse the Salsa infrastructure, any enhancement is welcome. cheers, -- Santiago signature.asc Description: PGP signature
Publishing multiple packages with aptly in Salsa CI
Hello, A few weeks ago, I asked about techniques for making new packages available to other new packages so that the autopkgtest job could be run successfully in a pipeline in the Salsa CI environment. Eventually, this was made to work by taking advantage of the aptly job defined in the standard Debian CI pipeline. To recap, the aptly job publishes the package built during the execution of a pipeline by creating a dedicated apt-compatible package repository just for that package. Such repositories can then be made available to various jobs in the pipeline of another package, allowing the successful installation of that package and its dependencies, including new packages, and the successful execution of jobs like autopkgtest. However, one other thing I wanted to achieve was to take the complete set of new packages and to publish them in a single package repository. This would allow people to install and test the built packages in a more convenient fashion than asking them to hunt down each built package from job artefacts or to build the packages themselves. Obviously, the aptly job in the standard Debian CI pipeline publishes a single package (or maybe a collection of packages built from a single source package), but I wanted to aggregate all packages published by a collection of aptly repositories. Fortunately, it seems that this is possible by augmenting the existing aptly job definition as shown in the following file: https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa-ci.yml Ignoring the "ls" command which was there to troubleshoot this rather opaque environment, one script adds repository definitions to the apt configuration, whereas the other performs the appropriate "apt source" and "apt download" commands, making the source and binary package files available for aptly to process. Consequently, aptly will include all the dependency packages in the final package repository. (Since the scripts reside in my package's debian directory, I also had to request the availability of the Git repository for the job. Generally, I have found it challenging to have these job definitions make effective use of scripts due to environmental inconsistencies.) I imagine that publishing packages like this is not particularly desirable, at least if done widely, but I hope it shows that it can be done relatively easily, at least if all the right incantations have been discovered. Paul
