Re: Gitlab support for Debian repositories (Was: Regarding the new "Debian User Repository")
On Mon, Aug 16, 2021 at 07:18:03PM +0200, Wouter Verhelst wrote: > Well, then we disagree (and that's fine). Personally, I'd rather have my > CI system try to find as many problems as possible, so I can fix them > *before* I upload, rather than after. I didn't try to build a CI system here, but rather a publishing system, which only publishes reproducible releases. For development, a CI system which finds problems, is nice. But I also want a system which can do releases and which only does them when they are reproducible. -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ It's climate crime, not climate change. signature.asc Description: PGP signature
Re: Gitlab support for Debian repositories (Was: Regarding the new "Debian User Repository")
On Mon, Aug 16, 2021 at 04:47:32PM +, Holger Levsen wrote:
> On Mon, Aug 16, 2021 at 03:59:50PM +0200, Wouter Verhelst wrote:
> > > because here, our focus would be to publish things :)
> > Sure. But also to find problems early rather than late, no?
>
> no.
Well, then we disagree (and that's fine). Personally, I'd rather have my
CI system try to find as many problems as possible, so I can fix them
*before* I upload, rather than after.
YMMV, of course.
--
w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}
Re: Gitlab support for Debian repositories (Was: Regarding the new "Debian User Repository")
On Mon, Aug 16, 2021 at 03:59:50PM +0200, Wouter Verhelst wrote: > > because here, our focus would be to publish things :) > Sure. But also to find problems early rather than late, no? no. -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ If it feels like we’re breaking climate records every year, it’s because we are. signature.asc Description: PGP signature
Re: Gitlab support for Debian repositories (Was: Regarding the new "Debian User Repository")
Hi Holger,
On Wed, Aug 11, 2021 at 05:12:54PM +, Holger Levsen wrote:
> Hi Wouter,
>
> sorry for the late reply but I think it's still relevant...
> (just thus rather leaving almost full quote as context.)
>
> On Thu, Jul 08, 2021 at 11:25:26AM +0200, Wouter Verhelst wrote:
> > On Mon, Jul 05, 2021 at 12:31:10PM +, Holger Levsen wrote:
> > > On Mon, Jul 05, 2021 at 02:09:36PM +0200, Mathieu Parent (Debian) wrote:
> > > > > Do you have plans to support publishing builds only if they've
> > > > > produced
> > > > > bit by bit identical results on several builders? IOW, do you plan to
> > > > > support reproducible builds? :)
> > > > There is no specific support for reproducible builds. Currently,
> [...]
> > > > But reproducibility can be tested in GItlab jobs, before the upload.
> > > that's nice, but rather theoretic (however common it is today) in
> > > practice :)
> > > It would be really interesting / a game changer, to have a publishing
> > > option
> > > which would only allow publishing of builds proven in practice to be
> > > identical.
> > It's actually fairly easy to do that:
> >
> > - Create two runners, with different tags (e.g., one tagged "build1",
> > and one tagged "build2"). One can be a docker runner, the other a
> > shell runner, just to keep things interesting.
> > - Create two jobs that build the same source in ways that might trigger
> > reproducability issues (I'm sure you're better at this than me). Make
> > sure that they don't store their artifacts in the same location (e.g.,
> > one job runs "dcmd mv ../*.changes products/build1/", and the other
> > one does "dcmd mv ../*.changes products/build2/").
> > - Have a third job that depends on both the above two jobs, and that
> > runs diffoscope over the artifacts of both jobs. If and only if the
> > diffoscope doesn't reveal any issues, run dput to upload the packages.
> >
> > I think the salsa-CI team can easily add support for this to their
> > generic pipeline...
>
> that would be really nice, thank you for explaining this idea so well!
>
> just one thing: here we do *not* want to trigger reproducibility issues,
> rather the opposite: if we manage to do two builds resulting in exactlty
> the same .deb(s), we are happy.
Yes, of course -- I didn't mean to say "you should make it fail", but
rather "I'm sure you know of ways in which it commonly fails that we
want to protect against".
> because here, our focus would be to publish things :)
Sure. But also to find problems early rather than late, no?
--
w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}
Re: Gitlab support for Debian repositories (Was: Regarding the new "Debian User Repository")
Hi Wouter, sorry for the late reply but I think it's still relevant... (just thus rather leaving almost full quote as context.) On Thu, Jul 08, 2021 at 11:25:26AM +0200, Wouter Verhelst wrote: > On Mon, Jul 05, 2021 at 12:31:10PM +, Holger Levsen wrote: > > On Mon, Jul 05, 2021 at 02:09:36PM +0200, Mathieu Parent (Debian) wrote: > > > > Do you have plans to support publishing builds only if they've produced > > > > bit by bit identical results on several builders? IOW, do you plan to > > > > support reproducible builds? :) > > > There is no specific support for reproducible builds. Currently, [...] > > > But reproducibility can be tested in GItlab jobs, before the upload. > > that's nice, but rather theoretic (however common it is today) in practice > > :) > > It would be really interesting / a game changer, to have a publishing option > > which would only allow publishing of builds proven in practice to be > > identical. > It's actually fairly easy to do that: > > - Create two runners, with different tags (e.g., one tagged "build1", > and one tagged "build2"). One can be a docker runner, the other a > shell runner, just to keep things interesting. > - Create two jobs that build the same source in ways that might trigger > reproducability issues (I'm sure you're better at this than me). Make > sure that they don't store their artifacts in the same location (e.g., > one job runs "dcmd mv ../*.changes products/build1/", and the other > one does "dcmd mv ../*.changes products/build2/"). > - Have a third job that depends on both the above two jobs, and that > runs diffoscope over the artifacts of both jobs. If and only if the > diffoscope doesn't reveal any issues, run dput to upload the packages. > > I think the salsa-CI team can easily add support for this to their > generic pipeline... that would be really nice, thank you for explaining this idea so well! just one thing: here we do *not* want to trigger reproducibility issues, rather the opposite: if we manage to do two builds resulting in exactlty the same .deb(s), we are happy. because here, our focus would be to publish things :) elsewhere we do have a well known setup to find problems... (=tests.r-b.o/debian) -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ When this virus is over, I still want some of y'all to stay away from me. signature.asc Description: PGP signature
Re: Gitlab support for Debian repositories (Was: Regarding the new "Debian User Repository")
On Mon, Jul 05, 2021 at 12:31:10PM +, Holger Levsen wrote:
> On Mon, Jul 05, 2021 at 02:09:36PM +0200, Mathieu Parent (Debian) wrote:
> > > Do you have plans to support publishing builds only if they've produced
> > > bit by bit identical results on several builders? IOW, do you plan to
> > > support reproducible builds? :)
> >
> > There is no specific support for reproducible builds. Currently,
> > buildinfo files can be uploaded and are kept, with the metadata stored
> > in the DB. but nothing is done yet with those.
>
> yeah :/
>
> > But reproducibility can be tested in GItlab jobs, before the upload.
>
> that's nice, but rather theoretic (however common it is today) in practice :)
> It would be really interesting / a game changer, to have a publishing option
> which would only allow publishing of builds proven in practice to be
> identical.
It's actually fairly easy to do that:
- Create two runners, with different tags (e.g., one tagged "build1",
and one tagged "build2"). One can be a docker runner, the other a
shell runner, just to keep things interesting.
- Create two jobs that build the same source in ways that might trigger
reproducability issues (I'm sure you're better at this than me). Make
sure that they don't store their artifacts in the same location (e.g.,
one job runs "dcmd mv ../*.changes products/build1/", and the other
one does "dcmd mv ../*.changes products/build2/").
- Have a third job that depends on both the above two jobs, and that
runs diffoscope over the artifacts of both jobs. If and only if the
diffoscope doesn't reveal any issues, run dput to upload the packages.
I think the salsa-CI team can easily add support for this to their
generic pipeline...
--
w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}
Re: Gitlab support for Debian repositories (Was: Regarding the new "Debian User Repository")
On Mon, Jul 05, 2021 at 02:09:36PM +0200, Mathieu Parent (Debian) wrote: > > Do you have plans to support publishing builds only if they've produced > > bit by bit identical results on several builders? IOW, do you plan to > > support reproducible builds? :) > > There is no specific support for reproducible builds. Currently, > buildinfo files can be uploaded and are kept, with the metadata stored > in the DB. but nothing is done yet with those. yeah :/ > But reproducibility can be tested in GItlab jobs, before the upload. that's nice, but rather theoretic (however common it is today) in practice :) It would be really interesting / a game changer, to have a publishing option which would only allow publishing of builds proven in practice to be identical. -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ No mas pobres en un pais rico! signature.asc Description: PGP signature
Re: Gitlab support for Debian repositories (Was: Regarding the new "Debian User Repository")
Le lun. 5 juil. 2021 à 11:46, Holger Levsen a écrit : > > Hi Mathieu, Hi Holger, > On Mon, Jul 05, 2021 at 10:37:31AM +0200, Mathieu Parent (Debian) wrote: > > [2]: https://docs.gitlab.com/ee/user/packages/debian_repository/ > > thanks, this looks nice and simple! Thanks. > Do you have plans to support publishing builds only if they've produced > bit by bit identical results on several builders? IOW, do you plan to > support reproducible builds? :) There is no specific support for reproducible builds. Currently, buildinfo files can be uploaded and are kept, with the metadata stored in the DB. but nothing is done yet with those. But reproducibility can be tested in GItlab jobs, before the upload. Cheers, -- Mathieu Parent
Re: Gitlab support for Debian repositories (Was: Regarding the new "Debian User Repository")
Hi Mathieu, On Mon, Jul 05, 2021 at 10:37:31AM +0200, Mathieu Parent (Debian) wrote: > [2]: https://docs.gitlab.com/ee/user/packages/debian_repository/ thanks, this looks nice and simple! Do you have plans to support publishing builds only if they've produced bit by bit identical results on several builders? IOW, do you plan to support reproducible builds? :) -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ There are no jobs on a dead planet. (Also many other things but people mostly seem to care about jobs.) signature.asc Description: PGP signature
Re: Gitlab support for Debian repositories (Was: Regarding the new "Debian User Repository")
Le sam. 3 juil. 2021 à 12:11, Simon McVittie a écrit : > > On Fri, 02 Jul 2021 at 20:04:45 +0200, Mathieu Parent wrote: > > On a related topic, I'm currently developing support for Debian > > repositories in Gitlab (and transitively Salsa). > > That's great news - being able to build packages in CI and make the results > easily installable seems like a big step forward, particularly for > fast-moving non-core packages. > > Given the other discussion in this thread, perhaps it should be labelled > "apt repositories" or ".deb repositories" or something else more > distro-neutral, to avoid implying Debian approval or official status, > while also making it obvious that if you want to build and publish > packages for Ubuntu or Linux Mint or some other Debian derivative, > this is also the right feature for those? I'm not sure. "Debian repository" is the official term for the format, as documented in the wiki [1]. And here, Debian is not in the product name. [1]; https://wiki.debian.org/DebianRepository/Format As the doc is now live [2],any ambiguous usage of the Debian term can be fixed, but I don't see any. [2]: https://docs.gitlab.com/ee/user/packages/debian_repository/ -- Mathieu Parent
Re: Gitlab support for Debian repositories (Was: Regarding the new "Debian User Repository")
On Fri, 02 Jul 2021 at 20:04:45 +0200, Mathieu Parent wrote: > On a related topic, I'm currently developing support for Debian > repositories in Gitlab (and transitively Salsa). That's great news - being able to build packages in CI and make the results easily installable seems like a big step forward, particularly for fast-moving non-core packages. Given the other discussion in this thread, perhaps it should be labelled "apt repositories" or ".deb repositories" or something else more distro-neutral, to avoid implying Debian approval or official status, while also making it obvious that if you want to build and publish packages for Ubuntu or Linux Mint or some other Debian derivative, this is also the right feature for those? smcv
Re: Gitlab support for Debian repositories (Was: Regarding the new "Debian User Repository")
On Fri, Jul 02, 2021 at 08:04:45PM +0200, Mathieu Parent wrote: > On a related topic, I'm currently developing support for Debian > repositories in Gitlab (and transitively Salsa). [...] wow, that's some very nice news! Thanks for sharing it here now. I'm looking forward to see it in production! :) -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ signature.asc Description: PGP signature
Gitlab support for Debian repositories (Was: Regarding the new "Debian User Repository")
Le ven. 2 juil. 2021 à 19:17, Stephan Lachnit a écrit : > > Today I discovered a relatively new project called "Debian User Repository" > [1]. > > It's similar to the AUR, and much more than just in principle. Hi, On a related topic, I'm currently developing support for Debian repositories in Gitlab (and transitively Salsa). Work on this started 9 month ago , and basic support will probably be shipped with Gitlab 14.1 behind a feature flag (i.e July 22). You can follow epic [1]. [1] https://gitlab.com/groups/gitlab-org/-/epics/6057#note_582697034 Cheers -- Mathieu Parent
