Bug#943743: ITP: golang-github-varlink-go -- Golang implementation of the Varlink protocol
Package: wnpp Severity: wishlist Owner: Dmitry Smirnov X-Debbugs-CC: debian-devel@lists.debian.org, pkg-go-maintain...@lists.alioth.debian.org Package name: golang-github-varlink-go Version: 0.0+git20191018 License: Apache-2.0 URL: https://github.com/varlink/go Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-varlink-go Description: Golang implementation of the Varlink protocol Golang library implementing the Varlink protocol. http://varlink.org/ signature.asc Description: This is a digitally signed message part.
Bug#943742: ITP: python-sop -- Framework for implementing the Stateless OpenPGP CLI in Python
Package: wnpp Severity: wishlist Owner: Daniel Kahn Gillmor * Package name: python-sop Version : 0.1.1 Upstream Author : Daniel Kahn Gillmor * URL : https://gitlab.com/dkg/python-sop * License : MIT Programming Lang: Python Description : Framework for implementing the Stateless OpenPGP CLI in Python The Stateless OpenPGP Command-Line Interface (or `sop`) is a specification that encourages OpenPGP implementators to provide a common, relatively simple command-line API for purposes of object security. This Python module helps implementers build such a CLI from any implementation accessible to the Python interpreter. It does *not* provide such an implementation itself -- this is just the scaffolding for the command line, which should make it relatively easy to supply a handful of Python functions as methods to a class. - This is a useful component in building out a `sop` implementation. I am hoping to convince the upstream maintainer of PGPy to build one, and to convince the GnuPG team to maintain one as part of GPGME. Having this framework available should make it easier to get wider adoption.
Re: [RFC] Proposal for new source format
Hi Sean,
On Sun, Oct 27, 2019 at 10:11:22AM -0700, Sean Whitton wrote:
> On Sat 26 Oct 2019 at 04:24PM -07, Russ Allbery wrote:
> > Hm, that's an interesting thought. I do generally include that sort of
> > information in the docuemntation of all packages for which I'm upstream,
> > but for Debian I've assumed the preferred way to propose changes is the
> > BTS. Now that's potentially changing with Salsa. I don't really mind
> > monitoring multiple input formats, but some people will.
>
> I think that README.source is a fine place for this sort of information.
Hell, no!
Having to read some arbitrary README.source slows down patch submission
excessively. You may consider this cost low, but if you try to file
thousands of patches across the whole archive, this adds up. Documenting
the preferred way of change submission in a machine-readable format
absolutely is a requirement for performing archive-wide changes. Our
present implementation of this requirement is "maintainers must consume
bugs filed via the BTS". I think this is less than ideal, but works
reasonably well from a submitter-pov. Changing this to "look up in
README.source" would make me stop contributing to Debian.
I think a good way to look at this is from a user-interface pov. I'm
attaching a script "reportpatch" to this mail. It takes the path to a
.debdiff and submits the change to the package maintainer. The current
implementation parses the package and version from the patched
debian/changelog and fires up reportbug. Filing patches becomes
reasonably quick this way.
Now if a different submission is preferred (e.g. salsa pr), a new
reportpatch should detect that and create a suitable pr at a suitable
project or fall back to the bts without me having to even think about
all of this.
I caution that it is not as easy. There are two more bits that are
easily overlooked. As a patch submitter, I don't want to work from an
arbitrary VCS snapshot that may or may not work. I want a tree that
precisely reproduces the failure seen in QA. In other words, I don't
want to deal with maintainer git trees. This is sad, but required to be
effective.
After submitting the change, we all know that Debian maintainers quickly
apply it. In Debian, quickly can mean years. Consequently, I need some
identifier that allows me to mechanically check whether the requested
change was applied to unstable (or declined). Yes, this is very
different from a salsa issue being closed. The version tracking in the
BTS quite reliably answers this question.
Enough people have complained about the BTS now that I'm convinced that
we need to somehow change the process for change submission. I'm equally
convinced, that documenting it in README.source is not the solution.
Helmut
#!/usr/bin/python3
import os
import re
import sys
import unidiff
def die(message):
sys.stderr.write("%s\n" % message)
sys.exit(1)
c = [f for f in unidiff.PatchSet.from_filename(sys.argv[1])
if re.match("^[^/]*/debian/changelog$", f.target_file)]
if len(c) < 1:
die("debian/changelog not patched")
elif len(c) > 1:
die("multiple debian/changelog??")
pkg, version = None, None
for h in c[0]:
for l in h:
if l.is_context:
m = re.match(r"^([a-z0-9.+-]+) \(([^()_ ]+)\) \S+;", l.value)
if not m:
continue
pkg, version = m.groups()
os.execlp("reportbug", "reportbug", "--from-buildd=%s_%s" % (pkg, version),
"--tag=patch")
Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)
Hi Ian, On Mon, Oct 28, 2019 at 05:53:00PM +, Ian Jackson wrote: > The sticking point, as I understand it, is that this still does not > allow dak to verify that the *contents* of the .dsc were as intended > by the uploading human. [0] > > In the tag2upload proposal, the conversion from git tag to dsc is > `merely' done by an official Debian service on an official Debian > machine, and is `merely' fully reproducible and auditable. > > But this is not good enough for some ftpmasters, who want that > verification to be done *by dak*. Various people attempted in the > previous thread on this topic to find out *why* this is thought > important, without apparent success. I fear I'll have to side with "some ftpmasters" here. As a user, I also want this verification work in both ways. Going from tag to upload is insufficient in my view. What I want is "apt source" with history. This is not debcheckout. I want the exact tree (tag) that matches unstable including its git history in a way that exactly reproduces the build failure seen on the source package. In other words, I want these formats (source package and tagged git tree) to be isomorphic (minus history). This requirement is too strong since not every source package will have a corresponding tag, but when there is a tag, I want to safely go from source package to tag and back again and arrive where I started from. This property allows me to start from a git tree that is authenticated by dak rather than a random git tree on a random git server of questionable origin. This backwards-connection seems to be missing thus far, but I do find it important for the reasons above. Adding it would easily allow dak to validate the signature on the tag. Helmut
Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)
On October 28, 2019 5:53:00 PM UTC, Ian Jackson
wrote:
>Scott Kitterman writes ("Re: Building Debian source packages
>reproducibly (was: Re: [RFC] Proposal for new source format)"):
>> Effectively tag2upload would replace DAK as the entry point for
>> packages into the archive (the equivalent to the current source
>> package signature verification being the git tag signature
>> verification). I don't think the discussion got to a point where a
>> path forward that was considered reasonable by both the tag2upload
>> developers and the FTP Masters was reached.
>
>The current tag2upload proposal includes providing dak with the signed
>git tag object so that it can re-verify the identity of the human DD
>who authorised the upload.
>
>The sticking point, as I understand it, is that this still does not
>allow dak to verify that the *contents* of the .dsc were as intended
>by the uploading human. [0]
>
>In the tag2upload proposal, the conversion from git tag to dsc is
>`merely' done by an official Debian service on an official Debian
>machine, and is `merely' fully reproducible and auditable.
>
>But this is not good enough for some ftpmasters, who want that
>verification to be done *by dak*. Various people attempted in the
>previous thread on this topic to find out *why* this is thought
>important, without apparent success.
>
>It would be nice to be able to work around this objection somehow by
>writing more code. Unfortunately, any alternative - such as that
>described by Didier earlier in this thread - has undesirable
>properties. In particular, it does not seem that it would be possible
>to do anything along these lines without continuing to burden the
>developer's working system with a whole pile of messing about with
>tarballs and quilt and so on.
>
>For example, you would not be able to do this:
> git clone salsa:something
> cd something
> make some straightforward change
> git tag# } [1]
> git push # }
>Instead you would have to download the .origs and so on, and wait
>while your machine crunched about unpacking and repacking tarballs,
>applying patches, etc.
>
>With tag2upload all that work is done for you on the tag2upload
>service, which of course has a fast network connection - and you don't
>need to wait for it.
>
>Ian.
>
>[0] Currently, of course, this requirement is not achieved for
>existing git based uploads. In practice, DDs use ad-hoc
>git-buildpackage runes on their local machine to convert git data into
>.dscs. That conversion is not controlled, not reproducible, and not
>practically auditable. I guess maybe those blocking tag2upload think
>it is sufficient that we can blame the DD if it is messed up or
>suborned ?
>
>[1] In practice with tag2upload you would use `git-debpush' instead of
>`git tag' + `git push' but this is a thin wrapper around `git tag' and
>does not involve downloads or indeed any network activity, etc.
And the talking past each other surely continues because I don't think that in
any way answers the objections. Repeating the same thing you've said before
isn't going to close the communication gap. I don't think it's possible to do
so right now. Also, I'm a mere FTP Assistant, so I'm not one of the ones you
have to convince.
Scott K
Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)
Didier 'OdyX' Raboud writes ("Building Debian source packages reproducibly
(was: Re: [RFC] Proposal for new source format)"):
> Where I'm coming from is that we were discussing the tag2upload
> problem at miniDebConf Vaumarcus. [...]
I appreciate your efforts to try to unstick all this.
> The hard part is not the packing and unpacking of the special tag; that's
> mostly just strings massaging. But building the exact same source package in
> different environments is harder than I expected.
Yes. I have code to do it for tag2upload, though. It's not released
yet because I stopped putting effort into this whole area after
getting discouraged.
> Of course, all of this can only work if we can have, or make the ".git to
> .dsc" conversion reproducible; hence my query.
>
> All-in-all; would this be a welcome mechanism?
I think by requiring the user to always have the tarballs on hand and
wait for them to be manipulated and maybe transferred, you are losing
a fair amount of the benefit of tag2upload.
But if you did want to do something along these lines, maybe you
should do it by adding code to git-debpush and the tag2upload service
rather than by reinventing the rest of the machinery ?
Regards,
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)
Scott Kitterman writes ("Re: Building Debian source packages reproducibly (was:
Re: [RFC] Proposal for new source format)"):
> Effectively tag2upload would replace DAK as the entry point for
> packages into the archive (the equivalent to the current source
> package signature verification being the git tag signature
> verification). I don't think the discussion got to a point where a
> path forward that was considered reasonable by both the tag2upload
> developers and the FTP Masters was reached.
The current tag2upload proposal includes providing dak with the signed
git tag object so that it can re-verify the identity of the human DD
who authorised the upload.
The sticking point, as I understand it, is that this still does not
allow dak to verify that the *contents* of the .dsc were as intended
by the uploading human. [0]
In the tag2upload proposal, the conversion from git tag to dsc is
`merely' done by an official Debian service on an official Debian
machine, and is `merely' fully reproducible and auditable.
But this is not good enough for some ftpmasters, who want that
verification to be done *by dak*. Various people attempted in the
previous thread on this topic to find out *why* this is thought
important, without apparent success.
It would be nice to be able to work around this objection somehow by
writing more code. Unfortunately, any alternative - such as that
described by Didier earlier in this thread - has undesirable
properties. In particular, it does not seem that it would be possible
to do anything along these lines without continuing to burden the
developer's working system with a whole pile of messing about with
tarballs and quilt and so on.
For example, you would not be able to do this:
git clone salsa:something
cd something
make some straightforward change
git tag# } [1]
git push # }
Instead you would have to download the .origs and so on, and wait
while your machine crunched about unpacking and repacking tarballs,
applying patches, etc.
With tag2upload all that work is done for you on the tag2upload
service, which of course has a fast network connection - and you don't
need to wait for it.
Ian.
[0] Currently, of course, this requirement is not achieved for
existing git based uploads. In practice, DDs use ad-hoc
git-buildpackage runes on their local machine to convert git data into
.dscs. That conversion is not controlled, not reproducible, and not
practically auditable. I guess maybe those blocking tag2upload think
it is sufficient that we can blame the DD if it is messed up or
suborned ?
[1] In practice with tag2upload you would use `git-debpush' instead of
`git tag' + `git push' but this is a thin wrapper around `git tag' and
does not involve downloads or indeed any network activity, etc.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Re: Init systems and docker
On Fri, Oct 11 2019, Scott Kitterman wrote: > I have been told by docker users (I'm not one) that systemd as provided on > Debian can't be used in docker. I have no idea if that's true or not. I try > really hard to know as little about init systems as possible and trust our > maintainers who work on such things. That is a popular misconception. Actually it is easier to make systemd work well under Docker than sysvinit, and it doesn't require --privileged. I have a whole series of Debian Docker images that work that way; see https://github.com/jgoerzen/docker-debian-base systemd nowadays even has explicit code to help with this. I made sysvint work in the stretch and jessie days, but it was a real pain and not at all friendly. Have a look at my debian-base-minimal code for how to make it work. - John
Re: Debian and our frenemies of containers and userland repos
On Mon, Oct 21 2019, Enrico Weigelt wrote: > On 05.10.19 03:31, Paul Wise wrote: >> On Fri, Oct 4, 2019 at 10:49 PM Enrico Weigelt wrote: >>> On 24.07.19 08:17, Marc Haber wrote: >>> Do we have a build technology that uses containers instead of chroots yet? >>> >>> Something like docker-buildpackage ? >> >> AFAICT, docker-buildpackage doesn't exist > > I'm pretty sure it does exist, since I wrote it :p > > https://github.com/metux/docker-buildpackage I should also mention https://packages.debian.org/buster/debocker - though I haven't used it. John > > > --mtx
Re: Building Debian source packages reproducibly
On 2019-10-28 10:05 +0100, Didier 'OdyX' Raboud wrote: > Le mercredi, 23 octobre 2019, 15.49:11 h CET Theodore Y. Ts'o a écrit : >> On Wed, Oct 23, 2019 at 11:18:24AM +1000, Russell Stuart wrote: >> > On Tue, 2019-10-22 at 16:52 -0700, Russ Allbery wrote: >> > > That seems excessively pessimistic. What about Git makes you think >> > > it's impossible to create a reproducible source package? >> > >> > Has it been done? Given this point has been raised several times >> > before if it hasn't been done by now I think it's reasonable to assume >> > it's difficult, and thinking that it's so is not excessively >> > pessimistic. >> >> Generating a reproducible source package given a particuar git commit >> is trivial. All you have to do is use "git archive". For example: > > When talking about upstream projects, sure. > > But generating Debian source packages (.dsc and friends) from a > `debian/master` (+ `pristine-tar`) reproducibly is not really, right? > > As far as I understand, `gbp buildpackage -S` is the closest we have, but so > far, I fail to get it to give me the bit-by-bit identical unsigned .dsc that > I'd like to get. What am I missing? Assuming format 3.0 (quilt): timestamps and permissions of files under the debian/ directory. The permissions of files in the git repository are different from user to user (mostly depending on their umask), and are propagated to the debian.tar.xz. When building from a fresh clone, timestamps of files in the debian.tar.xz should be set to the date of the latest debian/changelog entry, as dpkg-source will clamp their mtimes to that value. But in an existing git repository there will likely be files older than that, and their random mtime also propagates to the debian.tar.xz. Cheers, Sven
Bug#943726: ITP: golang-github-ssgelm-cookiejarparser -- A Go library that parses a curl (netscape) cookiejar file into a Go http.CookieJar
Package: wnpp Severity: wishlist Owner: Stephen Gelman * Package name: golang-github-ssgelm-cookiejarparser Version : 1.0.0-1 Upstream Author : Stephen Gelman * URL : https://github.com/ssgelm/cookiejarparser * License : Expat Programming Lang: Go Description : A Go library that parses a curl (netscape) cookiejar file into a Go http.CookieJar cookiejarparser cookiejarparser is a Go library that parses a curl (netscape) cookiejar file into a Go http.CookieJar. This is a new dependency for git-lfs
Summary: BLAS/LAPACK Ecosys Massive Update
Hi fellow developers, A good news especially for Debian scientific computing users. I shall call it a massive update, even if the whole update was decomposed into many tiny steps where some of them had already been finished 1 year ago. BLAS/LAPACK are two typical and classical dense linear algebra libraries, directly or indirectly used by most scientific computing software that involve vector & matrix operations, or basic linear algebra problems such as solving linear systems, solving linear least-square problems, or matrix factorization. A glance at the popcon and their reverse depends would demonsrate their importance: popcon > 0.1 million, and has notable rdeps such as numpy, scipy, octave, arpack, (julia), etc. TIPS: the typical performance bottle neck in a deep neural network is matrix multiplication (already mentioned here[1]). And that can be done by BLAS. For long time Debian's BLAS/LAPACK ecosystem lack a feature -- they should be compiled with different configurations due to different user demands. For example, a pthread application doesn't want to be linked against the library compiled in openmp settings; a supercomputer user may want to deal with a super large numerical array that cannot be indexed by 32-bit integer. That's the two massive updates we are going to talk about: (1) different threading flavours for BLAS/LAPACK implementations (2) different indexing (32bit v.s. 64bit) ... In 2018 I introduced intel-mkl (the fastest CPU-based BLAS/LAPACK implementation on x86 architecture, yet non-free) to our archive and registered it as an alternative of BLAS/LAPACK, and as the first alternative of BLAS64/LAPACK64. FYI, intel-mkl's magical runtime dispatching library libmkl_rt.so supports both (32-bit, 64-bit) indexing, and supports (gnu openmp, intel/llvm openmp, tbb, serial/sequential) threadding. Then I introduced blis (second fastest CPU-based free impl). It's a new package so it could be quite convenient for me to experiment the imagined ecosystem update: FYI: blis: (32bit,64bit) x (openmp,pthread,serial) Note that blis only provides BLAS/CBLAS implementation, not including LAPACK. Next I took part in GSoC2019/Gentoo, and introduced a BLAS/LAPACK runtime switching mechanism for Gentoo, fixing a long standing BLAS/LAPACK obstacle for Gentoo community: https://wiki.gentoo.org/wiki/Blas-lapack-switch After finishing the GSoC, I started to patch src:lapack which is the pivot package for the whole BLAS/LAPACK ecosys: FYI: src:lapack, (32bit,64bit)x(serial) It is the standard Fortran implementation. No multi-threading support. And subsequently src:openblas (fastest, free impl) FYI: openblas (32bit,64bit)x(pthread,openmp,serial) Just cleared NEW queue (experimental) several hours ago. So far all of my planned updates are basically finished (as long as openblas is uploaded to sid). We still need some time to test the features and make everything stablize, but our BLAS/LAPACK ecosys has already entered a new era. Notes for Debian Users == I recommend the following BLAS/LAPACK combinations, sorted by computation speed: * BLAS=libmkl-rt, LAPACK=libmkl-rt (non-free) * BLAS=openblas, LAPACK=openblas * BLAS=blis, LAPACK=lapack * BLAS=atlas, LAPACK=? (not tested) * BLAS=blas, LAPACK=lapack (standard, slow) I wrote a tool named "rover", a TUI frontend of update-alternatives, which is already present in stable/unstable. Initially it was written in order to conveniently debugging alternatives, however it is also useful enough and users can switch alternatives with it conveniently. Notes for Debian Developers === If you maintain packages depending on libblas.so|libblas-dev, or liblapack.so|liblapack-dev: Q: should I link my package against the 64-bit variant instead? A: don't do that if you are not sure. However, please investigate this problem if your package is intended for supercomputing or non-trivial numerical computation. ... I'll write a documentation about this whole thing and put it somewhere, to avoid a lengthy mail ... please look forward to the future update.. Currently we have this brief wiki page: https://wiki.debian.org/DebianScience/LinearAlgebraLibraries Acknowledgement === Many thanks to Sébastien Villemot who had been maintaining Debian's BLAS/LAPACK ecosys for many years. He reviewed nearly every of my step mentioned in this main, and helped me find many bugs and problems. FYI: Once upon a time when I was not a debian dev but merely a user, I had been deeply impressed by Sébastien's openblas packaging because it contains some carefully designed rules targets for the user to recompile the package locally, and that helped me at that time. That experience definitely further motivated my wish to become a DD. Thank you. Many thanks to Debian Science team that also provided a number of helpful feedbacks and suggestions. Many thanks to Aron
Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)
On Monday, October 28, 2019 9:45:36 AM EDT Theodore Y. Ts'o wrote: > On Mon, Oct 28, 2019 at 10:05:11AM +0100, Didier 'OdyX' Raboud wrote: > > Where I'm coming from is that we were discussing the tag2upload problem at > > miniDebConf Vaumarcus. The heart of the problem is that FTP-Master are > > (currently) not going to accept .dscs built reproducibly by a (even > > trusted) service. tag2upload is built on the idea that a signed git tag > > is the only needed thing (`git tag -s`) to trigger an upload, and that is > > not going to fly currently. > > Ah, now I understand the problem you're trying to solve; thanks for > the context. > > What are FTP Master's objections? Given that they *do* accept a > source-only upload, which is just a signed dsc plus the source/debian > tarballs, I would presume all that would be necessary is > demonstate that we have tools which can reliably translate between a > git commit and the dsc plus source tarball, and (b) that the git tree > is stored in Debian project infrastructure so we can be assured that > it can be stored with the same level of assurance as where we store > the source tar files. > > Do they have other concerns? If so, what are they? I would be > surprised that it has anything at all to do with reliable builds, > given the acceptance of source-only uploads today. My recollection of the discussion is that they key (pun intended) factor is signed by who. Currently all uploads are signed by an individual authorized to upload the package to the archive. The tag2upload proposal is premised on such keys being replaced by a single service based signing key. Effectively tag2upload would replace DAK as the entry point for packages into the archive (the equivalent to the current source package signature verification being the git tag signature verification). I don't think the discussion got to a point where a path forward that was considered reasonable by both the tag2upload developers and the FTP Masters was reached. There was a fair amount of discussion on this point in the tag2upload threads. Scott K
Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)
On Mon, Oct 28, 2019 at 10:05:11AM +0100, Didier 'OdyX' Raboud wrote: > Where I'm coming from is that we were discussing the tag2upload problem at > miniDebConf Vaumarcus. The heart of the problem is that FTP-Master are > (currently) not going to accept .dscs built reproducibly by a (even trusted) > service. tag2upload is built on the idea that a signed git tag is the only > needed thing (`git tag -s`) to trigger an upload, and that is not going to > fly > currently. Ah, now I understand the problem you're trying to solve; thanks for the context. What are FTP Master's objections? Given that they *do* accept a source-only upload, which is just a signed dsc plus the source/debian tarballs, I would presume all that would be necessary is (a) demonstate that we have tools which can reliably translate between a git commit and the dsc plus source tarball, and (b) that the git tree is stored in Debian project infrastructure so we can be assured that it can be stored with the same level of assurance as where we store the source tar files. Do they have other concerns? If so, what are they? I would be surprised that it has anything at all to do with reliable builds, given the acceptance of source-only uploads today. > The hard part is not the packing and unpacking of the special tag; that's > mostly just strings massaging. But building the exact same source package in > different environments is harder than I expected. Is there more than just (a) making sure the package can be built reproducibly in the first place, and (b) the information in the buildinfo file? Of course, the big problem is that not all packages are currently set up to be reproducibly built; for example if you try to compile using Link Optimization (LTO), you're completely out of luck. (I've since dropped use of LTO to deal with this issue.) [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=932098 But if it *is* reproducibly buildable, are there case where setting up a build environment using the information in buildinfo not enough? Cheers. - Ted
Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)
Hello, In fact what can be important is problem of downloading artifacts during build. At least in Java world given application can be small but be dependant on many libs which are downloaded during build. Program works, build is reproducible, but we can have no idea what it consist of. Best regards, Marek Mosiewicz W dniu pon, 28.10.2019 o godzinie 10∶05 +0100, użytkownik Didier 'OdyX' Raboud napisał: > Le mercredi, 23 octobre 2019, 15.49:11 h CET Theodore Y. Ts'o a écrit > : > > On Wed, Oct 23, 2019 at 11:18:24AM +1000, Russell Stuart wrote: > > > On Tue, 2019-10-22 at 16:52 -0700, Russ Allbery wrote: > > > > That seems excessively pessimistic. What about Git makes you > > > > think > > > > it's impossible to create a reproducible source package? > > > > > > Has it been done? Given this point has been raised several times > > > before if it hasn't been done by now I think it's reasonable to > > > assume > > > it's difficult, and thinking that it's so is not excessively > > > pessimistic. > > > > Generating a reproducible source package given a particuar git > > commit > > is trivial. All you have to do is use "git archive". For example: > > When talking about upstream projects, sure. > > But generating Debian source packages (.dsc and friends) from a > `debian/master` (+ `pristine-tar`) reproducibly is not really, right? > > As far as I understand, `gbp buildpackage -S` is the closest we have, > but so > far, I fail to get it to give me the bit-by-bit identical unsigned > .dsc that > I'd like to get. What am I missing? > > (A little digresssion…) > > Where I'm coming from is that we were discussing the tag2upload > problem at > miniDebConf Vaumarcus. The heart of the problem is that FTP-Master > are > (currently) not going to accept .dscs built reproducibly by a (even > trusted) > service. tag2upload is built on the idea that a signed git tag is the > only > needed thing (`git tag -s`) to trigger an upload, and that is not > going to fly > currently. > > The solution that seemed obvious during the discussion [0] is to > instead rely > on a local tool to produce a git tag with significantly more metadata > (such as > .dsc signature, _source.changes signature); and reconstruct the a > signed set > of .dsc and _source.changes automatically (as last pipeline step in > Gitlab > CI), which are then acceptable by the archive. > > In other words, its "tag2upload", but with a reproducible way to: > - build a source package on developer machine; > - sign it locally; > - create and push a special git tag > Then, in a different environment (such as a GitLab CI pipeline step), > given a > special git tag and a repository; > - build the exact unsigned same source package > - unpack the special git tag; > - apply the signatures to get the exact same signed source packages; > - dput to the archive. > > The hard part is not the packing and unpacking of the special tag; > that's > mostly just strings massaging. But building the exact same source > package in > different environments is harder than I expected. > > Some caveats: > - Q: if you built and signed the source package locally, why not > uploading? > A: Because you might want to only upload _after_ automated tests, > and in an > unsupervised manner. > - Q: If one can fit pgp signatures in a git tag; why not inlining the > complete > .dsc and _source.changes? > A: Indeed. You still need the debian.tar though. > - Q: What about Dgit: in the .dsc, or buildinfo files? > A: Currently optional; could just be left out for a prototype. > > Of course, all of this can only work if we can have, or make the > ".git to > .dsc" conversion reproducible; hence my query. > > All-in-all; would this be a welcome mechanism? > > > OdyX > > [0] It probably was already considered.
Re: [RFC] Proposal for new source format
On Wed, Oct 23, 2019 at 09:49:11AM -0400, Theodore Y. Ts'o wrote: > > > That seems excessively pessimistic. What about Git makes you think > > > it's impossible to create a reproducible source package? > > > > Has it been done? Given this point has been raised several times > > before if it hasn't been done by now I think it's reasonable to assume > > it's difficult, and thinking that it's so is not excessively > > pessimistic. > > Generating a reproducible source package given a particuar git commit > is trivial. All you have to do is use "git archive". For example: If you want bit-by-bit reproducible .tar.gz this assumes that git archive is reproducible (and that gzip -9n is reproducible). -- WBR, wRAR signature.asc Description: PGP signature
Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)
Le mercredi, 23 octobre 2019, 15.49:11 h CET Theodore Y. Ts'o a écrit : > On Wed, Oct 23, 2019 at 11:18:24AM +1000, Russell Stuart wrote: > > On Tue, 2019-10-22 at 16:52 -0700, Russ Allbery wrote: > > > That seems excessively pessimistic. What about Git makes you think > > > it's impossible to create a reproducible source package? > > > > Has it been done? Given this point has been raised several times > > before if it hasn't been done by now I think it's reasonable to assume > > it's difficult, and thinking that it's so is not excessively > > pessimistic. > > Generating a reproducible source package given a particuar git commit > is trivial. All you have to do is use "git archive". For example: When talking about upstream projects, sure. But generating Debian source packages (.dsc and friends) from a `debian/master` (+ `pristine-tar`) reproducibly is not really, right? As far as I understand, `gbp buildpackage -S` is the closest we have, but so far, I fail to get it to give me the bit-by-bit identical unsigned .dsc that I'd like to get. What am I missing? (A little digresssion…) Where I'm coming from is that we were discussing the tag2upload problem at miniDebConf Vaumarcus. The heart of the problem is that FTP-Master are (currently) not going to accept .dscs built reproducibly by a (even trusted) service. tag2upload is built on the idea that a signed git tag is the only needed thing (`git tag -s`) to trigger an upload, and that is not going to fly currently. The solution that seemed obvious during the discussion [0] is to instead rely on a local tool to produce a git tag with significantly more metadata (such as .dsc signature, _source.changes signature); and reconstruct the a signed set of .dsc and _source.changes automatically (as last pipeline step in Gitlab CI), which are then acceptable by the archive. In other words, its "tag2upload", but with a reproducible way to: - build a source package on developer machine; - sign it locally; - create and push a special git tag Then, in a different environment (such as a GitLab CI pipeline step), given a special git tag and a repository; - build the exact unsigned same source package - unpack the special git tag; - apply the signatures to get the exact same signed source packages; - dput to the archive. The hard part is not the packing and unpacking of the special tag; that's mostly just strings massaging. But building the exact same source package in different environments is harder than I expected. Some caveats: - Q: if you built and signed the source package locally, why not uploading? A: Because you might want to only upload _after_ automated tests, and in an unsupervised manner. - Q: If one can fit pgp signatures in a git tag; why not inlining the complete .dsc and _source.changes? A: Indeed. You still need the debian.tar though. - Q: What about Dgit: in the .dsc, or buildinfo files? A: Currently optional; could just be left out for a prototype. Of course, all of this can only work if we can have, or make the ".git to .dsc" conversion reproducible; hence my query. All-in-all; would this be a welcome mechanism? OdyX [0] It probably was already considered. signature.asc Description: This is a digitally signed message part.
