Re: [RFC] Proposal for new source format

2019-11-08 Thread Ian Jackson
gregor herrmann writes ("Re: [RFC] Proposal for new source format"): > On Thu, 31 Oct 2019 11:59:07 +, Ian Jackson wrote: > > * tag2upload service, or some related service: > > - determines that the maintainer is using a dgit-compatible git > > w

Re: [RFC] Proposal for new source format

2019-10-31 Thread gregor herrmann
On Thu, 31 Oct 2019 11:59:07 +, Ian Jackson wrote: > * tag2upload service, or some related service: > - determines that the maintainer is using a dgit-compatible git > workflow, by looking at the tags, and looks at some in-dsc > metadata to find the maintainer's repo > -

Re: [RFC] Proposal for new source format

2019-10-31 Thread Sean Whitton
Hello, On Thu 31 Oct 2019 at 11:59AM +00, Ian Jackson wrote: > Well, that's fair enough as far as it goes. But I think we could do > better. > > It would be possible to imagine some service that works like this: > [...] This would be cool! -- Sean Whitton signature.asc Description: PGP

Re: [RFC] Proposal for new source format

2019-10-31 Thread Ian Jackson
Sean Whitton writes ("Re: [RFC] Proposal for new source format"): > Sorry, I didn't phrase my suggestion carefully. I was assuming that we > will continue to expect maintainers to accept patches in the BTS, but > that if they *prefer* something else, they could document that

Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)

2019-10-30 Thread Sean Whitton
Hello, On Tue 29 Oct 2019 at 08:32AM +01, Tobias Frost wrote: >> For example, you would not be able to do this: >>git clone salsa:something >>cd something >>make some straightforward change >>git tag# } [1] >>git push # } >> Instead you would have to download the .origs

Re: [RFC] Proposal for new source format

2019-10-30 Thread Sean Whitton
Hello Helmut, On Mon 28 Oct 2019 at 09:35PM +01, Helmut Grohne wrote: > On Sun, Oct 27, 2019 at 10:11:22AM -0700, Sean Whitton wrote: >> On Sat 26 Oct 2019 at 04:24PM -07, Russ Allbery wrote: >> > Hm, that's an interesting thought. I do generally include that sort of >> > information in the

Re: [RFC] Proposal for new source format

2019-10-30 Thread Ian Jackson
Russ Allbery writes ("Re: [RFC] Proposal for new source format"): > Ian Jackson writes: > > Of course this means that the resulting source packages are not the "3.0 > > (quilt)" patch queue source packages that many people (even some people > > who like gi

Re: [RFC] Proposal for new source format

2019-10-29 Thread Russ Allbery
Ian Jackson writes: > Of course this means that the resulting source packages are not the "3.0 > (quilt)" patch queue source packages that many people (even some people > who like git) say is important to them. > A key design goal for dgit and my tag2upload proposal, is that (when > used in the

Re: [RFC] Proposal for new source format

2019-10-29 Thread Ian Jackson
Russ Allbery writes ("Re: [RFC] Proposal for new source format"): > And therefore the goal of this proposal is to define a source package > format that allows this to be done more easily than our current source > package format allows? Of course this means that the result

Re: [RFC] Proposal for new source format

2019-10-29 Thread Russ Allbery
Bastian Blank writes: > On Tue, Oct 29, 2019 at 12:19:03PM -0700, Russ Allbery wrote: >> Could you help me understand what this would look like? Is it something >> like this workflow? >> >> 1. tag2upload determines the local Git tree that should be uploaded as a >>new source package. >>

Re: [RFC] Proposal for new source format

2019-10-29 Thread Bastian Blank
Hi Russ On Tue, Oct 29, 2019 at 12:19:03PM -0700, Russ Allbery wrote: > Could you help me understand what this would look like? Is it something > like this workflow? > > 1. tag2upload determines the local Git tree that should be uploaded as a >new source package. > > 2. tag2upload locally

Re: [RFC] Proposal for new source format

2019-10-29 Thread Sam Hartman
> "Bastian" == Bastian Blank writes: >> I don't think this proposal is sufficiently well developed where >> you're going to get much good feedback on debian-devel. Bastian> What would be the correct location for it? I'm fairly frustrated that you snipped the key part of my mail

Re: [RFC] Proposal for new source format

2019-10-29 Thread Russ Allbery
Bastian Blank writes: > We had that discussion already, it is about the possibility of > reproducing the content of the upload. The tag2upload proposal said > they can't do it and everyone need to trust this service to do the right > thing. I like to solve this problem and allow such a

Re: [RFC] Proposal for new source format

2019-10-29 Thread Bastian Blank
On Tue, Oct 22, 2019 at 07:33:47AM -0400, Sam Hartman wrote: > My initial reaction is that this is additional complexity in a direction > that we don't need. It is not a question of complexity. It is a question of trust and who we want and need to trust. If we abolish the principle that we want

Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)

2019-10-29 Thread Ian Jackson
Helmut Grohne writes ("Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)"): > I think I'd trust the tag2upload service given the documentation you > presented about it. I'm less faithful in all dgit installations being > san

Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)

2019-10-29 Thread Bastian Blank
Hi Didier On Mon, Oct 28, 2019 at 10:05:11AM +0100, Didier 'OdyX' Raboud wrote: > Of course, all of this can only work if we can have, or make the ".git to > .dsc" conversion reproducible; hence my query. Now, please read the first mail of this thread again. Yes, maybe parts of it are unclear,

Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)

2019-10-29 Thread Helmut Grohne
Hi Ian, On Tue, Oct 29, 2019 at 12:54:57PM +, Ian Jackson wrote: > I wonder if I have misunderstood you, because: > > The tag2upload proposal is based on dgit, which already provides this. > dgit indeed defines an isomorphism between source packages and git > trees, and dgit clone gives a

Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)

2019-10-29 Thread Ian Jackson
Helmut Grohne writes ("Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)"): > In other words, I want these formats (source package and tagged git > tree) to be isomorphic (minus history). This requirement is too strong > sin

Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)

2019-10-29 Thread Tobias Frost
Hi Ian, On Mon, Oct 28, 2019 at 05:53:00PM +, Ian Jackson wrote: (...) > For example, you would not be able to do this: >git clone salsa:something >cd something >make some straightforward change >git tag# } [1] >git push # } > Instead you would have to download

Re: [RFC] Proposal for new source format

2019-10-28 Thread Helmut Grohne
Hi Sean, On Sun, Oct 27, 2019 at 10:11:22AM -0700, Sean Whitton wrote: > On Sat 26 Oct 2019 at 04:24PM -07, Russ Allbery wrote: > > Hm, that's an interesting thought. I do generally include that sort of > > information in the docuemntation of all packages for which I'm upstream, > > but for

Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)

2019-10-28 Thread Helmut Grohne
Hi Ian, On Mon, Oct 28, 2019 at 05:53:00PM +, Ian Jackson wrote: > The sticking point, as I understand it, is that this still does not > allow dak to verify that the *contents* of the .dsc were as intended > by the uploading human. [0] > > In the tag2upload proposal, the conversion from git

Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)

2019-10-28 Thread Scott Kitterman
On October 28, 2019 5:53:00 PM UTC, Ian Jackson wrote: >Scott Kitterman writes ("Re: Building Debian source packages >reproducibly (was: Re: [RFC] Proposal for new source format)"): >> Effectively tag2upload would replace DAK as the entry point for >> packages into

Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)

2019-10-28 Thread Ian Jackson
Didier 'OdyX' Raboud writes ("Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)"): > Where I'm coming from is that we were discussing the tag2upload > problem at miniDebConf Vaumarcus. [...] I appreciate your efforts to tr

Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)

2019-10-28 Thread Ian Jackson
Scott Kitterman writes ("Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)"): > Effectively tag2upload would replace DAK as the entry point for > packages into the archive (the equivalent to the current source > package signature

Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)

2019-10-28 Thread Scott Kitterman
On Monday, October 28, 2019 9:45:36 AM EDT Theodore Y. Ts'o wrote: > On Mon, Oct 28, 2019 at 10:05:11AM +0100, Didier 'OdyX' Raboud wrote: > > Where I'm coming from is that we were discussing the tag2upload problem at > > miniDebConf Vaumarcus. The heart of the problem is that FTP-Master are > >

Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)

2019-10-28 Thread Theodore Y. Ts'o
On Mon, Oct 28, 2019 at 10:05:11AM +0100, Didier 'OdyX' Raboud wrote: > Where I'm coming from is that we were discussing the tag2upload problem at > miniDebConf Vaumarcus. The heart of the problem is that FTP-Master are > (currently) not going to accept .dscs built reproducibly by a (even

Re: Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)

2019-10-28 Thread Marek Mosiewicz
Hello, In fact what can be important is problem of downloading artifacts during build. At least in Java world given application can be small but be dependant on many libs which are downloaded during build. Program works, build is reproducible, but we can have no idea what it consist of. Best

Re: [RFC] Proposal for new source format

2019-10-28 Thread Andrey Rahmatullin
On Wed, Oct 23, 2019 at 09:49:11AM -0400, Theodore Y. Ts'o wrote: > > > That seems excessively pessimistic. What about Git makes you think > > > it's impossible to create a reproducible source package? > > > > Has it been done? Given this point has been raised several times > > before if it

Building Debian source packages reproducibly (was: Re: [RFC] Proposal for new source format)

2019-10-28 Thread Didier 'OdyX' Raboud
Le mercredi, 23 octobre 2019, 15.49:11 h CET Theodore Y. Ts'o a écrit : > On Wed, Oct 23, 2019 at 11:18:24AM +1000, Russell Stuart wrote: > > On Tue, 2019-10-22 at 16:52 -0700, Russ Allbery wrote: > > > That seems excessively pessimistic. What about Git makes you think > > > it's impossible to

Re: [RFC] Proposal for new source format

2019-10-27 Thread Russ Allbery
Russell Stuart writes: > I don't believe that. I guess we are talking past each other. Out of > curiosity do you do maintain the changsets manually in git, or use > something like gquilt? I've tried a whole bunch of different things over the years, ranging from manually-maintained feature

Re: [RFC] Proposal for new source format

2019-10-27 Thread Russell Stuart
On Sun, 2019-10-27 at 20:29 -0700, Russ Allbery wrote: > If you modify the upstream source, then by definition you do not have > reproducibility of the upstream source, and you're now talking about > something else (review of the changes, which I called audit in my > previous message). I think

Re: [RFC] Proposal for new source format

2019-10-27 Thread Russ Allbery
Russell Stuart writes: > Harking back to the time we removed the randomness generator from > openssl, it's very nice to have a single patch say "it was removed > because it wasn't exercised in the tests. upstream didn't respond to > requests for comment" rather than having it interspersed with

Re: [RFC] Proposal for new source format

2019-10-27 Thread Russ Allbery
Russell Stuart writes: > That is a great definition of reproducibility if all you are interested > in is the Debian version of the package. It is not so great if you want > is the upstream version of the package - ie, it is important to you that > it behaves identically or at least diverges in

Re: [RFC] Proposal for new source format

2019-10-27 Thread Russell Stuart
On Wed, 2019-10-23 at 09:49 -0400, Theodore Y. Ts'o wrote: > Generating a reproducible source package given a particuar git commit > is trivial. All you have to do is use "git archive". For example: It is indeed. Almost a tautology. But it's not what I'm interested in doing. The focus is on

Re: [RFC] Proposal for new source format

2019-10-27 Thread Russell Stuart
On Tue, 2019-10-22 at 20:21 -0700, Russ Allbery wrote: > I define reproducibility as generating the same Debian source package > from a signed Git tag of my packaging repository plus, for non-native > packages, whatever release artifacts upstream considers canonical > (which may be a signed

Re: [RFC] Proposal for new source format

2019-10-27 Thread Sean Whitton
Hello, On Sat 26 Oct 2019 at 04:24PM -07, Russ Allbery wrote: >> It probably would also be useful if the metadata had some standardized >> way to indicate the preferred way to propose changes to either upstream >> or the debian packaging maintainer --- whether it's e-mail to a >> particular

Re: [RFC] Proposal for new source format

2019-10-27 Thread Simon Richter
Hi, On 27.10.19 01:20, Theodore Y. Ts'o wrote: > I think we will need to support the source tar.gz for the forseeable > future. At very least, *deprecating* the tar.gz/tar.gz.asc format > should be independent of question we also support a format that > involves a URL to a git repoistory plus a

Re: [RFC] Proposal for new source format

2019-10-26 Thread Russ Allbery
"Theodore Y. Ts'o" writes: > I do think, though that we should allow the specification of *multiple* > git repositories, with some kind of type specifier so it can be clear > whether a particular repository is just a read-only clone versus a > read/write "master" repository, and whether a

Re: [RFC] Proposal for new source format

2019-10-26 Thread Theodore Y. Ts'o
On Wed, Oct 23, 2019 at 11:40:06AM -0700, Russ Allbery wrote: > Marvin Renich writes: > > > The source package has historically (prior to the widespread use of VCS) > > also provided the basis for future development. Since most development > > these days is done using VCS, it's natural to try

Re: [RFC] Proposal for new source format

2019-10-23 Thread Bastian Blank
Hi Ansgar Thanks for filling in the gaps I left in my explanation. On Wed, Oct 23, 2019 at 10:15:16AM +0200, Ansgar wrote: > kernel.org uses a similar scheme: there are signatures for the > uncompressed tarballs by the maintainer (linux-*.tar.sign). In addition > there is a sha256sums.asc which

Re: [RFC] Proposal for new source format

2019-10-23 Thread Russ Allbery
Marvin Renich writes: > The source package has historically (prior to the widespread use of VCS) > also provided the basis for future development. Since most development > these days is done using VCS, it's natural to try to adapt the source > package to contain the VCS. I believe this is a

Re: [RFC] Proposal for new source format

2019-10-23 Thread Marvin Renich
I think this discussion has conflated two separate needs that should be kept distinct. The current source package provides a record of how the binary packages were built from source. This includes signatures and verifiability of source, and, more recently, reproducibility. It provides the

Re: [RFC] Proposal for new source format

2019-10-23 Thread Sean Whitton
Hello, On Wed 23 Oct 2019 at 09:49AM -04, Theodore Y. Ts'o wrote: > Generating a reproducible source package given a particuar git commit > is trivial. All you have to do is use "git archive". For example: > > #!/bin/bash > # > # Generate the e2fsprogs release tar ball > # > > commit=HEAD > >

Re: [RFC] Proposal for new source format

2019-10-23 Thread Theodore Y. Ts'o
On Wed, Oct 23, 2019 at 11:18:24AM +1000, Russell Stuart wrote: > On Tue, 2019-10-22 at 16:52 -0700, Russ Allbery wrote: > > That seems excessively pessimistic. What about Git makes you think > > it's impossible to create a reproducible source package? > > Has it been done? Given this point has

Re: [RFC] Proposal for new source format

2019-10-23 Thread Ian Jackson
Russ Allbery writes ("Re: [RFC] Proposal for new source format"): > That said, Bastian's point about what we should do if we find that the Git > repository contains something that isn't distributable is valid and needs > to be dealt with regardless. I think one of our poi

Re: [RFC] Proposal for new source format

2019-10-23 Thread Ansgar
Simon McVittie writes: > On Tue, 22 Oct 2019 at 05:22:57 +0200, Bastian Blank wrote: >> - Files need to be compressed and are recorded as such, which is a hard >> problem and give rise to tools like pristine-tar and such. > > My understanding is that this is deliberate: it means the only layer >

Re: [RFC] Proposal for new source format

2019-10-23 Thread Russ Allbery
Russell Stuart writes: > On Tue, 2019-10-22 at 20:21 -0700, Russ Allbery wrote: >> This history has at least one commit per upload, although ideally has >> the package maintainer's full revision history and upstream's full >> revision history. > I understand you like the history. A lot of

Re: [RFC] Proposal for new source format

2019-10-22 Thread Russell Stuart
On Tue, 2019-10-22 at 20:21 -0700, Russ Allbery wrote: > This history has at least one commit per upload, although ideally > has the package maintainer's full revision history and upstream's > full revision history. I understand you like the history. A lot of people do. But not everyone

Re: [RFC] Proposal for new source format

2019-10-22 Thread Sean Whitton
Hello, On Tue 22 Oct 2019 at 08:21PM -07, Russ Allbery wrote: > In looking this over, none of this precludes the source format 4.0 that > Bastian proposed, provided that there was some way to export that source > format easily and simply from point 4. Maybe it doesn't matter what's > published

Re: [RFC] Proposal for new source format

2019-10-22 Thread Russ Allbery
Russell Stuart writes: > Has it been done? Given this point has been raised several times before > if it hasn't been done by now I think it's reasonable to assume it's > difficult, and thinking that it's so is not excessively pessimistic. Oh, it's news to me that anyone has raised this before.

Re: [RFC] Proposal for new source format

2019-10-22 Thread Russell Stuart
On Tue, 2019-10-22 at 16:52 -0700, Russ Allbery wrote: > That seems excessively pessimistic. What about Git makes you think > it's impossible to create a reproducible source package? Has it been done? Given this point has been raised several times before if it hasn't been done by now I think

Re: [RFC] Proposal for new source format

2019-10-22 Thread Russ Allbery
Bastian Blank writes: > On Mon, Oct 21, 2019 at 09:29:05PM -0700, Russ Allbery wrote: >> If we're going to go to the trouble of defining a new source format, >> I'd prefer we embrace a VCS-based one rather than once again rolling >> our own idiosyncratic representation of a tree of files > I'm

Re: [RFC] Proposal for new source format

2019-10-22 Thread Thomas Goirand
Hi Bastian, thanks for this proposal, On 10/22/19 5:22 AM, Bastian Blank wrote: > During the tag2upload discussion, I think it got clear that it does not > fit anywhere. And my standing is, that we can't implement such a > service properly without some core changes to how our sources look like.

Re: [RFC] Proposal for new source format

2019-10-22 Thread Bastian Blank
Hi Russ On Mon, Oct 21, 2019 at 09:29:05PM -0700, Russ Allbery wrote: > If we're going to go to the trouble of defining a new source format, I'd > prefer we embrace a VCS-based one rather than once again rolling our own > idiosyncratic representation of a tree of files I'm not completely sure

Re: [RFC] Proposal for new source format

2019-10-22 Thread Simon McVittie
On Tue, 22 Oct 2019 at 05:22:57 +0200, Bastian Blank wrote: > - Files need to be compressed and are recorded as such, which is a hard > problem and give rise to tools like pristine-tar and such. My understanding is that this is deliberate: it means the only layer with the hard requirement to be

Re: [RFC] Proposal for new source format

2019-10-22 Thread Sam Hartman
Hi. My initial reaction is that this is additional complexity in a direction that we don't need. Like Russ, I generally assume that VCS-like things are the future. I understand there is complexity there. But I don't understand why this proposed format would be a step forward in a world where

Re: [RFC] Proposal for new source format

2019-10-22 Thread Bernd Zeimetz
On 10/22/19 6:29 AM, Russ Allbery wrote: > Bastian Blank writes: > >> I would like to have some comments on a large revision on the source >> format. It also needs modifications to dak to handle some parts of it. > >> - Source format version would be "4.0". >> - Each source includes an

Re: [RFC] Proposal for new source format

2019-10-21 Thread Russ Allbery
Bastian Blank writes: > I would like to have some comments on a large revision on the source > format. It also needs modifications to dak to handle some parts of it. > - Source format version would be "4.0". > - Each source includes an arbitrary number of "tar" layers, which are > applied