Bug#844431: policy: packages should be reproducible

2017-05-17 Thread Bill Allombert 
On Sun, May 14, 2017 at 11:15:26PM +, Holger Levsen wrote:
> On Mon, May 15, 2017 at 12:05:17AM +0200, Bill Allombert wrote:
> > On Sun, May 14, 2017 at 03:20:54PM +, Holger Levsen wrote:
> > > On Sun, May 14, 2017 at 05:05:36PM +0200, Bill Allombert wrote:
> > > a.) go to http://reproducible.debian.net/$srcpkg and see if its 
> > > reproducible today.
> > As I said, I would like to check that my package build is reproducible 
> > before
> > I upload it, not after, so I can be sure that any bug is fixed in the
> > upload.
>  
> b.), b.0), c.) and d.) were given as possible "tools" *to build twice with 
> (some) variation(s) and compare the results*.
> 
> "Reproducible Builds" (in the sense of bit by bit identicall builds) is
> really a rather new field in the era of software (well, not really, but 
> thats history and bit rotted until it was rediscovered in the early 2010s…)
> 
> What is trivial, if given, is to show that a package is *un*reproducible.
> 
> It's much harder to show that a package is reproducible.

We should avoid a terminological confusion...

Unreproducible means that "it will never be reproduced", which is quite
different from "it will always be reproduced".
Reproducible means that "it is possible to reproduce".

So in fact it is much easier to show that something is reproducible
than unreproducible.

There are situations where policy mandates that the build will be
different (for example setting DEB_BUILD_OPTIONS).

And actually, we do not need packages to build always identically.
Instead we need a reliable way to rebuild them identically, which is a
lower bar.

If (as it is planned) all packages are built by the autobuilders, then
we could provide a tool that rebuild a package (maybe by taking a
.buildinfo as input and downloading the same versions of the build
dependencies from snapshot.d.o) using the same setting as the autobuilders.

Then policy would cover the issues that could still lead to a different
build (for example using timestamp, hardcoding hardware characteristic
of the build machine , etc.).

Cheers,
-- 
Bill. 

Imagine a large red swirl here.

Bug#844431: policy: packages should be reproducible

2017-05-15 Thread Wouter Verhelst 
On Sun, May 14, 2017 at 11:15:26PM +, Holger Levsen wrote:
> On Mon, May 15, 2017 at 12:05:17AM +0200, Bill Allombert wrote:
> > On Sun, May 14, 2017 at 03:20:54PM +, Holger Levsen wrote:
> > > On Sun, May 14, 2017 at 05:05:36PM +0200, Bill Allombert wrote:
> > > a.) go to http://reproducible.debian.net/$srcpkg and see if its 
> > > reproducible today.
> > As I said, I would like to check that my package build is reproducible 
> > before
> > I upload it, not after, so I can be sure that any bug is fixed in the
> > upload.
>  
> b.), b.0), c.) and d.) were given as possible "tools" *to build twice with 
> (some) variation(s) and compare the results*.
> 
> "Reproducible Builds" (in the sense of bit by bit identicall builds) is
> really a rather new field in the era of software (well, not really, but 
> thats history and bit rotted until it was rediscovered in the early 2010s…)
> 
> What is trivial, if given, is to show that a package is *un*reproducible.
> 
> It's much harder to show that a package is reproducible.
> 
> And given that this is a new field I think it's ok, while somewhat 
> unsatisfying,
> that maybe some unreproducibility will only be detected by a more advanced
> tool, like reproducible.debian.net (which aint a,b,c nor d, but e.)
> after an upload has taken place.

I think it's probably not a good idea to (when we've moved to mandate
"packages must be reproducible") allow packages to become insta-buggy by
things that are out of their control and not clearly specified in
policy. That's not how we do things in Debian.

As such, I would favour the following approach:
- You guys (= the reproducible builds guys) come up with a list of
  things that commonly make a package nonreproducible today, and policy
  adds those as "should not"s. If I'm not mistaken, such a list already
  exists, you may simply need to generalize it a bit?
- Actually, I'm sure there may be things that packages failed to
  comply with in the past, but that are not a problem anymore today;
  we can make those "must not" rules already today.
- If you find new and interesting ways to make packages nonreproducible
  at some point in the future, those can be added (as "should" first,
  and as "must" later).

This would result in a section in policy of this form:

---
# Reproducible builds

Packages should generally be reproducible. That is, a package build
should result in a bit-by-bit identical package from one build to the
next.

Specifically, packages must not do any of the following things:
- non-reproducible thing A
- non-reproducible thing B
- ...

Moreover, while the following are not must rules yet, packages should
also not do any of the following things:
- still-in-the-wild non-reproducible thing A
- still-in-the-wild non-reproducible thing B
- ...
---

(wording may need some tweaking)

The above wording makes "bit-by-bit identical" a should (so packagers
are encouraged to reach that goal), but already allows you to file RC
bugs on some subset of "is not reproducible" package issues, and a
subset that will improve over time.

With that wording, I don't think we should ever make "bit-by-bit
identical" a must; I also don't think we would need to. As you say,
building packages nonreproducibly is difficult to define, and it
certainly is difficult to test for in a definite manner.

> > What parameters
> > are allowed to change need to be defined.
> 
> I sadly think this is impossible.

I agree that it will probably be a neverending effort, but I also think
it's the only way that it can reasonably be done.

-- 
< ron> I mean, the main *practical* problem with C++, is there's like a dozen
   people in the world who think they really understand all of its rules,
   and pretty much all of them are just lying to themselves too.
 -- #debian-devel, OFTC, 2016-02-12

Bug#844431: policy: packages should be reproducible

2017-05-14 Thread Holger Levsen 
On Mon, May 15, 2017 at 12:05:17AM +0200, Bill Allombert wrote:
> On Sun, May 14, 2017 at 03:20:54PM +, Holger Levsen wrote:
> > On Sun, May 14, 2017 at 05:05:36PM +0200, Bill Allombert wrote:
> > a.) go to http://reproducible.debian.net/$srcpkg and see if its 
> > reproducible today.
> As I said, I would like to check that my package build is reproducible before
> I upload it, not after, so I can be sure that any bug is fixed in the
> upload.
 
b.), b.0), c.) and d.) were given as possible "tools" *to build twice with 
(some) variation(s) and compare the results*.

"Reproducible Builds" (in the sense of bit by bit identicall builds) is
really a rather new field in the era of software (well, not really, but 
thats history and bit rotted until it was rediscovered in the early 2010s…)

What is trivial, if given, is to show that a package is *un*reproducible.

It's much harder to show that a package is reproducible.

And given that this is a new field I think it's ok, while somewhat unsatisfying,
that maybe some unreproducibility will only be detected by a more advanced
tool, like reproducible.debian.net (which aint a,b,c nor d, but e.)
after an upload has taken place.

This is one of the reasons we are aiming for "packages *should* be reproducible"
now, and not "*must* be".

> Some of my package were listed as reproducible for several months and
> then became unreproducible without any new upload. I do not mind that.

I guess this is because we introduced many more variations during 2014 and 2015.
During 2016 I don't recall us introducing many varitions, or rather many
causing many new unreproducibilty issues…

For 2017 there weren't any.

> However from a policy point of view, reproducible need to be defined
> precisely.

Yes!

> Generally speaking, reproducible means that the build will
> not change if some (but not all) parameters are changed.

Yes.

> What parameters
> are allowed to change need to be defined.

I sadly think this is impossible.

> One way is specify that would be to provide an authoritative tool to
> validate packages.

the tool to validate builds should be diff/sha256sum. a tool to simulate all 
possible
variations in the wild would probably need endless time to operate… 

> PS: I thanks you for your advices, I will reply to you privately if I
> need to.

While you surely can do so (and I will happily reply) I would even more happily
prefer if you could ask me on public list (and ping in private if you havent 
gotten a reply in whatever you think is appropriate)… a.) then more people can 
learn b.) you'll probably get faster *and better replies* (esp. on language
specific details) and c.) this helps me getting my inbox under control :-)


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#844431: policy: packages should be reproducible

2017-05-14 Thread Bill Allombert 
On Sun, May 14, 2017 at 03:20:54PM +, Holger Levsen wrote:
> On Sun, May 14, 2017 at 05:05:36PM +0200, Bill Allombert wrote:
> a.) go to http://reproducible.debian.net/$srcpkg and see if its reproducible 
> today.

As I said, I would like to check that my package build is reproducible before
I upload it, not after, so I can be sure that any bug is fixed in the
upload.

Some of my package were listed as reproducible for several months and
then became unreproducible without any new upload. I do not mind that.
However from a policy point of view, reproducible need to be defined
precisely. Generally speaking, reproducible means that the build will
not change if some (but not all) parameters are changed. What parameters
are allowed to change need to be defined.

One way is specify that would be to provide an authoritative tool to
validate packages.

Cheers,
PS: I thanks you for your advices, I will reply to you privately if I
need to.
-- 
Bill. 

Imagine a large red swirl here.

Bug#844431: policy: packages should be reproducible

2017-05-14 Thread Holger Levsen 
On Sun, May 14, 2017 at 09:58:12PM +0200, Guillem Jover wrote:
> On Sun, 2017-05-14 at 15:20:54 +, Holger Levsen wrote:
> > Bill, did you do this for your packages?

on re-reading what I wrote here, it occurred to me that this could be
read *hostile* despite me having *zero* intentions to be hostile… I
just wanted to be friendly and give helpful URLs to you, Bill… I'm
sorry if this came across differently!

> b.0.) use debrepro (from devscripts)
 
Thanks for this additional hint, Guillem!


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#844431: policy: packages should be reproducible

2017-05-14 Thread Guillem Jover 
On Sun, 2017-05-14 at 15:20:54 +, Holger Levsen wrote:
> On Sun, May 14, 2017 at 05:05:36PM +0200, Bill Allombert wrote:
> > OK, but how can I check that my package build is reproducible before 
> > uploading
> > it ?
> 
> in general you cannot find out with 100% certainity whether a given source 
> package
> will be reproducible. You can only find out with certainity if a package is 
> *not*
> reproducible…
> 
> that said
> 
> a.) go to http://reproducible.debian.net/$srcpkg and see if its reproducible 
> today.
> 
>   Bill, did you do this for your packages?
>   And then there is also 
> https://tests.reproducible-builds.org/debian/unstable/index_dd-list.html#ballo...@debian.org
>   which shows that half of your 26 packages in sid (main) are 
> unreproducible
>   with build path variation, though most of those unreproducible 
> ones
>   are reproducible without build path variation…
>   -> 
> https://tests.reproducible-builds.org/debian/testing/index_dd-list.html#ballo...@debian.org
>   only shows 4 unreproducible packages…

b.0.) use debrepro (from devscripts)

> b.) build it twice and compare using diffoscope
> c.) use reprotest

Thanks,
Guillem

Bug#844431: policy: packages should be reproducible

2017-05-14 Thread Holger Levsen 
On Sun, May 14, 2017 at 05:05:36PM +0200, Bill Allombert wrote:
> OK, but how can I check that my package build is reproducible before uploading
> it ?

in general you cannot find out with 100% certainity whether a given source 
package
will be reproducible. You can only find out with certainity if a package is 
*not*
reproducible…

that said

a.) go to http://reproducible.debian.net/$srcpkg and see if its reproducible 
today.

Bill, did you do this for your packages?
And then there is also 
https://tests.reproducible-builds.org/debian/unstable/index_dd-list.html#ballo...@debian.org
which shows that half of your 26 packages in sid (main) are 
unreproducible
with build path variation, though most of those unreproducible 
ones
are reproducible without build path variation…
-> 
https://tests.reproducible-builds.org/debian/testing/index_dd-list.html#ballo...@debian.org
only shows 4 unreproducible packages…

b.) build it twice and compare using diffoscope
c.) use reprotest


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#844431: policy: packages should be reproducible

2017-05-14 Thread Bill Allombert 
On Sun, May 14, 2017 at 02:58:27PM +, Holger Levsen wrote:
> On Sun, May 14, 2017 at 04:51:47PM +0200, Bill Allombert wrote:
> > > the official tool to build packages reproducible in sid is called
> > > "dpkg-buildpackage" (since dpkg 1.18.16 in sid since 2016-12-17).
> > So if your package builds with "dpkg-buildpackage" then the build is
> > reproducible and any bug report to the contrary is in error ? 
> 
> almost. 93% of the packages in stretch today can be re-build bit by bit
> identically. 

OK, but how can I check that my package build is reproducible before uploading
it ?

Cheers,
-- 
Bill. 

Imagine a large red swirl here.

Bug#844431: policy: packages should be reproducible

2017-05-14 Thread Holger Levsen 
On Sun, May 07, 2017 at 06:15:38PM +0100, Chris Lamb wrote:
> > unsurprisingly I'm also in favor of making this policy change, now.
> Actually, yes, why were we waiting for stretch to be released? :)

good question. I guess because of a mental barrier against doing changes
targeted post-stretch now :)
 
> > Last and least for now: the wording of
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=844431;filename=debian-policy.diff.txt;msg=17
> > IMO is almost good as it is, though I'll try to amend it to include the
> > definition of reproducible builds from reproducible-builds.org. 
> That seems the next concrete step.

indeed! Will see to work on this the next days…


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#844431: policy: packages should be reproducible

2017-05-14 Thread Holger Levsen 
On Sun, May 14, 2017 at 04:51:47PM +0200, Bill Allombert wrote:
> > the official tool to build packages reproducible in sid is called
> > "dpkg-buildpackage" (since dpkg 1.18.16 in sid since 2016-12-17).
> So if your package builds with "dpkg-buildpackage" then the build is
> reproducible and any bug report to the contrary is in error ? 

almost. 93% of the packages in stretch today can be re-build bit by bit
identically. 

that's why we're now aiming at "packages should be reproducible" and not
for "must be reproducible"… (but plan to later aim for "must be").


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#844431: policy: packages should be reproducible

2017-05-14 Thread Bill Allombert 
On Sun, May 14, 2017 at 02:36:46PM +, Holger Levsen wrote:
> On Thu, May 11, 2017 at 02:42:43PM +0200, Bill Allombert wrote:
> > I really think there should be an official tool to do build packages
> > reproducibly with an interface like cowbuilder. 
>  
> the official tool to build packages reproducible in sid is called
> "dpkg-buildpackage" (since dpkg 1.18.16 in sid since 2016-12-17).

So if your package builds with "dpkg-buildpackage" then the build is
reproducible and any bug report to the contrary is in error ? 

Cheers,
-- 
Bill. 

Imagine a large red swirl here.

Bug#844431: policy: packages should be reproducible

2017-05-14 Thread Holger Levsen 
On Thu, May 11, 2017 at 02:42:43PM +0200, Bill Allombert wrote:
> I really think there should be an official tool to do build packages
> reproducibly with an interface like cowbuilder. 
 
the official tool to build packages reproducible in sid is called
"dpkg-buildpackage" (since dpkg 1.18.16 in sid since 2016-12-17).


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#844431: policy: packages should be reproducible

2017-05-11 Thread Bill Allombert 
On Sun, May 07, 2017 at 03:35:00PM +, Holger Levsen wrote:
> hi,
> 
> unsurprisingly I'm also in favor of making this policy change, now.
> 
> I also believe there is quite a consensus (definitly a rough one…) in Debian
> for making this change, judging by the feedback we got at 3 DebConfs since 
> 2013,
> several mini Debconfs and other events, plus the general feedback in the form
> of code merges and uploads.
> 
> At the Reproducible Builds Hackathon in Hamburg we were reminded of the former
> DPL asking DDs to be "more bold" doing sensible changes forward, and as such
> we plan that starting with the development phase of "buster" we'll consider
> bugs about reproducible builds issues to be of severity "normal", not 
> "wishlist".

I really think there should be an official tool to do build packages
reproducibly with an interface like cowbuilder. 

Currently, there are too much uncertainty about the process for bug
reports to be of severity normal.

Cheers,
-- 
Bill. 

Imagine a large red swirl here.

Bug#844431: policy: packages should be reproducible

2017-05-07 Thread Chris Lamb 
Hi Holger,

> unsurprisingly I'm also in favor of making this policy change, now.

Actually, yes, why were we waiting for stretch to be released? :)

> Last and least for now: the wording of
> https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=844431;filename=debian-policy.diff.txt;msg=17
> IMO is almost good as it is, though I'll try to amend it to include the
> definition of reproducible builds from reproducible-builds.org. 

That seems the next concrete step.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#844431: policy: packages should be reproducible

2017-05-07 Thread Holger Levsen 
hi,

unsurprisingly I'm also in favor of making this policy change, now.

I also believe there is quite a consensus (definitly a rough one…) in Debian
for making this change, judging by the feedback we got at 3 DebConfs since 2013,
several mini Debconfs and other events, plus the general feedback in the form
of code merges and uploads.

At the Reproducible Builds Hackathon in Hamburg we were reminded of the former
DPL asking DDs to be "more bold" doing sensible changes forward, and as such
we plan that starting with the development phase of "buster" we'll consider
bugs about reproducible builds issues to be of severity "normal", not 
"wishlist".

This shall be announced on d-d-a soon & given there is no disagrement on this
procedure on this bug.

Last and least for now: the wording of
https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=844431;filename=debian-policy.diff.txt;msg=17
IMO is almost good as it is, though I'll try to amend it to include the
definition of reproducible builds from reproducible-builds.org. 


-- 
cheers,
Holger


signature.asc
Description: Digital signature