package removed (Re: package uploaded to our repo)

2017-02-19 Thread Holger Levsen
On Tue, Feb 14, 2017 at 06:21:11PM +, Holger Levsen wrote:
> I really want to see a bug number here.
 
I would have loved to send this mail to such a bug…

> > > also, you uploaded python2.7_2.7.13-3~reproducible1 for amd64, do you 
> > > plan to
> > > do upload the other architectures as well?
> > Let's just start with amd64 for now and see what happens. 
 
so, this happened:

https://tests.reproducible-builds.org/debian/unstable/amd64/stats_pkg_state.png

because, from #debian-reproducible a few minutes ago:

 | bah
 | i've found the cause…
 | Err http://reproducible.alioth.debian.org/debian ./ 
python2.7-minimal 2.7.13-3~reproducible1
 |   Hash Sum mismatch
 | eg in 
https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/python-tooz.html
 uh…
 ah, so maybe the depwait are not so related to the other thing.
 | python-tooz is depwait because of exactly this
 | oh
 | come on
 | those files are also mode 644
   * | h01ger could move them away \o/
 could you please write a mail too? (following up on the previous 
thread?)
   * | h01ger is still cleaning up the mess atm
 | triggering maint+setup jobs

IOW: I've moved these python2.7 packages to 
/home/groups/reproducible/python2.7_7_2.7.13-3
on alioth…


-- 
cheers,
Holger


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: package uploaded to our repo

2017-02-14 Thread Holger Levsen
On Tue, Feb 14, 2017 at 04:52:22PM +1300, Chris Lamb wrote:
> No. This upload to our repo adds new codepaths to Python 2.7 which are then
> enabled on the "b" build on jenkins.debian.net; see 
> <1486933699.96290.878650304.07373...@webmail.messagingengine.com> for the
> background.
 
ic, thanks.

> Let me know if there is anything specific that you are unsure of in the
> above referenced message as I'm not sure what remains unclear.

I really want to see a bug number here.

> > also, you uploaded python2.7_2.7.13-3~reproducible1 for amd64, do you plan 
> > to
> > do upload the other architectures as well?
> Let's just start with amd64 for now and see what happens. 

No. For once, this needs a bug somewhere for tracking. Secondly, we should
either do this on all architectures or on none at all.

> Perhaps it will
> not be of any value, so preparing other uploads or making long-term plans
> for the patch would be premature.

IMO it's also premature to deploy this in our setup, if it's so unclear
whether it's useful. We have worked long and hard to be able to say "we are
testing 100% unmodified pure Debian", we should spoil this for something
premature.


-- 
cheers,
Holger


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: package uploaded to our repo

2017-02-13 Thread Chris Lamb
Holger Levsen wrote:

> does jenkins.d.n.git change make this upload mood?

No. This upload to our repo adds new codepaths to Python 2.7 which are then
enabled on the "b" build on jenkins.debian.net; see 
<1486933699.96290.878650304.07373...@webmail.messagingengine.com> for the
background.

Let me know if there is anything specific that you are unsure of in the
above referenced message as I'm not sure what remains unclear.

> also, you uploaded python2.7_2.7.13-3~reproducible1 for amd64, do you plan to
> do upload the other architectures as well?

Let's just start with amd64 for now and see what happens. Perhaps it will
not be of any value, so preparing other uploads or making long-term plans
for the patch would be premature.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds


Re: package uploaded to our repo

2017-02-13 Thread Chris Lamb
Holger Levsen wrote:

> does jenkins.d.n.git change make this upload mood?

No. This upload to our repo adds new codepaths to Python 2.7 which are then
enabled on the "b" build on jenkins.debian.net; see 
<1486933699.96290.878650304.07373...@webmail.messagingengine.com> for the
background.

Let me know if there is anything specific that you are unsure of in the
above referenced message as I'm not sure what remains unclear.

> also, you uploaded python2.7_2.7.13-3~reproducible1 for amd64, do you plan to
> do upload the other architectures as well?

Let's just start with amd64 for now and see what happens. Perhaps it will
not be of any value, so preparing other uploads or making long-term plans
for the patch would be premature.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds


Re: package uploaded to our repo

2017-02-13 Thread Holger Levsen
On Tue, Feb 14, 2017 at 09:29:37AM +1300, Chris Lamb wrote:
> > > python2.7_2.7.13-3~reproducible1.dsc has just been uploaded to 
> > > https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain
> > can you please explain why you did this?
> See my other mail re. PYTHONREVERSEDICTKEYORDER; the commit message
> I expanded at your request would appear to cover it. If not, please
> reply there.

if (?) you are refering to the git commit msg for this for jenkins.d.n.git
then I'm afraid this wasnt clear, please explain. 

(and if its something else please explain as well. also please use message-ids
when refering to oher mails, that make things a lot easier in async
communication.)

also, you uploaded python2.7_2.7.13-3~reproducible1 for amd64, do you plan to
do upload the other architectures as well? and how long do you propose we
keep this delta?

(or does jenkins.d.n.git change make this upload mood? if so, I would really
have appreciated to have this explained in the first place! :)
 

-- 
cheers,
Holger


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: package uploaded to our repo

2017-02-13 Thread Chris Lamb
Ximin Luo wrote:

> > can you please explain why you did this?
> 
> Yes, it'd be good to cite some real-world cases where you've
> seen this type of non-determinism happen

It's *extremely* common; I've sent 100s of patches solving non-determinstic
Python dict ordering patches to the BTS. This way just ensures the
non-determinism is "deterministic" in our test framework.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds


Re: package uploaded to our repo

2017-02-13 Thread Chris Lamb
Holger Levsen wrote:

> > python2.7_2.7.13-3~reproducible1.dsc has just been uploaded to 
> > https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain
> 
> can you please explain why you did this?

See my other mail re. PYTHONREVERSEDICTKEYORDER; the commit message
I expanded at your request would appear to cover it. If not, please
reply there.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds


Re: package uploaded to our repo

2017-02-13 Thread Ximin Luo
Holger Levsen:
> Hi Chris,
> 
> On Sun, Feb 12, 2017 at 09:09:54PM +, Chris Lamb wrote:
>> python2.7_2.7.13-3~reproducible1.dsc has just been uploaded to 
>> https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain
> 
> can you please explain why you did this?
> 

Yes, it'd be good to cite some real-world cases where you've seen this type of 
non-determinism happen. My intuition is that this is not worth maintaining in 
the long run - are you going to backport these patches to all future versions 
of python?

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds


Re: package uploaded to our repo

2017-02-13 Thread Holger Levsen
Hi Chris,

On Sun, Feb 12, 2017 at 09:09:54PM +, Chris Lamb wrote:
> python2.7_2.7.13-3~reproducible1.dsc has just been uploaded to 
> https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

can you please explain why you did this?


-- 
cheers,
Holger


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: package uploaded to our repo

2016-09-21 Thread Holger Levsen
Hi,

Mattia, thanks a lot for this great description what you did why!
Really awesome.

On Wed, Sep 21, 2016 at 01:35:27PM +, Mattia Rizzolo wrote:
> well, why, considering a single-archive world, is Source+Version fields
> in .buildinfo not enough to link the binaries to the source?

well, if this reproducible builds effort is also ment to improve the
security of Debian, it's very proper not only to record what the label
says it should contain (src pkg + version) but also something so it's
later possible to check whether "your src pkg + version" is the same
"I" later build… ;) (IOW: to not only record the label but also a hash
of the contents.)


-- 
cheers,
Holger


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: package uploaded to our repo

2016-09-21 Thread Chris Lamb
Dear Mattia,

> They told me it was not totally clear what happens here, why I did this
> upload, what triggered the chanegs I did, and why last night.

[..]

Wow, so much detail! Was not expecting that, thank you. Will try and
introduce this into this week's blog post :)


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds


Re: package uploaded to our repo

2016-09-21 Thread Mattia Rizzolo
On Wed, Sep 21, 2016 at 01:24:46PM +, Jérémy Bobbio wrote:
> Mattia Rizzolo:
> > Next steps:
> > it would be great if somebody could figure what's the real gain of
> > having .dsc in Checksums-Sha256.
> 
> This is how .buildinfo files have linked a source package in an
> unambiguous manner to the resulting binaries. I see how the current
> specification is giving you trouble, but it's kinda their “raison d'être”.

well, why, considering a single-archive world, is Source+Version fields
in .buildinfo not enough to link the binaries to the source?

I concur the assumptions of it are not that easy to live with, but it's
not so needed for e.g. the limited use case of recording the env a build
is made within Debian.

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: package uploaded to our repo

2016-09-21 Thread Jérémy Bobbio
Mattia Rizzolo:
> Next steps:
> it would be great if somebody could figure what's the real gain of
> having .dsc in Checksums-Sha256.

This is how .buildinfo files have linked a source package in an
unambiguous manner to the resulting binaries. I see how the current
specification is giving you trouble, but it's kinda their “raison d'être”.

-- 
Lunar.''`.
lunar at debian.org : :Ⓐ :  # apt-get install anarchism
`. `'`
  `-



signature.asc
Description: OpenPGP digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: package uploaded to our repo

2016-09-21 Thread Mattia Rizzolo
They told me it was not totally clear what happens here, why I did this
upload, what triggered the chanegs I did, and why last night.

On Tue, Sep 20, 2016 at 10:55:55PM +, Mattia Rizzolo wrote:
> dpkg_1.18.10.0~reproducible1.dsc has just been uploaded to 
> https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

The changelog against 1.18.10.0~reproducible0 (i.e. what we have been
running in the last months) is this:
  * quite a lot cleanup of the .buildinfo implementation by guillem
  * rewrite of the "deterministic modes in control.tar" by guillem
  * temporary backward compatibility for --buildinfo-id after guillem
renamed the option so we can transition over our tools
  * "rewrote" (within quotes as it's really silly) the "disable
Environment field from .buildinfo" thing, after the big modification
happened to dpkg-genbuildinfo.

The trigger:
yesterday evening a bystander I don't know about came in the IRC
channel, noticing how the diffoscope report of newly built packages in
the last days was totally incomplete, only showing the diff between the
.buildinfo files instead of unpacking over into the .debs.
After some minutes of debugging I discovered this was caused by a change
in pbuilder, which started to behave sanely¹.
The buildinfo spec says² that a .dsc must be included in
Checksums-Sha256 if that one is present.  Regardless of whether this is
a good choice, this is broken in multiple ways:
 * on the idea:
   + a .dsc could not be present for a binary build
   + until recently source builds were not reproducible, so an already
 present .dsc would be overwritten during a full build, and the
 .buildinfo would record the new one instead of the original
 * lexically: that field contains a list of built artifacts that have
   been built and distributed, putting a .dsc in it goes against this
   definition.
The problem is that we do a binary-only build (-b), so the changes file
does not contain the .dsc, but that one was referenced in .buildinfo
nonetheless.  Until pbuilder 0.226 the .dsc would have been copied over
even if not referenced in .changes; starting with 0.226 it's not copied
anymore.
This broke diffoscope, as it considers a .buildinfo referencing a
non-existing file as an invalid DotBuildinfoFile, and therefor falling
back to TextFile.

The fix:
I went to the our dpkg sources and see why it does that, and
individuated the interesting part.  Then I remembered that guillem did a
bunch of work on it too, and thought about looking whether some of the
stuff could be merged in our tree (mostly for wider testing).  Turned
out that he found such thing weird too, and so guarded that part of code
with an `if` that would get executed only in the case of a source build.
Hence I got in touch with him, had his patches rebased, fixed a couple
of glitches, and incorporated.  I didn't want to squash the commits, as
I would still like to keep some history of the evolution of the thing
still.

What I did:
* rolled back the history like we do with every new dpkg release
* substituted the patches for the "deterministic modes in control.tar"
  with one wrote by guillem
* appended the patches to the .buildinfo implementation from guillem
* "rewrote" (within quotes as it's really silly) the "disable
  Environment field from .buildinfo" thing, after the big modification
  happened to dpkg-genbuildinfo (the original one was from ntyni)
* added back a temporary --buildinfo-identifier flag in
  dpkg-buildpackage after guillem renamed id.
* built, tested, rebuilt, uploaded to our repo.


Next steps:
it would be great if somebody could figure what's the real gain of
having .dsc in Checksums-Sha256.  Also consider that within the context
of a single sane archive (as in: files once landed don't change) like
Debian's such trick is not needed, as a source package can already be
identified by other information already stored in .buildinfo like Source
and Version.



Thanks for reading this far,
Mattia (hoping to don't have to write any such long email for at least
some hours…)

¹ IOW: https://bugs.debian.org/492312
  https://anonscm.debian.org/git/pbuilder/pbuilder.git/commit/?id=806db12
² 
https://wiki.debian.org/ReproducibleBuilds/BuildinfoSpecification#buildinfo_field_descriptions

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds