Re: [PATCH] reproducible Debian: Reverse Python 2.x dict order in the second build.
Holger Levsen wrote:
> i'm confused: uploaded what to where?
[…]
> the commit has a little too little information why this is useful
Updated commit to cover both concerns:
commit a48e8e4685c204bea66d440014687ef662fdfcc5
Author: Chris Lamb
Date: Mon Feb 13 10:00:18 2017 +1300
reproducible Debian: Reverse Python 2.x dict order in the second build.
We would like to find any code that is relying on non-determinstic or
undefined behaviour. In Python 2.x, this includes dict ordering.
It would therefore be convenient to be able to specify an environment
variable that would reverse the usual ordering; we would then run a build
of packages both with and without this flag and compare the resulting
binaries as before.
I wrote a patch for both 3.x and 2.x but they are unlikely to be accepted
upstream (and the the 3.x one may be meaningless due to language changes),
hence uploading to our custom reproducible toolchain.
This change to Jenkins enables the "PYTHONREVERSEDICTKEYORDER" behaviour
in
our second build. It changes the behaviour of:
* for x in d:
* d.popitem()
* d.items() & d.iteritems()
* d.values() & d.itervalues()
* print(d)
* repr(dx)
* PyDict_Next & _PyDict_Next
It does not change the behaviour of collections.OrderedDict.
Signed-off-by: Chris Lamb
bin/reproducible_build.sh | 1 +
bin/reproducible_common.sh | 1 +
2 files changed, 2 insertions(+)
You can also merge from the "export-PYTHONREVERSEDICTKEYORDER-pbuilderrc"
branch of
https://github.com/lamby/jenkins.debian.net if that is more convenient.
Regards,
--
,''`.
: :' : Chris Lamb
`. `'` la...@debian.org / chris-lamb.co.uk
`-
From a48e8e4685c204bea66d440014687ef662fdfcc5 Mon Sep 17 00:00:00 2001
From: Chris Lamb
Date: Mon, 13 Feb 2017 10:00:18 +1300
Subject: [PATCH] reproducible Debian: Reverse Python 2.x dict order in the
second build.
We would like to find any code that is relying on non-determinstic or
undefined behaviour. In Python 2.x, this includes dict ordering.
It would therefore be convenient to be able to specify an environment
variable that would reverse the usual ordering; we would then run a build
of packages both with and without this flag and compare the resulting
binaries as before.
I wrote a patch for both 3.x and 2.x but they are unlikely to be accepted
upstream (and the the 3.x one may be meaningless due to language changes),
hence uploading to our custom reproducible toolchain.
This change to Jenkins enables the "PYTHONREVERSEDICTKEYORDER" behaviour in
our second build. It changes the behaviour of:
* for x in d:
* d.popitem()
* d.items() & d.iteritems()
* d.values() & d.itervalues()
* print(d)
* repr(dx)
* PyDict_Next & _PyDict_Next
It does not change the behaviour of collections.OrderedDict.
Signed-off-by: Chris Lamb
---
bin/reproducible_build.sh | 1 +
bin/reproducible_common.sh | 1 +
2 files changed, 2 insertions(+)
diff --git a/bin/reproducible_build.sh b/bin/reproducible_build.sh
index 1ebc786f..4e66acfe 100755
--- a/bin/reproducible_build.sh
+++ b/bin/reproducible_build.sh
@@ -689,6 +689,7 @@ export TZ="/usr/share/zoneinfo/Etc/GMT-14"
export LANG="$locale.UTF-8"
export LC_ALL="$locale.UTF-8"
export LANGUAGE="$locale:$language"
+export PYTHONREVERSEDICTKEYORDER=1
umask 0002
EOF
# build path is not yet varied on testing
diff --git a/bin/reproducible_common.sh b/bin/reproducible_common.sh
index d6ab6978..a0d39028 100755
--- a/bin/reproducible_common.sh
+++ b/bin/reproducible_common.sh
@@ -377,6 +377,7 @@ write_variation_table() {
fi
if [ "$1" != "FreeBSD" ] && [ "$1" != "Arch Linux" ] && [ "$1" != "fedora-23" ] ; then
write_page "env CAPTURE_ENVIRONMENTnot setCAPTURE_ENVIRONMENT=\"I capture the environment\""
+ write_page "env PYTHONREVERSEDICTKEYORDERnot seton amd64: PYTHONREVERSEDICTKEYORDER=\"1\""
fi
write_page "env TZTZ=\"/usr/share/zoneinfo/Etc/GMT+12\"TZ=\"/usr/share/zoneinfo/Etc/GMT-14\""
if [ "$1" = "debian" ] ; then
--
2.11.0
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [PATCH] reproducible Debian: Reverse Python 2.x dict order in the second build.
On Sun, Feb 12, 2017 at 10:43:22AM +0100, Mattia Rizzolo wrote: > On Sun, Feb 12, 2017 at 10:27:02PM +1300, Chris Lamb wrote: > > Technically true, but I've only uploaded a python2.7 with my > > PYTHONREVERSEDICTKEYORDER patch for amd64 so far. > I didn't notice the upload, as it was not announced because you didn't > upload the sources; please also upload the source packages (then other > people can provide builds for the other architectures, also). i'm confused: uploaded what to where? > > Force-pushed as 899a13de to `export-PYTHONREVERSEDICTKEYORDER-pbuilderrc`. > ↑ Holger, pls? the commit has a little too little information why this is useful, relevant in the real world, etc, can you please explain? (best in the commit message, so it can be easily found later.) -- cheers, Holger signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [PATCH] reproducible Debian: Reverse Python 2.x dict order in the second build.
Mattia Rizzolo wrote: > I didn't notice the upload, as it was not announced because you didn't > upload the sources; please also upload the source packages Rebuilding and uploading now. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `- ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [PATCH] reproducible Debian: Reverse Python 2.x dict order in the second build.
On Sun, Feb 12, 2017 at 10:27:02PM +1300, Chris Lamb wrote: > Technically true, but I've only uploaded a python2.7 with my > PYTHONREVERSEDICTKEYORDER patch for amd64 so far. I didn't notice the upload, as it was not announced because you didn't upload the sources; please also upload the source packages (then other people can provide builds for the other architectures, also). > Force-pushed as 899a13de to `export-PYTHONREVERSEDICTKEYORDER-pbuilderrc`. ↑ Holger, pls? -- regards, Mattia Rizzolo GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`. more about me: https://mapreri.org : :' : Launchpad user: https://launchpad.net/~mapreri `. `'` Debian QA page: https://qa.debian.org/developer.php?login=mattia `- signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [PATCH] reproducible Debian: Reverse Python 2.x dict order in the second build.
Mattia Rizzolo wrote: > > > Umh, could you please set that variable in the pbuilderrc of the second > > > build instead of using the hooks (which I don't understand how that > > > would ever work > > > > I just grepped for existing changes (eg. "I capture the environment"). If > > these things don't work as you claim, please could you remove them so they > > are not confusing? > > I don't know, did that ever worked? I assumed it did otherwise it would not be in the repo. I suggest removing all of that code to prevent any further confusion... > > - write_page "env CAPTURE_ENVIRONMENTnot > > setCAPTURE_ENVIRONMENT=\"I capture the > > environment\"" > > + write_page "env CAPTURE_ENVIRONMENTnot > > setCAPTURE_ENVIRONMENT=\"I capture the environment\"on > > amd64: PYTHONREVERSEDICTKEYORDER=1" > > This would be everywhere, not just amd64. Technically true, but I've only uploaded a python2.7 with my PYTHONREVERSEDICTKEYORDER patch for amd64 so far. To save another patch round, feel free to remove that bit before committing that you disagree; I'm easy either way. > Plus, I think it would look better if added a different line on the > table, separated from CAPTURE_ENVIRONMENT. Force-pushed as 899a13de to `export-PYTHONREVERSEDICTKEYORDER-pbuilderrc`. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `- ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [PATCH] reproducible Debian: Reverse Python 2.x dict order in the second build.
On Sun, Feb 12, 2017 at 01:18:56PM +1300, Chris Lamb wrote: > > Umh, could you please set that variable in the pbuilderrc of the second > > build instead of using the hooks (which I don't understand how that > > would ever work > > I just grepped for existing changes (eg. "I capture the environment"). If > these things don't work as you claim, please could you remove them so they > are not confusing? I don't know, did that ever worked? A grep for CAPTURE_ENVIRONMENT in the text diffoscope reports returns nothing. I'm not claiming anything, as I didn't add and tested that variation (as I suppose whoever added it did), but with my pbuilder knowledge I'd be very surprised if it does. > > PS: was it you that triggered a admin password reset? > > Yes. (The button is misleading, I didn't think it would actually reset > the password but rather give me immediate ML admin access.) Well, the button on https://alioth.debian.org/mail/admin/?group_id=100870 says "Reset admin password", not sure how that can be misleading. > - write_page "env CAPTURE_ENVIRONMENTnot > setCAPTURE_ENVIRONMENT=\"I capture the environment\"" > + write_page "env CAPTURE_ENVIRONMENTnot > setCAPTURE_ENVIRONMENT=\"I capture the environment\"on > amd64: PYTHONREVERSEDICTKEYORDER=1" This would be everywhere, not just amd64. Plus, I think it would look better if added a different line on the table, separated from CAPTURE_ENVIRONMENT. -- regards, Mattia Rizzolo GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`. more about me: https://mapreri.org : :' : Launchpad user: https://launchpad.net/~mapreri `. `'` Debian QA page: https://qa.debian.org/developer.php?login=mattia `- signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [PATCH] reproducible Debian: Reverse Python 2.x dict order in the second build.
> Umh, could you please set that variable in the pbuilderrc of the second
> build instead of using the hooks (which I don't understand how that
> would ever work
I just grepped for existing changes (eg. "I capture the environment"). If
these things don't work as you claim, please could you remove them so they
are not confusing? Perhaps I'm missing something as that particular one
isn't even in pbuilderrc
> 1 line change is better than changing dozens of files anyway, I suppose.
Exactly ^ :)
> So just dump it in reproducible_build.sh, line 682 onwards
Sure:
commit 086d2938f25331c0635537d3445db9f1ba29774f
Author: Chris Lamb
Date: Sun Feb 12 10:03:02 2017 +1300
reproducible Debian: Reverse Python 2.x dict order in the second build.
Signed-off-by: Chris Lamb
bin/reproducible_build.sh | 1 +
bin/reproducible_common.sh | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
You can also merge from the "export-PYTHONREVERSEDICTKEYORDER-pbuilderrc"
branch of
https://github.com/lamby/jenkins.debian.net if that is more convenient.
> PS: was it you that triggered a admin password reset?
Yes. (The button is misleading, I didn't think it would actually reset
the password but rather give me immediate ML admin access.)
Regards,
--
,''`.
: :' : Chris Lamb
`. `'` la...@debian.org / chris-lamb.co.uk
`-
From 086d2938f25331c0635537d3445db9f1ba29774f Mon Sep 17 00:00:00 2001
From: Chris Lamb
Date: Sun, 12 Feb 2017 10:03:02 +1300
Subject: [PATCH] reproducible Debian: Reverse Python 2.x dict order in the
second build.
Signed-off-by: Chris Lamb
---
bin/reproducible_build.sh | 1 +
bin/reproducible_common.sh | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/bin/reproducible_build.sh b/bin/reproducible_build.sh
index 1ebc786f..4e66acfe 100755
--- a/bin/reproducible_build.sh
+++ b/bin/reproducible_build.sh
@@ -689,6 +689,7 @@ export TZ="/usr/share/zoneinfo/Etc/GMT-14"
export LANG="$locale.UTF-8"
export LC_ALL="$locale.UTF-8"
export LANGUAGE="$locale:$language"
+export PYTHONREVERSEDICTKEYORDER=1
umask 0002
EOF
# build path is not yet varied on testing
diff --git a/bin/reproducible_common.sh b/bin/reproducible_common.sh
index d6ab6978..63c0cb34 100755
--- a/bin/reproducible_common.sh
+++ b/bin/reproducible_common.sh
@@ -376,7 +376,7 @@ write_variation_table() {
write_page "domainname is not yet varied between rebuilds of $1."
fi
if [ "$1" != "FreeBSD" ] && [ "$1" != "Arch Linux" ] && [ "$1" != "fedora-23" ] ; then
- write_page "env CAPTURE_ENVIRONMENTnot setCAPTURE_ENVIRONMENT=\"I capture the environment\""
+ write_page "env CAPTURE_ENVIRONMENTnot setCAPTURE_ENVIRONMENT=\"I capture the environment\"on amd64: PYTHONREVERSEDICTKEYORDER=1"
fi
write_page "env TZTZ=\"/usr/share/zoneinfo/Etc/GMT+12\"TZ=\"/usr/share/zoneinfo/Etc/GMT-14\""
if [ "$1" = "debian" ] ; then
--
2.11.0
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [PATCH] reproducible Debian: Reverse Python 2.x dict order in the second build.
On Sun, Feb 12, 2017 at 12:46:42AM +0100, Mattia Rizzolo wrote: > On Sun, Feb 12, 2017 at 10:08:54AM +1300, Chris Lamb wrote: > > Hi, > > > > (Attachment was too large for Alioth apparently, so re-sending) > > Yes (I discarded the email from the moderation queue now). BTW, yes, remote to pull from is better anyway than a patch; anyhow, I increased a tad the limit, as 40 KB are pretty small these days (you sent a ~53 KB email) PS: was it you that triggered a admin password reset? -- regards, Mattia Rizzolo GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`. more about me: https://mapreri.org : :' : Launchpad user: https://launchpad.net/~mapreri `. `'` Debian QA page: https://qa.debian.org/developer.php?login=mattia `- signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [PATCH] reproducible Debian: Reverse Python 2.x dict order in the second build.
On Sun, Feb 12, 2017 at 10:08:54AM +1300, Chris Lamb wrote: > Hi, > > (Attachment was too large for Alioth apparently, so re-sending) Yes (I discarded the email from the moderation queue now). > commit 02d4d20ae8eab1440b0d9fba8eda3df860d259d8 > Author: Chris Lamb > Date: Sun Feb 12 10:03:02 2017 +1300 > > reproducible Debian: Reverse Python 2.x dict order in the second build. > > Signed-off-by: Chris Lamb > >bin/reproducible_common.sh | 2 > +- >hosts/bbx15-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 > +++ >hosts/bpi0-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 > +++ Umh, could you please set that variable in the pbuilderrc of the second build instead of using the hooks (which I don't understand how that would ever work, considering the hooks are *executed* inside the chroot, and not sourced, so variables exported from there shouldn't be seen during the build, in theory; does that variation even work?) So just dump it in reproducible_build.sh, line 682 onwards. 1 line change is better than changing dozens of files anyway, I suppose. > Please merge from the "export-PYTHONREVERSEDICTKEYORDER" branch of > https://github.com/lamby/jenkins.debian.net -- regards, Mattia Rizzolo GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`. more about me: https://mapreri.org : :' : Launchpad user: https://launchpad.net/~mapreri `. `'` Debian QA page: https://qa.debian.org/developer.php?login=mattia `- signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[PATCH] reproducible Debian: Reverse Python 2.x dict order in the second build.
Hi, (Attachment was too large for Alioth apparently, so re-sending) commit 02d4d20ae8eab1440b0d9fba8eda3df860d259d8 Author: Chris Lamb Date: Sun Feb 12 10:03:02 2017 +1300 reproducible Debian: Reverse Python 2.x dict order in the second build. Signed-off-by: Chris Lamb bin/reproducible_common.sh | 2 +- hosts/bbx15-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/bpi0-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/cb3a-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../cbxi4a-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../cbxi4b-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/ff2a-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/ff2b-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/ff4a-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/hb0-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../jenkins-test-vm/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/jenkins/etc/pbuilder/rebuild-hooks/D01_modify_environment| 3 +++ hosts/jtk1a-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/odu3a-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/odxu4-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../odxu4b-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../odxu4c-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/opi2a-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/opi2b-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/opi2c-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ .../etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/rpi2b-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/rpi2c-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/wbd0-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ hosts/wbq0-armhf-rb/etc/pbuilder/rebuild-hooks/D01_modify_environment | 3 +++ 42 files changed, 124 insertions(+), 1 deletion(-) Please merge from the "export-PYTHONREVERSEDICTKEYORDER" branch of https://github.com/lamby/jenkins.debian.net Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `- ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
