Your message dated Tue, 22 Feb 2022 14:43:05 +0000
with message-id <[email protected]>
and subject line Bug#1004558: fixed in python3.10 3.10.2-2
has caused the Debian Bug report #1004558,
regarding python3.10: reproducible pyc files -- please consider applying this
backported patch
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
1004558: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1004558
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Source: python3.10
Version: 3.10.2-1
Severity: normal
Tags: patch
User: [email protected]
Usertags: randomness
X-Debbugs-Cc: [email protected], [email protected]
Hi,
currently, pyc files generated after installation are not reproducible.
Namely, the elements in frozen sets are not ordered reproducibly. This
means that to create a bit-by-bit reproducible Debian chroot including
Python one has to remove all *.pyc files:
https://gitlab.tails.boum.org/tails/tails/-/blob/stable/config/chroot_local-hooks/99-remove_pyc
https://salsa.debian.org/live-team/live-build/-/blob/master/share/hooks/normal/0170-remove-python-py.hook.chroot
Fortunately, there is a patch that is already accepted upstream that
fixes this problem:
https://github.com/python/cpython/pull/27926
https://bugs.python.org/issue37596
That patch also cleanly applies to Python 3.10 as well as to 3.9. I
tested it for both versions. Please consider carrying the attached patch
for 3.9 and 3.10 so that we can have reproducible Debian chroots before
the Python 3.11 release.
Thanks!
cheers, josch
>From 36ae9beb04763d498df2114657bfbbcfe58bf913 Mon Sep 17 00:00:00 2001
From: Brandt Bucher <[email protected]>
Date: Mon, 23 Aug 2021 18:34:17 -0700
Subject: [PATCH] Serialize frozenset elements deterministically
---
Lib/test/test_marshal.py | 25 +++++++++++++++
.../2021-08-23-21-39-59.bpo-37596.ojRcwB.rst | 2 ++
Python/marshal.c | 32 +++++++++++++++++++
3 files changed, 59 insertions(+)
create mode 100644
Misc/NEWS.d/next/Library/2021-08-23-21-39-59.bpo-37596.ojRcwB.rst
diff --git a/Lib/test/test_marshal.py b/Lib/test/test_marshal.py
index 7bcf8e8..8252be3 100644
--- a/Lib/test/test_marshal.py
+++ b/Lib/test/test_marshal.py
@@ -318,6 +318,31 @@ class BugsTestCase(unittest.TestCase):
for i in range(len(data)):
self.assertRaises(EOFError, marshal.loads, data[0: i])
+ def test_deterministic_sets(self):
+ # bpo-37596: To support reproducible builds, sets and frozensets need
to
+ # have their elements serialized in a consistent order (even when they
+ # have been scrambled by hash randomization):
+ for kind in ("set", "frozenset"):
+ for elements in (
+ "float('nan'), b'a', b'b', b'c', 'x', 'y', 'z'",
+ # Also test for bad interactions with backreferencing:
+ "('string', 1), ('string', 2), ('string', 3)",
+ ):
+ s = f"{kind}([{elements}])"
+ with self.subTest(s):
+ # First, make sure that our test case still has different
+ # orders under hash seeds 0 and 1. If this check fails, we
+ # need to update this test with different elements:
+ args = ["-c", f"print({s})"]
+ _, repr_0, _ = assert_python_ok(*args, PYTHONHASHSEED="0")
+ _, repr_1, _ = assert_python_ok(*args, PYTHONHASHSEED="1")
+ self.assertNotEqual(repr_0, repr_1)
+ # Then, perform the actual test:
+ args = ["-c", f"import marshal; print(marshal.dumps({s}))"]
+ _, dump_0, _ = assert_python_ok(*args, PYTHONHASHSEED="0")
+ _, dump_1, _ = assert_python_ok(*args, PYTHONHASHSEED="1")
+ self.assertEqual(dump_0, dump_1)
+
LARGE_SIZE = 2**31
pointer_size = 8 if sys.maxsize > 0xFFFFFFFF else 4
diff --git a/Misc/NEWS.d/next/Library/2021-08-23-21-39-59.bpo-37596.ojRcwB.rst
b/Misc/NEWS.d/next/Library/2021-08-23-21-39-59.bpo-37596.ojRcwB.rst
new file mode 100644
index 0000000..81fdfeb
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2021-08-23-21-39-59.bpo-37596.ojRcwB.rst
@@ -0,0 +1,2 @@
+Ensure that :class:`set` and :class:`frozenset` objects are always
+:mod:`marshalled <marshal>` reproducibly.
diff --git a/Python/marshal.c b/Python/marshal.c
index 4125240..6bd4537 100644
--- a/Python/marshal.c
+++ b/Python/marshal.c
@@ -502,9 +502,41 @@ w_complex_object(PyObject *v, char flag, WFILE *p)
W_TYPE(TYPE_SET, p);
n = PySet_GET_SIZE(v);
W_SIZE(n, p);
+ // bpo-37596: To support reproducible builds, sets and frozensets need
+ // to have their elements serialized in a consistent order (even when
+ // they have been scrambled by hash randomization). To ensure this, we
+ // use an order equivalent to sorted(v, key=marshal.dumps):
+ PyObject *pairs = PyList_New(0);
+ if (pairs == NULL) {
+ p->error = WFERR_NOMEMORY;
+ return;
+ }
while (_PySet_NextEntry(v, &pos, &value, &hash)) {
+ PyObject *dump = PyMarshal_WriteObjectToString(value, p->version);
+ if (dump == NULL) {
+ p->error = WFERR_UNMARSHALLABLE;
+ goto anyset_done;
+ }
+ PyObject *pair = PyTuple_Pack(2, dump, value);
+ Py_DECREF(dump);
+ if (pair == NULL || PyList_Append(pairs, pair)) {
+ p->error = WFERR_NOMEMORY;
+ Py_XDECREF(pair);
+ goto anyset_done;
+ }
+ Py_DECREF(pair);
+ }
+ if (PyList_Sort(pairs)) {
+ p->error = WFERR_NOMEMORY;
+ goto anyset_done;
+ }
+ for (Py_ssize_t i = 0; i < n; i++) {
+ PyObject *pair = PyList_GET_ITEM(pairs, i);
+ value = PyTuple_GET_ITEM(pair, 1);
w_object(value, p);
}
+ anyset_done:
+ Py_DECREF(pairs);
}
else if (PyCode_Check(v)) {
PyCodeObject *co = (PyCodeObject *)v;
--
2.33.0
--- End Message ---
--- Begin Message ---
Source: python3.10
Source-Version: 3.10.2-2
Done: Matthias Klose <[email protected]>
We believe that the bug you reported is fixed in the latest version of
python3.10, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to [email protected],
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Matthias Klose <[email protected]> (supplier of updated python3.10 package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing [email protected])
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Format: 1.8
Date: Tue, 22 Feb 2022 14:57:42 +0100
Source: python3.10
Architecture: source
Version: 3.10.2-2
Distribution: unstable
Urgency: medium
Maintainer: Matthias Klose <[email protected]>
Changed-By: Matthias Klose <[email protected]>
Closes: 1004558 1005761 1006219
Changes:
python3.10 (3.10.2-2) unstable; urgency=medium
.
* Make test suite support Expat >=2.4.5. Closes: #1006219.
* Build again using readline instead of libedit. Closes: #1005761.
* Serialize frozenset elements deterministically, taken from the trunk.
Closes: #1004558.
Checksums-Sha1:
16965d4aecc7b3b3996fb2e4516f534212847772 3500 python3.10_3.10.2-2.dsc
f314fdac6d5d1973bc262d04adddcb05e9fba695 213904
python3.10_3.10.2-2.debian.tar.xz
ea1f7154367be0608a0c9f6467c2b239c66c9b1b 9694
python3.10_3.10.2-2_source.buildinfo
Checksums-Sha256:
c73c292188e620a18666019b764076c8303190be0ca4721530fa0167eb66e696 3500
python3.10_3.10.2-2.dsc
2a57f709175ee19e31ed26ab1bef4ff213320e8654f4e7129588524d43bf6a62 213904
python3.10_3.10.2-2.debian.tar.xz
49ec4aebab69cac5c0fd4ab64e0b6b8327ab28b1895ea037d9700ea2cd2ef2b7 9694
python3.10_3.10.2-2_source.buildinfo
Files:
5fb2ac914f24149c34804c9366a6e122 3500 python optional python3.10_3.10.2-2.dsc
e2863f1cb9f1485ca17b584b66deb166 213904 python optional
python3.10_3.10.2-2.debian.tar.xz
e5bb4d2e03457bbea69c304e22e5d8cf 9694 python optional
python3.10_3.10.2-2_source.buildinfo
-----BEGIN PGP SIGNATURE-----
iQJEBAEBCAAuFiEE1WVxuIqLuvFAv2PWvX6qYHePpvUFAmIU7sYQHGRva29AZGVi
aWFuLm9yZwAKCRC9fqpgd4+m9bYVEACRwaLV+6dkBXtZMDmRxsW861N2Oo7Fzu9X
Ne5IPdCafWVstZuM5NRreV3eiLjaIjS0JBJjpG7/u+bpEKABl9qUjANjcTMIYkJ+
a6lPtmTc3OKKthxatOVVVtnu70qB75Hg5h4J5j5gQWHdYkkDtSQ62v/jOk/76EaQ
x7bRW+3GjajM8Lhu4tpikSlVMC13F6d4E5wbF+3paSpvGxKpO+0GmEb2DRr+nMC1
lznqA18hiHaFhPBQg2m0FJJUTpx+m3j1GXYmtWopOG8yYv1sLRCorSohrHxMm/vB
yXKlzEH6ppboL3iajoCcPFe36r7S9P4GLRJrtLE10QPFM9bIxIn/rmRnGUARxO8t
o7vFf0v97D41a1qfjBXFYQu+4fcKHGaET8CSJoBDbZ6Ux4GhoTFAQH9hR0J40oKZ
iOHuR5CZo8BDNVFI7t6ggRXqPIsclE2jcOLWEZ2eB095BAmSwdSznnRoawLGvV+I
HdCbcNWmRSImO/YlPlUtgL7y98u78NeUGT2XO0DJNRX0Jnp7uOmKnYJXAaB2lKG2
98yyj2DCZKIGqSvnWMHqej9MsLNA14LTruzVwqc5Ak0IQsKxm6bhz4E8zMtBuP5b
Lr5BBGDDyS0PTXFpSmg10l9i/1v/6xzpZOkY+YmHw7mAz7oPmqJcqNvZVDAvEXLs
vyhKpWwuTA==
=GCVU
-----END PGP SIGNATURE-----
--- End Message ---
