bug#22533: Python bytecode reproducibility

2019-02-04 Thread Ludovic Courtès
Ricardo Wurmus skribis: > Ricardo Wurmus writes: > >> Now that we’re using Python 3.7 and this version supports hash-based pyc >> files, is this still an issue? Do we need to do anything to enable >> hash-based pyc compilation? >> >> See: >> https://docs.python.org/3/whatsnew/3.7.html#pep-552

bug#22533: Python bytecode reproducibility

2019-02-03 Thread Ricardo Wurmus
Ricardo Wurmus writes: > Now that we’re using Python 3.7 and this version supports hash-based pyc > files, is this still an issue? Do we need to do anything to enable > hash-based pyc compilation? > > See: > https://docs.python.org/3/whatsnew/3.7.html#pep-552-hash-based-pyc-files > https:/

bug#22533: Python bytecode reproducibility

2019-01-14 Thread Ricardo Wurmus
Now that we’re using Python 3.7 and this version supports hash-based pyc files, is this still an issue? Do we need to do anything to enable hash-based pyc compilation? See: https://docs.python.org/3/whatsnew/3.7.html#pep-552-hash-based-pyc-files https://www.python.org/dev/peps/pep-0552/ --

bug#22533: Python bytecode reproducibility

2018-03-08 Thread Gábor Boskovits
2018-03-04 20:18 GMT+01:00 Ricardo Wurmus : > I have applied this patch locally: > > > diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm > index 5f701701a..0d1ecc3c6 100644 > --- a/gnu/packages/python.scm > +++ b/gnu/packages/python.scm > @@ -359,8 +359,42 @@ data types.") >

bug#22533: Python bytecode reproducibility

2018-03-06 Thread Gábor Boskovits
2018-03-06 15:43 GMT+01:00 Ricardo Wurmus : > > Ricardo Wurmus writes: > > > Marius Bakke writes: > > > >> I suppose we'll have to set PYTHONHASHSEED somewhere in > >> python-build-system as well. Did you check if that makes a difference > >> for numpy? Perhaps it's enough to set it if we add

bug#22533: Python bytecode reproducibility

2018-03-06 Thread Ricardo Wurmus
Ricardo Wurmus writes: > Marius Bakke writes: > >> I suppose we'll have to set PYTHONHASHSEED somewhere in >> python-build-system as well. Did you check if that makes a difference >> for numpy? Perhaps it's enough to set it if we add an auto-compilation >> step? > > Right, I’m going to test t

bug#22533: Python bytecode reproducibility

2018-03-06 Thread Ricardo Wurmus
Marius Bakke writes: > The only remark I have is: is introducing a new variable necessary? > SOURCE_DATE_EPOCH implies that the user wants a deterministic build; > the upstream patch doesn't actually honor it outside of making the > hashing method deterministic. So, I think it might be enough t

bug#22533: Python bytecode reproducibility

2018-03-05 Thread Marius Bakke
Ricardo Wurmus writes: > I have applied this patch locally: > > diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm > index 5f701701a..0d1ecc3c6 100644 > --- a/gnu/packages/python.scm > +++ b/gnu/packages/python.scm > @@ -359,8 +359,42 @@ data types.") >

bug#22533: Python bytecode reproducibility

2018-03-05 Thread Ricardo Wurmus
Ricardo Wurmus writes: > Ricardo Wurmus writes: > >> I have applied this patch locally: >> >> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm >> index 5f701701a..0d1ecc3c6 100644 >> --- a/gnu/packages/python.scm >> +++ b/gnu/packages/python.scm >> @@ -359,8 +359,42 @@ data types.

bug#22533: Python bytecode reproducibility

2018-03-05 Thread Ricardo Wurmus
Ricardo Wurmus writes: > Ricardo Wurmus writes: > >> Unfortunately, this doesn’t fix all reproducibility problems with numpy: >> >> --8<---cut here---start->8--- >> Binary files >> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check/lib/pyt

bug#22533: Python bytecode reproducibility

2018-03-05 Thread Ricardo Wurmus
Gábor Boskovits writes: > 2018-03-05 16:36 GMT+01:00 Gábor Boskovits : > >> 2018-03-05 1:05 GMT+01:00 Ricardo Wurmus : >> >>> >>> Ricardo Wurmus writes: >>> >>> > Unfortunately, this doesn’t fix all reproducibility problems with numpy: >>> > >>> > --8<---cut here---start

bug#22533: Python bytecode reproducibility

2018-03-05 Thread Gábor Boskovits
2018-03-05 16:36 GMT+01:00 Gábor Boskovits : > 2018-03-05 1:05 GMT+01:00 Ricardo Wurmus : > >> >> Ricardo Wurmus writes: >> >> > Unfortunately, this doesn’t fix all reproducibility problems with numpy: >> > >> > --8<---cut here---start->8--- >> > Binary files /

bug#22533: Python bytecode reproducibility

2018-03-05 Thread Gábor Boskovits
2018-03-05 1:05 GMT+01:00 Ricardo Wurmus : > > Ricardo Wurmus writes: > > > Unfortunately, this doesn’t fix all reproducibility problems with numpy: > > > > --8<---cut here---start->8--- > > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw > cc-python-num

bug#22533: Python bytecode reproducibility

2018-03-05 Thread Ludovic Courtès
Hello! Ricardo Wurmus skribis: > Is it a bad idea to override the timestamps in the generated binaries? > I think that we could avoid the recency check then, which was an > obstacle to resetting the timestamps of the source files. I think it’s good if we can fix Python itself to honor SOURCE_DA

bug#22533: Python bytecode reproducibility

2018-03-04 Thread Ricardo Wurmus
Ricardo Wurmus writes: > Unfortunately, this doesn’t fix all reproducibility problems with numpy: > > --8<---cut here---start->8--- > Binary files > /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/distut

bug#22533: Python bytecode reproducibility

2018-03-04 Thread Ricardo Wurmus
Ricardo Wurmus writes: > I have applied this patch locally: > > diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm > index 5f701701a..0d1ecc3c6 100644 > --- a/gnu/packages/python.scm > +++ b/gnu/packages/python.scm > @@ -359,8 +359,42 @@ data types.") >

bug#22533: Python bytecode reproducibility

2018-03-04 Thread Ricardo Wurmus
I have applied this patch locally: diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index 5f701701a..0d1ecc3c6 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -359,8 +359,42 @@ data types.") "Lib/ctypes/test/test_win32.py" ; fails

bug#22533: Python bytecode reproducibility

2018-03-04 Thread Gábor Boskovits
2018-03-04 13:46 GMT+01:00 Ricardo Wurmus : > > Hi Gábor, > > > Nix had this issue, it seems they have a python 3.5 solution, which > > should be easy to adopt: https://github.com/NixOS/nixpkgs/issues/22570. > > WDYT? > > Here’s the patch for Nix: > > https://patch-diff.githubusercontent.com/raw

bug#22533: Python bytecode reproducibility

2018-03-04 Thread Ricardo Wurmus
Hi Gábor, > Nix had this issue, it seems they have a python 3.5 solution, which > should be easy to adopt: https://github.com/NixOS/nixpkgs/issues/22570. > WDYT? Here’s the patch for Nix: https://patch-diff.githubusercontent.com/raw/NixOS/nixpkgs/pull/22585.diff Here are the relevant changes

bug#22533: Python bytecode reproducibility

2018-03-04 Thread Gábor Boskovits
2018-03-03 23:37 GMT+01:00 Ricardo Wurmus : > Hi Guix, > > Marius Bakke writes: > > > It would be great to revive this longstanding bug! > > Indeed. > > Here’s another attempt. As far as I understand, the timestamp in the > pyc files only affects the header. > > Up until Python 3.6 (incl) the he

bug#22533: Python bytecode reproducibility

2018-03-03 Thread Ricardo Wurmus
Hi Guix, Marius Bakke writes: > It would be great to revive this longstanding bug! Indeed. Here’s another attempt. As far as I understand, the timestamp in the pyc files only affects the header. Up until Python 3.6 (incl) the header looks like this: magic | timestamp | size Since Python

bug#22533: Python bytecode reproducibility

2017-05-26 Thread Marius Bakke
Hello! I stumbled across this bug after re-discovering that Python bytecode is not reproducible (through "glib"). Just sharing some notes.. Nix recently made an effort to fix this. AFAICT the ".pyc" files are still a problem, but at least they got the interpreters building reproducibly: https://