bug#30688: hdparm hash mismatch

[Clément Lassieur]

Björn Höfling  writes:

> On Sat, 03 Mar 2018 21:52:03 +0100
> Clément Lassieur  wrote:
>
>> Hello,
>> 
>> I got this error while reconfiguring, did the source change upstream?
>> 
>> Starting download
>> of /gnu/store/4gn3ksv0hrglymrhh11474s13h4x6wcr-hdparm-9.54.tar.gz
>> >From
>> >http://downloads.sourceforge.net/project/hdparm/hdparm/hdparm-9.54.tar.gz...
>> >  
>> following redirection to
>> `https://downloads.sourceforge.net/#!/project/hdparm/hdparm/hdparm-9.54.tar.gz'...
>>  ...54.tar.gz
>> 652B   797KiB/s 00:00 [##] 100.0%
>> sha256 hash mismatch for output path
>> `/gnu/store/4gn3ksv0hrglymrhh11474s13h4x6wcr-hdparm-9.54.tar.gz'
>> expected: 0ghnhdj7wfw6acfyhdawpfa5n9kvkvzgi1fw6i7sghgbjx5nhyjd
>> actual:   1grhp5i29v1krrn1m36n68ls6vakvbw4x60im26b7h55rkvx61vs cannot
>> build derivation
>> `/gnu/store/75g4xr84spyxr6i7h5w1flflzd5gx1z2-hdparm-9.54.drv': 1
>> dependencies couldn't be built cannot build derivation
>> `/gnu/store/wkacz7nvhc31w6iifiw02v7yqjrdm6hg-tlp-1.1.drv': 1
>> dependencies couldn't be built guix system: error: build failed:
>> build of `/gnu/store/wkacz7nvhc31w6iifiw02v7yqjrdm6hg-tlp-1.1.drv'
>> failed
>> 
>> Clément
>
> SourceForge is currently (since weeks) restructuring their site, which
> unfortunately causes these kind of hiccups:
>
> $> guix build -S --no-substitutes hdparm
> $> tail -n 5 \ 
> /gnu/store/mr2ksp6kf5n3mw9g446ff1ivmlh897mg-hdparm-9.53.tar.gz
>
>  
> We're sorry -- the Sourceforge site is currently in Disaster Recovery mode, 
> and currently requires
> the use of javascript to function.  Please check back later.
> 
> 
>
> (I'm lacking a version behind, but the result is the same)
>
> Either wait, get substitutes from somewhere, or get the source code
> from somewhere.
>
> Björn

It works again now.  It's the redirection from
http://downloads.sourceforge.net/ to a mirror that didn't work.

Thanks
Clément

bug#22533: Python bytecode reproducibility

[Gábor Boskovits]

2018-03-03 23:37 GMT+01:00 Ricardo Wurmus :

> Hi Guix,
>
> Marius Bakke  writes:
>
> > It would be great to revive this longstanding bug!
>
> Indeed.
>
> Here’s another attempt.  As far as I understand, the timestamp in the
> pyc files only affects the header.
>
> Up until Python 3.6 (incl) the header looks like this:
>
>   magic | timestamp | size
>
> Since Python 3.7 the header may either contain a timestamp or a hash:
>
>   magic |  | timestamp | size
>   magic | 0001 | hash  | size
>
> This means we likely won’t have this problem any more with Python 3.7.
> For Python 3.6 I guess we could add a final build phase that overwrites
> the timestamp in the *binary*.  This needs to happen before any of the
> compiled files are wrapped up in a wheel.
>
> Should we just wait for Python 3.7 which is expected to be released in
> June 2018?  We’d still have to deal with this problem in Python 2,
> though.
>
> Is it a bad idea to override the timestamps in the generated binaries?
> I think that we could avoid the recency check then, which was an
> obstacle to resetting the timestamps of the source files.

--
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net
>
>
Nix had this issue, it seems they have a python 3.5 solution, which
should be easy to adopt: https://github.com/NixOS/nixpkgs/issues/22570.
WDYT?

bug#22533: Python bytecode reproducibility

[Gábor Boskovits]

2018-03-04 13:46 GMT+01:00 Ricardo Wurmus :

>
> Hi Gábor,
>
> > Nix had this issue, it seems they have a python 3.5 solution, which
> > should be easy to adopt: https://github.com/NixOS/nixpkgs/issues/22570.
> > WDYT?
>
> Here’s the patch for Nix:
>
>   https://patch-diff.githubusercontent.com/raw/
> NixOS/nixpkgs/pull/22585.diff
>
> Here are the relevant changes to the Python packages:
>
> * Python 3.4
>
>   substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']"
> "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"
>   substituteInPlace "Lib/importlib/_bootstrap.py" --replace "source_mtime
> = int(source_stats['mtime'])" "source_mtime = 1"
>
> * Python 3.5
>
>   substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']"
> "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"
>   substituteInPlace "Lib/importlib/_bootstrap_external.py" --replace
> "source_mtime = int(st['mtime'])" "source_mtime = 1"
>
> * Python 3.6
>   substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']"
> "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"
>   substituteInPlace "Lib/importlib/_bootstrap_external.py" --replace
> "source_mtime = int(st['mtime'])" "source_mtime = 1"
>
>
>
Nice, thanks for the summary.
Can we adopt this as is?
Do we need the 3.4 and 3.5 fix or the 3.6 one is enough?


> For all packages they set these environment variables:
>
>   - set PYTHONHASHSEED=0 (for hashes of str, bytes and datetime objects)
>
>   - set DETERMINISTIC_BUILD; for conditional patching of the timestamp
> for package builds.  The timestamp is not patched in ad-hoc
> environments, because that would mess with Python’s ability to
> determine whether to compile source files.
>
>
Should we set these in python-build-system? What about python booststrap?
I guess we use gnu-build-system there, so bootstrap packages might need to
set these explicitly?


> They also rebuild all bytecode (with the exception of lib2to3 because it
> is Python 2 code) three times, once for each optimization level.
>
> --8<---cut here---start->8---
> +# Determinism: rebuild all bytecode
> +# We exclude lib2to3 because that's Python 2 code which fails
> +# We rebuild three times, once for each optimization level
> +find $out -name "*.py" | $out/bin/python -m compileall -q -f -x
> "lib2to3" -i -
> +find $out -name "*.py" | $out/bin/python -O -m compileall -q -f -x
> "lib2to3" -i -
> +find $out -name "*.py" | $out/bin/python -OO -m compileall -q -f -x
> "lib2to3" -i -
> --8<---cut here---end--->8---
>
>
Do we also have to do this, or should we settle with one optimization
level? Which one?


> --
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net
>
>
>

bug#22533: Python bytecode reproducibility

[Ricardo Wurmus]

Hi Gábor,

> Nix had this issue, it seems they have a python 3.5 solution, which
> should be easy to adopt: https://github.com/NixOS/nixpkgs/issues/22570.
> WDYT?

Here’s the patch for Nix:

  https://patch-diff.githubusercontent.com/raw/NixOS/nixpkgs/pull/22585.diff

Here are the relevant changes to the Python packages:

* Python 3.4

  substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']" "(1 
if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"
  substituteInPlace "Lib/importlib/_bootstrap.py" --replace "source_mtime = 
int(source_stats['mtime'])" "source_mtime = 1"

* Python 3.5

  substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']" "(1 
if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"
  substituteInPlace "Lib/importlib/_bootstrap_external.py" --replace 
"source_mtime = int(st['mtime'])" "source_mtime = 1"

* Python 3.6
  substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']" "(1 
if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"
  substituteInPlace "Lib/importlib/_bootstrap_external.py" --replace 
"source_mtime = int(st['mtime'])" "source_mtime = 1"


For all packages they set these environment variables:

  - set PYTHONHASHSEED=0 (for hashes of str, bytes and datetime objects)

  - set DETERMINISTIC_BUILD; for conditional patching of the timestamp
for package builds.  The timestamp is not patched in ad-hoc
environments, because that would mess with Python’s ability to
determine whether to compile source files.

They also rebuild all bytecode (with the exception of lib2to3 because it
is Python 2 code) three times, once for each optimization level.

--8<---cut here---start->8---
+# Determinism: rebuild all bytecode
+# We exclude lib2to3 because that's Python 2 code which fails
+# We rebuild three times, once for each optimization level
+find $out -name "*.py" | $out/bin/python -m compileall -q -f -x "lib2to3" 
-i -
+find $out -name "*.py" | $out/bin/python -O -m compileall -q -f -x 
"lib2to3" -i -
+find $out -name "*.py" | $out/bin/python -OO -m compileall -q -f -x 
"lib2to3" -i -
--8<---cut here---end--->8---

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

bug#30655: closed (Re: bug#30655: xset(1) not accepting symlink to font path)

[Ludovic Courtès]

Hi,

Marco van Hulten  skribis:

> Je  2 mrt 12:46 skribis GNU:
>> > diff --git a/doc/guix.texi b/doc/guix.texi
>> > index 24db16761..35221ff94 100644
>> > --- a/doc/guix.texi
>> > +++ b/doc/guix.texi
>> > @@ -1572,7 +1572,7 @@ To be able to use such full names for the
>> > TrueType fonts installed in your Guix profile, you need to extend the
>> > font path of the X server: 
>> >  @example
>> > -xset +fp ~/.guix-profile/share/fonts/truetype
>> > +xset +fp $(dirname $(readlink -f 
>> > ~/.guix-profile/share/fonts/truetype/fonts.dir))
>> >  @end example  
>> 
>> Actually commit 07ec349229 that I just pushed fixes this very line.  So
>> I guess we’re done?
>
> I don't think so, because I just did a 'git pull', and found that the
> old patch had been applied (the one with `readlink... that doesn't
> work, instead of the above with $(dirname $(readlink...) which works.

Oops, really fixed now (will push shortly.)

> Apropos, how often are the HTML pages generated from the repository and
> put online?

At each new release.

Thanks,
Ludo’.

bug#22533: Python bytecode reproducibility

[Ricardo Wurmus]

I have applied this patch locally:

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 5f701701a..0d1ecc3c6 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -359,8 +359,42 @@ data types.")
   "Lib/ctypes/test/test_win32.py" ; fails on aarch64
   "Lib/test/test_fcntl.py")) ; fails on aarch64
   #t
-(arguments (substitute-keyword-arguments (package-arguments python-2)
- ((#:tests? _) #t)))
+(arguments
+ (substitute-keyword-arguments (package-arguments python-2)
+   ((#:tests? _) #t)
+   ((#:phases phases)
+`(modify-phases ,phases
+   (add-after 'unpack 'patch-timestamp-for-pyc-files
+ (lambda _
+   ;; We set DETERMINISTIC_BUILD to only override the mtime when
+   ;; building with Guix, lest we break auto-compilation in
+   ;; environments.
+   (setenv "DETERMINISTIC_BUILD" "1")
+   (substitute* "Lib/py_compile.py"
+ (("source_stats\\['mtime'\\]")
+  "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"))
+
+   ;; Use deterministic hashes for strings, bytes, and datetime
+   ;; objects.
+   (setenv "PYTHONHASHSEED" "0")
+
+   ;; Reset mtime when validating bytecode header.
+   (substitute* "Lib/importlib/_bootstrap_external.py"
+ (("source_mtime = int\\(source_stats\\['mtime'\\]\\)")
+  "source_mtime = 1"))
+   #t))
+   (add-after 'unpack 'disable-timestamp-tests
+ (lambda _
+   (substitute* "Lib/test/test_importlib/source/test_file_loader.py"
+ (("test_bad_marshal")
+  "disable_test_bad_marshal")
+ (("test_no_marshal")
+  "disable_test_no_marshal")
+ (("test_non_code_marshal")
+  "disable_test_non_code_marshal"))
+   #t))
+   (add-before 'check 'allow-non-deterministic-compilation
+ (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t))
 (native-search-paths
  (list (search-path-specification
 (variable "PYTHONPATH")

It allows me to build python-six and python-sip reproducibly.  It does
not fix problems with Python 2, and I haven’t yet tested if it causes
any new problems.

It’s a little worrying that I had to disable three more tests that I
think shouldn’t have failed.

What do you think?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

bug#22533: Python bytecode reproducibility

[Ricardo Wurmus]

Ricardo Wurmus  writes:

> Unfortunately, this doesn’t fix all reproducibility problems with numpy:
>
> --8<---cut here---start->8---
> Binary files 
> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config__.cpython-36.pyc
>  and 
> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config__.cpython-36.pyc
>  differ
> Binary files 
> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_command.cpython-36.pyc
>  and 
> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_command.cpython-36.pyc
>  differ
> Binary files 
> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_info.cpython-36.pyc
>  and 
> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_info.cpython-36.pyc
>  differ
> Binary files 
> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36.pyc
>  and 
> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36.pyc
>  differ
> Binary files 
> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc
>  and 
> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc
>  differ
> Binary files 
> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache__/utils.cpython-36.pyc
>  and 
> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache__/utils.cpython-36.pyc
>  differ
> --8<---cut here---end--->8---

Here’s what diffoscope says:

--8<---cut here---start->8---
diffoscope 
/gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0{-check,}/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc
--- 
/gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc
+++ 
/gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc
@@ -1,8 +1,8 @@
-: 330d 0d0a fa87 9c5a 2601  e300   3..Z&...
+: 330d 0d0a c485 9c5a 2601  e300   3..Z&...
 0010:     0001  0040   .@..
 0020: 0073 2000  6400 5a00 6400 5a01 6400  .s ...d.Z.d.Z.d.
 0030: 5a02 6401 5a03 6402 5a04 6504 731c 6502  Z.d.Z.d.Z.e.s.e.
 0040: 5a01 6403 5300 2904 7a06 312e 3134 2e30  Z.d.S.).z.1.14.0
 0050: da28 3639 3134 6262 3431 6630 6662 3363  .(6914bb41f0fb3c
 0060: 3162 6135 3030 6261 6534 6537 6436 3731  1ba500bae4e7d671
 0070: 6461 3935 3336 3738 3666 544e 2905 da0d  da9536786fTN)...
--8<---cut here---end--->8---

In other words: this is the timestamp field of the pyc file.

Maybe this can be avoided by setting DETERMINISTIC_BUILD in the
python-build-system?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

bug#30706: Nginx service fails

[Ricardo Wurmus]

Andreas Enge  writes:

> Well, I am turning this into a bug report, since it still occurs with
> the latest git commit ac1a9ce8b07f3b80900ee08436ff6e683e8dc195 .
>
> This is the result of "./pre-inst-env guix system reconfigure ...",
> where "..." is my configuration file:
>
> ...
> creating nginx log directory '/var/log/nginx'
> creating nginx run directory '/var/run/nginx'
> creating nginx temp directories 
> '/var/run/nginx/{client_body,proxy,fastcgi,uwsgi,scgi}_temp'
> nginx: [alert] could not open error log file: open() 
> "/gnu/store/pp71iff1qxwhh82vm34g18h9kmn0xrg5-nginx-1.13.9/logs/error.log" 
> failed (2: No such file or directory)
> nginx: the configuration file 
> /gnu/store/5ixkryw6jl32cm6d1g9jb8dm9rbz8csc-nginx.conf syntax is ok
> nginx: configuration file 
> /gnu/store/5ixkryw6jl32cm6d1g9jb8dm9rbz8csc-nginx.conf test is successful
> `/gnu/store/zchh8s3r1bbmia3zfxsyhsz3c4b9fmps-openssh-authorized-keys/root' -> 
> `/etc/ssh/authorized_keys.d/root'
> `/gnu/store/zchh8s3r1bbmia3zfxsyhsz3c4b9fmps-openssh-authorized-keys/andreas' 
> -> `/etc/ssh/authorized_keys.d/andreas'
> guix system: loading new services: user-homes term-auto nginx...
> shepherd: Evaluating user expression (register-services (primitive-load 
> "/gnu/st?") ?).
> guix system: error: exception caught while executing 'eval' on service 'root':
> find-long-options: unbound variable

I had the same error when updating my i686 netbook after a long while.
After a reboot everything seemed to be fine, though.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

bug#30706: Nginx service fails

[Andreas Enge]

On Mon, Mar 05, 2018 at 08:23:18AM +0100, Ricardo Wurmus wrote:
> I had the same error when updating my i686 netbook after a long while.
> After a reboot everything seemed to be fine, though.

Ah, thanks for the information! A reboot made things worse in my case -
I rebooted the virtual machine, and now I cannot ssh into it any more.
So it looks like I will have to set it up from scratch again...

In my case, the problem occurred between February 28 and March 4.

Andreas

bug#30522: Guile-SSH retries upon “guile --listen” failure to start

[Artyom Poptsov]

Hello Ludovic and Marius,

I implemented a fix for the looping problem; could you please check if
the patch works for you?

The changes are committed in e24401f1ef56b1f3d7ef847c54fb0c2fe9cf3cf8 on
the 'master' branch.

Thanks!

- Artyom

-- 
Artyom V. Poptsov 
Home page: http://poptsov-artyom.narod.ru/
CADR Hackerspace co-founder: https://cadrspace.ru/
GPG: D0C2 EAC1 3310 822D 98DE  B57C E9C5 A2D9 0898 A02F


signature.asc
Description: PGP signature