Re: reproducible .pyc files (& python-for-android)

[Felix C. Stegerman]

* "Bernhard M. Wiedemann"  [2021-01-04 12:48]:
> This is not a timestamp issue, though. If those are varying, they are in
> the header (first 12 bytes) of the .pyc.
> 
> 
> │  00f0: 6d5a 0d62 6469 7374 5f77 696e 696e 7374  mZ.bdist_wininst
> │ -0100: 5a05 6368 6563 6b5a 0675 706c 6f61 644e  Z.checkZ.uploadN
> │ +0100: da05 6368 6563 6b5a 0675 706c 6f61 644e  ..checkZ.uploadN
> 
> 
> I have seen this before and remember something about python string
> reference counters being dumped into these pickle files and that varied
> from ordering, so that
> py_compile py1.py py2.py
> produced different results than
> py_compile py2.py py1.py
> 
> One way to get reproducible results is to delete and recreate all .pyc
> files with
> find -type f -a -name "*.py" -print0 |
>   sort -z |
>   xargs -0 $python_binary -m py_compile
> 
> 
> Maybe related: creating .pyc files on i586 and x86_64 (with identical
> toolchain) always produced different results for me.

I was finally able to reproduce this on another machine.  This
particular difference is caused by whether the system has liblzma-dev
installed or not (when Python is built).

- Felix

Re: reproducible .pyc files (& python-for-android)

[Felix C. Stegerman]

Hi Frederik,

* Frederik Rietdijk  [2021-01-04 14:48]:
> Recently I spent some time again as well on making the Python interpreters
> in Nixpkgs build reproducibly. The following Nix expression results in
> deterministic builds of the 3.x interpreters we have. Search for
> `determinis` and you will see the changes we do to get there. Maybe it's of
> help to you.
> 
> https://github.com/NixOS/nixpkgs/blob/f19ad43d674e5bbaa70a000f65031ab09a32a338/pkgs/development/interpreters/python/cpython/default.nix

Thanks for the link!  I don't really see anything significantly
different to what I'm doing though.

The issue I had w/ .pyc files was probably caused by something
specific to the GitHub Actions Ubuntu image.

I'm still kind of curious as to what that is, but I don't think it's
currently worth investigating since everything works as expected using
a Debian buster container/VM.

- Felix

Re: reproducible .pyc files (& python-for-android)

[Leo Wandersleb]

> In Debian, Holger Levsen uploaded 540 packages ...

I sort of get what this means I think ... Maybe this could be dumbed down a bit,
especially as the in-depth links are there.


On 1/2/21 11:25 AM, Felix C. Stegerman wrote:
> Hi!
>
> I'm trying to get python-for-android (p4a) to build reproducible
> .APKs [1].
>
> At the moment I'm trying to figure out why I'm getting different .pyc
> files (python bytecode) on Github Actions vs on a local VM (both
> Ubuntu 20.04).
>
> p4a compiles those with "hostpython -OO -m compileall -b -f" (where
> hostpython is the cross-compiled Python for the target -- arm64-v8a or
> armeabi-v7a -- which is thus definitely the same version on both
> machines).
>
> diffoscope outputs e.g.
>
> ├── distutils/command/__init__.pyc
> │ @@ -10,17 +10,17 @@
> │  0090: 6962 5a0f 696e 7374 616c 6c5f 6865 6164  ibZ.install_head
> │  00a0: 6572 735a 0f69 6e73 7461 6c6c 5f73 6372  ersZ.install_scr
> │  00b0: 6970 7473 5a0c 696e 7374 616c 6c5f 6461  iptsZ.install_da
> │  00c0: 7461 5a05 7364 6973 74da 0872 6567 6973  taZ.sdist..regis
> │  00d0: 7465 725a 0562 6469 7374 5a0a 6264 6973  terZ.bdistZ.bdis
> │  00e0: 745f 6475 6d62 5a09 6264 6973 745f 7270  t_dumbZ.bdist_rp
> │  00f0: 6d5a 0d62 6469 7374 5f77 696e 696e 7374  mZ.bdist_wininst
> │ -0100: 5a05 6368 6563 6b5a 0675 706c 6f61 644e  Z.checkZ.uploadN
> │ -0110: 2901 da07 5f5f 616c 6c5f 5fa9 0072 0400  )...__all__..r..
> │ -0120:  7204  00fa af2f 686f 6d65 2f72  ..r../home/r
> │ +0100: da05 6368 6563 6b5a 0675 706c 6f61 644e  ..checkZ.uploadN
> │ +0110: 2901 da07 5f5f 616c 6c5f 5fa9 0072 0500  )...__all__..r..
> │ +0120:  7205  00fa af2f 686f 6d65 2f72  ..r../home/r
> │  0130: 756e 6e65 722f 776f 726b 2f6a 6974 656e  unner/work/jiten
> │  0140: 2f6a 6974 656e 2f61 6e64 726f 6964 2f2e  /jiten/android/.
> │  0150: 6275 696c 646f 7a65 722f 616e 6472 6f69  buildozer/androi
> │  0160: 642f 706c 6174 666f 726d 2f62 7569 6c64  d/platform/build
> │  0170: 2d61 726d 3634 2d76 3861 2f62 7569 6c64  -arm64-v8a/build
> │  0180: 2f6f 7468 6572 5f62 7569 6c64 732f 7079  /other_builds/py
> │  0190: 7468 6f6e 332f 6172 6d36 342d 7638 615f  thon3/arm64-v8a_
>
> I'm hoping someone on this list knows how to fix this or can point me
> in the right direction.
>
> Thanks.
>
> - Felix
>
> [1] https://github.com/kivy/python-for-android/pull/2390

Re: reproducible .pyc files (& python-for-android)

[Leo Wandersleb]

The section

> Last month we reported  on a
fork of the official German Corona App called /Corona Warn App/
. Since then, the
application is now available on the F-Droid free-software app store
 (without integration with Google-operated services).
However, the version on /F-Droid/ also supports reproducible builds, and
instructions on how to rebuild the package

are available from the upstream Git repository
.
 (FSFE’s
announcement. )

could be more clear. I'm confused. Is this a working Corona App or is in
incompatible with the "official" one? If not, it's kind of not good news for
reproducible builds so this detail matters.
Also the first sentence is not sufficiently clear about whether the link is to
the upstream repo or to the fork repo.

On 1/2/21 11:25 AM, Felix C. Stegerman wrote:
> Hi!
>
> I'm trying to get python-for-android (p4a) to build reproducible
> .APKs [1].
>
> At the moment I'm trying to figure out why I'm getting different .pyc
> files (python bytecode) on Github Actions vs on a local VM (both
> Ubuntu 20.04).
>
> p4a compiles those with "hostpython -OO -m compileall -b -f" (where
> hostpython is the cross-compiled Python for the target -- arm64-v8a or
> armeabi-v7a -- which is thus definitely the same version on both
> machines).
>
> diffoscope outputs e.g.
>
> ├── distutils/command/__init__.pyc
> │ @@ -10,17 +10,17 @@
> │  0090: 6962 5a0f 696e 7374 616c 6c5f 6865 6164  ibZ.install_head
> │  00a0: 6572 735a 0f69 6e73 7461 6c6c 5f73 6372  ersZ.install_scr
> │  00b0: 6970 7473 5a0c 696e 7374 616c 6c5f 6461  iptsZ.install_da
> │  00c0: 7461 5a05 7364 6973 74da 0872 6567 6973  taZ.sdist..regis
> │  00d0: 7465 725a 0562 6469 7374 5a0a 6264 6973  terZ.bdistZ.bdis
> │  00e0: 745f 6475 6d62 5a09 6264 6973 745f 7270  t_dumbZ.bdist_rp
> │  00f0: 6d5a 0d62 6469 7374 5f77 696e 696e 7374  mZ.bdist_wininst
> │ -0100: 5a05 6368 6563 6b5a 0675 706c 6f61 644e  Z.checkZ.uploadN
> │ -0110: 2901 da07 5f5f 616c 6c5f 5fa9 0072 0400  )...__all__..r..
> │ -0120:  7204  00fa af2f 686f 6d65 2f72  ..r../home/r
> │ +0100: da05 6368 6563 6b5a 0675 706c 6f61 644e  ..checkZ.uploadN
> │ +0110: 2901 da07 5f5f 616c 6c5f 5fa9 0072 0500  )...__all__..r..
> │ +0120:  7205  00fa af2f 686f 6d65 2f72  ..r../home/r
> │  0130: 756e 6e65 722f 776f 726b 2f6a 6974 656e  unner/work/jiten
> │  0140: 2f6a 6974 656e 2f61 6e64 726f 6964 2f2e  /jiten/android/.
> │  0150: 6275 696c 646f 7a65 722f 616e 6472 6f69  buildozer/androi
> │  0160: 642f 706c 6174 666f 726d 2f62 7569 6c64  d/platform/build
> │  0170: 2d61 726d 3634 2d76 3861 2f62 7569 6c64  -arm64-v8a/build
> │  0180: 2f6f 7468 6572 5f62 7569 6c64 732f 7079  /other_builds/py
> │  0190: 7468 6f6e 332f 6172 6d36 342d 7638 615f  thon3/arm64-v8a_
>
> I'm hoping someone on this list knows how to fix this or can point me
> in the right direction.
>
> Thanks.
>
> - Felix
>
> [1] https://github.com/kivy/python-for-android/pull/2390

Re: reproducible .pyc files (& python-for-android)

[Frederik Rietdijk]

Recently I spent some time again as well on making the Python interpreters
in Nixpkgs build reproducibly. The following Nix expression results in
deterministic builds of the 3.x interpreters we have. Search for
`determinis` and you will see the changes we do to get there. Maybe it's of
help to you.

https://github.com/NixOS/nixpkgs/blob/f19ad43d674e5bbaa70a000f65031ab09a32a338/pkgs/development/interpreters/python/cpython/default.nix

On Mon, Jan 4, 2021 at 2:41 PM Felix C. Stegerman  wrote:

> Hi Bernhard (& Chris),
>
> * "Bernhard M. Wiedemann"  [2021-01-04 12:48]:
> > Am 04.01.21 um 11:23 schrieb Chris Lamb:
> > >> p4a compiles those with "hostpython -OO -m compileall -b -f" (where
> > >> hostpython is the cross-compiled Python for the target -- arm64-v8a or
> > >> armeabi-v7a -- which is thus definitely the same version on both
> > >> machines).
> > >
> > > As I understand it, recent versions Python can use SOURCE_DATE_EPOCH
> > > in its internal py_compile routine to ensure that .pyc files are
> > > reproducible.
> >
> > This is not a timestamp issue, though. If those are varying, they are in
> > the header (first 12 bytes) of the .pyc.
>
> Indeed.  I should perhaps have mentioned that I am indeed setting
> SOURCE_DATE_EPOCH (though that is noted in the PR I [1] linked).  For
> reference: the Python version is 3.9.1.
>
> > │  00f0: 6d5a 0d62 6469 7374 5f77 696e 696e 7374  mZ.bdist_wininst
> > │ -0100: 5a05 6368 6563 6b5a 0675 706c 6f61 644e  Z.checkZ.uploadN
> > │ +0100: da05 6368 6563 6b5a 0675 706c 6f61 644e  ..checkZ.uploadN
> >
> > I have seen this before and remember something about python string
> > reference counters being dumped into these pickle files and that varied
> > from ordering, so that
> > py_compile py1.py py2.py
> > produced different results than
> > py_compile py2.py py1.py
>
> That's interesting.  But p4a is running "hostpython -OO -m compileall
> -b -f $DIRECTORY" -- and compileall walks the directory tree in sorted
> order -- so I don't think that's the problem here.
>
> I have currently sidestepped the issue by using a Debian buster docker
> container (with usrmerge installed) in GitHub Actions (instead of
> the default Ubuntu image).  I now get identical .apks there and on a
> local buster VM (with the same build path) \o/
>
> If anyone is interested in the modifications I had to make to p4a to
> get it to build reproducibly: see the PR [1].  It also has comments on
> what does and does not (yet) work.
>
> Thanks.
>
> - Felix
>
> [1] https://github.com/kivy/python-for-android/pull/2390
>
> P.S. @Chris as a longtime Debian sid user I would also consider that
> my 'home' distribution :)
>

Re: reproducible .pyc files (& python-for-android)

[Felix C. Stegerman]

Hi Bernhard (& Chris),

* "Bernhard M. Wiedemann"  [2021-01-04 12:48]:
> Am 04.01.21 um 11:23 schrieb Chris Lamb:
> >> p4a compiles those with "hostpython -OO -m compileall -b -f" (where
> >> hostpython is the cross-compiled Python for the target -- arm64-v8a or
> >> armeabi-v7a -- which is thus definitely the same version on both
> >> machines).
> >
> > As I understand it, recent versions Python can use SOURCE_DATE_EPOCH
> > in its internal py_compile routine to ensure that .pyc files are
> > reproducible.
>
> This is not a timestamp issue, though. If those are varying, they are in
> the header (first 12 bytes) of the .pyc.

Indeed.  I should perhaps have mentioned that I am indeed setting
SOURCE_DATE_EPOCH (though that is noted in the PR I [1] linked).  For
reference: the Python version is 3.9.1.

> │  00f0: 6d5a 0d62 6469 7374 5f77 696e 696e 7374  mZ.bdist_wininst
> │ -0100: 5a05 6368 6563 6b5a 0675 706c 6f61 644e  Z.checkZ.uploadN
> │ +0100: da05 6368 6563 6b5a 0675 706c 6f61 644e  ..checkZ.uploadN
>
> I have seen this before and remember something about python string
> reference counters being dumped into these pickle files and that varied
> from ordering, so that
> py_compile py1.py py2.py
> produced different results than
> py_compile py2.py py1.py

That's interesting.  But p4a is running "hostpython -OO -m compileall
-b -f $DIRECTORY" -- and compileall walks the directory tree in sorted
order -- so I don't think that's the problem here.

I have currently sidestepped the issue by using a Debian buster docker
container (with usrmerge installed) in GitHub Actions (instead of
the default Ubuntu image).  I now get identical .apks there and on a
local buster VM (with the same build path) \o/

If anyone is interested in the modifications I had to make to p4a to
get it to build reproducibly: see the PR [1].  It also has comments on
what does and does not (yet) work.

Thanks.

- Felix

[1] https://github.com/kivy/python-for-android/pull/2390

P.S. @Chris as a longtime Debian sid user I would also consider that
my 'home' distribution :)

Re: reproducible .pyc files (& python-for-android)

[Chris Lamb]

Hi Felix,

> p4a compiles those with "hostpython -OO -m compileall -b -f" (where
> hostpython is the cross-compiled Python for the target -- arm64-v8a or
> armeabi-v7a -- which is thus definitely the same version on both
> machines).

As I understand it, recent versions Python can use SOURCE_DATE_EPOCH
in its internal py_compile routine to ensure that .pyc files are
reproducible.

You may find more info here:

  https://bugs.python.org/issue29708

In the first instance, could you try exporting SOURCE_DATE_EPOCH
and/or confirming which Python versions you are working with?

I suspect that Bernhard (who worked on the above issue and associated
PR for cpython (#296) may be able to shed more light on this. I
personally do have any much insight on .pyc reproducibility beyond
this as Debian, my 'home' distribution, does not ship these files in
its binary artifacts. PEP 552 may also be useful but, again, this is
not quite my area.


Best wishes,

--
  o
⬋   ⬊  Chris Lamb
   o o reproducible-builds.org 
⬊   ⬋
  o