Bug#1055919: python-ansible-pygments: please make the build reproducible
Control: clone 1055919 -1
Control: reassign -1 dh-python 5.20230130+deb12u1
Making a clone of this bug for dh-python to track it getting fixed
there; MR to come.
Context:
On Thu, 16 Nov 2023 17:33:58 +1100 Stuart Prescott
wrote:
> tldr: smells like a dh-python bug - I'll look at it more and reassign
> etc later.
>
>
> Staring at some build logs some more:
>
> * the dirs are generated always
> * they get copied from .../.pybuild to ../debian/$package/ always
> * they are supposed to get removed by dh_python3
> * that removal is not always working
>
> A common theme of the failures is that there are _two_
> /usr/lib/python3.11/dist-packages/.foo directories to remove and only
> one of them is being removed. For python-ansible-pygments, .pytest_cache
> was being removed by dh-python3 but .test-results was not.
>
> Adding in PYBUILD_VERBOSE=1 and some breakpoints into dh-python
> (specifically /usr/share/dh-python/dhpython/fs.py), I think there's a
> subtle bug about altering `dirs` while inside a loop from `os.walk`:
>
> for name in dirs:
> if name in ('test', 'tests') or name.startswith('.'):
> log.debug('removing dist-packages/%s', name)
> rmtree(join(root, name))
> dirs.remove(name)
>
> Removing `name` from `dirs` means that the next item is accidentally
> skipped. A classic "don't change the object you're iterating through
> while you are iterating through it" issue:
>
> In [1]: L = list(range(0, 10))
>
> In [2]: for i in L:
> ...: print(i)
> ...: L.remove(i)
> ...:
> 0
> 2
> 4
> 6
> 8
>
> Which item is not processed in the next iteration by dh_python3 now
> depends on the order in which `os.walk` lists the directories. That is
> presumably dependent on all manner of things (filesystem? collation?
> luck?). On the r-b setup and building by hand I get different results to
> within sbuild (and on the buildd).
>
> In sbuild on ext4, `find -type d` (I have memory that this reflects disk
> order?) has an order in
> ./debian/python3-ansible-pygments/usr/lib/python3.11/dist-packages of:
>
> .test-results ansible_pygments .pytest_cache
>
> while building by hand on tmpfs, I get
>
> ansible_pygments .test-results .pytest_cache
>
> For the former, accidentally skipping the directory after the one that
> gets removed isn't an issue, but for the latter it is.
--
Stuart Prescott http://www.nanonanonano.net/ stu...@nanonanonano.net
Debian Developer http://www.debian.org/ stu...@debian.org
GPG fingerprint 90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7
Bug#1055919: python-ansible-pygments: please make the build reproducible
Stuart Prescott wrote: > I think there's a subtle bug about altering `dirs` while > inside a loop from `os.walk`: I'm quite impressed that you managed to hunt this down — that's very nice work. :) > Which item is not processed in the next iteration by dh_python3 now > depends on the order in which `os.walk` lists the directories. That is > presumably dependent on all manner of things (filesystem? collation? > luck?). It will be dependent entirely on the filesystem being used because `os.walk` merely passes on the underlying filesystem ordering. (The same is true for GNU find as you later query.) Entirely anecdotally, ext4 will be "more orderful", whilst btrfs, possibly due to the way it rebalances its tree data structures, tends to be less so. I raise it only because it can help explain why different folks will get different results locally despite using the same building tools. Building under tmpfs will have different properties as well. Anyway, thank you again. Once this is addressed in dh-python (even as a proposed patch), I'll go through the last few months of bugs that I've filed and see whether they can be closed. Best wishes, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#1055919: python-ansible-pygments: please make the build reproducible
tldr: smells like a dh-python bug - I'll look at it more and reassign
etc later.
Staring at some build logs some more:
* the dirs are generated always
* they get copied from .../.pybuild to ../debian/$package/ always
* they are supposed to get removed by dh_python3
* that removal is not always working
A common theme of the failures is that there are _two_
/usr/lib/python3.11/dist-packages/.foo directories to remove and only
one of them is being removed. For python-ansible-pygments, .pytest_cache
was being removed by dh-python3 but .test-results was not.
Adding in PYBUILD_VERBOSE=1 and some breakpoints into dh-python
(specifically /usr/share/dh-python/dhpython/fs.py), I think there's a
subtle bug about altering `dirs` while inside a loop from `os.walk`:
for name in dirs:
if name in ('test', 'tests') or name.startswith('.'):
log.debug('removing dist-packages/%s', name)
rmtree(join(root, name))
dirs.remove(name)
Removing `name` from `dirs` means that the next item is accidentally
skipped. A classic "don't change the object you're iterating through
while you are iterating through it" issue:
In [1]: L = list(range(0, 10))
In [2]: for i in L:
...: print(i)
...: L.remove(i)
...:
0
2
4
6
8
Which item is not processed in the next iteration by dh_python3 now
depends on the order in which `os.walk` lists the directories. That is
presumably dependent on all manner of things (filesystem? collation?
luck?). On the r-b setup and building by hand I get different results to
within sbuild (and on the buildd).
In sbuild on ext4, `find -type d` (I have memory that this reflects disk
order?) has an order in
./debian/python3-ansible-pygments/usr/lib/python3.11/dist-packages of:
.test-results ansible_pygments .pytest_cache
while building by hand on tmpfs, I get
ansible_pygments .test-results .pytest_cache
For the former, accidentally skipping the directory after the one that
gets removed isn't an issue, but for the latter it is.
--
Stuart Prescott http://www.nanonanonano.net/ stu...@nanonanonano.net
Debian Developer http://www.debian.org/ stu...@debian.org
GPG fingerprint 90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7
Bug#1055919: python-ansible-pygments: please make the build reproducible
On Wed, Nov 15, 2023 at 01:31:26PM +, Chris Lamb wrote: > I would be more than willing to conclude that this is an issue in > tests.reproducible-builds.org setup. However, I am actually seeing > these test files when I build locally as well — and my patch > consequently fixes the "problem". Moreover, I am not using the same > setup as tests.reproducible-builds.org at all. (Can you confirm whether > you see them when building locally?) tests.r-b.o is using pbuilder, while the buildds use sbuild. what do you use? -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ Privacy is a Human Right. (Universal Declaration of Human Rights, article 12.) signature.asc Description: PGP signature
Bug#1055919: python-ansible-pygments: please make the build reproducible
[[ looping reproducible-bui...@lists.alioth.debian.org in on this issue ]] Stuart Prescott wrote: > Is there something about the r-b setup that would cause these > directories to be created and appear in the package? Hm, you know, I think there must be. In fact, connecting a few dots in my head, that now makes three recent reproducibility bugs that the buildds are not, if you can excuse the expression, reproducing: 1. https://bugs.debian.org/1055919 (this bug) 2. https://bugs.debian.org/1053353 3. https://bugs.debian.org/1000401 I would be more than willing to conclude that this is an issue in tests.reproducible-builds.org setup. However, I am actually seeing these test files when I build locally as well — and my patch consequently fixes the "problem". Moreover, I am not using the same setup as tests.reproducible-builds.org at all. (Can you confirm whether you see them when building locally?) I won't file any more until this has been more generally resolved. Hopefully I haven't been filing similar issues recently (#1055920?) and not realising it. (Has a build-dependency changed and then the buildd chroots are out of sync? Is that even possible? Am I?) Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#1055919: python-ansible-pygments: please make the build reproducible
Hi Chris, Whilst working on the Reproducible Builds effort [0], we noticed that python-ansible-pygments could not be built reproducibly. This is because the binary package included .pytest_cache and .test-results directories. Patch attached that removes these after running the tests. I see those directories in the packages built on the r-b machines, but I don't see them in the package in the archive or when rebuilding the package locally. The files exist within the build tree but in /.pybuild/cpython3_3.11/build/ which does not get them installed. Is there something about the r-b setup that would cause these directories to be created and appear in the package? thanks Stuart $ apt download -t sid python3-ansible-pygments $ dpkg-deb --fsys-tarfile python3-ansible-pygments_0.1.1-6_all.deb | tar t ./ ./usr/ ./usr/lib/ ./usr/lib/python3/ ./usr/lib/python3/dist-packages/ ./usr/lib/python3/dist-packages/ansible_pygments/ ./usr/lib/python3/dist-packages/ansible_pygments/__init__.py ./usr/lib/python3/dist-packages/ansible_pygments/lexers.py ./usr/lib/python3/dist-packages/ansible_pygments/styles.py ./usr/lib/python3/dist-packages/ansible_pygments-0.1.1.dist-info/ ./usr/lib/python3/dist-packages/ansible_pygments-0.1.1.dist-info/METADATA ./usr/lib/python3/dist-packages/ansible_pygments-0.1.1.dist-info/RECORD ./usr/lib/python3/dist-packages/ansible_pygments-0.1.1.dist-info/WHEEL ./usr/lib/python3/dist-packages/ansible_pygments-0.1.1.dist-info/entry_points.txt ./usr/share/ ./usr/share/doc/ ./usr/share/doc/python3-ansible-pygments/ ./usr/share/doc/python3-ansible-pygments/changelog.Debian.gz ./usr/share/doc/python3-ansible-pygments/copyright ./usr/share/doc/python3-ansible-pygments/examples/ ./usr/share/doc/python3-ansible-pygments/examples/lexer_test.py -- Stuart Prescott http://www.nanonanonano.net/ stu...@nanonanonano.net Debian Developer http://www.debian.org/ stu...@debian.org GPG fingerprint 90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7
Bug#1055919: python-ansible-pygments: please make the build reproducible
Source: python-ansible-pygments Version: 0.1.1-6 Severity: wishlist Tags: patch User: reproducible-bui...@lists.alioth.debian.org Usertags: timestamps X-Debbugs-Cc: reproducible-b...@lists.alioth.debian.org Hi, Whilst working on the Reproducible Builds effort [0], we noticed that python-ansible-pygments could not be built reproducibly. This is because the binary package included .pytest_cache and .test-results directories. Patch attached that removes these after running the tests. [0] https://reproducible-builds.org/ Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `- --- a/debian/rules 2023-11-14 09:25:37.237065158 + --- b/debian/rules 2023-11-14 09:28:11.611381883 + @@ -4,3 +4,6 @@ %: dh $@ --buildsystem=pybuild + +execute_after_dh_auto_test: + find .pybuild -type d \( -name .pytest_cache -or -name .test-results \) -print0 | xargs -0r rm -rfv
