Bug#1036214: Tracking gets confused by using copysrc on binnmu'ed packages
For Kali Linux I just ran "reprepro retrack", which was enough to get rid of all the unwanted packages. The command took 15 minutes to run, and the Kali repo went from 806G to 439G -- not bad! I will run this command manually every few weeks, and if it proves to be reliable, I will probably let cron run it once a week or something like that. -- Arnaud Rebillout / OffSec / Kali Linux Developer
Bug#1036214: Tracking gets confused by using copysrc on binnmu'ed packages
Hello, On Wed, 17 May 2023, Christoph Berg wrote: > Ever since, I've seen warnings from reprepro that the tracking > database was missing references when copysrc'ing a new version into a > distribution when the previous version in there had been binnmued. But > since everything worked as expected, I didn't pay attention. > > Now I have been inspecting the pool directory more closely, and I > found a lot of +b1.deb files that should long have been removed since > they were replaced by newer versions, e.g.: We have the same problem in Kali, our mirror is growing in size due to files that are not removed properly. Many of the files were bin-NMU but not exclusively. We also have the issue with the binary packages from android-platform-tools for example. And looking at this source package we can see that there's a mismatch between the source version and the binary version. The rules files uses `dh_gencontrol -p$PKG -- -v1:$(DEB_VERSION)` to add an epoch to the binary packages which is not present in the source package. https://salsa.debian.org/android-tools-team/android-platform-tools/-/blob/master/debian/rules?ref_type=heads#L217 So the problem seems to be generalized to all packages that have a different version compared to the source packages (a few more packages affected on our side: binutils-bpf, binutils-arm-none-eabi, many more binutils-* in fact, linux with bpftool, bsdutils, llvm-defaults, gcc, etc.) > The problem seems to never happen on *-pgdg-testing (which gets filled > by processincoming), only on *-pgdg (which gets filled by copysrc), so > I suspect the problem to be in that area. On our side, we have a kali-rolling distribution that is built manually out of kali-dev with many "reprepro copy" and "reprepro removefilter" calls (the set of commands to run is built out of analysis of the current distribution and the comparison with the "heidi" file generated by britney which defines the desired end-state). That it seems likely that the "copy" command is affected/involved as well. In our case, both kali-rolling and kali-dev have "Tracking: minimal includechanges includebyhand includebuildinfos" in conf/distributions. kali-dev is updated with a "pull" rule that combines a mirror of debian testing (created with an "update" rule) and of kali specific packages (which are uploaded and installed via processincoming). It would be nice to see this fixed. I'm happy to pitch a bounty or suggest anyone to apply for https://salsa.debian.org/freexian-team/project-funding to tackle a few long standing bugs/feature request on reprepro. Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS
Bug#1036214: Tracking gets confused by using copysrc on binnmu'ed packages
Package: reprepro Version: 5.3.0-1.2 Severity: normal Hi, we are using reprepro for apt.postgresql.org. It's been running very stable over the past years, so thanks for that. The workflow is that every distribution exists twice: sid-pgdg sid-pgdg-testing bullseye-pgdg bullseye-pgdg-testing ... Uploaded packages go into *-pgdg-testing (reprepro processincoming on .changes files) and after inspection that everything still works as expected, they get copied over to *-pgdg: reprepro copysrc sid-pgdg sid-pgdg-testing postgresql-foo (This works even in the presence of binary cruft, PostgreSQL extension packages are compiled per PG major version. Once some PG version NN goes out of support, src:postgresql-foo will no longer build bin:postgresql-NN-foo, but we keep them around in the repo as long as they are still installable.) Ever since, I've seen warnings from reprepro that the tracking database was missing references when copysrc'ing a new version into a distribution when the previous version in there had been binnmued. But since everything worked as expected, I didn't pay attention. Now I have been inspecting the pool directory more closely, and I found a lot of +b1.deb files that should long have been removed since they were replaced by newer versions, e.g.: postgresql-11_11.5-3.pgdg110+1+b1_amd64.deb <-- old postgresql-11_11.20-1.pgdg110+1_amd64.deb <-- current The old version is not listed by "reprepro ls postgresql-11". It is reported by "reprepro reportcruft bullseye-pgdg": binaries-without-source bullseye-pgdg postgresql-10 10.10-2.pgdg110+1+b1 But since that list also includes "proper" cruft (binaries that are no longer built from a source package) that we want to keep, this isn't the best command to find these. What works is to use removetrack: $ reprepro --verbose removetrack bullseye-pgdg postgresql-11 11.5-3.pgdg110+1+b1 Removed postgresql-11_11.5-3.pgdg110+1+b1 from bullseye-pgdg. Deleting files no longer referenced... deleting and forgetting pool/main/p/postgresql-11/postgresql-11-dbgsym_11.5-3.pgdg110+1+b1_amd64.deb deleting and forgetting pool/main/p/postgresql-11/postgresql-11-dbgsym_11.5-3.pgdg110+1+b1_ppc64el.deb ... The problem seems to never happen on *-pgdg-testing (which gets filled by processincoming), only on *-pgdg (which gets filled by copysrc), so I suspect the problem to be in that area. Christoph
