Bug#893802: diffoscope: missing Depends: python3-distutils
Package: diffoscope Version: 91 Severity: serious User: helm...@debian.org Usertags: rebootstrap In a fresh sid debootstrap, after apt-get install --no-install-recommends diffoscope, you get: $ diffoscope --help Traceback (most recent call last): File "/usr/bin/diffoscope", line 33, in from diffoscope.main import main File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 32, in from .tools import tool_prepend_prefix, tool_required, OS_NAMES, get_current_os File "/usr/lib/python3/dist-packages/diffoscope/tools.py", line 29, in from distutils.spawn import find_executable ModuleNotFoundError: No module named 'distutils' $ This is due to python3.6/3.6.5~rc1-2 having dropped its dependency on python3-distutils. I guess we need to add disutils to install_requires in setup.py to fix this issue. Helmut ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#891867: diffoscope: improve .changes diffs
Control: tags -1 - patch On Thu, Mar 01, 2018 at 09:00:27PM +, Chris Lamb wrote: > Why not .strip() them all instead, out of interest? Feel free to do so. I just felt that changing the type from str to List(str) would be non-trivial and I couldn't immediately figure out whether that works as intended. It would have the additional benefit of also removing the indentation. When removing the first line you get the very same problem with indentation rather than newline. Diff these against one another: "line1\n line2\n line3" "line1\n line2" "line2\n line3" So yeah, we need a better solution here. Helmut ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#891867: diffoscope: improve .changes diffs
Package: diffoscope
Version: 91
Severity: wishlist
Tags: patch
Hi awesome reproducible team!
I'm working on adding more build profiles and diffoscope + reproducible
builds turn out to be an awesome tool in validating that my profiles
only drop packages and never change package contents. :)
Now I've run into a strange corner case with diffing .changes files. If
the last entry in a Checksums-Sha256 field is removed, new last entry is
removed and added. For example:
├── Files
│ @@ -1,12 +1,8 @@
│
│ a6434caf7853f0e84013e6004b934351 10496 debug optional
cracklib-runtime-dbgsym_2.9.2-5.1_amd64.deb
│ 93d547da959c0080bb45cc05d354539f 148972 admin optional
cracklib-runtime_2.9.2-5.1_amd64.deb
│ - 608dbc69affb8158f7f203e3c83e5595 11084 libs optional
cracklib2_2.9.2-5.1_amd64.buildinfo
│ + 1941a0fd2dbf8b3df8536caab1895993 9042 libs optional
cracklib2_2.9.2-5.1_amd64.buildinfo
│ b2010c8e3498d469ae3b5bf5e5469da1 21416 debug optional
libcrack2-dbgsym_2.9.2-5.1_amd64.deb
│ f099f5629a8b0cad9e5fba5ded9f88d9 32052 libdevel extra
libcrack2-dev_2.9.2-5.1_amd64.deb
│ e3b100bcf99561c965cc5da8ef58e206 120108 debian-installer optional
libcrack2-udeb_2.9.2-5.1_amd64.udeb
│ - e04c3f5f14a2c8eb2022ba8eb1d5d915 54788 libs optional
libcrack2_2.9.2-5.1_amd64.deb
│ - 85fa278ff743cf93ead1075664df42f2 12880 debug optional
python-cracklib-dbgsym_2.9.2-5.1_amd64.deb
│ - 674e3f26f4301f5c253fddb0366a3832 23392 python optional
python-cracklib_2.9.2-5.1_amd64.deb
│ - c1bceeb542a7dbed7d2989fd93bba60b 14160 debug optional
python3-cracklib-dbgsym_2.9.2-5.1_amd64.deb
│ - b9868203bdda021286129428b5eb3e55 23424 python optional
python3-cracklib_2.9.2-5.1_amd64.deb
│ + e04c3f5f14a2c8eb2022ba8eb1d5d915 54788 libs optional
libcrack2_2.9.2-5.1_amd64.deb
The DotChangesFile comaprator performs a super call into DebControlFile,
which pulls the Checksums-Sha256 field out of the parsed deb822 file. In
doing so it strips the trailing newline and this what makes for the
difference above. I propose working around the issue with the following
patch:
--- a/diffoscope/comparators/debian.py
+++ b/diffoscope/comparators/debian.py
@@ -142,16 +142,16 @@
# Compare Files as string
if self.deb822.get('Files'):
differences.append(Difference.from_text(
-self.deb822.get_as_string('Files'),
-other.deb822.get_as_string('Files'),
+self.deb822.get_as_string('Files') + "\n",
+other.deb822.get_as_string('Files') + "\n",
self.path,
other.path,
source='Files',
))
else:
differences.append(Difference.from_text(
-self.deb822.get_as_string('Checksums-Sha256'),
-other.deb822.get_as_string('Checksums-Sha256'),
+self.deb822.get_as_string('Checksums-Sha256') + "\n",
+other.deb822.get_as_string('Checksums-Sha256') + "\n",
self.path,
other.path,
source='Checksums-Sha256',
Many thanks to Mattia Rizzolo for pointing precisely at the relevant
source code.
Helmut
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#887180: diffoscope should depend on e2fsprogs explicitly
Package: diffoscope Version: 90 User: helm...@debian.org Usertags: nonessentiale2fsprogs Dear maintainer, We want to make removing e2fsprogs from installations possible. For standard installations this is not useful, but embedded applications and chroots benefit from such an option. For getting there all packages that use e2fsprogs must be identified and gain a dependency on it as e2fsprogs currently is essential. diffoscope was identified as potentially needing such a dependency, because it mentions tool names from e2fsprogs in the following files: /usr/lib/python3/dist-packages/diffoscope/comparators/directory.py contains lsattr. According to file it is a C++ source, UTF-8 Unicode text /usr/lib/python3/dist-packages/diffoscope/external_tools.py contains lsattr. According to file it is a UTF-8 Unicode text Please investigate whether these cases are actually uses of a tool from e2fsprogs. Care has been taken to shrink the number of candidates as much as possible, but a few false positives will remain. After doing so, do one of the following: * Add e2fsprogs to Depends. * Add e2fsprogs to Recommends. * Close this bug explaining why e2fsprogs is not used by this package. Once e2fsprogs drops the "Essential: yes" flag, this bug will be upgraded to RC severity. Please note that lintian will warn about such a dependency before lintian 2.5.56. Thanks for your help Helmut ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#865660: diffoscope uses 35GB ram for comparing influxdb_1.1.1+dfsg1-4_arm64.changes
Hi Chris, On Fri, Jun 23, 2017 at 03:50:54PM +0100, Chris Lamb wrote: > > Unfortunately, I cannot provide any more details at this point. Sorry. > > Can you at least provide the version of diffoscope you are using? It wasn't me. It was jenkins.debian.net. So I actually tried looking up the version, but I was too late and the process finished. As far as I can tell, the reproducible chroots always use the latest diffoscope. Most likely they were using 83. Hope that helps. If it doesn't, close it. Helmut ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#831443: dh-strip-nondeterminism: "Out of memory" (on 4G RAM) while while processing a png in glib2.0
Package: dh-strip-nondeterminism Version: 0.019-1 Severity: important User: helm...@debian.org Usertags: rebootstrap I noticed that building glib2.0 (native or cross) started to fail for me on some systems. The relevant part of the build log always is: | dh_strip_nondeterminism -plibglib2.0-tests | Out of memory! | /usr/share/cdbs/1/rules/debhelper.mk:302: recipe for target 'binary-strip-IMPL/libglib2.0-tests' failed | make: *** [binary-strip-IMPL/libglib2.0-tests] Error 1 | dpkg-buildpackage: error: fakeroot debian/rules binary-arch gave error exit status 2 So I reran dh_strip_nondeterminism under strace and saw mmap(NULL, 4294971392, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) as the last action before printing the "Out of memory" message. The last file opened was debian/libglib2.0-tests/usr/lib/glib2.0/installed-tests/glib/thumbnails/huge-chunk-size.png which has a size of 512 bytes. If you try reproducing the issue, "ulimit -v $((1024*1024*4))" can be useful. I'll be using ln -sf /bin/true /usr/bin/dh_strip_nondeterminism for now unless you can propose something better. I was also thinking that since these files are deliberately broken, maybe glib2.0 should explicitly --exclude them from dh_strip_nondeterminism? Helmut ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#826700: strip-nondeterminism: missing Multi-Arch: foreign, breaks cross building of gdbm
Package: strip-nondeterminism
Version: 0.018-1
Severity: normal
Tags: patch
User: helm...@debian.org
Usertags: rebootstrap
Hi reproducible folks,
gdbm started Build-Depending on strip-nondeterminism. Since that package
is Arch:all + Multi-Arch:no (implicitly), the Build-Depends is never
satisfiable in a cross compilation setting.
There are two ways to fix this:
1) Mark strip-nondeterminism Multi-Arch: foreign.
2) Remove Build-Depends: strip-nondeterminism from gdbm.
There are strong clues that the first option is correct:
* strip-nondeterminism essentially does the same as
dh-strip-nondeterminism and the latter is a dependency of debhelper,
which is Multi-Arch foreign. So from the perspective of most packages
the functionality of strip-nondeterminism is implicitly treated as if
it were Multi-Arch: foreign already.
* strip-nondeterminism is architecture-independent and uses only
architecture-independent perl modules. It does not have maintainer
scripts either (which would have been a common source for being not
Multi-Arch: foreign).
I therefore attach a patch to add that marking. If you disagree, please
reassign this bug to src:gdbm to have that Build-Depends removed.
Helmut
diff --minimal -Nru strip-nondeterminism-0.018/debian/changelog
strip-nondeterminism-0.018/debian/changelog
--- strip-nondeterminism-0.018/debian/changelog 2016-05-30 21:07:34.0
+0200
+++ strip-nondeterminism-0.018/debian/changelog 2016-06-08 05:55:14.0
+0200
@@ -1,3 +1,11 @@
+strip-nondeterminism (0.018-1.1) UNRELEASED; urgency=medium
+
+ * Non-maintainer upload.
+ * Mark dh-strip-nondeterminism and strip-nondeterminism Multi-Arch: foreign
+(Closes: #-1)
+
+ -- Helmut Grohne <hel...@subdivi.de> Wed, 08 Jun 2016 05:54:22 +0200
+
strip-nondeterminism (0.018-1) unstable; urgency=medium
* New upstream release:
diff --minimal -Nru strip-nondeterminism-0.018/debian/control
strip-nondeterminism-0.018/debian/control
--- strip-nondeterminism-0.018/debian/control 2016-05-30 21:07:03.0
+0200
+++ strip-nondeterminism-0.018/debian/control 2016-06-08 05:53:02.0
+0200
@@ -40,6 +40,7 @@
libfile-stripnondeterminism-perl (= ${binary:Version}),
${misc:Depends},
${perl:Depends},
+Multi-Arch: foreign
Description: file non-deterministic information stripper — stand-alone tool
StripNondeterminism is a library for stripping non-deterministic
information, such as timestamps and file system order, from files. It
@@ -59,6 +60,7 @@
libtimedate-perl,
${misc:Depends},
${perl:Depends},
+Multi-Arch: foreign
Description: file non-deterministic information stripper — Debhelper add-on
StripNondeterminism is a library for stripping non-deterministic
information, such as timestamps and file system order, from files. It
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#813052: Bug#813052: Bug#813052: diffoscope takes more than an hour on foreign arch libc6
Hi Lunar, On Fri, Feb 05, 2016 at 01:00:51PM +0100, Jérémy Bobbio wrote: > Still, it was indeed looking at all files in the tree. Could you try the > attached patch and see if it helps? As I said, I cannot reproduce the issue outside profitbricks-build4-amd64.d.n. Still I have a high confidence that the recursion is indeed the cause and would thus go for the patch. Given the currently large number of regressions in unstable, I have to give regression fixes high priority in order to avoid being blinded. So I'd rather implement a workaround than try patching diffoscope there. Sorry, testing this is infeasible at the moment. Helmut ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#813052: Bug#813052: diffoscope takes more than an hour on foreign arch libc6
Hi Lunar, On Fri, Jan 29, 2016 at 03:11:55PM +0100, Jérémy Bobbio wrote: > Helmut Grohne: > > Even though I cannot reproduce the issue at hand, I think that the code > > adding automatic debug symbols looks fishy to me. It appears to recurse > > over /tmp here and that looks very wrong to me. > > I don't understand what you mean by that. Could you provide be (at least > some) of the `--debug` output? What I mean is that diffoscope takes the directory that contains the first debian package and then recursively looks at all contained files. If that tree happens to be big, bad things can happen. So I finally managed to reproduce that bit and I'll give a recipe and a --debug log. chroot into a fresh sid bootstrap. # apt-get --no-install-recommends install diffoscope binutils-multiarch $ cd /tmp $ mkdir -p buildd/diffoscope/should/not/be/looking snapshot $ echo no > buildd/diffoscope/should/not/be/looking/here $ cd buildd Now obtain a full set of binary packages from an arch-only glibc build. $ cd ../snapshot Obtain the corresponding libc6_*.deb from snapshot.d.o. $ cd ../buildd $ diffoscope --debug --text out ./libc6_*.deb /tmp/snapshot/libc6_*.deb 2>debug The run finishes quickly (< 3 minutes) and the debug log contains: | DEBUG Looking for a dbgsym package for Build Id ... | DEBUG Using TextFile for ./diffoscope/should/not/be/looking/here Now for the profitbricks node, what diffoscope looks at is a build tree for glibc. Recursively. Note that I said "./libc6_*.deb" above. If you drop the "./", diffoscope doesn't look where it's not supposed to look. Helmut debug.xz Description: Binary data ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#813052: Bug#813052: diffoscope takes more than an hour on foreign arch libc6
Hi Holger, On Fri, Jan 29, 2016 at 02:08:53AM +0100, Holger Levsen wrote: > to be clear: there is nothing running on profitbricks-build4-amd64 except > Helmut's jobs, which I assume haven't changed on the 24th??? Correct. I'm watching top atm and see diffoscope exploding in memory. 100MB resident after a minute. 600MB resident after 4 minutes. 1.7G after 6 minutes. 2.2G after 8 minutes. 2.7G after 10 minutes. So yeah, diffoscope really is the culprit here. Helmut ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#813052: diffoscope takes more than an hour on foreign arch libc6
Package: diffoscope Version: 47 User: helm...@debian.org Usertags: rebootstrap Hi reproducible team, I wish I could give you a detailed bug report, but instead you get a very vague one. Sorry about that. Since the 24th I am observing that all rebootstrap_*_diffoscope jobs on jenkins.d.n fail no later than running diffoscope on libc6. Given the date, we can be pretty sure that the regression was introduced precisely in version 47. The actual failure is rather unspecific. diffoscope is killed by timeout with SIGKILL after running for more than an hour. You can see an example failure log at https://jenkins.debian.net/job/rebootstrap_arm64_gcc5_supported_diffoscope/27/console One also sees that it works just fine on libc6-pic for instance. So I tried to copy one of those packages off the host profitbricks-build4-amd64.debian.net to run a local diffoscope and inspect the issue. Instead the very same diffoscope in a very similar amd64 chroot was done analyzing within just 2 minutes. In reality, I couldn't reproduce the issue at all, but it still persists on that jenkins slave. What else could I tell? It's using the --text output mode. It's always run on an amd64 system and always on non-amd64 binary packages. Recommends are turned off, but binutils-multiarch and vim-common are explicitly installed. I also found looking into munin a bit interesting: https://jenkins.debian.net/munin/debian.net/profitbricks-build4-amd64.debian.net/index.html Since precisely the 24th, it started swapping a lot more and there are large spikes in the resident memory usage. There is a chance that we can attribute this to diffoscope. Then I looked through the commits between 46 and 47 and the only thing that caught my eye was the automatic use detached debug symbols. In my own tests, I was never able to make that work (according to --debug output), so maybe the feature works on profitbricks-build4-amd64.debian.net? Even though I cannot reproduce the issue at hand, I think that the code adding automatic debug symbols looks fishy to me. It appears to recurse over /tmp here and that looks very wrong to me. /tmp is small here, but on the jenkins slave, /tmp is big. Any further ideas for debugging this? Helmut ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Cross-building across architectures (reproducibly?)
Hi Steven, On Sat, Jan 02, 2016 at 09:03:21PM +, Steven Chamberlain wrote: > I've begun to analyze this now, using diffoscope to compare a stage1 > debootstrapped chroot of linux-i386, from official Debian packages vs. > my cross-built ones from kfreebsd-amd64. Note that there are public jenkins jobs for a very similar task already. On https://jenkins.debian.net/view/rebootstrap/ the "*_diffoscope" jobs do compare the cross build packages to the archive versions. The resulting text diffs are embedded into the build logs, so apart from the missing cross kernel part (because kfreebsd-any doesn't cover release architectures atm and jenkins.d.n doesn't have any kfreebsd-any slaves), we do get these diffs for almost a year already. > Unfortunately all the ELF binaries have differences, but most other > files' contents are reproducible. There are lots of reasons for these differences. The top reasons most likely are: * Different gcc version * Different build path (and thus different build ids) * Different configure check results due to cross compilation > The biggest problem was that the cross toolchain's linker used a > different target ABI: > > ??? ??? ??? -OS: Linux, ABI: 3.2.0 > ??? ??? ??? +OS: Linux, ABI: 2.6.32 > > and that leads to huge differences in the output binaries, such as not > using .init_array/.fini_array sections. This is surprising as glibc explicitly requests 3.2 via --enable-kernel. I also cannot confirm this in the (linux-only) diffoscope logs mentioned above. There all the cross built packages use the 3.2 ABI. Maybe some configure check went wrong here? In general, wrong configure results are a big problem to cross compilation and are the main reason why I added support for running diffoscope early. Some wrong checks simply make the build fail, but many go unnoticed until you diff the binary. The root cause is that autotools tends to be pessimistic rather than optimistic. When you cross build and a particular check cannot be run, autotools tends to assume the worst possible outcome. To solve this, we need an organized way to preseed these check results to configure scripts. Currently, some check results are bundled into dpkg-cross (not in jessie or stretch), but ideally these would be distributed to the packages that are being checked (e.g. glibc when you test whether malloc(0) returns NULL). While my focus has been on making cross building work at all, I welcome and support efforts to make it reproducible as well. You can find interesting pieces in the rebootstrap_*_diffoscope jobs today for packages like libtool, nettle-dev, bash, libfreetype6-dev, libexpat1-dev, libicu-dev, libxml2-dev and libdebian-installer4-dev. Please Cc me in your replies. Helmut ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#792867: FTCBFS: uses wrong architecture Python
Source: bsdmainutils Version: 9.0.6 Tags: patch User: helm...@debian.org Usertags: rebootstrap Dear bsdmainutils maintainers, While trying to cross build bsdmainutils, I noticed that it was installing the host architecture python and python-hdate. It then tries to execute that Python as a build tool, which fails, because the host architecture is usually not executable during cross compilation. Since python and python-hdate are only used as build tools, the correct solution is to always install them for the build architecture. This can be reflected in the Build-Depends by annotating those packages with :native. The attached patch implements that proposal. Please consider applying it. The :native is supported in apt and dpkg since wheezy. If in doubt, don't hesitate to ask for details on the identified problem or the proposed solution. As an aside note, I question the use of python-hdate to compute the Jewish calendar: * Since the build date is used to select the year of the calendar file, the package is not reproducible. (Ccing the reproducible people) * The last build happened in 2014, so the installed calendar is no longer useful for day to day use (neither in stable nor in unstable). Helmut diff -Nru bsdmainutils-9.0.6/debian/changelog bsdmainutils-9.0.6+nmu1/debian/changelog --- bsdmainutils-9.0.6/debian/changelog 2014-10-17 15:42:56.0 +0200 +++ bsdmainutils-9.0.6+nmu1/debian/changelog2015-07-19 15:37:08.0 +0200 @@ -1,3 +1,11 @@ +bsdmainutils (9.0.6+nmu1) UNRELEASED; urgency=medium + + * Non-maintainer upload. + * Add :native annotationes to Python-related Build-Depends to facilitate +cross dependency satisfiability (Closes: #-1) + + -- Helmut Grohne hel...@subdivi.de Sun, 19 Jul 2015 15:36:38 +0200 + bsdmainutils (9.0.6) unstable; urgency=medium * Add Ubuntu 13.04 release date to calendar.ubuntu. diff -Nru bsdmainutils-9.0.6/debian/control bsdmainutils-9.0.6+nmu1/debian/control --- bsdmainutils-9.0.6/debian/control 2014-08-15 11:49:55.0 +0200 +++ bsdmainutils-9.0.6+nmu1/debian/control 2015-07-19 15:36:33.0 +0200 @@ -3,7 +3,7 @@ Priority: important Maintainer: Debian Bsdmainutils Team pkg-bsdmainut...@teams.debian.net Uploaders: Giacomo Catenazzi c...@debian.org, Michael Meskes mes...@debian.org -Build-Depends: debhelper (= 7), libncurses5-dev, quilt (= 0.40), python, python-hdate +Build-Depends: debhelper (= 7), libncurses5-dev, quilt (= 0.40), python:native, python-hdate:native Standards-Version: 3.9.5 Vcs-Git: git://anonscm.debian.org/bsdmainutils/bsdmainutils.git Vcs-Browser: http://anonscm.debian.org/gitweb/?p=bsdmainutils/bsdmainutils.git ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#781280: --text crashes when encountering empty files
Package: debbindiff
Version: 11
Tags: patch
User: helm...@debian.org
Usertags: rebootstrap
debbindiff --text crashes when it encounters an empty file.
Example input package triggering the error: libreadline6
Example traceback being fixed:
Traceback (most recent call last):
File /usr/bin/debbindiff, line 120, in module
sys.exit(main())
File /usr/bin/debbindiff, line 114, in main
output_text(differences, print_func=print_func)
File /usr/lib/python2.7/dist-packages/debbindiff/presenters/text.py, line
71, in output_text
print_details(difference, print_func)
File /usr/lib/python2.7/dist-packages/debbindiff/presenters/text.py, line
62, in print_details
print_details(detail, new_print_func)
File /usr/lib/python2.7/dist-packages/debbindiff/presenters/text.py, line
62, in print_details
print_details(detail, new_print_func)
File /usr/lib/python2.7/dist-packages/debbindiff/presenters/text.py, line
62, in print_details
print_details(detail, new_print_func)
File /usr/lib/python2.7/dist-packages/debbindiff/presenters/text.py, line
59, in print_details
print_difference(detail, print_func)
File /usr/lib/python2.7/dist-packages/debbindiff/presenters/text.py, line
36, in print_difference
if not difference.lines2[-1].endswith('\n'):
IndexError: list index out of range
I think the fix is pretty straight forward.
Helmut
From 8697e25016f0cd2786563cc5c5aa440fa99ef62d Mon Sep 17 00:00:00 2001
From: Helmut Grohne hel...@subdivi.de
Date: Thu, 26 Mar 2015 06:58:20 +0100
Subject: [PATCH] fix IndexError from --text with empty files
Example input package triggering the error: libreadline6
Example traceback being fixed:
Traceback (most recent call last):
File /usr/bin/debbindiff, line 120, in module
sys.exit(main())
File /usr/bin/debbindiff, line 114, in main
output_text(differences, print_func=print_func)
File /usr/lib/python2.7/dist-packages/debbindiff/presenters/text.py, line 71, in output_text
print_details(difference, print_func)
File /usr/lib/python2.7/dist-packages/debbindiff/presenters/text.py, line 62, in print_details
print_details(detail, new_print_func)
File /usr/lib/python2.7/dist-packages/debbindiff/presenters/text.py, line 62, in print_details
print_details(detail, new_print_func)
File /usr/lib/python2.7/dist-packages/debbindiff/presenters/text.py, line 62, in print_details
print_details(detail, new_print_func)
File /usr/lib/python2.7/dist-packages/debbindiff/presenters/text.py, line 59, in print_details
print_difference(detail, print_func)
File /usr/lib/python2.7/dist-packages/debbindiff/presenters/text.py, line 36, in print_difference
if not difference.lines2[-1].endswith('\n'):
IndexError: list index out of range
---
debbindiff/presenters/text.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/debbindiff/presenters/text.py b/debbindiff/presenters/text.py
index 5cda5ce..ca60af8 100644
--- a/debbindiff/presenters/text.py
+++ b/debbindiff/presenters/text.py
@@ -30,10 +30,10 @@ def print_difference(difference, print_func):
for line in difference.comment.split('\n'):
print_func(uââ %s % line)
if difference.lines1 or difference.lines2:
-if not difference.lines1[-1].endswith('\n'):
+if difference.lines1 and not difference.lines1[-1].endswith('\n'):
difference.lines1[-1] = difference.lines1[-1] + '\n'
difference.lines1.append('No newline at the end\n')
-if not difference.lines2[-1].endswith('\n'):
+if difference.lines2 and nand ot difference.lines2[-1].endswith('\n'):
difference.lines2[-1] = difference.lines2[-1] + '\n'
difference.lines2.append('No newline at the end\n')
g = difflib.unified_diff(difference.lines1, difference.lines2)
--
2.1.4
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#778641: closed by Jérémy Bobbio lu...@debian.org (Bug#778641: fixed in debbindiff 11)
Control: reopen -1 Control: severity -1 important Raising severity to important as it now fails for way more invocations than initially (i.e. I no longer find any successful input). On Mon, Mar 23, 2015 at 09:21:13PM +, Debian Bug Tracking System wrote: * Error out when encoding is not Unicode-compatible for text output. (Closes: #778641) I'm sorry to reopen again, but your fix is bogus. 1) debbindiff already forces LC_CTYPE=C.UTF-8. Setting LC_CTYPE in any way has absolutely no effect on debbindiff. The error message is wrong. 2) The error message claims that a non-utf8 console would be a problem, even when the output is directed to a file and nothing is written to the console. 3) When the except block issuing the error is discarded, one obtains the following more useful traceback: Traceback (most recent call last): File /usr/bin/debbindiff, line 119, in module sys.exit(main()) File /usr/bin/debbindiff, line 113, in main output_text(differences, print_func=print_func) File /usr/lib/python2.7/dist-packages/debbindiff/presenters/text.py, line 70, in output_text print_details(difference, print_func) File /usr/lib/python2.7/dist-packages/debbindiff/presenters/text.py, line 54, in print_details print_func(u### %s % detail.source1) File /usr/bin/debbindiff, line 67, in print_func print(*args, **kwargs) UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128) Here detail1.source1 == control.tar.gz (a str, not unicode). I am seriously wondering whether fixing this properly is the best way forward. The continued unavailability of debbindiffs from rebootstrap is unfortunate. Maybe you can still apply the patch originally sent to this bug report to make it just work? Helmut ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#778641: closed by Jérémy Bobbio lu...@debian.org (Bug#778641: fixed in debbindiff 10)
Control: reopen -1 Control: found -1 debbindiff/10 On Fri, Mar 20, 2015 at 01:06:13PM +, Debian Bug Tracking System wrote: * Attempt to sort out multiple decoding and encoding issues. (Closes: #778641) I'm sorry to say that this attempt failed: https://jenkins.debian.net/job/rebootstrap_i386_gcc49_supported_debbindiff/9/console In fact it fails on the very first package being tested. Can we just apply the fix that makes it work for now? I think that having these diffs is more important than waiting for someone to fix this properly some day. Helmut ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#778641: debbindiff --text crashes for xz-utils with a UnicodeDecodeError
Control: tags -1 + patch
It seems that currently the lines1 and lines2 lists in a Difference
object can contain str and unicode. This means that:
* The Difference class has poor API documentation and does not explain
how it is supposed to be used.
* Consequently, functions that generate Difference objects do not agree
what it should contain and choose different interpretations.
* In particular:
+ control files tend to be unicode.
+ readelf output tends to be str.
The bug results from issuing .encode(utf8) on a str object. When doing
so, it is automatically decoded to unicode using the
sys.getdefaultencoding() (which happens to default to ascii). This
class of bugs can be found by:
* Running debbindiff in the python3 interpreter (if that is feasible
already).
* Issuing: import sys; reload(sys); sys.setdefaultencoding(invalid)
after importing all modules. This causes implicit encoding and
encoding to always fail and it breaks importing and using certain
builtin modules (which is why setdefaultencoding is normally
deleted).
Attaching a minimal patch that makes it work somewhat.
Helmut
diff -Nru debbindiff-9/debbindiff/presenters/text.py
debbindiff-9+nmu1/debbindiff/presenters/text.py
--- debbindiff-9/debbindiff/presenters/text.py 2015-02-15 12:13:51.0
+0100
+++ debbindiff-9+nmu1/debbindiff/presenters/text.py 2015-02-24
00:50:38.0 +0100
@@ -34,7 +34,9 @@
for line in g:
if line.startswith('--- ') or line.startswith('+++ '):
continue
-print_func(â %s % line.encode('utf-8'), end='')
+if isinstance(line, unicode):
+line = line.encode('utf-8')
+print_func(â %s % line, end='')
def print_details(difference, print_func):
if not difference.details:
diff -Nru debbindiff-9/debian/changelog debbindiff-9+nmu1/debian/changelog
--- debbindiff-9/debian/changelog 2015-02-15 12:13:51.0 +0100
+++ debbindiff-9+nmu1/debian/changelog 2015-02-24 08:49:58.0 +0100
@@ -1,3 +1,10 @@
+debbindiff (9+nmu1) UNRELEASED; urgency=low
+
+ * Non-maintainer upload.
+ * Work around UnicodeDecodeError in --text. (Closes: #778641)
+
+ -- Helmut Grohne hel...@subdivi.de Tue, 24 Feb 2015 08:49:08 +0100
+
debbindiff (9) unstable; urgency=medium
[ Chris Lamb ]
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#778423: support a plain text output mode
Package: debbindiff
Version: 8
Severity: wishlist
Tags: patch
User: helm...@debian.org
Usertags: rebootstrap
A limitation of rebootstrap currently is that it can only output a build
log. Thus I embed the debbindiffs into the log. This is cumbersome to
read, since they are unrendered html. Having a plain text output mode
would make reading these logs way easier. Not as easy as the html
rendered presentation, but still more useful.
I therefore ask to add a plain text output mode.
Oh and here is a patch. :)
Helmut
diff -Nru debbindiff-8/debbindiff/presenters/text.py
debbindiff-8+nmu1/debbindiff/presenters/text.py
--- debbindiff-8/debbindiff/presenters/text.py 1970-01-01 00:00:00.0
+
+++ debbindiff-8+nmu1/debbindiff/presenters/text.py 2015-02-14
20:08:37.0 +
@@ -0,0 +1,17 @@
+import sys
+import difflib
+
+def output_text(differences):
+for difference in differences:
+print(difference between %s and %s %
+ (difference.source1, difference.source2))
+if difference.comment:
+print(comment: %s % difference.comment)
+if difference.lines1 and difference.lines2:
+for line in difflib.unified_diff(difference.lines1,
+difference.lines2, difference.source1, difference.source2):
+sys.stdout.write(line)
+if difference.details:
+print(descending into details)
+ output_text(difference.details)
+print(returning from details)
diff -Nru debbindiff-8/debbindiff.py debbindiff-8+nmu1/debbindiff.py
--- debbindiff-8/debbindiff.py 2015-02-10 02:03:06.0 +
+++ debbindiff-8+nmu1/debbindiff.py 2015-02-14 20:03:12.0 +
@@ -26,6 +26,7 @@
from debbindiff import logger, VERSION
import debbindiff.comparators
from debbindiff.presenters.html import output_html
+from debbindiff.presenters.text import output_text
def create_parser():
@@ -38,6 +39,9 @@
default=False, help='display debug messages')
parser.add_argument('--html', metavar='output', dest='html_output',
help='write HTML report to given file')
+parser.add_argument('--text', dest='text_output', default=False,
+action='store_true',
+help='write plain text output to stdout')
parser.add_argument('--max-report-size', metavar='BYTES',
dest='max_report_size', type=int,
help='maximum bytes written in report')
@@ -62,6 +66,8 @@
print(*args, **kwargs)
output_html(differences, css_url=parsed_args.css_url,
print_func=print_func,
max_page_size=parsed_args.max_report_size)
+if len(differences) 0 and parsed_args.text_output:
+output_text(differences)
if len(differences) 0:
sys.exit(1)
diff -Nru debbindiff-8/debian/changelog debbindiff-8+nmu1/debian/changelog
--- debbindiff-8/debian/changelog 2015-02-10 02:03:06.0 +
+++ debbindiff-8+nmu1/debian/changelog 2015-02-14 20:10:12.0 +
@@ -1,3 +1,10 @@
+debbindiff (8+nmu1) UNRELEASED; urgency=medium
+
+ * Non-maintainer upload.
+ * Add a text presentation based on unified diffs. (Closes: #-1)
+
+ -- Helmut Grohne hel...@subdivi.de Sat, 14 Feb 2015 20:09:42 +
+
debbindiff (8) unstable; urgency=medium
[ Peter De Wachter ]
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
