[cygport] enabling a replacement for "objdump -d -l"
Cygport uses "objdump -d -l" to extract the list of source files that
need to be copied into the debuginfo package. This operation triggers
some O(N²) or even higher complexity and in addition has been getting
slower in recent binutils releases due to more and more information
being put into the object files. For gcc-11 extracting the debug source
files takes up to 45 minutes per executable (up from about 15 minutes
until 2.39) and for gcc-13 (with about 1.5 times the number of lines to
extract) it is already taking more than two hours. So if you just
package gcc-13 using a single thread you'd be looking on the order of 20
hours wall clock time, which is unacceptable.
The deassembly implied by the "-d" (which is not the part that has the
superlinear complexity btw, but produces a baseline of 2 hours single
thread runtime all by itself) is also unnecessary to extract just the
filenames of the source files as we throw away the location information
anyway and so I've written a small parser that works on the DWARF dump
instead (which can be produced in linear time with a very small scaling
factor, so practically constant time even for very large executables).
Unfortunately binutils does not yet offer a machine readable format for
these dumps, but parsing the text is not too difficult even though the
format is undocumented. The DWARF-5 documentation isn't the most
enjoyable read, but it was helpful enough to figure it all out. I've
also integrated the filtering of unrelated source file information (from
system headers and external libraries). The end result is the same
runtime as before on small object files, a factor up to 100 speedup for
medium sized object files and speedups in the several thousands range
for large sized ones (or a total single-thread runtime of less than 20
seconds for gcc-13).
dwarf-parse.-pl
--8<---cut here---start->8---
#!perl -w
use common::sense;
use List::Util qw( sum );
my $filter = shift @ARGV
or die "not enough arguments";
my $obj = shift @ARGV
or die "not enough arguments";
my @objdump = qw( /usr/bin/objdump -WNl );
open my $DWARF, "-|", @objdump, $obj
or die "can't invoke objdump\n$!";
my ( @dirs, @files, %fn, %rn );
while (<$DWARF>) {
if (/^ The Directory Table/../^$/) {
if (/^ \d+/) {
my ( $entry, $dir ) = m/^ (\d+)\t.+: (.+)$/;
$dir = "$dirs[0]/$dir" if ($dir =~ m:\A[^/]:);
push @dirs, $dir;
}
}
if (/^ The File Name Table/../^$/) {
if (/^ \d+/) {
my ( $idx, $fn, undef ) = m/^ \d+\t(\d+)\t.+: (.+)$/;
$rn{"$dirs[$idx]/$fn"}++;
push @files, "$dirs[$idx]/$fn";
}
}
if (my $rc = /^ Line Number Statements/../^ Offset:/) {
$fn{"$files[0]"}++ if ($rc == 1);
$fn{"$files[$1]"}++ if m/ Set File Name to entry (\d+) in the File Name
Table/;
@files = () if ($rc =~ m/E0$/);
@dirs = () if ($rc =~ m/E0$/);
}
if (/^ No Line Number Statements./../^$/) {
@files = ();
@dirs = ();
}
}
foreach my $fn (grep m:^$filter:, sort keys %fn) {
say sprintf "%s", $fn;
}
say STDERR sprintf "\tLNS: %6d (%6d locations) <=> FNT: %6d ( %6d locations)",
0+grep( m:^$filter:, keys %fn ), sum( values %fn ),
0+grep( m:^$filter:, keys %rn ), sum( values %rn )
if (0);
close $DWARF
or die "failed to close objdump\n$!";
--8<---cut here---end--->8---
Integration into cygport is made configurable via a variable to be set
in .cygportrc for instance in order to easily revert back to the
original objdump invocation if necessary. I've been producing packages
with that setup for a while now and have not noticed any errors. In
principle the new parser actually produces more complete output as there
can be multiple line number statements and hence source files per
location, but objdump only lists one of them in the disassembly (at
least sometimes). In practise I haven't found a package until now where
the final list (after filtering) is different.
https://repo.or.cz/cygport/rpm-style.git/commitdiff_plain/7ab8b26aaefb8a6ce050a196ddc97ce416ebe7a9
--8<---cut here---start->8---
lib/src_postinst.cygpart: use DWARF_PARSE optionally instead of objdump -dl
---
diff --git a/lib/src_postinst.cygpart b/lib/src_postinst.cygpart
index f06004e4..3dd6e893 100644
--- a/lib/src_postinst.cygpart
+++ b/lib/src_postinst.cygpart
@@ -1096,7 +1096,12 @@ __prepstrip_one() {
else
dbg="/usr/lib/debug/${exe}.dbg";
- lines=$(${objdump} -d -l "${exe}" 2>/dev/null | sed -ne
"s|.*\(/usr/src/debug/${PF}/.*\):[0-9]*$|\1|gp" | sort -u | tee -a
${T}/.dbgsrc.out.${oxt} | wc -l);
+ if defined DWARF_PARSE
+ then
+ lines=$(${DWARF_PARSE} /usr/src/debug/${PF}/ "${exe}" |
tee -a ${T}/.dbgsrc.out.${oxt} | wc -l);
+
[PATCH cygport] Add repro-check command
This could be used to check whether a package is possibly reproducible.
Then it could make sense to add a reasonable SOURCE_DATE_EPOCH value to
the cygport file.
Example:
$ export SOURCE_DATE_EPOCH=$(date +%s)
$ cygport project.cygport all repro-check
...
*** Info: Build reproducibility test succeeded
$ TZ=UTC cygport project.cygport repro-check
...
*** Info: Build reproducibility test succeeded
$ unset SOURCE_DATE_EPOCH
$ cygport project.cygport repro-check
...
*** ERROR: Build reproducibility test failed
--
Regards,
Christian
From 97f518478dac722647b8a423068f2a5461c82f19 Mon Sep 17 00:00:00 2001
From: Christian Franke
Date: Sun, 18 Feb 2024 16:33:07 +0100
Subject: [PATCH] Add repro-check command
This command checks for reproducibility of distribution packages.
The source package from the dist directory is unpacked to the
temp directory. A nested rebuild of the packages is run there.
If successful, original and rebuild packages are compared and the
result is reported.
---
README | 1 +
bin/cygport.in | 8
lib/help.cygpart| 1 +
lib/pkg_pkg.cygpart | 42 +-
4 files changed, 51 insertions(+), 1 deletion(-)
diff --git a/README b/README
index fd16df6b..fec46b13 100644
--- a/README
+++ b/README
@@ -163,6 +163,7 @@ Other COMMANDs are meant primarily for maintainers:
diff - write a patch file capturing changes to source in the working
directory
stage- as upload, but don't request processing of uploaded packages
announce - compose and send a package announcement
+repro-check - check whether a rebuild produces binary identical packages
The standard arguments --help or --version may also be passed to cygport.
diff --git a/bin/cygport.in b/bin/cygport.in
index 5fc89eaf..6acbc85b 100755
--- a/bin/cygport.in
+++ b/bin/cygport.in
@@ -29,6 +29,10 @@ set -e;
#
+# Preserve original environment for repro-check command
+declare -r _cygport_orig_env=$(export)
+declare -r _cygport_orig_pwd=$(pwd)
+
# for regexes, sort, etc.
export LC_COLLATE=C
@@ -784,6 +788,10 @@ do
test ${PIPESTATUS[0]} -eq 0
_status=$?;
;;
+ repro-check)
+ __pkg_repro_check
+ _status=$?
+ ;;
help)
__show_help;
exit 0;
diff --git a/lib/help.cygpart b/lib/help.cygpart
index a7f30f7a..d851762e 100644
--- a/lib/help.cygpart
+++ b/lib/help.cygpart
@@ -56,6 +56,7 @@ __show_help() {
finishdelete the working directory
all run prep, compile, install and package
all-test run prep, compile, install and package-test
+ repro-check check whether a rebuild produces binary
identical packages
See the included README file for further documentation.
diff --git a/lib/pkg_pkg.cygpart b/lib/pkg_pkg.cygpart
index 756a687c..719ffcd1 100644
--- a/lib/pkg_pkg.cygpart
+++ b/lib/pkg_pkg.cygpart
@@ -992,6 +992,46 @@ _EOF
fi
}
+__pkg_repro_check() {
+ local rc srcpkg t_cygport t_spkgdir
+
+ srcpkg=${distdir}/${PN}/${PF}-src.tar.${TAR_COMPRESSION_EXT}
+ t_spkgdir=${T}/${spkgdir##*/}
+
+ echo
+ __stage "Checking reproducibility of"
+
+ echo
+ __step "Unpacking ${srcpkg}"
+ [ -f "${srcpkg}" ] || error "Packages not built yet"
+ tar xf ${srcpkg} -C ${T} || error "tar xf ${srcpkg} -C ${T} failed"
+
+ echo
+ __step "Rebuilding in ${t_spkgdir}"
+ t_cygport="cygport ${cygportfile} finish all"
+ echo "${_cygport_orig_env}" > ${T}/.cygport_orig_env
+ __step "=== Start: ${t_cygport} ="
+
+ # Start nested cygport with original environment in temp directory
+ rc=0
+ env --chdir=${_cygport_orig_pwd} --ignore-environment /bin/bash -c \
+ "source ${T}/.cygport_orig_env && cd ${t_spkgdir} &&
${t_cygport}" \
+ || rc=$?
+
+ __step "=== Done: ${t_cygport} (exit $rc) ="
+ echo
+ [ $rc = 0 ] || error "Rebuild in ${t_spkgdir} failed"
+
+ __step "Comparing original and rebuilt packages"
+ if ! diff -qr ${distdir} ${t_spkgdir}/${PF}.${ARCH}/dist
+ then
+ echo
+ error "Build reproducibility test failed"
+ fi
+ echo
+ inform "Build reproducibility test succeeded"
+}
+
# protect functions
readonly -f __pkg_binpkg __pkg_diff __gpg_sign __pkg_srcpkg __pkg_dist \
-__squeeze_whitespace __tar
+__pkg_repro_check __squeeze_whitespace __tar
--
2.43.0
Re: [PATCH cygport] git.cygclass: Suppress the depth option
On 16/02/2024 11:59, Daisuke Fujimura via Cygwin-apps wrote:
Thank you for merging.
I have confirmed that this modification has resulted in the intended behaviour.
[...]
$ head -3 agbsum-15-1bl1.cygport
HOMEPAGE="https://mandelbrot.dk/${PN};
GIT_URI="https://mandelbrot.dk/${PN};
GIT_TAG="${PV}"
$ cygport agbsum-15-1bl1.cygport fetch
*** Info: Trying to enable case sensitivity on /tmp/agbsum/agbsum-15-1bl1.x86_64
git clone --depth 1 --branch 15 --no-checkout
https://mandelbrot.dk/agbsum agbsum
Cloning into 'agbsum'...
fatal: dumb http transport does not support shallow capabilities
*** Warning: git clone failed, retrying without --depth option
git clone --branch 15 --no-checkout https://mandelbrot.dk/agbsum agbsum
Cloning into 'agbsum'...
Fetching objects: 251, done.
git checkout tags/15
HEAD is now at bef1780 Rename source directory: 'src' => 'source';
Update naming convention; Update copyright holder name; Update code
style;
```
Thank you very much for testing!
Re: [ITA] libid3tag
On 17/02/2024 13:43, Marco Atzeri via Cygwin-apps wrote: On 17/02/2024 04:18, Takashi Yano via Cygwin-apps wrote: I would like to adopt libid3tag. $ git diff | grep ^+ +++ b/cygwin-pkg-maint +libid3tag Takashi Yano +libmad Takashi Yano +taglib Takashi Yano +taglib-extras Takashi Yano +utf8cpp Takashi Yano Thanks Marco FWIW, I took a look at these cygports and couldn't find anything to comment on. Good job! Thanks for adopting these.
