[Rpm-maint] [PATCH 5/7] Add build-id links to rpm for all ELF files.

2016-06-14 Thread Mark Wielaard 
This patch moves the main ELF file build-id symlinks from the
debuginfo package into the main package. And uses different
base directories for the main ELF file build-id symlink.
For the main build-id use /usr/lib/.build-id and for the debug
build-id use /usr/lib/debug/.build-id.

There are two reasons for doing this. The main package and the
debuginfo package might get out of sync, or the debuginfo package
might not be installed at all. In which case finding the main ELF
file through the build-id symlink becomes impossible. Secondly by
moving the main ELF build-id symlink in its own directory the
/usr/lib/debug directory gets populated with only debuginfo files
which is convenient if the user might want to have that directory
populated through a network mountpoint.

To support the new logic the symlink code has been moved from
find-debuginfo.sh to build/files.c.

This also includes support for a new config %_build_id_links that
defaults to compat. The other settings are none, alldebug (the old
style) and separate. compat is like separate, but adds a compatibility
link under /usr/lib/debug/.build-id for the main build-id symlink.

There are several new testcases added to test the various settings
using the new keyword "buildid".

Signed-off-by: Mark Wielaard 
---
 build/Makefile.am  |   4 +
 build/files.c  | 375 
 configure.ac   |  15 +
 macros.in  |  28 ++
 scripts/find-debuginfo.sh  |  60 
 tests/Makefile.am  |   1 +
 tests/data/SPECS/hello2cp.spec |  64 
 tests/data/SPECS/hello2ln.spec |  63 
 tests/rpmbuildid.at| 761 +
 tests/rpmtests.at  |   1 +
 10 files changed, 1312 insertions(+), 60 deletions(-)
 create mode 100644 tests/data/SPECS/hello2cp.spec
 create mode 100644 tests/data/SPECS/hello2ln.spec
 create mode 100644 tests/rpmbuildid.at

diff --git a/build/Makefile.am b/build/Makefile.am
index 8318cc8..f0f55a9 100644
--- a/build/Makefile.am
+++ b/build/Makefile.am
@@ -27,3 +27,7 @@ librpmbuild_la_LIBADD = \
@LTLIBICONV@ \
@WITH_POPT_LIB@ \
@WITH_MAGIC_LIB@
+
+if LIBDW
+librpmbuild_la_LIBADD += @WITH_LIBELF_LIB@ @WITH_LIBDW_LIB@
+endif
diff --git a/build/files.c b/build/files.c
index 07bc94e..b3d821f 100644
--- a/build/files.c
+++ b/build/files.c
@@ -14,6 +14,11 @@
 #include 
 #endif
 
+#if HAVE_LIBDW
+#include 
+#include 
+#endif
+
 #include 
 #include 
 #include 
@@ -1544,6 +1549,368 @@ exit:
 return rc;
 }
 
+#if HAVE_LIBDW
+/* How build id links are generated.  See macros.in for description.  */
+#define BUILD_IDS_NONE 0
+#define BUILD_IDS_ALLDEBUG 1
+#define BUILD_IDS_SEPARATE 2
+#define BUILD_IDS_COMPAT   3
+
+static int addNewIDSymlink(FileList fl,
+  char *targetpath, char *idlinkpath,
+  int isDbg, int isCompat)
+{
+const char *linkerr = _("failed symlink");
+int rc = 0;
+int nr = 0;
+char *origpath, *linkpath;
+
+if (isDbg)
+   rasprintf(, "%s.debug", idlinkpath);
+else
+   linkpath = idlinkpath;
+origpath = linkpath;
+
+while (faccessat(AT_FDCWD, linkpath, F_OK, AT_SYMLINK_NOFOLLOW) == 0) {
+   if (nr > 0)
+   free(linkpath);
+   nr++;
+   rasprintf(, "%s.%d%s", idlinkpath, nr,
+ isDbg ? ".debug" : "");
+}
+
+char *symtarget = targetpath;
+if (nr > 0 && isCompat)
+   rasprintf (, "%s.%d", targetpath, nr);
+
+if (symlink(symtarget, linkpath) < 0) {
+   rc = 1;
+   rpmlog(RPMLOG_ERR, "%s: %s -> %s: %m\n",
+  linkerr, linkpath, symtarget);
+} else {
+   fl->cur.isDir = 0;
+   rc = addFile(fl, linkpath, NULL);
+}
+
+/* Don't warn (again) if this is a compat id-link, we retarget it. */
+if (nr > 0 && !isCompat) {
+   /* Lets see why there are multiple build-ids. If the original
+  targets are hard linked, then it is OK, otherwise warn
+  something fishy is going on. Would be nice to call
+  something like eu-elfcmp to see if they are really the same
+  ELF file or not. */
+   struct stat st1, st2;
+   if (stat (origpath, ) != 0) {
+   rpmlog(RPMLOG_WARNING, _("Duplicate build-id, stat %s: %m\n"),
+  origpath);
+   } else if (stat (linkpath, ) != 0) {
+   rpmlog(RPMLOG_WARNING, _("Duplicate build-id, stat %s: %m\n"),
+  linkpath);
+   } else if (!(S_ISREG(st1.st_mode) && S_ISREG(st2.st_mode)
+ && st1.st_nlink > 1 && st2.st_nlink == st1.st_nlink
+ && st1.st_ino == st2.st_ino && st1.st_dev == st2.st_dev)) {
+   char *rpath1 = realpath(origpath, NULL);
+   char *rpath2 = realpath(linkpath, NULL);
+   rpmlog(RPMLOG_WARNING, _("Duplicate build-ids %s and %s\n"),
+  rpath1, rpath2);
+   free(rpath1);
+

[Rpm-maint] [PATCH 6/7] Make it possible to have unique build-ids across build versions/releases.

2016-06-14 Thread Mark Wielaard 
Introduce a new macro _unique_build_ids that when set will pass the
version and release to find-debuginfo.sh and debugedit to recalculate
the build-id of ELF files.

Includes two new testcases to make sure the new setting works as expected
both when set and unset.

Signed-off-by: Mark Wielaard 
---
 macros.in  |  8 +++-
 scripts/find-debuginfo.sh  | 20 -
 tests/data/SPECS/hello-r2.spec | 58 +
 tests/rpmbuildid.at| 96 +-
 tools/debugedit.c  | 24 ++-
 5 files changed, 201 insertions(+), 5 deletions(-)
 create mode 100644 tests/data/SPECS/hello-r2.spec

diff --git a/macros.in b/macros.in
index 508c50f..c79cfcc 100644
--- a/macros.in
+++ b/macros.in
@@ -178,7 +178,7 @@
 #  the script.  See the script for details.
 #
 %__debug_install_post   \
-   %{_rpmconfigdir}/find-debuginfo.sh 
%{?_missing_build_ids_terminate_build:--strict-build-id} 
%{?_include_minidebuginfo:-m} %{?_find_debuginfo_dwz_opts} 
%{?_find_debuginfo_opts} "%{_builddir}/%{?buildsubdir}"\
+   %{_rpmconfigdir}/find-debuginfo.sh 
%{?_missing_build_ids_terminate_build:--strict-build-id} 
%{?_include_minidebuginfo:-m} %{?_unique_build_ids:--ver-rel 
"%{version}-%{release}"} %{?_find_debuginfo_dwz_opts} %{?_find_debuginfo_opts} 
"%{_builddir}/%{?buildsubdir}"\
 %{nil}
 
 #  Template for debug information sub-package.
@@ -481,6 +481,12 @@ package or when debugging this package.\
 #   ELF /usr/lib/debug/.build-id/xx/yyy -> /usr/lib/.build-id/xx/yyy
 %_build_id_links compat
 
+# Whether build-ids should be made unique between package version/releases
+# when generating debuginfo packages. If set to 1 this will pass
+# --ver-rel "%{version}-%{release}" to find-debuginfo.sh which will pass it
+# onto debugedit --build-id-seed to be used to prime the build-id note hash.
+%_unique_build_ids 1
+
 #
 # Use internal dependency generator rather than external helpers?
 %_use_internal_dependency_generator1
diff --git a/scripts/find-debuginfo.sh b/scripts/find-debuginfo.sh
index c93b246..9783715 100644
--- a/scripts/find-debuginfo.sh
+++ b/scripts/find-debuginfo.sh
@@ -6,6 +6,7 @@
 # [-o debugfiles.list]
 # [--run-dwz] [--dwz-low-mem-die-limit N]
 # [--dwz-max-die-limit N]
+# [--ver-rel VERSION-RELEASE]
 # [[-l filelist]... [-p 'pattern'] -o debuginfo.list]
 # [builddir]
 #
@@ -26,6 +27,12 @@
 # if available, and --dwz-low-mem-die-limit and --dwz-max-die-limit
 # provide detailed limits.  See dwz(1) -l and -L option for details.
 #
+# If --ver-rel VERSION-RELEASE is given then debugedit is called to
+# update the build-ids it finds adding the VERSION-RELEASE string as
+# seed to recalculate the build-id hash.  This makes sure the
+# build-ids in the ELF files are unique between versions and releases
+# of the same package.
+#
 # All file names in switches are relative to builddir (. if not given).
 #
 
@@ -49,6 +56,9 @@ run_dwz=false
 dwz_low_mem_die_limit=
 dwz_max_die_limit=
 
+# Version and release of the spec. Given by --ver-rel
+ver_rel=
+
 BUILDDIR=.
 out=debugfiles.list
 nout=0
@@ -68,6 +78,10 @@ while [ $# -gt 0 ]; do
 dwz_max_die_limit=$2
 shift
 ;;
+  --ver-rel)
+ver_rel=$2
+shift
+;;
   -g)
 strip_g=true
 ;;
@@ -238,8 +252,12 @@ while read nlinks inum f; do
   fi
 
   echo "extracting debug info from $f"
+  build_id_seed=
+  if [ ! -z "$ver_rel" ]; then
+build_id_seed="--build-id-seed=$ver_rel"
+  fi
   id=$(${lib_rpm_dir}/debugedit -b "$RPM_BUILD_DIR" -d /usr/src/debug \
- -i -l "$SOURCEFILE" "$f") || exit
+ -i $build_id_seed -l "$SOURCEFILE" "$f") || exit
   if [ $nlinks -gt 1 ]; then
 eval linkedid_$inum=\$id
   fi
diff --git a/tests/data/SPECS/hello-r2.spec b/tests/data/SPECS/hello-r2.spec
new file mode 100644
index 000..ca5091d
--- /dev/null
+++ b/tests/data/SPECS/hello-r2.spec
@@ -0,0 +1,58 @@
+Summary: hello -- hello, world rpm
+Name: hello
+Version: 1.0
+Release: 2
+Group: Utilities
+License: GPL
+Distribution: RPM test suite.
+Vendor: Red Hat Software
+Packager: Red Hat Software 
+URL: http://www.redhat.com
+Source0: hello-1.0.tar.gz
+Patch0: hello-1.0-modernize.patch
+Excludearch: lsi
+Excludeos: cpm
+Provides: hi
+Conflicts: goodbye
+Obsoletes: howdy
+Prefix: /usr
+
+%description
+Simple rpm demonstration.
+
+%prep
+%setup -q
+%patch0 -p1 -b .modernize
+
+%build
+make
+
+%install
+rm -rf $RPM_BUILD_ROOT
+mkdir -p $RPM_BUILD_ROOT/usr/local/bin
+make DESTDIR=$RPM_BUILD_ROOT install
+
+%clean
+rm -rf $RPM_BUILD_ROOT
+
+%pre
+
+%post
+
+%preun
+
+%postun
+
+%files
+%defattr(-,root,root)
+%doc   FAQ
+#%readme README
+#%license COPYING
+%attr(0751,root,root)  /usr/local/bin/hello
+
+%changelog
+* Wed Jun  8 2016 Mark Wielaard

[Rpm-maint] [PATCH 2/7] Add dwz debuginfo compression support.

2016-06-14 Thread Mark Wielaard 
Support for dwz compression has been in Fedora since a couple of years.
https://fedoraproject.org/wiki/Features/DwarfCompressor

The original find-debuginfo.sh patch was written by Jakub Jelinek.
https://bugzilla.redhat.com/show_bug.cgi?id=833311
The new testcase using the macros.debug was added by me.

Signed-off-by: Mark Wielaard 
---
 macros.debug | 13 
 macros.in|  2 +-
 scripts/find-debuginfo.sh| 53 ++
 tests/data/SPECS/hello2.spec | 62 +++
 tests/rpmbuild.at| 78 
 5 files changed, 207 insertions(+), 1 deletion(-)
 create mode 100644 tests/data/SPECS/hello2.spec

diff --git a/macros.debug b/macros.debug
index bb2c02f..6a8432e 100644
--- a/macros.debug
+++ b/macros.debug
@@ -28,3 +28,16 @@
 
 # Should missing buildids terminate a build?
 %_missing_build_ids_terminate_build1
+
+# Number of debugging information entries (DIEs) above which
+# dwz will stop considering file for multifile optimizations
+# and enter a low memory mode, in which it will optimize
+# in about half the memory needed otherwise.
+%_dwz_low_mem_die_limit 1000
+# Number of DIEs above which dwz will stop processing
+# a file altogether.
+%_dwz_max_die_limit 5000
+
+%_find_debuginfo_dwz_opts --run-dwz\\\
+   --dwz-low-mem-die-limit %{_dwz_low_mem_die_limit}\\\
+   --dwz-max-die-limit %{_dwz_max_die_limit}
diff --git a/macros.in b/macros.in
index 94d0aa8..e94f476 100644
--- a/macros.in
+++ b/macros.in
@@ -178,7 +178,7 @@
 #  the script.  See the script for details.
 #
 %__debug_install_post   \
-   %{_rpmconfigdir}/find-debuginfo.sh 
%{?_missing_build_ids_terminate_build:--strict-build-id} 
%{?_include_minidebuginfo:-m} %{?_find_debuginfo_opts} 
"%{_builddir}/%{?buildsubdir}"\
+   %{_rpmconfigdir}/find-debuginfo.sh 
%{?_missing_build_ids_terminate_build:--strict-build-id} 
%{?_include_minidebuginfo:-m} %{?_find_debuginfo_dwz_opts} 
%{?_find_debuginfo_opts} "%{_builddir}/%{?buildsubdir}"\
 %{nil}
 
 #  Template for debug information sub-package.
diff --git a/scripts/find-debuginfo.sh b/scripts/find-debuginfo.sh
index 5c2c381..8de7bad 100644
--- a/scripts/find-debuginfo.sh
+++ b/scripts/find-debuginfo.sh
@@ -4,6 +4,8 @@
 #
 # Usage: find-debuginfo.sh [--strict-build-id] [-g] [-r] [-m]
 # [-o debugfiles.list]
+# [--run-dwz] [--dwz-low-mem-die-limit N]
+# [--dwz-max-die-limit N]
 # [[-l filelist]... [-p 'pattern'] -o debuginfo.list]
 # [builddir]
 #
@@ -20,6 +22,10 @@
 # The -p argument is an grep -E -style regexp matching the a file name,
 # and must not use anchors (^ or $).
 #
+# The --run-dwz flag instructs find-debuginfo.sh to run the dwz utility
+# if available, and --dwz-low-mem-die-limit and --dwz-max-die-limit
+# provide detailed limits.  See dwz(1) -l and -L option for details.
+#
 # All file names in switches are relative to builddir (. if not given).
 #
 
@@ -35,6 +41,11 @@ include_minidebug=false
 # Barf on missing build IDs.
 strict=false
 
+# DWZ parameters.
+run_dwz=false
+dwz_low_mem_die_limit=
+dwz_max_die_limit=
+
 BUILDDIR=.
 out=debugfiles.list
 nout=0
@@ -43,6 +54,17 @@ while [ $# -gt 0 ]; do
   --strict-build-id)
 strict=true
 ;;
+  --run-dwz)
+run_dwz=true
+;;
+  --dwz-low-mem-die-limit)
+dwz_low_mem_die_limit=$2
+shift
+;;
+  --dwz-max-die-limit)
+dwz_max_die_limit=$2
+shift
+;;
   -g)
 strip_g=true
 ;;
@@ -302,6 +324,37 @@ while read nlinks inum f; do
   fi
 done || exit
 
+# Invoke the DWARF Compressor utility.
+if $run_dwz && type dwz >/dev/null 2>&1 \
+   && [ -d "${RPM_BUILD_ROOT}/usr/lib/debug" ]; then
+  dwz_files="`cd "${RPM_BUILD_ROOT}/usr/lib/debug"; find -type f -name 
\*.debug`"
+  if [ -n "${dwz_files}" ]; then
+
dwz_multifile_name="${RPM_PACKAGE_NAME}-${RPM_PACKAGE_VERSION}-${RPM_PACKAGE_RELEASE}.${RPM_ARCH}"
+dwz_multifile_suffix=
+dwz_multifile_idx=0
+while [ -f 
"${RPM_BUILD_ROOT}/usr/lib/debug/.dwz/${dwz_multifile_name}${dwz_multifile_suffix}"
 ]; do
+  let ++dwz_multifile_idx
+  dwz_multifile_suffix=".${dwz_multifile_idx}"
+done
+dwz_multfile_name="${dwz_multifile_name}${dwz_multifile_suffix}"
+dwz_opts="-h -q -r -m .dwz/${dwz_multifile_name}"
+mkdir -p "${RPM_BUILD_ROOT}/usr/lib/debug/.dwz"
+[ -n "${dwz_low_mem_die_limit}" ] \
+  && dwz_opts="${dwz_opts} -l ${dwz_low_mem_die_limit}"
+[ -n "${dwz_max_die_limit}" ] \
+  && dwz_opts="${dwz_opts} -L ${dwz_max_die_limit}"
+( cd "${RPM_BUILD_ROOT}/usr/lib/debug" && dwz $dwz_opts $dwz_files )
+# Remove .dwz directory if empty
+rmdir "${RPM_BUILD_ROOT}/usr/lib/debug/.dwz" 2>/dev/null
+if [ -f "${RPM_BUILD_ROOT}/usr/lib/debug/.dwz/${dwz_multifile_name}" ]; 
then
+  id="`readelf -Wn

Re: [Rpm-maint] [PATCH 3/4] Add sepdebugcrcfix to fixup old style gnu_debuglink CRC checksum.

2016-06-14 Thread Mark Wielaard 
Hi Thierry,

On Mon, 2016-06-13 at 14:33 +0200, Thierry Vignaud wrote:
> Could you provide a small example? From the bug report it isn't clear if
> > it really is a bug with that particular fix or that it is caused by bad
> > usage of attributes. If we have a small example we can create a test
> > case for it.
> 
> Here's one example attached.
> With current FC patch applied to rpm (aka uncommenting the patch at
> http://svnweb.mageia.org/packages/cauldron/rpm/current/SPECS/rpm.spec?revision=1018103=markup#l116)
> 
> With "%define debug_package %{nil}", "ll /bin/test2" shows:
> -rwsr-xr-x 1 root root 6128 Eve  13 13:56 /bin/test2*
> With "#define debug_package %{nil}", "ll /bin/test2" shows:
> -rwxr-xr-x 1 root root 7184 Eve  13 13:55 /bin/test2*
> 
> When the rpm patch is not applied, both cases works OK (aka ls -[ol]
> reports the suid bit)
 
Aha, ok. I see what is happening. Since you don't set the attributes
explicitly you rely on the suid bit being picked up. But since we change
the file the suid flag gets reset. The fix is similar to how other helpers
get and reset all mods. I added a testcase for this issue making sure that
it triggers all helpers and still has the suid binaries coming out as
expected. I also rearranged the patchset a little, to make sure the test
actually tests the right binary by bringing forward the "Don't use
hardcoded paths to tools/scripts in find-debuginfo.sh." fix.

 [PATCH 1/7] Add find-debuginfo.sh -m minisymtab support.
 [PATCH 2/7] Add dwz debuginfo compression support.
 [PATCH 3/7] Don't use hardcoded paths to tools/scripts in
 [PATCH 4/7] Add sepdebugcrcfix to fixup old style gnu_debuglink CRC
 [PATCH 5/7] Add build-id links to rpm for all ELF files.
 [PATCH 6/7] Make it possible to have unique build-ids across build
 [PATCH 7/7] Make adding GDB index sections configurable.

Thanks,

Mark
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint