[Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)
There are (at least) 3 new FSE compression formats that have been released in the past few years: zstd (Facebook), brotli (Google), lzfse (Apple). The new compressors have better compression ratios and speed than gzip/zlib. And while then compression rations is still not as good as XZ/LZMA, the decompression speed (which is mostly rate determining for RPM installation speed), of zest is considerably faster than XZ/LZMA. One or more of these implementations might be useful for RPM. (aside) There's an implementation for RPM+ZSTD available here: [http://rpm5.org/files/rpm/rpm-5.4/SNAPSHOT/rpm-5.4.18-0.20170718.src.rpm](url) that can be adapted to RPM4. I can/will do the patch if there is sufficient interest. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/256___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)
Here are URL's for zstd, brotli, and lzfse" - [https://github.com/facebook/zstd.git](url) - [https://github.com/google/brotli.git](url) - [https://github.com/lzfse/lzfse.git](url) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/256#issuecomment-316037161___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [PATCH] find-debuginfo.sh: Add --keep-section and --remove-section for eu-strip.
Thanks! Pushed.
Florian
On 07/17/2017 12:58 PM, Mark Wielaard wrote:
> Use --keep-section SECTION or --remove-section SECTION to explicitly
> keep a (non-allocated) section in the main executable or explicitly
> remove it into the .debug file. SECTION is an extended wildcard pattern.
> Both options can be given more than once.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1465997
>
> Signed-off-by: Mark Wielaard
> ---
> scripts/find-debuginfo.sh | 21 +++--
> 1 file changed, 19 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/find-debuginfo.sh b/scripts/find-debuginfo.sh
> index 5613391..e19ce9b 100755
> --- a/scripts/find-debuginfo.sh
> +++ b/scripts/find-debuginfo.sh
> @@ -3,6 +3,7 @@
> #for inclusion in an rpm spec file.
> #
> # Usage: find-debuginfo.sh [--strict-build-id] [-g] [-r] [-m] [-i] [-n]
> +# [--keep-section SECTION] [--remove-section SECTION]
> # [-j N]
> # [-o debugfiles.list]
> # [-S debugsourcefiles.list]
> @@ -15,9 +16,14 @@
> # [builddir]
> #
> # The -g flag says to use strip -g instead of full strip on DSOs or EXEs.
> +# The -r flag says to use eu-strip --reloc-debug-sections.
> +# Use --keep-section SECTION or --remove-section SECTION to explicitly
> +# keep a (non-allocated) section in the main executable or explicitly
> +# remove it into the .debug file. SECTION is an extended wildcard pattern.
> +# Both options can be given more than once.
> +#
> # The --strict-build-id flag says to exit with failure status if
> # any ELF binary processed fails to contain a build-id note.
> -# The -r flag says to use eu-strip --reloc-debug-sections.
> # The -m flag says to include a .gnu_debugdata section in the main binary.
> # The -i flag says to include a .gdb_index section in the .debug file.
> # The -n flag says to not recompute the build-id.
> @@ -65,6 +71,9 @@ strip_g=false
> # with -r arg, pass --reloc-debug-sections to eu-strip.
> strip_r=false
>
> +# keep or remove arguments to eu-strip.
> +keep_remove_args=
> +
> # with -m arg, add minimal debuginfo to binary.
> include_minidebug=false
>
> @@ -158,6 +167,14 @@ while [ $# -gt 0 ]; do
>-r)
> strip_r=true
> ;;
> + --keep-section)
> +keep_remove_args="${keep_remove_args} --keep-section $2"
> +shift
> +;;
> + --remove-section)
> +keep_remove_args="${keep_remove_args} --remove-section $2"
> +shift
> +;;
>-j)
> n_jobs=$2
> shift
> @@ -215,7 +232,7 @@ strip_to_debug()
>application/x-sharedlib*) g=-g ;;
>application/x-executable*) g=-g ;;
>esac
> - eu-strip --remove-comment $r $g -f "$1" "$2" || exit
> + eu-strip --remove-comment $r $g ${keep_remove_args} -f "$1" "$2" || exit
>chmod 444 "$1" || exit
> }
>
>
--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham,
Michael O'Neill
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Correctly detect Transmeta Crusoe as i686 (#254)
Thanks! Pushed. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/254#issuecomment-316064237___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Correctly detect Transmeta Crusoe as i686 (#254)
Closed #254. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/254#event-1168337543___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)
Projects that use autoconf are expected to detect build prerequisites and libraries. This usually leads to a set of ad hoc de facto m4 macros that deal with various details like API and path incompatibilities, much of which is platform and os and distro dependent. Various standard tools (like pkgconfig) to find these details have been devised; meanwhile there is still a great deal of diversity, particularly between platforms like linux and *BSD, that often makes portability more difficult than it needs to be. One useful approach -- particularly when maintaining a project like RPM -- is RPM_CHECK_LIB that does all the usual operations and permits a compact specification of 3rd part library details. Here is the blurb from the RPM_CHECK_LIB m4 macro (I will send along the complete addition to acinclude.m4 later today): ``` dnl ## dnl ## NAME: dnl ##RPM_CHECK_LIB -- Check for third-party libraries dnl ## dnl ## COPYRIGHT dnl ##Copyright (c) 2007 Ralf S. Engelschall dnl ## dnl ## DESCRIPTION: dnl ##This is a rather complex Autoconf macro for sophisticated dnl ##checking the availability of third-party libraries and dnl ##extending the build environment for correctly building dnl ##against it. dnl ## dnl ##It especially supports the following particular features: dnl ##- is aware of old-style [lib]-config style scripts dnl ##- is aware of new-style pkg-config(1) [lib].pc configuration files dnl ##- searches under standard sub-directories "include", "lib", etc. dnl ##- searches under arbitrary sub-areas of a tree like ".libs", etc. dnl ##- searches in standard system locations (implicitly) dnl ##- supports searching for function in multiple libraries dnl ##- supports searching for multiple headers dnl ##- supports multiple search locations (fallbacks!) dnl ## dnl ## USAGE: dnl ## - configure.in: dnl ##RPM_CHECK_LIB( dnl ##,-- [$1] e.g. GNU bzip2 dnl ##, -- [$2] e.g. bzip2 dnl ##,-- [$3] e.g. bz2 dnl ##,-- [$4] e.g. BZ2_bzlibVersion dnl ##, -- [$5] e.g. bzlib.h dnl ##[,-- [$6] e.g. yes,external:internal:none dnl ##[, -- [$7] e.g. lib/bzip2:include:src dnl ##[, -- [$8] e.g. AC_DEFINE(USE_BZIP2, 1, [...]) dnl ##-- [$9] e.g. AC_MSG_ERROR([...]) dnl ##]]]) dnl ## dnl ## - Makefile.in: dnl ##top_srcdir = @top_srcdir@ dnl ##srcdir = @srcdir@ dnl ##WITH_= @WITH_@ dnl ##WITH__SUBDIR = @WITH__SUBDIR@ dnl ##CPPFLAGS = @CPPFLAGS@ dnl ##CFLAGS = @CFLAGS@ dnl ##LDFLAGS= @LDFLAGS@ dnl ##LIBS = @LIBS@ dnl ## dnl ## - CLI: dnl ##$ ./configure \ dnl ## --with-[=] dnl ## [...] dnl ## dnl ## SYNTAX: dnl ## ::= | dnl ## ::= "," dnl ## ::= "yes" | "no" dnl ## ::= ":" dnl ## | dnl ## | "system" dnl ## | "external" dnl ## | "internal" dnl ## | "none" dnl ## ::= [...] /* valid arg for test(1) option "-d" */ dnl ## dnl ## ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/257___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)
The RPM_CHECK_LIB macro: add to acinclude.m4. [rpm_check_lib.m4.gz](https://github.com/rpm-software-management/rpm/files/1156106/rpm_check_lib.m4.gz) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/257#issuecomment-316076403___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)
Here is a moderately complex real world example of how to use: detecting ZSTD either externally or internally. In configure.ac: ``` dnl # Facebook Zstd RPM_CHECK_LIB( [Facebook Zstd], [zstd], [zstd], [ZSTD_versionNumber], [zstd.h], [no,external:internal:none], [zstd:lib:lib], [ ], []) ``` When the generated configure is invoked with --with-zstd=internal (with a github zstd git checkout in the top level directory) succeeds, then these values are available in Makefile's for adding to the usual libtool variables: ``` WITH_ZSTD = yes WITH_ZSTD_CPPFLAGS = -I$(top_srcdir)/$(WITH_ZSTD_SUBDIR)/lib -I$(top_builddir)/$(WITH_ZSTD_SUBDIR)/lib -I$(top_srcdir)/$(WITH_ZSTD_SUBDIR) -I$(top_builddir)/$(WITH_ZSTD_SUBDIR) WITH_ZSTD_LDFLAGS = -L$(top_builddir)/$(WITH_ZSTD_SUBDIR)/lib -L$(top_builddir)/$(WITH_ZSTD_SUBDIR) WITH_ZSTD_LIBS = -lzstd WITH_ZSTD_SUBDIR = zstd ``` The equivalent Makefile values for configure --with-zstd=external success: ``` WITH_ZSTD = yes WITH_ZSTD_CPPFLAGS = WITH_ZSTD_LDFLAGS = WITH_ZSTD_LIBS = WITH_ZSTD_SUBDIR = ... LIBS = ... -lzstd ``` And config.h of course contains ``` /* Define as 1 if building with Facebook Zstd library */ #define WITH_ZSTD 1 ... /* Define to 1 if you have the `zstd' library (-lzstd). */ #define HAVE_LIBZSTD 1 ... /* Define to 1 if you have the header file. */ #define HAVE_ZSTD_H 1 ``` Note: I've left out the definition's for HAVE_LIBZSTD and HAVE_ZSTD_H when using --with-zstd=internal for simplicity of explanation. In practice, these definitions are seldom needed with an internal zest source tree. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/257#issuecomment-316087315___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)
@n3npq Ooh, this looks very nice! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/256#issuecomment-316095214___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Adding namespaces for macros (#246)
After some further thought: Using an embedded '_' character as a strong hint (e.g. "_python_sitelib") for namespace membership, and adding a filter for those macros that do not have conformant naming, is likelier a far less disruptive deployment than adding Brand New! Better! Bestest! explicit syntax everywhere. todo++. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/246#issuecomment-316098932___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)
Projects that use autoconf are expected to detect build prerequisites and libraries. This usually leads to a set of ad hoc de facto m4 macros that deal with various details like API and path incompatibilities, much of which is platform and os and distro dependent. Various standard tools (like pkgconfig) to find these details have been devised; meanwhile there is still a great deal of diversity, particularly between platforms like linux and *BSD, that often makes portability more difficult than it needs to be. One useful approach -- particularly when maintaining a project like RPM -- is RPM_CHECK_LIB that does all the usual operations and permits a compact specification of 3rd part library details. Here is the blurb from the RPM_CHECK_LIB m4 macro (I will send along the complete addition to acinclude.m4 later today): ``` dnl ## dnl ## NAME: dnl ##RPM_CHECK_LIB -- Check for third-party libraries dnl ## dnl ## COPYRIGHT dnl ##Copyright (c) 2007 Ralf S. Engelschall dnl ## dnl ## DESCRIPTION: dnl ##This is a rather complex Autoconf macro for sophisticated dnl ##checking the availability of third-party libraries and dnl ##extending the build environment for correctly building dnl ##against it. dnl ## dnl ##It especially supports the following particular features: dnl ##- is aware of old-style [lib]-config style scripts dnl ##- is aware of new-style pkg-config(1) [lib].pc configuration files dnl ##- searches under standard sub-directories "include", "lib", etc. dnl ##- searches under arbitrary sub-areas of a tree like ".libs", etc. dnl ##- searches in standard system locations (implicitly) dnl ##- supports searching for function in multiple libraries dnl ##- supports searching for multiple headers dnl ##- supports multiple search locations (fallbacks!) dnl ## dnl ## USAGE: dnl ## - configure.in: dnl ##RPM_CHECK_LIB( dnl ##,-- [$1] e.g. GNU bzip2 dnl ##, -- [$2] e.g. bzip2 dnl ##,-- [$3] e.g. bz2 dnl ##,-- [$4] e.g. BZ2_bzlibVersion dnl ##, -- [$5] e.g. bzlib.h dnl ##[,-- [$6] e.g. yes,external:internal:none dnl ##[, -- [$7] e.g. lib/bzip2:include:src dnl ##[, -- [$8] e.g. AC_DEFINE(USE_BZIP2, 1, [...]) dnl ##-- [$9] e.g. AC_MSG_ERROR([...]) dnl ##]]]) dnl ## dnl ## - Makefile.in: dnl ##top_srcdir = @top_srcdir@ dnl ##srcdir = @srcdir@ dnl ##WITH_= @WITH_@ dnl ##WITH__SUBDIR = @WITH__SUBDIR@ dnl ##CPPFLAGS = @CPPFLAGS@ dnl ##CFLAGS = @CFLAGS@ dnl ##LDFLAGS= @LDFLAGS@ dnl ##LIBS = @LIBS@ dnl ## dnl ## - CLI: dnl ##$ ./configure \ dnl ## --with-[=] dnl ## [...] dnl ## dnl ## SYNTAX: dnl ## ::= | dnl ## ::= "," dnl ## ::= "yes" | "no" dnl ## ::= ":" dnl ## | dnl ## | "system" dnl ## | "external" dnl ## | "internal" dnl ## | "none" dnl ## ::= [...] /* valid arg for test(1) option "-d" */ dnl ## dnl ## ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/258___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] RPM NDB and chroot failure
While testing my previous patches to allow for an NDB only build of RPM, I ran across an issue using the --root argument. When passing --root, the strace log shows that after the chroot call, a stat() of the Packages.db absolute path fails, which triggers a reopen, which fails as it attempts open() using the absolute path. This manifests in lib/backend/ndb/rpmpkg.c: rpmpkgLock rpmpkgGetLock is_correct_db reopen_db open(pkgdb->filename, ...) The pkgdb->filename is not being updated after chroot. I'm looking into how to fix this, but I'm still a bit green on the RPM DB abstractions. Should the ndb implementation be using the rpmbd_internal API? For example, rather than accessing pkgdb->filename directly, should the ndb implementation have a call which provides it with the chroot adjusted path to the RPM DB? Something using rpmdbHome() for example? -- Darren Hart VMware Open Source Technology Center ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
