Re: [Rpm-maint] [rpm-software-management/rpm] CMakeLists.txt: eliminate floating dependencies (PR #2914)
> That said... This does make the cmake file a bit noisier. I wonder what the > actual use cases are for disabling these (versus just not installing the > given build dependency in the first place, assuming a clean build system or > container is set up for the build). The problem with not installing items is that one can't be certain if they are indeed not installed. They can be pulled in indirectly through some other dependency, or appear later on after a system update. That happens quietly, and will change the way rpm builds. The cmake option on the other hand guarantees that the feature will not be enabled (or conversely, if it is enabled via the option, then the build will fail if dependencies are absent). -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2914#issuecomment-1978826316 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] CMakeLists.txt: eliminate floating dependencies (PR #2914)
Prior to this, the respective build options would get enabled depending on whether or not these components were found on the build system. I believe its better to make them strict (add REQUIRED parameter), so that rpm builds are always deterministic, subject to cmake options. As a follow-up, some of these newly hardened dependencies can be wrapped in cmake options (like was just done for readline support), if there are situations where they should be disabled. Resolves: https://github.com/rpm-software-management/rpm/issues/2855 You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/2914 -- Commit Summary -- * CMakeLists.txt: eliminate floating dependencies -- File Changes -- M CMakeLists.txt (12) M docs/CMakeLists.txt (2) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/2914.patch https://github.com/rpm-software-management/rpm/pull/2914.diff -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2914 You are receiving this because you are subscribed to this thread. Message ID: rpm-software-management/rpm/pull/2...@github.com ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Improve build determinism (replace soft dependencies with strict ones) (Issue #2855)
This is a followup to https://github.com/rpm-software-management/rpm/pull/2852/ There is still a number of soft dependencies in https://github.com/rpm-software-management/rpm/blob/master/CMakeLists.txt: ``` find_package(BZip2) find_package(Iconv) ... pkg_check_modules(ZSTD IMPORTED_TARGET libzstd>=1.3.8) pkg_check_modules(LIBELF IMPORTED_TARGET libelf) pkg_check_modules(LIBDW IMPORTED_TARGET libdw) pkg_check_modules(LIBLZMA IMPORTED_TARGET liblzma>=5.2.0) ``` Then, the respective build options get enabled depending on whether or not these components were found on the build system. I believe it's better to make them strict (add REQUIRED parameter), and (optionally) wrap them in cmake options (like was just done for readline support), if there are situations where they should be disabled. I can write the patch, but wanted first to confirm what would be preferred. -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/2855 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] CMakeLists.txt: restore readline support as an explicit option (PR #2852)
This was lost in autotools - cmake transition. The particular reason to make it explicit is that readline is gpl version 3 licensed, and in some builds components under that license need to be excluded. You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/2852 -- Commit Summary -- * CMakeLists.txt: restore readline support as an explicit option -- File Changes -- M CMakeLists.txt (7) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/2852.patch https://github.com/rpm-software-management/rpm/pull/2852.diff -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2852 You are receiving this because you are subscribed to this thread. Message ID: rpm-software-management/rpm/pull/2...@github.com ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] 4.19: GLOB_BRACE is not portable (Issue #2844)
FWIW, I think it's most easily fixable by relying on gnulib implementation? https://github.com/coreutils/gnulib/blob/master/lib/glob.c -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/2844#issuecomment-1893554648 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] 4.19: GLOB_BRACE is not portable (Issue #2844)
For completeness, we're going to patch rpm to ignore the braces if GLOB_BRACE
is absent, and if failures occur on musl systems, then they would be separately
fixed - so far none have been found:
```
>From f78e05544fb5ae9ef688963f19666f1af34c3d5c Mon Sep 17 00:00:00 2001
From: Alexander Kanavin
Date: Tue, 16 Jan 2024 09:59:26 +0100
Subject: [PATCH] rpmio/rpmglob.c: avoid using GLOB_BRACE if undefined by C
library
This addresses musl failures; if there is code out there relying on
those braces, it needs to be fixed when used on musl.
This is unlikely to be trivially fixable upstream.
Upstream-Status: Inappropriate [reported at
https://github.com/rpm-software-management/rpm/issues/2844]
Signed-off-by: Alexander Kanavin
---
rpmio/rpmglob.c | 6 ++
1 file changed, 6 insertions(+)
diff --git a/rpmio/rpmglob.c b/rpmio/rpmglob.c
index 243568766..43c27074a 100644
--- a/rpmio/rpmglob.c
+++ b/rpmio/rpmglob.c
@@ -33,6 +33,12 @@
#include "debug.h"
+/* Don't fail if the standard C library
++ * doesn't provide brace expansion */
+#ifndef GLOB_BRACE
+#define GLOB_BRACE 0
+#endif
+
/* Return 1 if pattern contains a magic char, see glob(7) for a list */
static int ismagic(const char *pattern)
{
```
--
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/2844#issuecomment-1893487061
You are receiving this because you are subscribed to this thread.
Message ID: ___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] 4.19: GLOB_BRACE is not portable (Issue #2844)
When building 4.19.1 with musl C library, the following error happens: ``` /home/pokybuild/yocto-worker/musl-qemux86-64/build/build/tmp/work/core2-64-poky-linux-musl/rpm/4.19.1/git/rpmio/rpmglob.c: In function 'rpmGlobPath': | /home/pokybuild/yocto-worker/musl-qemux86-64/build/build/tmp/work/core2-64-poky-linux-musl/rpm/4.19.1/git/rpmio/rpmglob.c:84:15: error: 'GLOB_BRACE' undeclared (first use in this function); did you mean 'GLOB_NOSPACE'? |84 | gflags |= GLOB_BRACE; | | ^~ | | GLOB_NOSPACE ``` I appreciate that fixing this isn't easy (unless one wants to break all usages of braces in calls to the function). Full log (it's large, and doesn't contain anything else that would be relevant): https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/8466/steps/11/logs/stdio -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/2844 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Remove the internal OpenPGP parser (Issue #2414)
> @kanavin Are all of the RPMs used also built locally? In that case disabling > signature checking is fine. Yes of course. Yocto is fully self-contained, except for the bootstrap items mentioned above. It builds components from source, then makes its own packages from the binaries, then makes a system image from those packages. You can also pick what package format you prefer: ipk/deb/rpm, or all three. -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/2414#issuecomment-1829372875 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Remove the internal OpenPGP parser (Issue #2414)
> > Using host distro tools in cross-compilation builds is problematic, as we > > don't have control over what versions we're going to get, and how they are > > built and configured. To ensure things work in a reproducible manner, yocto > > builds its own rpm executable that can run on the build machine. > > What about fetching a correctly configured binary RPM and verifying its hash > before using it? Possible, but comes with a significant support burden - someone needs to write and support the code that does this, and provide and update the binaries, for all of the architectures that yocto builds can run on. There needs to be tests and documentation too. And a possibility to opt out of it, and build rpm binary locally anyway. When there are alternatives, we pick one which puts the least pressure on our very limited maintainer resources, or ideally makes it less than it was. The standard approach is to build the tools locally, and we'd rather stick with it. > > FWIW, the only host tools allowed to bootstrap the yocto build are python, > > gcc, wget, tar, git and various (de)compression utilities - things you need > > to fetch the sources, and bootstrap a cross-compiler. > > Do you also need tools to cryptographically verify the downloaded sources? Or > is that done in Python? For tarballs, their checksums are verified against local record of what they should be with python. For git checkouts we trust the git executable that the specified commit id will result in the correct tree, or there will be an error. -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/2414#issuecomment-1829319201 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Remove the internal OpenPGP parser (Issue #2414)
> > So Yocto can accept that regression in package security, we'll make sure to > > place warnings where appropriate. > > Another option would be to use the host system’s RPM for verifying the > packages. Using host distro tools in cross-compilation builds is problematic, as we don't have control over what versions we're going to get, and how they are built and configured. To ensure things work in a reproducible manner, yocto builds its own rpm executable that can run on the build machine. FWIW, the only host tools allowed to bootstrap the yocto build are python, gcc, wget, tar, git and various (de)compression utilities - things you need to fetch the sources, and bootstrap a cross-compiler. -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/2414#issuecomment-1829301952 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Remove the internal OpenPGP parser (Issue #2414)
> > We can live with rpm verification disabled too. > > This is a terrible idea from a security perspective. In embedded linux world, production systems are rarely if ever updated from package feeds by a package manager. Rather, the whole root filesystem gets overwritten from an image file. Package manager is used to compose that root filesystem from local packages in a controlled CI environment (where package-level security isn't needed), and to allow developers to install additional items into a running system on their desks used for development and testing (where there's no need to sign packages either). So Yocto can accept that regression in package security, we'll make sure to place warnings where appropriate. -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/2414#issuecomment-1826247188 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Remove the internal OpenPGP parser (Issue #2414)
Just wanted to add the Yocto perspective: we don't have anything against sequoia, except its build dependencies. It needs both rust and clang (via one of the crates), rust and cland are both extremely heavy items to build, and we can't inject them into the core build sequence because it would lengthen builds far too much, even if you have a 96 core epyc machine to build on. So outsourcing the crypto to external gpg executable would be very welcome. We can live with rpm verification disabled too. @Conan-Kudo FYI -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/2414#issuecomment-1825991703 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Parallel rpmfc thread-safety - libmagic may fork helpers (#756)
I dug a bit deeper: the issue happens when libmagic calls out to external executables, which isn't implemented in a thread-safe way, and locks up. Enabling internal compression support (done via linking to libraries) makes the issue go away. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/756#issuecomment-711430288___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] rpmplugins.c: call dlerror() prior to dlsym() (#998)
It came up when testing a combination of dnf, musl C library, and newer elfutils (with libdebuginfod disabled). libdw from elfutils tries to dlopen() a non-existing libdebuginfod, but never fetches the error via dlerrror(), so when rpm later loads a plugin (successfully) and issues dlerror(), this happens: > Downloading Packages: > (1/5): busybox-udhcpc-1.31.1-r0.core2_32.rpm2.2 MB/s | 8.0 kB 00:00 > > (2/5): update-alternatives-opkg-0.4.2-r0.core2_ 3.5 MB/s | 8.5 kB 00:00 > > (3/5): run-postinsts-1.0-r10.noarch.rpm 5.2 MB/s | 8.3 kB 00:00 > > (4/5): busybox-1.31.1-r0.core2_32.rpm31 MB/s | 356 kB 00:00 > > (5/5): musl-1.1.23+git0+2c2477da9a-r0.core2_32. 27 MB/s | 361 kB 00:00 > > > Total49 MB/s | 742 kB 00:00 > > Running transaction check > error: Failed to resolve symbol ima_hooks: Error loading shared library > libdebuginfod.so: No such file or directory > The downloaded packages were saved in cache until the next successful > transaction. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/998#issuecomment-572002072___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] rpmplugins.c: call dlerror() prior to dlsym() (#998)
This is the recommended way in the manpage; if there is a lingering error from an unrelated dl*() call that was never obtained via dlerror(), it needs to be cleared prior to calling dlsym(). You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/998 -- Commit Summary -- * rpmplugins.c: call dlerror() prior to dlsym() -- File Changes -- M lib/rpmplugins.c (2) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/998.patch https://github.com/rpm-software-management/rpm/pull/998.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/998 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Cannot set Build Date from changelog anymore (#932)
Possibly, but the patch is still valid, as it makes the data thread-safe. I really have no idea about where and how changelong date is used, so you need to figure out why it isn't working anymore by reading code. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/932#issuecomment-551838648___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Parallel rpmfc thread-safety - libmagic may fork helpers (#756)
Let's work together and sort the problem. If you would like me to run additional diagnostics, I'd be happy to help. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/756#issuecomment-545841096___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Parallel rpmfc thread-safety - libmagic may fork helpers (#756)
TBH, this seems like a serious enough issue in 4.15.0 release, that parallel file classification should be simply disabled for the time being, with a 4.15.0.1 issued to include that. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/756#issuecomment-545471841___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Parallel rpmfc thread-safety - libmagic may fork helpers (#756)
Sadly, I am seeing this when rpm is linked against file 5.37 as well. Here's a
couple tracebacks obtained by attaching gdb to running (hanging) rpm processes:
0x7fc5a9ee22a1 in ?? () from /usr/lib/x86_64-linux-gnu/libgomp.so.1
(gdb) bt
#0 0x7fc5a9ee22a1 in ?? () from /usr/lib/x86_64-linux-gnu/libgomp.so.1
#1 0x7fc5ab0579be in ?? () from
/home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/elfutils/0.177-r0/recipe-sysroot-native/usr/bin/../lib/librpmbuild.so.9
#2 0x7fc5a9ed6ecf in GOMP_parallel () from
/usr/lib/x86_64-linux-gnu/libgomp.so.1
#3 0x7fc5ab058b3c in rpmfcClassify () from
/home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/elfutils/0.177-r0/recipe-sysroot-native/usr/bin/../lib/librpmbuild.so.9
#4 0x7fc5ab0598e6 in ?? () from
/home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/elfutils/0.177-r0/recipe-sysroot-native/usr/bin/../lib/librpmbuild.so.9
#5 0x7fc5ab04aa3b in ?? () from
/home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/elfutils/0.177-r0/recipe-sysroot-native/usr/bin/../lib/librpmbuild.so.9
#6 0x7fc5ab044f74 in ?? () from
/home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/elfutils/0.177-r0/recipe-sysroot-native/usr/bin/../lib/librpmbuild.so.9
#7 0x562e400c1ad1 in ?? ()
#8 0x562e400c1c33 in ?? ()
#9 0x562e400c0da3 in ?? ()
#10 0x7fc5aa7d3ddb in __libc_start_main () from
/home/alexander/development/poky/build-x86-64/tmp/sysroots-uninative/x86_64-linux/lib/libc.so.6
#11 0x562e400c107a in ?? ()
0x7f2220f000ee in waitpid () from
/home/alexander/development/poky/build-x86-64/tmp/sysroots-uninative/x86_64-linux/lib/libc.so.6
(gdb) bt
#0 0x7f2220f000ee in waitpid () from
/home/alexander/development/poky/build-x86-64/tmp/sysroots-uninative/x86_64-linux/lib/libc.so.6
#1 0x7f221ff53f46 in ?? () from
/home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/util-linux/2.34-r0/recipe-sysroot-native/usr/lib/rpm/.././libmagic.so.1
#2 0x7f221ff54478 in ?? () from
/home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/util-linux/2.34-r0/recipe-sysroot-native/usr/lib/rpm/.././libmagic.so.1
#3 0x7f221ff57b16 in ?? () from
/home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/util-linux/2.34-r0/recipe-sysroot-native/usr/lib/rpm/.././libmagic.so.1
#4 0x7f221ff48e54 in ?? () from
/home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/util-linux/2.34-r0/recipe-sysroot-native/usr/lib/rpm/.././libmagic.so.1
#5 0x7f2221223bb9 in ?? () from
/home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/util-linux/2.34-r0/recipe-sysroot-native/usr/lib/rpm/../librpmbuild.so.9
#6 0x7f221f894ecf in GOMP_parallel () from
/usr/lib/x86_64-linux-gnu/libgomp.so.1
#7 0x7f2221224b3c in rpmfcClassify () from
/home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/util-linux/2.34-r0/recipe-sysroot-native/usr/lib/rpm/../librpmbuild.so.9
#8 0x556ed5245571 in ?? ()
#9 0x7f2220e5eddb in __libc_start_main () from
/home/alexander/development/poky/build-x86-64/tmp/sysroots-uninative/x86_64-linux/lib/libc.so.6
#10 0x556ed524579a in ?? ()
Process tree shows:
rpmbuild.real─┬─2*[bzip2]
└─31*[{rpmbuild.real}]
rpmdeps.real─┬─29*[xz]
└─31*[{rpmdeps.real}]
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/756#issuecomment-545470525___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] No tag in git for 4.15 final (#871)
Also no link to tarball on the download page on the website. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/871___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
@pmatilai I trust that the last two commits in my set (make string pool operations thread-safe; avoid use of static data) are already merged into master in some form? Don't see those fixes in #695. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-490518945___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
@ignatenkobrain The reason we see a lot of speed up in Yocto with this patchset is that building the sources happens outside of rpm, and we only use rpm to do packaging of binaries. If the bulk of the above time is spend building texlive from source then of course the final difference won't be as much. Can you investigate a little bit where the time goes? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-325977144___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
@ignatenkobrain Sorry, forgot to push the fix. Now fixed. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-325659110___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
@Conan-Kudo @ignatenkobrain I have rebased this against master now. I checked that it builds, but did not run it against any spec files. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-325649868___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Do not hardcode "lib/rpm" as the installation path for default configuration and macros. (#298)
@Conan-Kudo I would rather not introduce special non-standard --rpmconfigdir --rpmlibexecdir flags to ./configure, where the defaults continue to violate established standards and practice. I'd rather go with using $libexecdir (alternatively, $libdir) and $datadir, or simply continue to carry a custom patch, with the explanation/link that upstream has rejected it. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/298#issuecomment-32024___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Do not hardcode "lib/rpm" as the installation path for default configuration and macros. (#298)
@Conan-Kudo Yes, I forgot to check that 2.3 is not in fact the latest FHS. That would be my preferred solution: helper binaries go to $libexecdir/rpm, data goes to $datadir/rpm, any software that makes different assumptions and becomes broken because of the change gets fixed. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/298#issuecomment-320181692___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Do not hardcode "lib/rpm" as the installation path for default configuration and macros. (#298)
The way I read the FHS is that it does not allow you to hardcode $prefix/lib, and it implicitly stipulates that helper executables go to $libdir/$name. The way I read GNU coding standards is that helper executables should go to $libexecdir/$name. I believe systemd, rpm, etc. violate both, perhaps unintentionally. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/298#issuecomment-320073541___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Do not hardcode "lib/rpm" as the installation path for default configuration and macros. (#298)
@pmatilai You are selectively quoting the FHS. Just a bit further it says: > /usr/lib : Alternate format libraries (optional) > Purpose > > /usr/lib performs the same role as /usr/lib for an alternate binary > format, > So no, you cannot hardcode a literal /usr/lib/whatever anywhere in source code. It's up to the build environment to decide that. Not to mention that arch-independent data should not be in /usr/lib/ at all, it should be in /usr/share - as per the FHS. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/298#issuecomment-319982787___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Do not hardcode "lib/rpm" as the installation path for default configuration and macros. (#298)
It is a problem for multilib environments which are very common in embedded space. Yocto has had full support for it for a long time, but it does require that nothing hardcodes the /usr/lib assumption, and instead installs its stuff where it is told to by the build environment. http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#combining-multiple-versions-library-files-into-one-image -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/298#issuecomment-319654697___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Do not hardcode "lib/rpm" as the installation path for default configuration and macros. (#298)
I just want to settle this question going forward: should these references be hardcoded to /usr/lib, which is less pain for existing distributions, but more pain for Yocto, as we have to fix every instance of such hard coding (that also includes the rest of dnf stack, and various other software), and then carry/rebase the fixes forever as vendor patches. Or should they be made flexible, as an upstream policy. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/298#issuecomment-319650516___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Do not hardcode "lib/rpm" as the installation path for default configuration and macros. (#298)
This can be named in various different ways, especially in cross-compilation environments, so let's take it from the build setup. This is a much simpler and less invasive version of an earlier patch: it simply changes references to $(prefix)/lib/rpm, so that they become $(libdir)/rpm. Hopefully this is less controversial or problematic. You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/298 -- Commit Summary -- * Do not hardcode "lib/rpm" as the installation path for default configuration and macros. -- File Changes -- M configure.ac (2) M rpm.am (6) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/298.patch https://github.com/rpm-software-management/rpm/pull/298.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/298 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Do not hardcode "lib/rpm" as the installation path for default config… (#263)
@pmatilai Okay, splitting things into two places is bad. How about not hardcoding the 'lib' part? Should I open a separate PR for it? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/263#issuecomment-319639985___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Do not hardcode "lib/rpm" as the installation path for default config… (#263)
@Conan-Kudo in Yocto the generated platform macros are not used, as there is a hardcoded list of those platforms, which is not flexible enough for us. So we directly override settings from the default macros file at runtime. However, for some reason _libdir is not overriden in the same way, and is instead configured at build time - so you have a point, I should change that. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/263#issuecomment-319638245___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Do not hardcode "lib/rpm" as the installation path for default config… (#263)
@Conan-Kudo In Yocto project the libdir can be lib, lib32, lib64, or anything else. You cannot assume anything about it, and when building the software, the value for it is typically passed to ./configure as a command line switch. On the other hand, %_libdir macro hardcodes the value to be /usr/lib, and then that macro is used all over the place in rpm. So I am removing that hardcoding, and setting the macro dynamically at build time from the build environment settings. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/263#issuecomment-319627324___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Do not hardcode "lib/rpm" as the installation path for default config… (#263)
Previously it was set this way:
```
%_lib lib
%_libdir%{_exec_prefix}/%{_lib}
```
So basically 'lib' was hardcoded via %_lib. If you show me how to avoid that in
a different way than my patch, I can do it.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/263#issuecomment-319375522___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Do not hardcode "lib/rpm" as the installation path for default config… (#263)
…uration and macros. This can be named in various different ways, especially in cross-compilation environments, so let's take it from the build setup. You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/263 -- Commit Summary -- * Do not hardcode "lib/rpm" as the installation path for default configuration and macros. -- File Changes -- M configure.ac (2) M macros.in (2) M rpm.am (4) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/263.patch https://github.com/rpm-software-management/rpm/pull/263.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/263 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
I have now updated the patchset again, addressing the crashes with old gcc, and Jeff's comment above. And this time they should be rebased correctly :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-308014402___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
There you go, I force pushed the reworked patches. It is 100% pure openmp now :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#issuecomment-307121770___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
kanavin commented on this pull request. > @@ -14,6 +14,8 @@ #include #include +#include/* NSPR thread pools */ Alright, I'll rework the patches with openmp, and re-do the pull request. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#discussion_r120109813___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
kanavin commented on this pull request. > @@ -14,6 +14,8 @@ #include #include +#include/* NSPR thread pools */ rpm already has a hard dependency on nss, and nss has a hard dependency on nspr, so it seemed obvious to utilize the thread pool API from nspr. Why you are reluctant? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226#discussion_r119710103___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)
@proyvind @simotek There you go, pull request created :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/211#issuecomment-305532544___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)
Please see here for background information and rationale: https://github.com/rpm-software-management/rpm/issues/211 Caveats: - this work was based on an older commit from master that Yocto is currently using. I have rebased it to latest upstream master, but did not test that the rebased version still works. People have asked me to publish the code as quickly as possible, so here it is. You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/226 -- Commit Summary -- * Split binary package building into a separate function * Run binary package creation via thread pools. * rpmstrpool.c: make operations over string pools thread-safe * build/pack.c: remove static local variables from buildHost() and getBuildTime() -- File Changes -- M build/pack.c (202) M rpmio/rpmstrpool.c (51) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/226.patch https://github.com/rpm-software-management/rpm/pull/226.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/226 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)
I actually have this working now. The improvements for us (the Yocto project) are quite dramatic, together with the switchover to multithreaded xz compression the process takes several times less for many bigger packages. I had to do these things: 1) split the binary packaging loop body into its own function 2) set up the task execution and joining with appropriate data structures for handling each task 3) fix a few data races (like Jeff predicted): particularly the shared string pool structure was completely thread unsafe, and there a couple of other issues around copying header data between packages, and usage of static local variables in a couple of places. I'm going to polish the patches, and will provide a link in a couple of days. If there's interest from upstream maintainers, I can turn it into a pull request. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/211#issuecomment-305258563___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)
NSPR includes thread pools, so I can make an experimental implementation using that: https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSPR/Reference/Thread_Pools Not really keen on messing about with the low-level pthreads API, it's 2017. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/211#issuecomment-302048447___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)
Hello, I'd like to ask your opinion on the idea of creating binary rpms in parallel. It seems possible: I looked at packageBinaries() and it has a straightforward for loop with no obvious dependencies between iterations. Each iteration could be in its own thread via pthreads. The reason I would like to do this is that we use rpm in Yocto project, and a) making binary rpms is a part of image building process in Yocto; there's generally no common package feed, so a lot of time is spent on building rpms on the local machine. b) some of the source packages produce a lot of very small binary packages; for instance we package glibc locales, or perl modules separately. In embedded world constraining disk space usage is important, so this has to be done. This makes rpm building really slow - several minutes or more, as it iterates over them one by one. Opinions? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/211___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
