Re: [Rpm-maint] How to get RPM requires quickly
On 09/13/2014 04:13 PM, Richard W.M. Jones wrote:
We have a program[1] that needs to take a list of installed RPM
package names, and quickly generate all of the installed dependencies
(recursively).
*Note* this is NOT a question about yum/dnf/zypper/etc depsolving.
It's about doing 'rpm -qR' quickly on installed packages only.
At the moment we run the following command:
rpm -qR current set of packages |
awk '{print $1}' |
xargs rpm -q --qf '%%{name}\n' --whatprovides |
grep -v 'no package provides' |
sort -u
The output is added to the current set, and we repeat the process
until the set of packages reaches a fixpoint.
To give you an idea, on a typical starting set this takes around 20 seconds,
which is over half the total running time of the program.
Does anyone have any suggestions about how to do this more quickly
and/or cleverly? Maybe using rpmlib (which unfortunately seems to
lack any documentation)?
That kind of usage is pretty much worst case behavior for rpm as it
defies almost all caching etc. For immediate speed gain with no other
changes, add --nosignature to stop rpm from re-re-re-re-revalidating
header signatures over and over again. In the average 'rpm -qa' time
--nodigest doesn't help much but the cumulative effect might be worth it
here.
To get rpm do a bit more work for you, use 'rpm -e --test' to make rpm
resolve the deps. Basically
rpm -e --test current set of packages 21 |
awk '/needed by/{print $NF}' |
sort -u
...and feed the output to the next cycle. --nosignature and --nodigest
will help here too, but to lesser degree.
For more speed, you'd need to use the rpmlib API, pointers to
documentation can be found at http://rpm.org/wiki/Docs
- Panu -
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] How to get RPM requires quickly
On 09/15/2014 02:42 PM, Panu Matilainen wrote: On 09/13/2014 04:13 PM, Richard W.M. Jones wrote: We have a program[1] that needs to take a list of installed RPM package names, and quickly generate all of the installed dependencies (recursively). To get rpm do a bit more work for you, use 'rpm -e --test' to make rpm resolve the deps. Basically I don't think these are same things. 'rpm -e' gives the packages requiring the package in question while I think the original question was for the packages required *by* the package. Florian -- Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Charles Peters ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] How to get RPM requires quickly
On Mon, Sep 15, 2014 at 03:42:37PM +0300, Panu Matilainen wrote:
On 09/13/2014 04:13 PM, Richard W.M. Jones wrote:
We have a program[1] that needs to take a list of installed RPM
package names, and quickly generate all of the installed dependencies
(recursively).
*Note* this is NOT a question about yum/dnf/zypper/etc depsolving.
It's about doing 'rpm -qR' quickly on installed packages only.
At the moment we run the following command:
rpm -qR current set of packages |
awk '{print $1}' |
xargs rpm -q --qf '%%{name}\n' --whatprovides |
grep -v 'no package provides' |
sort -u
The output is added to the current set, and we repeat the process
until the set of packages reaches a fixpoint.
To give you an idea, on a typical starting set this takes around 20 seconds,
which is over half the total running time of the program.
Does anyone have any suggestions about how to do this more quickly
and/or cleverly? Maybe using rpmlib (which unfortunately seems to
lack any documentation)?
That kind of usage is pretty much worst case behavior for rpm as it
defies almost all caching etc.
I never claimed it was clever :-)
For immediate speed gain with no
other changes, add --nosignature to stop rpm from
re-re-re-re-revalidating header signatures over and over again. In
the average 'rpm -qa' time --nodigest doesn't help much but the
cumulative effect might be worth it here.
That helps a bit. Just adding --nosignature --nodigest to every
invocation of RPM has reduced the time from 20 seconds to 17 seconds.
I guess the next step is rpmlib ...
Thanks,
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [PATCH] perl.{prov,req} improvments
Hi
Mageia just switched to internal deps generator (at least).
In the process, I unforked as most scripts as possible.
Here's some old fixes:
skip-plain-regular-comments.diff:
just skip plain, regular perl comments...
support-for-_-in-perl-module-version.diff:
support for _ in perl module version
commit 2af13ef2dccbe7c3dc88b4f2b5a5329f0e135a99
Author: Jérôme Quelin jque...@mandriva.org
Date: Fri Mar 12 17:22:23 2010 +
skip plain, regular comments. how comes this has not been spotted earlier?
(sigh)
diff --git a/scripts/perl.prov b/scripts/perl.prov
index 8f5d53b..3113cb2 100755
--- a/scripts/perl.prov
+++ b/scripts/perl.prov
@@ -117,7 +117,7 @@ sub process_file {
$inover = 0;
}
-if ($incomment || $inover) {
+if ($incomment || $inover || m/^\s*#/) {
next;
}
commit 0f1dd7bebbea6916dc5d698099a8a4cbfb829e81
Author: Jerome Quelin jque...@mageia.org
Date: Sat Feb 26 12:04:53 2011 +
1.118 - support for _ in perl module version
diff --git a/scripts/perl.prov b/scripts/perl.prov
index a6d68bb..85b8e81 100755
--- a/scripts/perl.prov
+++ b/scripts/perl.prov
@@ -171,7 +171,7 @@ sub process_file {
if (m/\$Revision: (\d+[.0-9]+)/) {
$version = $1;
- } elsif (m/[']?(\d+[.0-9]+)[']?/) {
+ } elsif (m/[']?(\d+[._0-9]+)[']?/) {
# look for a static number hard coded in the script
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] speeding up internal deps generator
Hi
We've recently switched to internal deps generator.
However one of the deps scripts we use [1] is very slow b/c of this[2]:
+for path in \
+$(for tlpath in \
+$(find ${RPM_BUILD_ROOT}/usr/lib64 ${RPM_BUILD_ROOT}/usr/lib
/usr/lib64 /usr/lib -name '*.typelib'); do
+dirname $tlpath; done | uniq ); do
+export GI_TYPELIB_PATH=$GI_TYPELIB_PATH:$path
+done
Previously, it was run once (with so called external deps generator).
With internal deps generator, rpm forks it for every matching file.
When lots of files match, this is very slow as it runs tons of find
for every matching file.
This could be workarounded if internal deps generator could offer a
new flag (instead of just exeonly magic_and_path ) that, if
found in the .attr file, would make rpm to call the script just once
with the whole list of files instead of calling it once per file.
It could be named all_in_one, fork_once_only, or whatever.
This would be helpfull for most packages builds as most of standard
prov/req scripts already handle more than on file name and thus could
use this new flag.
WDYT?
[1]
https://build.opensuse.org/package/view_file/openSUSE:Factory/gobject-introspection/gi-find-deps.sh?expand=1
[2]
https://build.opensuse.org/package/rdiff/openSUSE:Factory/gobject-introspection?linkrev=baserev=36
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] speeding up internal deps generator
On Mon, Sep 15, 2014 at 05:34:48PM +0200, Thierry Vignaud wrote:
We've recently switched to internal deps generator.
However one of the deps scripts we use [1] is very slow b/c of this[2]:
+for path in \
+$(for tlpath in \
+$(find ${RPM_BUILD_ROOT}/usr/lib64 ${RPM_BUILD_ROOT}/usr/lib
/usr/lib64 /usr/lib -name '*.typelib'); do
+dirname $tlpath; done | uniq ); do
+export GI_TYPELIB_PATH=$GI_TYPELIB_PATH:$path
+done
Previously, it was run once (with so called external deps generator).
With internal deps generator, rpm forks it for every matching file.
When lots of files match, this is very slow as it runs tons of find
for every matching file.
This could be workarounded if internal deps generator could offer a
new flag (instead of just exeonly magic_and_path ) that, if
found in the .attr file, would make rpm to call the script just once
with the whole list of files instead of calling it once per file.
It could be named all_in_one, fork_once_only, or whatever.
This would be helpfull for most packages builds as most of standard
prov/req scripts already handle more than on file name and thus could
use this new flag.
WDYT?
In that case the script needs to annotate the output in some way.
It has to transport the information, which files generated the dependency.
Otherwise things like 'rpm -q --filerequire' would no longer work.
Cheers,
Michael.
--
Michael Schroeder m...@suse.de
SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg
main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
