On 2018-06-15 Denys Vlasenko wrote:
> I noticed that one of my build scripts has this:
>
> find $RPM_BUILD_ROOT/lib/modules/ -type f -name '*.ko' | xargs xz
>
> A rather typical usage, right?
>
> In my case, this takes several minutes to complete,
> as there are 4000+ modules. The quick check confirms that xz
> runs single-threaded.
I hope the following quote from the xz man page is helpful:
Parallel compression of many files
On GNU and *BSD, find(1) and xargs(1) can be used to paral‐
lelize compression of many files:
find . -type f \! -name '*.xz' -print0 \
| xargs -0r -P4 -n16 xz -T1
The -P option to xargs(1) sets the number of parallel xz pro‐
cesses. The best value for the -n option depends on how many
files there are to be compressed. If there are only a couple
of files, the value should probably be 1; with tens of thou‐
sands of files, 100 or even more may be appropriate to reduce
the number of xz processes that xargs(1) will eventually cre‐
ate.
The option -T1 for xz is there to force it to single-threaded
mode, because xargs(1) is used to control the amount of par‐
allelization.
nproc(1) from GNU coreutils can be used to determine a value for the -P
option. This isn't mentioned on the man page, but perhaps it should
be, although it's GNU-only. Perhaps xz should have --info-threads or
such which would print the value that is internally available for
--threads=0.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
