Re: [xz-devel] Only single core used when trying multi-thread compression

[Matt Parlane]

On Wed, Apr 13, 2016 at 8:54 AM, Anna Henningsen  wrote:

> Note how the first lines of your attached strace outputs are different;
> one time, you execute /usr/bin/xz, and one time /usr/local/bin/xz is run.

Gah! So obvious!

Sorry for wasting your time. I will now hang my head in shame.

Matt

Re: [xz-devel] Only single core used when trying multi-thread compression

[Anna Henningsen]

On 12.04.2016 22:47, Matt Parlane wrote:
> On Tue, Apr 12, 2016 at 9:51 PM, Lasse Collin  
> wrote:
>> If something like SELinux blocks reading such files, then the core
>> count detection won't work. Compare the strace output in the two
>> situations:
>>
>> strace -o /home/foo/strace.txt xz -T0 < /dev/null > /dev/null
> 
> I've attached the two outputs. Let me know if I can provide anything else.

Note how the first lines of your attached strace outputs are different;
one time, you execute /usr/bin/xz, and one time /usr/local/bin/xz is run.

You should be able to solve your problem by either directly invoking
/usr/local/bin/xz explicitly or adding something like
PATH=/usr/local/bin:/bin:/usr/bin at the top of your crontab.

Re: [xz-devel] Only single core used when trying multi-thread compression

[Lasse Collin]

On 2016-04-12 Matt Parlane wrote:
> I have compiled xz from source (rev ac398c3), and I am trying to use
> multithread compression by using -T 0 in the command line.
> 
> What I am finding is that when run straight from the command line, it
> uses multiple cores as expected. However, when running via a shell
> script via cron, only one core is utilised.
> 
> Any ideas? I'm running exactly the same command on the same file, and
> I would really love to make full use of my system.

Check if the environment variables XZ_DEFAULTS and XZ_OPT differ
between the two situations. I guess it's unlikely, but it's easy to
check, so I mentioned this first.

If the above didn't help: Does it work if you specify the number of
threads explicitly, for example, by using -T4 instead of -T0? If it
does, then the problem must be in the detection of the number of CPU
cores.

Since the detection works outside cron, perhaps there are some
restrictions that prevent it from working under cron. You didn't
mention your operating system or distribution, but I'm guessing
it's GNU/Linux. There xz uses sysconf(_SC_NPROCESSORS_ONLN) which makes
glibc read from /sys/devices/system/cpu/online. (I think it used to
read /proc/cpuinfo in the past.)

If something like SELinux blocks reading such files, then the core
count detection won't work. Compare the strace output in the two
situations:

strace -o /home/foo/strace.txt xz -T0 < /dev/null > /dev/null

If it is a SELinux issue (or similar), it would be good to report it to
the distribution maintainers.

-- 
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode