On 2021-01-10 Sebastian Andrzej Siewior wrote:
> I hope for sane defaults :)

I hope so too. So far I have felt that the suggested solutions have
significant flaws or downsides, and I'm not able to see what is a good
enough compromise. As a result the discussion hasn't progressed much
and I feel it's partly my fault, sorry. I will try again:

I have understood that *in practice* the problem with the xz command
line tool is limited to "xz -T0" usage so fixing this use case is
enough for most people. Please correct me if I missed something.

The change in XZ Utils 5.2.5 helps a little with 32-bit xz running
under 64-bit kernel but only if one specifies a memory usage limit like
-M90% together with -T0. To make plain -T0 work too, in an earlier
email I suggested that -T0 could also imply a memory usage limit if no
limit was otherwise specified (a preliminary patch was included too). I
have been hesitant to make changes to the defaults of the memory usage
limiter but this solution would only affect a very specific situation
and thus I feel it would be fine. Comments would be appreciated.

The problem with applications using liblzma and running out of address
space sounds harder to fix. As I explained in another email, making
liblzma more robust with memory allocation failures is not a perfect
fix and can still result in severe problems depending on how the
application as a whole works (with some apps it could be enough).

An alternative "fix" for the liblzma case could be adding a simple API
function that would scale down the number of threads in a lzma_mt
structure based on a memory usage limit and if the application is 32
bits. Currently the thread count and LZMA2 settings adjusting code is
in xz, not in liblzma.

> Anyway. Not to overcompilcate things: On Linux you can obtain the
> available system memory which I would cap to 2 or 2.5 GiB by default.
> Nobody should be hurt by that.

If full 4 GiB of address space is available, capping to 2 GiB to 2.5 GiB
when the available memory isn't known would mean fewer threads than
with the 4020 MiB limit. Obviously this is less bad than failing due to
running out of address space but it still makes me feel that if
available memory is used on Linux, it should be ported to other OSes

The idea for the current 4020 MiB special limit is based on a patch
that was in use in FreeBSD to solve the problem of 32-bit xz on 64-bit
kernel. So at least FreeBSD should be supported to not make 32-bit xz
worse under 64-bit FreeBSD kernel.

In liblzma, if a new function is added to reduce the thread count based
on a memory usage limit, a capping the limit to 2 to 3 GiB on 32-bit
applications could be fine even if there is more available memory. Being
conservative means fewer threads but it would make it more likely that
things keep working if the application allocates memory after liblzma
has already done so.

Oh well. :-( I think I still made this sound like a mess. In any case,
let's at least try to find some solution to the "xz -T0" case. It would
be nice to hear if my suggestion makes any sense. Thanks.

Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode

Reply via email to