On 2021-01-18 23:52:50 [+0200], Lasse Collin wrote:
> On 2021-01-10 Sebastian Andrzej Siewior wrote:
> > I hope for sane defaults :)
>
> I hope so too. So far I have felt that the suggested solutions have
> significant flaws or downsides, and I'm not able to see what is a good
> enough compromise. As a result the discussion hasn't progressed much
> and I feel it's partly my fault, sorry. I will try again:
>
> I have understood that *in practice* the problem with the xz command
> line tool is limited to "xz -T0" usage so fixing this use case is
> enough for most people. Please correct me if I missed something.
Correct.
> The change in XZ Utils 5.2.5 helps a little with 32-bit xz running
> under 64-bit kernel but only if one specifies a memory usage limit like
> -M90% together with -T0. To make plain -T0 work too, in an earlier
> email I suggested that -T0 could also imply a memory usage limit if no
> limit was otherwise specified (a preliminary patch was included too). I
> have been hesitant to make changes to the defaults of the memory usage
> limiter but this solution would only affect a very specific situation
> and thus I feel it would be fine. Comments would be appreciated.
In the parallel decompress I added code on Linux to query the
available memory. I would prefer that as an upper limit on 64bit if no
limit is given. The reason is that *this* amount of memory is safe to
use without over-committing / involving swap.
For 32bit applications I would cap that limit to 2.5 GiB or so. The
reason is that the *normal* case is to run 32bit application on a 32bit
kernel and so likely only 3GiB can be addressed at most (minus a few
details like linked in libs, NULL page, guard pages and so on).
The 32bit application on 64bit kernel is probably a shortcut where
something is done a 32bit chroot - like building a package.
I'm not sure what a sane upper limit is on other OSes. Limitting it on
32bit does probably more good than bad if there is no -M parameter.
> The problem with applications using liblzma and running out of address
> space sounds harder to fix. As I explained in another email, making
> liblzma more robust with memory allocation failures is not a perfect
> fix and can still result in severe problems depending on how the
> application as a whole works (with some apps it could be enough).
Yes. For liblzma you get the memory limitation from the caller. I've
seen in Debian's dpkg to use physmem/2 with 2GiB as upper limit on
32bit. That works ;)
> An alternative "fix" for the liblzma case could be adding a simple API
> function that would scale down the number of threads in a lzma_mt
> structure based on a memory usage limit and if the application is 32
> bits. Currently the thread count and LZMA2 settings adjusting code is
> in xz, not in liblzma.
It might help. dpkg checks the memlimit with
lzma_stream_encoder_mt_memusage() and decreases the memory limit until
it fits. It looks simpler compared to rpm's attempt and various
exceptions.
> > Anyway. Not to overcompilcate things: On Linux you can obtain the
> > available system memory which I would cap to 2 or 2.5 GiB by default.
> > Nobody should be hurt by that.
>
> If full 4 GiB of address space is available, capping to 2 GiB to 2.5 GiB
> when the available memory isn't known would mean fewer threads than
> with the 4020 MiB limit. Obviously this is less bad than failing due to
> running out of address space but it still makes me feel that if
> available memory is used on Linux, it should be ported to other OSes
> too.
I didn't understand that last sentance.
> The idea for the current 4020 MiB special limit is based on a patch
> that was in use in FreeBSD to solve the problem of 32-bit xz on 64-bit
> kernel. So at least FreeBSD should be supported to not make 32-bit xz
> worse under 64-bit FreeBSD kernel.
Is this a common case?
While poking around, Linux has this personality() syscall/function.
There is a flag called PER_LINUX32_3GB and PER_LINUX_32BIT which are set
if the command is invoked with `linux32' say
linux32 xz
then it would set that flag set and could act. It is not set by starting
a 32bit application on a 64bit kernel on its own or on a 32bit kernel.
I don't know if this is common practise but I use this in my chroots. So
commands like `uname -m' return `i686' instead of `x86_64'.
If other chroot environments do it as well then it could be used as a
hack to assume that it is run on 64bit kernel. That is if we want that
ofcourse :)
> In liblzma, if a new function is added to reduce the thread count based
> on a memory usage limit, a capping the limit to 2 to 3 GiB on 32-bit
> applications could be fine even if there is more available memory. Being
> conservative means fewer threads but it would make it more likely that
> things keep working if the application allocates memory after liblzma
> has already done so.
>
> Oh well. :-( I think I still made this sound like a mess. In any case,
> let's at least try to find some solution to the "xz -T0" case. It would
> be nice to hear if my suggestion makes any sense. Thanks.
oki.
Sebastian
