On 2021-01-18 23:52:50 [+0200], Lasse Collin wrote: > On 2021-01-10 Sebastian Andrzej Siewior wrote: > > I hope for sane defaults :) > > I hope so too. So far I have felt that the suggested solutions have > significant flaws or downsides, and I'm not able to see what is a good > enough compromise. As a result the discussion hasn't progressed much > and I feel it's partly my fault, sorry. I will try again: > > I have understood that *in practice* the problem with the xz command > line tool is limited to "xz -T0" usage so fixing this use case is > enough for most people. Please correct me if I missed something.
Correct. > The change in XZ Utils 5.2.5 helps a little with 32-bit xz running > under 64-bit kernel but only if one specifies a memory usage limit like > -M90% together with -T0. To make plain -T0 work too, in an earlier > email I suggested that -T0 could also imply a memory usage limit if no > limit was otherwise specified (a preliminary patch was included too). I > have been hesitant to make changes to the defaults of the memory usage > limiter but this solution would only affect a very specific situation > and thus I feel it would be fine. Comments would be appreciated. In the parallel decompress I added code on Linux to query the available memory. I would prefer that as an upper limit on 64bit if no limit is given. The reason is that *this* amount of memory is safe to use without over-committing / involving swap. For 32bit applications I would cap that limit to 2.5 GiB or so. The reason is that the *normal* case is to run 32bit application on a 32bit kernel and so likely only 3GiB can be addressed at most (minus a few details like linked in libs, NULL page, guard pages and so on). The 32bit application on 64bit kernel is probably a shortcut where something is done a 32bit chroot - like building a package. I'm not sure what a sane upper limit is on other OSes. Limitting it on 32bit does probably more good than bad if there is no -M parameter. > The problem with applications using liblzma and running out of address > space sounds harder to fix. As I explained in another email, making > liblzma more robust with memory allocation failures is not a perfect > fix and can still result in severe problems depending on how the > application as a whole works (with some apps it could be enough). Yes. For liblzma you get the memory limitation from the caller. I've seen in Debian's dpkg to use physmem/2 with 2GiB as upper limit on 32bit. That works ;) > An alternative "fix" for the liblzma case could be adding a simple API > function that would scale down the number of threads in a lzma_mt > structure based on a memory usage limit and if the application is 32 > bits. Currently the thread count and LZMA2 settings adjusting code is > in xz, not in liblzma. It might help. dpkg checks the memlimit with lzma_stream_encoder_mt_memusage() and decreases the memory limit until it fits. It looks simpler compared to rpm's attempt and various exceptions. > > Anyway. Not to overcompilcate things: On Linux you can obtain the > > available system memory which I would cap to 2 or 2.5 GiB by default. > > Nobody should be hurt by that. > > If full 4 GiB of address space is available, capping to 2 GiB to 2.5 GiB > when the available memory isn't known would mean fewer threads than > with the 4020 MiB limit. Obviously this is less bad than failing due to > running out of address space but it still makes me feel that if > available memory is used on Linux, it should be ported to other OSes > too. I didn't understand that last sentance. > The idea for the current 4020 MiB special limit is based on a patch > that was in use in FreeBSD to solve the problem of 32-bit xz on 64-bit > kernel. So at least FreeBSD should be supported to not make 32-bit xz > worse under 64-bit FreeBSD kernel. Is this a common case? While poking around, Linux has this personality() syscall/function. There is a flag called PER_LINUX32_3GB and PER_LINUX_32BIT which are set if the command is invoked with `linux32' say linux32 xz then it would set that flag set and could act. It is not set by starting a 32bit application on a 64bit kernel on its own or on a 32bit kernel. I don't know if this is common practise but I use this in my chroots. So commands like `uname -m' return `i686' instead of `x86_64'. If other chroot environments do it as well then it could be used as a hack to assume that it is run on 64bit kernel. That is if we want that ofcourse :) > In liblzma, if a new function is added to reduce the thread count based > on a memory usage limit, a capping the limit to 2 to 3 GiB on 32-bit > applications could be fine even if there is more available memory. Being > conservative means fewer threads but it would make it more likely that > things keep working if the application allocates memory after liblzma > has already done so. > > Oh well. :-( I think I still made this sound like a mess. In any case, > let's at least try to find some solution to the "xz -T0" case. It would > be nice to hear if my suggestion makes any sense. Thanks. oki. Sebastian