On 2021-01-10 Sebastian Andrzej Siewior wrote: > I hope for sane defaults :)
I hope so too. So far I have felt that the suggested solutions have significant flaws or downsides, and I'm not able to see what is a good enough compromise. As a result the discussion hasn't progressed much and I feel it's partly my fault, sorry. I will try again: I have understood that *in practice* the problem with the xz command line tool is limited to "xz -T0" usage so fixing this use case is enough for most people. Please correct me if I missed something. The change in XZ Utils 5.2.5 helps a little with 32-bit xz running under 64-bit kernel but only if one specifies a memory usage limit like -M90% together with -T0. To make plain -T0 work too, in an earlier email I suggested that -T0 could also imply a memory usage limit if no limit was otherwise specified (a preliminary patch was included too). I have been hesitant to make changes to the defaults of the memory usage limiter but this solution would only affect a very specific situation and thus I feel it would be fine. Comments would be appreciated. The problem with applications using liblzma and running out of address space sounds harder to fix. As I explained in another email, making liblzma more robust with memory allocation failures is not a perfect fix and can still result in severe problems depending on how the application as a whole works (with some apps it could be enough). An alternative "fix" for the liblzma case could be adding a simple API function that would scale down the number of threads in a lzma_mt structure based on a memory usage limit and if the application is 32 bits. Currently the thread count and LZMA2 settings adjusting code is in xz, not in liblzma. > Anyway. Not to overcompilcate things: On Linux you can obtain the > available system memory which I would cap to 2 or 2.5 GiB by default. > Nobody should be hurt by that. If full 4 GiB of address space is available, capping to 2 GiB to 2.5 GiB when the available memory isn't known would mean fewer threads than with the 4020 MiB limit. Obviously this is less bad than failing due to running out of address space but it still makes me feel that if available memory is used on Linux, it should be ported to other OSes too. The idea for the current 4020 MiB special limit is based on a patch that was in use in FreeBSD to solve the problem of 32-bit xz on 64-bit kernel. So at least FreeBSD should be supported to not make 32-bit xz worse under 64-bit FreeBSD kernel. In liblzma, if a new function is added to reduce the thread count based on a memory usage limit, a capping the limit to 2 to 3 GiB on 32-bit applications could be fine even if there is more available memory. Being conservative means fewer threads but it would make it more likely that things keep working if the application allocates memory after liblzma has already done so. Oh well. :-( I think I still made this sound like a mess. In any case, let's at least try to find some solution to the "xz -T0" case. It would be nice to hear if my suggestion makes any sense. Thanks. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
