Hello! Sorry for the two-week silence, I was ill. It will take a few days for me to catch up with the emails.
On 2020-12-26 Sebastian Andrzej Siewior wrote: > On 2020-12-26 09:33:04 [+0300], Vitaly Chikunov wrote: > > This wasn't working, because `memlimit_compress` initialized with > > zero, thus memory limit is never lowered for 32-bit address space, > > causing `Cannot allocate memory' error (in `lzma_outq_init()'). For > > example, when `-T0' is used on 32 CPUs with compression level > > higher than `-6'. > > That is one way. It might be that hardware_init() should pass > `total_ram' to hardware_memlimit_set() instead of 0. > hardware_memlimit_get() treats 0 as unlimited but I don't think it > makes sense since memory is never unlimited. 0 means disabled, that is, xz is expected to behave just like most other programs that might allocate a lot of memory but don't have any internal memory usage limiting. Memory isn't unlimited but many programs sort of behave as if it were and fail hard if allocation fails. That's not robust but it seems to work most of the time and many seem find this to be acceptable behavior in general. The whole limiter feature exist because I felt it was good to have a mechanism to control the memory usage, especially when decompressing since a .xz file may cause xz to allocate 4 GiB of memory for a single thread. However, I think few people think the same and thus the limiter is off by default for both compression and decompression. > Also, 32bit with almost 4GiB as a limit is working. If you increase > your input (the example from your previous email) then you also end > up "can not allocate memory) simply because 32bit can not allocate > 4GiB of memory. I'm not sure if the actual memory limit is exported. > It is usually at around 3GiB but there architectures which allow less > than that (not to mention kernel configurations). I don't know if Linux makes it possible for userspace applications to know the available address space. It can indeed vary depending on the kernel config. xz also needs to be portable to many other kernels. The 4020 MiB hack works with 64-bit kernels running 32-bit applications since in that case many kernels provide 4 GiB of address space. There are also resource limits that may also be somewhat OS-specific. On GNU/Linux one can use "ulimit -v LIM" where LIM is the virtual memory limit in KiB. Trying to exceed it will result in ENOMEM just like when running out of address space. Trying to figure out the various limits doesn't sound practical especially if it is supposed to work with kernels other than Linux. Simply trying to allocate a lot of memory (to test if it works) is more realistic but I think it's still dumb. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
