On 2021-01-08 18:40:18 [+0200], Lasse Collin wrote: > Hello! Hi, > Sorry for the two-week silence, I was ill. It will take a few days for > me to catch up with the emails.
No worries. Take your time to get better. > On 2020-12-26 Sebastian Andrzej Siewior wrote: > > On 2020-12-26 09:33:04 [+0300], Vitaly Chikunov wrote: > > > This wasn't working, because `memlimit_compress` initialized with > > > zero, thus memory limit is never lowered for 32-bit address space, > > > causing `Cannot allocate memory' error (in `lzma_outq_init()'). For > > > example, when `-T0' is used on 32 CPUs with compression level > > > higher than `-6'. > > > > That is one way. It might be that hardware_init() should pass > > `total_ram' to hardware_memlimit_set() instead of 0. > > hardware_memlimit_get() treats 0 as unlimited but I don't think it > > makes sense since memory is never unlimited. > > 0 means disabled, that is, xz is expected to behave just like most > other programs that might allocate a lot of memory but don't have any > internal memory usage limiting. Memory isn't unlimited but many > programs sort of behave as if it were and fail hard if allocation > fails. That's not robust but it seems to work most of the time and > many seem find this to be acceptable behavior in general. I hope for sane defaults :) > The whole limiter feature exist because I felt it was good to have a > mechanism to control the memory usage, especially when decompressing > since a .xz file may cause xz to allocate 4 GiB of memory for a single > thread. However, I think few people think the same and thus the limiter > is off by default for both compression and decompression. The memory limiter sounds reasonable - no doubts. > > Also, 32bit with almost 4GiB as a limit is working. If you increase > > your input (the example from your previous email) then you also end > > up "can not allocate memory) simply because 32bit can not allocate > > 4GiB of memory. I'm not sure if the actual memory limit is exported. > > It is usually at around 3GiB but there architectures which allow less > > than that (not to mention kernel configurations). > > I don't know if Linux makes it possible for userspace applications to > know the available address space. It can indeed vary depending on the > kernel config. xz also needs to be portable to many other kernels. The > 4020 MiB hack works with 64-bit kernels running 32-bit applications > since in that case many kernels provide 4 GiB of address space. That is kind of a pain. I'm not aware of anything that reports the possible address limit other than some test-and-error hacks. Debian had a 2:2 split and was "forced" to switch to a 3:1 split because some java applications expected / required a larger virtual address space. I think that every distro ships a 3:1 32bit kernel now. You can also have architecture level limitations. If I remember correctly there was (is) a MIPS achitecture which can not assign more than 2GiB of address space to a single application. Anyway. Not to overcompilcate things: On Linux you can obtain the available system memory which I would cap to 2 or 2.5 GiB by default. Nobody should be hurt by that. > There are also resource limits that may also be somewhat OS-specific. > On GNU/Linux one can use "ulimit -v LIM" where LIM is the virtual > memory limit in KiB. Trying to exceed it will result in ENOMEM just > like when running out of address space. > > Trying to figure out the various limits doesn't sound practical > especially if it is supposed to work with kernels other than Linux. > Simply trying to allocate a lot of memory (to test if it works) is more > realistic but I think it's still dumb. Oh, I though that this isn't Linux only. xz could query it but for liblzma it is imposible since it is part of something bigger. If the user is able to set ulimit, it is reasonable to assume that he can also use -M. While I observed it a few times that a script invoked "xz -T0" as somepoint which led to bad outcome on big iron. Sebastian
