Hi, On Fri, Jan 08, 2021 at 06:40:18PM +0200, Lasse Collin wrote: > On 2020-12-26 Sebastian Andrzej Siewior wrote: > > On 2020-12-26 09:33:04 [+0300], Vitaly Chikunov wrote: > > > This wasn't working, because `memlimit_compress` initialized with > > > zero, thus memory limit is never lowered for 32-bit address space, > > > causing `Cannot allocate memory' error (in `lzma_outq_init()'). For > > > example, when `-T0' is used on 32 CPUs with compression level > > > higher than `-6'. > > > > That is one way. It might be that hardware_init() should pass > > `total_ram' to hardware_memlimit_set() instead of 0. > > hardware_memlimit_get() treats 0 as unlimited but I don't think it > > makes sense since memory is never unlimited. > > 0 means disabled, that is, xz is expected to behave just like most > other programs that might allocate a lot of memory but don't have any > internal memory usage limiting. Memory isn't unlimited but many > programs sort of behave as if it were and fail hard if allocation > fails. That's not robust but it seems to work most of the time and > many seem find this to be acceptable behavior in general.
Few percent of the people report bugs or misfeatures, most just adapt or abandon usage. For example, 1. We added -T0 to our rpm building system for xz call, and when it's failed (tests) the change is just reverted. Yes, we didn't gain speed up, but things are working again. How much people did the same? 2. More important problem is not just with xz tool, but with liblzma xz mt compression. For example, in Fedora's RPM they added multi-thread xz compression in (note: because it's usage is configurable by other means this commit does not mean they stated use it at the time): https://github.com/rpm-software-management/rpm/commit/7740d1098642cd42f725fb9a2a3819ceaf51a80b then, guess what, it didn't worked on 32-bit systems. Did you have bug report about it? People tried to solve it by themselves with so complicated code (which we found out lateris not always effective): https://github.com/rpm-software-management/rpm/commit/a60f36a55cff3331e8bef3a1ab95c87d313911bb Then they optimized it to use 4 threads on 32-bit architectures. https://github.com/rpm-software-management/rpm/commit/4dfb381742c27449aafcd907c05f3a84015ffb54 (I think, they switched to zstd after all.) In our rpmbuild we are using lzma, but wanted to speed up compression and use multi-threaded xz, so we optimistically merged these upstream changes: http://git.altlinux.org/gears/r/rpm-build.git?p=rpm-build.git;a=commit;h=a4bace177116163e8f6cb9ce6e31f94252e06775 Again, we got the same problem as them, and thought to reserve 128MiB will be enough: http://git.altlinux.org/gears/r/rpm-build.git?p=rpm-build.git;a=commitdiff;h=0b18c0498e1caa14149369c168440ede6aa664b2 This turned out to be not enough, so we increased to 256MiB: http://git.altlinux.org/gears/r/rpm-build.git?p=rpm-build.git;a=commitdiff;h=afe6605589afaf241b360c32d6e03d4aa0c271cd We have to build and test re-build thousands of packages on many architectures every day, some of them got compression failures So, there is new solution - increase memory limit to 512MiB: http://git.altlinux.org/gears/r/rpm-build.git?p=rpm-build.git;a=commitdiff;h=f5fcb8f43d99f718bc339c5eb1dbecefc717b706 It tuned out to be not enough for some rare (and huge) packages (like llvm). So, there is proposed new ugly fix: http://git.altlinux.org/people/vt/packages/?p=rpm-build.git;a=commitdiff;h=ca3433a6aabf98a837e69f78f7f3426940c23ec5 This pass all our tests, yet. (What else we could do not changing liblzma and still wanting to not abandon T0?) So, this is not like happy careless life out there. > The whole limiter feature exist because I felt it was good to have a > mechanism to control the memory usage, especially when decompressing > since a .xz file may cause xz to allocate 4 GiB of memory for a single > thread. However, I think few people think the same and thus the limiter > is off by default for both compression and decompression. > > > Also, 32bit with almost 4GiB as a limit is working. If you increase > > your input (the example from your previous email) then you also end > > up "can not allocate memory) simply because 32bit can not allocate > > 4GiB of memory. I'm not sure if the actual memory limit is exported. > > It is usually at around 3GiB but there architectures which allow less > > than that (not to mention kernel configurations). > > I don't know if Linux makes it possible for userspace applications to > know the available address space. It can indeed vary depending on the > kernel config. xz also needs to be portable to many other kernels. The > 4020 MiB hack works with 64-bit kernels running 32-bit applications > since in that case many kernels provide 4 GiB of address space. Knowing available memory is _not different_ from guard malloc(3) call, but even worse, because it does not guarantee malloc(3) will succeed (by so many reason). Only proper solution, of course, is to make liblzma mt code robust on per-thread memory allocation. > There are also resource limits that may also be somewhat OS-specific. > On GNU/Linux one can use "ulimit -v LIM" where LIM is the virtual > memory limit in KiB. Trying to exceed it will result in ENOMEM just > like when running out of address space. > > Trying to figure out the various limits doesn't sound practical > especially if it is supposed to work with kernels other than Linux. > Simply trying to allocate a lot of memory (to test if it works) is more > realistic but I think it's still dumb. My ugly and dumb hack does not meant to be merged as is, but (RFC) invitation to discussion and would show your attitude to the problem. Thanks, > > -- > Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
