Lasse,

On Sat, Dec 26, 2020 at 05:03:48PM +0200, Lasse Collin wrote:
> On 2020-12-24 Vitaly Chikunov wrote:
> > Add guard call to allocate the memory before calling get_thread to
> > prevent memory failures (LZMA_MEM_ERROR).
> > 
> > This would make simple `xz -T0' more robust on 32-bit architectures.
> > 
> > Rationale: simple `-T0' is hard to use portably in scripts on
> > different platforms, because there is always different amount of RAM
> > and CPU, causing unexpected "xz: Cannot allocate memory" errors.
> 
> Yes, -T0 is a bit broken as on systems with many cores it may attempt
> to use ridiculous amount of resources. Enabling a memory usage limit
> can help, and starting with xz 5.2.5 there is a small hack that makes
> memory usage limiting work better with 32-bit xz binaries when running
> under a 64-bit kernel. From the NEWS file:
> 
>     - Added a special case for 32-bit xz: If --memlimit-compress is
>       used to specify a limit that exceeds 4020 MiB, the limit will
>       be set to 4020 MiB. The values "0" and "max" aren't affected
>       by this and neither is decompression. This hack can be
>       helpful when a 32-bit xz has access to 4 GiB address space
>       but the specified memlimit exceeds 4 GiB. This can happen
>       e.g. with some scripts.
> 
> Thus one can use "xz -T0 --memlimit-compress=100%" or 90% or whatever
> number one finds good. If that percentage results in a value over 4020
> MiB with a 32-bit xz, the limit is reduced to 4020 MiB. This should
> make things work quite often.
> 
> Have you tried something like the above or does it have some other
> problem?

Thanks for the suggestion! I will experiment more with this approach.

> > +           // Test memory availability.
> > +           void *mem = lzma_alloc(coder->block_size * 6, allocator);
> > +           if (!mem) {
> > +                   coder->threads_max = coder->threads_initialized;
> > +                   return LZMA_OK;
> > +           } else
> > +                   lzma_free(mem, allocator);
> > +
> 
> This is a too ugly hack. It makes assumptions that aren't always true.
> A better way would be to add an option to make the encoder tolerate
> allocation failures and be able to continue with fewer threads.

I thought about this and even experimented with the code. But, it
requires much rework, because thread that will fail to allocate memory
at some later stage may already got (asynchronously) much of input data,
that, after failure, should be returned back to pipe to some other thread.

Thanks,


> However, I haven't heard complaints about this outside xz -T0 context
> and I think the existing hack isn't horrible.
> 
> -- 
> Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode

Reply via email to