On Sat, Dec 26, 2020 at 05:04:02PM +0200, Lasse Collin wrote:
> On 2020-12-26 Vitaly Chikunov wrote:
> > This wasn't working, because `memlimit_compress` initialized with
> > zero, thus memory limit is never lowered for 32-bit address space,
> > causing `Cannot allocate memory' error (in `lzma_outq_init()'). For
> > example, when `-T0' is used on 32 CPUs with compression level higher
> > than `-6'.
> Zero means disabling the limiter. That is the default. The patch would
> change this and enable a memory usage limit by default on 32-bit
> platforms.
> If you want to enable a limiter by default, you may use the environment
> variables XZ_DEFAULTS or XZ_OPT (see the man page).
> The whole memory usage limiting feature is a bit controversial and I'm
> hesitant to enable anything by default (although with threaded
> decompression there may need to be some default *soft* limit). Your
> suggestion is reasonable but someone else may prefer that xz gives a
> hard error if too high settings are specified instead of xz trying to
> outsmart the user and change the settings automatically. Perhaps that
> sounds silly but I don't have energy for endless arguments where both
> sides are kind of right but both wishes cannot be fulfilled at the
> same time.
> I cannot make everyone happy.

Wow, that's philosophical! I think, we should solve this fundamental
problem first. -- Even if we cannot satisfy everybody, better than
satisfying just one party and make other unhappy, we can give users
choice. If that's approach is accepted we can rework patch to make it

Then, I think, better approach is 'sane defaults', thus more use cases
are working correctly out of the box. (Most users don't need to test
memory subsystem they need compression from xz.)

Users who want (by hard to imagine unknown reason) xz to fail on 32-bit
arches would use `--no-adjust` -- we could (rework the patch to) disable
memory auto-fitting logic if this option is used.  Does it sound good?

Enabling (robust behavior) by default is better than not, because -- you
do not shift burden to users: 1) unexpected consequences (of xz not
working in some corner cases, somewhere), 2) then, user would have hard
time solving the issue (man, googling, reading sources). -- By what
combination of options of xz to solve this (if she is not expert in xz
usage hacks). For example, percentage memory limit on 32-bit systems is
calculated against whole memory and not against 'physical' 4MiB limit --
user should somehow find this, probably by trial and error, wasting her

So, to make `xz -T0` "just work" on all possible systems, single portable
approach is just disabling `-T0` and wasting opportinity to speed up
compression real time.
Second option is to make correct xz option generator depending on system
configuration -- she should understand virtual memory and how xz works,
and still may miss something.

By this, I think its always better that program works by default.

Your suggestion (in previous mail) to use `--memlimit-compress=100%`
seems to offset this patch. But, still, need rare expert to know about
this hack. Also, it would be easier for users (I'm also suggested be
colleague) if `--memlimit-compress=100%` is setting enabled by default,
and if someone wants to disable limiter she would use

I reason like this: Setting [non-zero value to] `--memlimit-compress=` 
_increases_ use cases by avoiding memory errors, in comparison to not
setting it [or setting it to 0]. So it should be enabled by default.


> -- 
> Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode

Reply via email to