Re: xz: (stdin): Cannot allocate memory

[Lampshade]

I figured out that my default is:
ulimit -d

1572864

echo "1572864/1024" | bc
1536

Value, which lets me compress using this setting is
between 1682864 and 1672864 kilobytes.

I have also discovered command line option for xz --memlimit=
Now my command looks that:

cat archive.tar | xz -zf --memlimit=1600MiB  \
--format=xz -9e --threads=2 - > archive.tar.xz

but I get:
xz: Adjusted the number of threads from 2 to 1 to not exceed the memory
usage limit of 1600 MiB

1600 is clearly larger than 674*2=1348

In the end I can compress, but I think that something is wrong.

Od: "Christian Weisgerber" <na...@mips.inka.de>
Do: "Lampshade" <lampsh...@poczta.fm>;
Wysłane: 16:25 Sobota 2016-01-30
Temat: Re: xz: (stdin): Cannot allocate memory

> Lampshade:
>
> > I have following error:
> > cat archive.tar | xz -zf --format=xz -9e --threads=2 - >
archive.tar.xz
> > xz: (stdin): Cannot allocate memory
>
> You are using the most extreme compression setting, which requires
> about 674 MB per thread according to the xz(1) man page.  This
> causes you to bump against the data size limit (ulimit -d, see
> ksh(1)).
>
> You need to raise the limit or use a less greedy compression setting.
>
> --
> Christian "naddy" Weisgerber  na...@mips.inka.de

Re: xz: (stdin): Cannot allocate memory

[Lampshade]

This xz command worked in past so I think something must
have been changed in past. Indeed, this command worked
when I had 4G of DDR3@1333Mhz RAM. Now I have 6GB DDR3
on the same laptop so I have even more.
I will look at ulimit -d this evening. I didn't changed them manually, so they
must have been changed during upgrade from current to current.

Re: xz: (stdin): Cannot allocate memory

[Brian Conway]

Have you read the xz man page, specifically on memory requirements?
9-extreme is probably not what you want - it's fairly insane/placebo. Full
dmesg with memory information is needed beyond that.

Brian Conway


On Jan 30, 2016 7:18 AM, "Lampshade" <lampsh...@poczta.fm> wrote:

> Hello
> I have this OS with packages as of yesterday (Jan 29):
> kern.version=OpenBSD 5.9-beta (GENERIC.MP) #1865: Thu Jan 28 20:18:15 MST
> 2016
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> and also tested on with packages around Jan 17:
> kern.version=OpenBSD 5.9-beta (GENERIC.MP) #1846: Sun Jan 17 02:34:54 MST
> 2016
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> I have following error:
> cat archive.tar | xz -zf --format=xz -9e --threads=2 - > archive.tar.xz
> xz: (stdin): Cannot allocate memory

Re: xz: (stdin): Cannot allocate memory

[Christian Weisgerber]

Lampshade:

> I have following error:
> cat archive.tar | xz -zf --format=xz -9e --threads=2 - > archive.tar.xz 
> xz: (stdin): Cannot allocate memory

You are using the most extreme compression setting, which requires
about 674 MB per thread according to the xz(1) man page.  This
causes you to bump against the data size limit (ulimit -d, see
ksh(1)).

You need to raise the limit or use a less greedy compression setting.

-- 
Christian "naddy" Weisgerber  na...@mips.inka.de

xz: (stdin): Cannot allocate memory

[Lampshade]

Hello
I have this OS with packages as of yesterday (Jan 29):
kern.version=OpenBSD 5.9-beta (GENERIC.MP) #1865: Thu Jan 28 20:18:15 MST 2016
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP

and also tested on with packages around Jan 17:
kern.version=OpenBSD 5.9-beta (GENERIC.MP) #1846: Sun Jan 17 02:34:54 MST 2016
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP

I have following error:
cat archive.tar | xz -zf --format=xz -9e --threads=2 - > archive.tar.xz 
xz: (stdin): Cannot allocate memory

Re: xz: (stdin): Cannot allocate memory

[Christian Weisgerber]

On 2016-01-30, Lampshade  wrote:

> xz: Adjusted the number of threads from 2 to 1 to not exceed the memory
> usage limit of 1600 MiB
>
> 1600 is clearly larger than 674*2=1348

A closer reading of the man page reveals that memory consumption
is even higher in multi-threaded mode.

In multi-threaded mode about three times _size_ bytes will be
allocated in each thread for buffering input and output.  The
default _size_ is three times the LZMA2 dictionary size or 1 MiB,
whichever is more.

At -9, the dictionary size is 64 MB, so this adds another 576 MB
per thread.  A quick check with top(1) confirms that xz in
multi-threaded mode allocates 1250 MB per thread at compression
level 9.

> In the end I can compress, but I think that something is wrong.

The last relevant change is that we actually enabled multi-threading.
Before that, the -T option was silently ignored.  However that
change was ten months ago and is already in the 5.8 packages.

Anyway, there is nothing wrong, and as I said before:
You need to raise the limit or use a less greedy compression setting.

-- 
Christian "naddy" Weisgerber  na...@mips.inka.de