Re: [xz-devel] [PATCH] xz: Avoid warnings due to memlimit if threads are in auto mode.

On 2024-02-28 13:00:08 [+0200], Lasse Collin wrote: > > > There are also messages that are shown when memory limit does affect > > > compressed output (switching to single-threaded mode and LZMA2 > > > dictionary size adjustment). The verbosity requirement of these > > > messages isn't being

Re: [xz-devel] [PATCH] xz: Avoid warnings due to memlimit if threads are in auto mode.

On 2024-02-27 19:17:48 [+0200], Lasse Collin wrote: > Thanks for the patch! We discussed a bit on IRC and everyone thinks > it's on the right track but we are pondering the implementation details > still. > > The thread count messages are shown in situations which don't affect > the compressed

[xz-devel] [PATCH] xz: Avoid warnings due to memlimit if threads are in auto mode.

From: Sebastian Andrzej Siewior If threads are automatically selected then it is possible that their number needs to get reduced in order not to exceed the current memory limit. This "reducing" forces a warning which is printed on stderr to inform the user. The information i

Re: [xz-devel] Testing LZMA_RANGE_DECODER_CONFIG

On 2024-02-18 22:35:03 [+0200], Lasse Collin wrote: > The balance between the hottest locations in the decompressor code > varies depending on the input file. Linux kernel source compresses very > well (ratio is about 0.10). This reduces the benefit of branchless > code. On my main computer I

[xz-devel] Testing LZMA_RANGE_DECODER_CONFIG

Hi, I did some testing on !x86. I changed LZMA_RANGE_DECODER_CONFIG to different values run a test and looked at the MiB/s value. xz_0 means LZMA_RANGE_DECODER_CONFIG was 0, xz_1 means the define was set to 1. I touched src/liblzma/lzma/lzma_decoder.c and rebuilt xz. I pinned the shell to a

Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64

On 2022-11-23 21:12:53 [+0100], To Lasse Collin wrote: > > It is fine to build *static* liblzma with --disable-symbol-versions on > > all archs. Debian-specific workaround is fine in the short term but > > this should be fixed upstream. One method would be to disable the extra > > symbols on ia64

Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64

On 2022-11-23 22:26:39 [+0200], Lasse Collin wrote: > On 2022-11-23 John Paul Adrian Glaubitz wrote: > > Well, Debian builds both the static and dynamic libraries in separate > > steps, so I'm not sure whether the autotools build system would be > > able to detect that. > > I would assume the

Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64

On 2022-11-23 14:33:39 [+0100], John Paul Adrian Glaubitz wrote: > @Sebastian: Can you do that? Does anything speak against that? No, let me do that. > Adrian Sebastian

Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64

On 2022-11-22 18:51:49 [+0100], John Paul Adrian Glaubitz wrote: > Hello! Hi, > [ 36%] Linking CXX shared module ha_archive.so > cd /<>/builddir/storage/archive && /usr/bin/cmake -E > cmake_link_script CMakeFiles/archive.dir/link.txt --verbose=1 > /usr/bin/c++ -fPIC -g -O2

Re: [xz-devel] XZ Utils 5.3.3alpha

On 2022-09-30 20:23:06 [+0300], Lasse Collin wrote: > On 2022-09-29 Guillem Jover wrote: > > On Wed, 2022-09-28 at 21:41:59 +0800, Jia Tan wrote: > > > […] The > > > interface for liblzma and xz for the multi threaded decoder does not > > > have any planned changes, so things could probably be

Re: [xz-devel] XZ Utils 5.3.3alpha

On 2022-09-27 21:29:07 [+0800], Jia Tan wrote: > > Are there any open issues? If not, what needs to be done before the > > final release can happen? > > The 5.4.0 release that will contain the multi threaded decoder is > planned for December. The list of open issues related to 5..4.0 in > general

Re: [xz-devel] XZ Utils 5.3.3alpha

On 2022-08-22 19:52:09 [+0300], Lasse Collin wrote: > XZ Utils 5.3.3alpha is available at . Here is an > extract from the NEWS file: > > - Added threaded .xz decompressor lzma_stream_decoder_mt(). > It can use multiple threads with .xz files that have

Re: [xz-devel] [PATCH v3] liblzma: Add multi-threaded decoder

On 2022-03-11 21:59:19 [+0800], Jia Tan wrote: > > This: > > > > diff --git a/src/liblzma/common/stream_decoder_mt.c > > b/src/liblzma/common/stream_decoder_mt.c > > --- a/src/liblzma/common/stream_decoder_mt.c > > +++ b/src/liblzma/common/stream_decoder_mt.c > > @@ -786,7 +786,7 @@

Re: [xz-devel] [PATCH v3] liblzma: Add multi-threaded decoder

On 2022-03-10 22:16:27 [+0800], Jia Tan wrote: > I started writing tests in the new framework and I found one bug and > one issue. If you want to check out the tests I have so far, here is a > link to check out my progress: >

Re: [xz-devel] [PATCH v3] liblzma: Add multi-threaded decoder

On 2022-03-07 01:08:40 [+0200], Lasse Collin wrote: > Hello! Hi, > I committed something. The liblzma part shouldn't need any big changes, > I hope. There are a few FIXMEs but some of them might actually be fine > as is. The xz side is just an initial commit, there isn't even >

Re: [xz-devel] [PATCH v3] liblzma: Add multi-threaded decoder

On 2022-02-07 01:46:32 [+0200], Lasse Collin wrote: > I now plan to use memlimit_threading and memlimit_stop in the lzma_mt > structure. Documentation is still needed but hopefully those are a bit > more obvious. oki … [ long update ] > I added support for lzma_get_progress(). It is needed to

Re: [xz-devel] [PATCH v3] liblzma: Add multi-threaded decoder

On 2021-12-15 23:33:58 [+0200], Lasse Collin wrote: > Yes. It's fairly simple from implementation point of view but is it > clear enough for the users, I'm not sure. > > I suppose the alternative is having just one limit value and a flag to > tell if it is a soft limit (so no limit for

Re: [xz-devel] [PATCH v3] liblzma: Add multi-threaded decoder

On 2021-11-30 00:25:11 [+0200], Lasse Collin wrote: > Hello! Hello, > On 2021-02-05 Sebastian Andrzej Siewior wrote: > > - Added enum `lzma_memlimit_opt' to lzma_stream_decoder_mt() as an > > init parameter. The idea is to specify how to obey the memory limit > > so

[xz-devel] Multithreaded decompression for XZ Utils.

Hi, just spotted that Christmas is around the corner. I *think* that I've been a good boy over the year. I plan to keep it that way just to be sure. Not trying to push my luck here but what are my chances to find parallel decompression in xz-utils under the christmas tree? Sebastian

Re: [xz-devel] [PATCH v3] liblzma: Add multi-threaded decoder

On 2021-07-20 02:09:31 [+0200], Guillem Jover wrote: > Hi! Hi Guillem, > On Fri, 2021-02-05 at 00:11:57 +0100, Sebastian Andrzej Siewior wrote: > > From: Sebastian Andrzej Siewior > > I've only skimmer very quickly over the patch, but I've been running > it on my system in

Re: [xz-devel] [PATCH v2] liblzma: Add multi-threaded decoder

On 2021-04-11 21:48:39 [+0300], Lasse Collin wrote: > > Yes, sorry, I didn't read carefully enough. no worries. > After the XZ for Java 1.9 release I wanted a short break but it's > already been a month. Outside the Java version there are multiple XZ > things to do (not all on xz-devel) and I

[xz-devel] [PATCH v3] liblzma: Add multi-threaded decoder

From: Sebastian Andrzej Siewior Changes since v2: - use outq - The block is considered as decompressed after LZMA_STREAM_END (not if one of the buffer reach their limit). - If the in-buffer would produce more than allocated for the out-buffer then LZMA_DATA_ERROR is returned

[xz-devel] [PATCH] partial update, end with LZMA_STREAM_END error

From: Sebastian Andrzej Siewior This is an incremental update against the outq patch. I can send an all-in-one patch against the current master branch, I hope this makes the review easier. I wanted to address all issues raised in the previous mail. The "no progress" state is hopefull

Re: [xz-devel] [PATCH v2] liblzma: Add multi-threaded decoder

On 2021-01-24 23:56:15 [+0200], Lasse Collin wrote: > Hello! Hi, > I haven't made much progress with this still, I'm sorry. :-( Below are > comments about a few small details. It's not much but I will (slowly) > keep reading and testing. Thank you. > (1) Segfault due to thr->outbuf == NULL > >

Re: [xz-devel] [RFC 2/2] Add xxHash, XX3 (128bit) for hashing.

On 2021-01-20 00:37:06 [+0100], Sebastian Andrzej Siewior wrote: > So this is better than crc64 and close to none while doing something ;) xz -tv -T0 with crc64 reports: 100 % 10,2 GiB / 40,0 GiB = 0,255 1,1 GiB/s 0:35 and the same archive with xxh3: 100 % 10,2

Re: [xz-devel] [PATCH 1/2] Add support openssl's SHA256 implementation

On 2021-01-20 00:30:08 [+0100], Sebastian Andrzej Siewior wrote: > From: Sebastian Andrzej Siewior > > I created a test file via > dd if=/dev/zero bs=1024k count=1024 | xz -v -0 -Csha256 Another data point: xz -tv -T0 went from 100 % 10,2 GiB / 40,0 GiB = 0,255

[xz-devel] [RFC 2/2] Add xxHash, XX3 (128bit) for hashing.

From: Sebastian Andrzej Siewior After seeing the numbers of of openssl's sha256 vs intree, vs crc64 vs none I decided to look at xxHash 16byte version which got stable in the 0.8 version (given I understood the signs right): | Performance counter stats for './src/xz/.libs/xz -t xxh3.xz' (5 runs

[xz-devel] [PATCH 1/2] Add support openssl's SHA256 implementation

From: Sebastian Andrzej Siewior I created a test file via dd if=/dev/zero bs=1024k count=1024 | xz -v -0 -Csha256 and compared the in-tree sha256 implementation on a Ryzen (CPU acceleration available): | Performance counter stats for 'xz --test sha256.xz' (5 runs

Re: [xz-devel] [PATCH] xz: Fix setting memory limit on 32-bit systems

On 2021-01-18 23:52:50 [+0200], Lasse Collin wrote: > On 2021-01-10 Sebastian Andrzej Siewior wrote: > > I hope for sane defaults :) > > I hope so too. So far I have felt that the suggested solutions have > significant flaws or downsides, and I'm not able to see what is a good &

[xz-devel] [PATCH] Use outq

From: Sebastian Andrzej Siewior This is against the MT decoder to use the outq instead the self cooked one buffer/thread to illustrate what had to be done. It appears to work. The numbers changed from xz -tv 100 % 10,2 GiB / 40,0 GiB = 0,255 958 MiB/s 0:42 xz -dv

Re: [xz-devel] [PATCH v2] liblzma: Add multi-threaded decoder

On 2021-01-09 22:21:23 [+0200], Lasse Collin wrote: > Hello! Hi, > This sounds great but unfortunately I still haven't been able to > properly read it yet. It may take a few more days. I apologize. :-( no need to apologize. > As a minor update from my side, I committed lzma_outq changes that I

Re: [xz-devel] [PATCH v2] xz: Ignore hard link count if not deleting

On 2021-01-09 18:21:45 [+0200], Lasse Collin wrote: > > Ignore hard link count on input. > > I'm fine with the patch since the suggested behavior makes sense to me > too. There's always a risk that some specific use case can break due to > this change but I guess that is acceptably low risk. > >

Re: [xz-devel] [PATCH] xz: Fix setting memory limit on 32-bit systems

On 2021-01-08 18:40:18 [+0200], Lasse Collin wrote: > Hello! Hi, > Sorry for the two-week silence, I was ill. It will take a few days for > me to catch up with the emails. No worries. Take your time to get better. > On 2020-12-26 Sebastian Andrzej Siewior wrote: > > On 2

[xz-devel] [PATCH v2] liblzma: Add multi-threaded decoder

From: Sebastian Andrzej Siewior Changes since v1: - Use the `timeout' parameter / mythread_cond_timedwait() in the main thread. - A little bit of doxygen documentation. - Added enum `lzma_memlimit_opt' to lzma_stream_decoder_mt() as an init parameter. The idea is to specify how to obey

[xz-devel] [PATCH v2] xz: Ignore hard link count if not deleting

From: Sebastian Andrzej Siewior xz refuses to decompress a file which has more than one hard link. It can be reproduced by (as per Vincent): |$ echo foo > file1 |$ xz file1 |$ ln file1.xz file2.xz |$ xz -dk file1.xz |xz: file1.xz: Input file has more than one hard link, skipping This behavi

[xz-devel] Re: [PATCH] xz: Don't care if input has more than one hard link.

On 2020-12-27 18:14:51 [+0100], Vincent Lefevre wrote: > > On 2020-12-27 15:47:05 +0100, Sebastian Andrzej Siewior wrote: > > > I traced it back in `gzip' to the 90s but the change was not documented > > > as such. It was moved, altered but not documented. At so

[xz-devel] Re: [PATCH] xz: Don't care if input has more than one hard link.

On 2020-12-27 18:02:44 [+0100], Vincent Lefevre wrote: > On 2020-12-27 15:47:05 +0100, Sebastian Andrzej Siewior wrote: > > xz refuses to decompress a file which has more than one hard link. It > > can be reproduced byi (as per Vincent): > > |$ echo foo > file1 > >

[xz-devel] [PATCH] xz: Don't care if input has more than one hard link.

From: Sebastian Andrzej Siewior xz refuses to decompress a file which has more than one hard link. It can be reproduced byi (as per Vincent): |$ echo foo > file1 |$ xz file1 |$ ln file1.xz file2.xz |$ xz -d file1.xz |xz: file1.xz: Input file has more than one hard link, skipping This behavi

Re: [xz-devel] [PATCH] xz: Fix setting memory limit on 32-bit systems

On 2020-12-26 09:33:04 [+0300], Vitaly Chikunov wrote: > This wasn't working, because `memlimit_compress` initialized with zero, > thus memory limit is never lowered for 32-bit address space, causing > `Cannot allocate memory' error (in `lzma_outq_init()'). For example, > when `-T0' is used on 32

[xz-devel] [PATCH] xzdiff: Trap SIGPIPE

From: Sebastian Andrzej Siewior The `cmp' command will return early if a difference is found while the shell script is still invoking the decompressor which writes into the closed FD. This results in SIGPIPE / exit code 141. By ignoring SIGPIPE the real return code from `cmp' is observed which

Re: [xz-devel] [RFC PATCH] liblzma: Test memory availability for lzma_stream_encoder_mt

On 2020-12-24 00:51:58 [+0300], Vitaly Chikunov wrote: > Add guard call to allocate the memory before calling get_thread to > prevent memory failures (LZMA_MEM_ERROR). > > This would make simple `xz -T0' more robust on 32-bit architectures. > > Rationale: simple `-T0' is hard to use portably in

Re: [xz-devel] [RFC WIP] liblzma: Add multi-threaded decoder

On 2020-12-17 00:12:06 [+0200], Lasse Collin wrote: > When decoding one can get smoother output by copying decompressed data > out in smaller chunks. Your code does this but it's still a single > mutex for all threads. With many threads that is easily thousands or > even tens of thousands of

[xz-devel] [PATCH] liblzma: Add multi-threaded decoder

From: Sebastian Andrzej Siewior Changes since last post: - Options are considered - Memory limits are considered. The limit may get exceeded if we get close to it and then the existing threads enlarge their in-buffer. - Blocks with no size information in the header can be decompressed

Re: [xz-devel] [RFC WIP] liblzma: Add multi-threaded decoder

On 2020-12-13 23:19:25 [+0200], Lasse Collin wrote: > > Threads, which finished decoding, remain idle until their output > > buffer has been fully consumed. The output buffer once allocated > > remains allocated until the thread is cleaned up. This saved 5 secs > > in the example above compared to

[xz-devel] [RFC WIP] liblzma: Add multi-threaded decoder

From: Sebastian Andrzej Siewior This is WIP, the decoder appears to work based on: |$ xz -dv < buster-pl.xz | openssl sha1 | 100 % 10,2 GiB / 40,0 GiB = 0,255 114 MiB/s 6:00 |(stdin)= 5eb4e2a3ce2253a6ec3fc86ee7ad8db0a5395959 | |vs | |$ ./src/xz/.libs/xz -dv < buster