Re: [xz-devel] XZ Utils 5.3.3alpha

[Sebastian Andrzej Siewior]

On 2022-09-30 20:23:06 [+0300], Lasse Collin wrote:
> On 2022-09-29 Guillem Jover wrote:
> > On Wed, 2022-09-28 at 21:41:59 +0800, Jia Tan wrote:
> > > […] The
> > > interface for liblzma and xz for the multi threaded decoder does not
> > > have any planned changes, so things could probably be developed and
> > > tested using 5.3.3.  
> > 
> > Ah, thanks, that's reassuring then. It's one of the things I was
> > worried about when having to decide whether to merge the patch I've
> > got implementing this support into dpkg. So, once the alpha version
> > has been packaged for Debian experimental, I'll test the patch and
> > commit it.
> 
> There are no planned changes but that isn't a *promise* that there won't
> be any changes before 5.4.0.
>
> I don't track API or ABI compatibility within development releases and
> thus binaries linked against shared liblzma from one alpha/beta release
> won't run with liblzma from the next alpha/beta *if* they depend on
> unstable symbols (symbol versioning stops it). This includes the xz
> binary itself and would include dpkg too if it uses the threaded
> decoder.

That should be no problem. The last alpha version has been uploaded to
debian experimental and it is exactly the place for such things. So dpkg
could be linked against that version in experimental and will never
enter an official release.

Sebastian

Re: [xz-devel] XZ Utils 5.3.3alpha

[Lasse Collin]

On 2022-09-29 Guillem Jover wrote:
> On Wed, 2022-09-28 at 21:41:59 +0800, Jia Tan wrote:
> > […] The
> > interface for liblzma and xz for the multi threaded decoder does not
> > have any planned changes, so things could probably be developed and
> > tested using 5.3.3.  
> 
> Ah, thanks, that's reassuring then. It's one of the things I was
> worried about when having to decide whether to merge the patch I've
> got implementing this support into dpkg. So, once the alpha version
> has been packaged for Debian experimental, I'll test the patch and
> commit it.

There are no planned changes but that isn't a *promise* that there won't
be any changes before 5.4.0.

I don't track API or ABI compatibility within development releases and
thus binaries linked against shared liblzma from one alpha/beta release
won't run with liblzma from the next alpha/beta *if* they depend on
unstable symbols (symbol versioning stops it). This includes the xz
binary itself and would include dpkg too if it uses the threaded
decoder.

Sometimes it can be worked around with distro-specific patches but
that's extra hassle and can go wrong too. Please don't end up with a
similar result that happened with RHEL/CentOS 7 which ended up
affecting users of other distributions too (this is included in 5.2.7):


https://git.tukaani.org/?p=xz.git;a=commitdiff;h=913ddc5572b9455fa0cf299be2e35c708840e922

So while I encourage testing, one needs to be careful when it can
affect critical tools in the operating system. :-)

-- 
Lasse Collin

Re: [xz-devel] XZ Utils 5.3.3alpha

[Lasse Collin]

On 2022-09-28 Jia Tan wrote:
> On 2022-09-27 Sebastian Andrzej Siewior wrote:
> > Okay, so that is what you are tracking. I remember that there was a
> > stall in the decoding but I don't remember how it played out.
> >
> > I do remember that I had something for memory allocation/ limit but
> > I don't remember if we settled on something or if discussion is
> > needed. Also how many decoding threads make sense, etc.  
> 
> We ended up changing xz to use (total_ram / 4) as the default "soft
> limit". If the soft limit is reached, xz will decode single threaded.
> The "hard limit" shares the same environment variable and xz option
> (--memlimit-decompress).

There is also the 1400 MiB cap for 32-bit executables.

The memory limiting in threaded decompression (two separate limits in
parallel) is one thing where feedback would be important as after the
liblzma API, ABI and xz tool syntax are in a stable release, backward
compatibility has to be maintained.

Another thing needing feedback is the new behavior of -T0 when no
memlimit has been specified. Now it has a default soft limit. I hope it
is an improvement but quite possibly it could be improved. Your
suggestion to use MemAvailable on Linux is one thing that could be
included if people think it is a good way to go as a Linux-specific
behavior (having more benefits than downsides).

These are documented on the xz man page. I hope it is clear enough. It
feels a bit complicated, which is a bad sign but on the other hand I
feel the underlying problem isn't as trivial as it seems on the surface.

So far Jia Tan and I have received no feedback about these things at
all. I would prefer to hear the complaints before 5.4.0 is out. :-)

> > This reminds me that I once posted a patch to use openssl for the
> > sha256.
> > https://www.mail-archive.com/xz-devel@tukaani.org/msg00429.html
> >
> > Some distro is using sha256 instead crc64 by default, I don't
> > remember which one… Not that I care personally ;)  
> 
> I am unsure if we will have time to include your sha256 patch, but if
> we finish all the tasks with extra time it may be considered.

There's more to this than available time. 5.1.2alpha added support for
using SHA-256 from the OS base libraries (not OpenSSL) but starting with
5.2.3 it is disabled by default. Some OS libs use (or used to use) the
same symbol names for SHA-256 functions as OpenSSL while having
incompatible ABI. This lead to weird problems when an application
needed both liblzma and OpenSSL as liblzma ended up calling OpenSSL
functions. Plus, some of the OS-specific implementations were slower
than the C code in liblzma (OpenSSL would be faster).

OpenSSL's license has compatibility questions with GNU GPL. If I
remember correctly, some distributions consider OpenSSL to be part of
the core operating system and thus avoid the compatibility problem with
the GPL. I'm not up to date how distros handle it in 2022 but perhaps
it should be taken into account so that apps depending on liblzma won't
get legally unacceptable OpenSSL linkage. So if OpenSSL support is
added it likely should be disabled by default in configure.ac.

> > > This is everything currently planned.

Translations need to be updated too once the strings and man pages are
close to final. A development release needs to be sent to the
Translation Project at some point. If people want to translate the man
pages too, they will need quite a bit of time.

-- 
Lasse Collin

Re: [xz-devel] XZ Utils 5.3.3alpha

[Guillem Jover]

On Wed, 2022-09-28 at 21:41:59 +0800, Jia Tan wrote:
> […] The
> interface for liblzma and xz for the multi threaded decoder does not
> have any planned changes, so things could probably be developed and
> tested using 5.3.3.

Ah, thanks, that's reassuring then. It's one of the things I was
worried about when having to decide whether to merge the patch I've got
implementing this support into dpkg. So, once the alpha version has been
packaged for Debian experimental, I'll test the patch and commit it.

> This would actually help us because having people
> test and give us feedback on both performance and the interface would
> help before committing to things in the stable release.

Given that this will be disabled at configure time (until the support
is in Debian unstable), I'm not sure we'll have many people testing
this, but I guess it will make it possible for people wanting to test
it to do it more easily. And there's always the option to do that over
the entire Debian archive or similar.

Thanks,
Guillem

Re: [xz-devel] XZ Utils 5.3.3alpha

[Jia Tan]

> Okay, so that is what you are tracking. I remember that there was a
> stall in the decoding but I don't remember how it played out.
>
> I do remember that I had something for memory allocation/ limit but I
> don't remember if we settled on something or if discussion is needed.
> Also how many decoding threads make sense, etc.

We ended up changing xz to use (total_ram / 4) as the default "soft
limit". If the soft limit is reached, xz will decode single threaded.
The "hard limit" shares the same environment variable and xz option
(--memlimit-decompress).

> > - New ARM64 filter needs to be properly coordinated to other xz
> > implementations and documented.
> > - Converting tests to the new tuktest framework. Most of the tests
> > have been written, but they still need to be reviewed.
> > - liblzma and xz functionality to convert a string into a filter
> > chain. A draft of this is on the mailing list already, but the syntax
> > needs finalizing and the code was not polished.
> > - A patch for .lz support needs review.
> > - A patch for crc64 optimizations needs review.
>
> This reminds me that I once posted a patch to use openssl for the
> sha256.
> https://www.mail-archive.com/xz-devel@tukaani.org/msg00429.html
>
> Some distro is using sha256 instead crc64 by default, I don't remember
> which one… Not that I care personally ;)

I am unsure if we will have time to include your sha256 patch, but if
we finish all the tasks with extra time it may be considered.

> > - Misc. minor bug fixes.
> >
> > This is everything currently planned. Most things are done and just
> > needs review and minor improvements. Don't worry, multi threaded
> > decompression will be coming to xz in a stable release very soon!
>
> Okay. That is good to hear. I would like to get it in Debian and have
> dpkg support for the upcomming stable release. The earlier the better
> since this affects quite a large part of the system. The toolchain
> freeze is in January and I think that dpkg is part of it (or people will
> probably get very nervous if such a change gets integrated later in the
> cycle).

Thank you for notifying us about the January freeze. I hope this is
the extra bit of motivation needed for us to release 5.4.0 as soon as
it is ready.

Lasse is confident that we will have the release by December. I will
see what we can do to make it as early December as possible since I
understand not wanting to make large changes just before a freeze. The
interface for liblzma and xz for the multi threaded decoder does not
have any planned changes, so things could probably be developed and
tested using 5.3.3. This would actually help us because having people
test and give us feedback on both performance and the interface would
help before committing to things in the stable release.

Thanks again Sebastian for your contributions to both xz and Debian's use of xz!

Jia Tan

Re: [xz-devel] XZ Utils 5.3.3alpha

[Sebastian Andrzej Siewior]

On 2022-09-27 21:29:07 [+0800], Jia Tan wrote:
> > Are there any open issues? If not, what needs to be done before the
> > final release can happen?
> 
> The 5.4.0 release that will contain the multi threaded decoder is
> planned for December. The list of open issues related to 5..4.0 in
> general that I am tracking are:
> 
> - Final tweaks to multi threaded decoder (error handling may need
> improvements since the worker threads stay running in some cases when
> they should not).

Okay, so that is what you are tracking. I remember that there was a
stall in the decoding but I don't remember how it played out.

I do remember that I had something for memory allocation/ limit but I
don't remember if we settled on something or if discussion is needed.
Also how many decoding threads make sense, etc.

> - New ARM64 filter needs to be properly coordinated to other xz
> implementations and documented.
> - Converting tests to the new tuktest framework. Most of the tests
> have been written, but they still need to be reviewed.
> - liblzma and xz functionality to convert a string into a filter
> chain. A draft of this is on the mailing list already, but the syntax
> needs finalizing and the code was not polished.
> - A patch for .lz support needs review.
> - A patch for crc64 optimizations needs review.

This reminds me that I once posted a patch to use openssl for the
sha256. 
https://www.mail-archive.com/xz-devel@tukaani.org/msg00429.html

Some distro is using sha256 instead crc64 by default, I don't remember
which one… Not that I care personally ;)

> - Misc. minor bug fixes.
>
> This is everything currently planned. Most things are done and just
> needs review and minor improvements. Don't worry, multi threaded
> decompression will be coming to xz in a stable release very soon!

Okay. That is good to hear. I would like to get it in Debian and have
dpkg support for the upcomming stable release. The earlier the better
since this affects quite a large part of the system. The toolchain
freeze is in January and I think that dpkg is part of it (or people will
probably get very nervous if such a change gets integrated later in the
cycle).

> Jia Tan

Sebastian

Re: [xz-devel] XZ Utils 5.3.3alpha

[Sam James]

> On 27 Sep 2022, at 14:29, Jia Tan  wrote:
> 
>> Are there any open issues? If not, what needs to be done before the
>> final release can happen?
> 
> The 5.4.0 release that will contain the multi threaded decoder is
> planned for December. The list of open issues related to 5..4.0 in
> general that I am tracking are:
> 
> - Final tweaks to multi threaded decoder (error handling may need
> improvements since the worker threads stay running in some cases when
> they should not).
> - New ARM64 filter needs to be properly coordinated to other xz
> implementations and documented.
> - Converting tests to the new tuktest framework. Most of the tests
> have been written, but they still need to be reviewed.
> - liblzma and xz functionality to convert a string into a filter
> chain. A draft of this is on the mailing list already, but the syntax
> needs finalizing and the code was not polished.
> - A patch for .lz support needs review.
> - A patch for crc64 optimizations needs review.
> - Misc. minor bug fixes.
> 
> This is everything currently planned. Most things are done and just
> needs review and minor improvements. Don't worry, multi threaded
> decompression will be coming to xz in a stable release very soon!
> 

Fantastic news! Thank you! I'm testing out the alpha and all is well so far.

> Jia Tan
> 

Best,
sam


signature.asc
Description: Message signed with OpenPGP

Re: [xz-devel] XZ Utils 5.3.3alpha

[Jia Tan]

> Are there any open issues? If not, what needs to be done before the
> final release can happen?

The 5.4.0 release that will contain the multi threaded decoder is
planned for December. The list of open issues related to 5..4.0 in
general that I am tracking are:

- Final tweaks to multi threaded decoder (error handling may need
improvements since the worker threads stay running in some cases when
they should not).
- New ARM64 filter needs to be properly coordinated to other xz
implementations and documented.
- Converting tests to the new tuktest framework. Most of the tests
have been written, but they still need to be reviewed.
- liblzma and xz functionality to convert a string into a filter
chain. A draft of this is on the mailing list already, but the syntax
needs finalizing and the code was not polished.
- A patch for .lz support needs review.
- A patch for crc64 optimizations needs review.
- Misc. minor bug fixes.

This is everything currently planned. Most things are done and just
needs review and minor improvements. Don't worry, multi threaded
decompression will be coming to xz in a stable release very soon!

Jia Tan

Re: [xz-devel] XZ Utils 5.3.3alpha

[Sebastian Andrzej Siewior]

On 2022-08-22 19:52:09 [+0300], Lasse Collin wrote:
> XZ Utils 5.3.3alpha is available at . Here is an
> extract from the NEWS file:
> 
> - Added threaded .xz decompressor lzma_stream_decoder_mt().
>   It can use multiple threads with .xz files that have multiple
>   Blocks with size information in Block Headers. The threaded
>   encoder in xz has always created such files.

Are there any open issues? If not, what needs to be done before the
final release can happen?

Sebasttian