Bug#829100: lintian: [patch] Warn about over-eagerly xz-compressed data.tar.xz

[Chris Lamb]

tags 829100 + pending
thanks

Thanks for all your input. To prevent this issue going stale, I've
gone ahead and implemented the most basic regex-based checking. No
need to be "too" clever, right?

  
https://anonscm.debian.org/git/lintian/lintian.git/commit/?id=1f8051cb211c53a66c6c526828eee3fa90847cc9

We can naturally re-open this later if there is scope for
improvement or refinement. :)


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#829100: lintian: [patch] Warn about over-eagerly xz-compressed data.tar.xz

[Guillem Jover]

Hi!

On Thu, 2016-06-30 at 17:23:46 +0200, Christoph Biedl wrote:
> Package: lintian
> Version: 2.5.45
> Severity: wishlist
> Tags: patch

> as not known to everybody, xz's higher compression levels have -
> besides improving compression of big files - the side effect of taking
> a lot of memory for the dictionary, even when unpacking. There is
> however no sense in using a compression level that (roughly) takes
> more DictSize than the size of the uncompressed file. [1] has a
> discussion on this,

Thanks for filing this, I had it on my TODO after I posted it on
debian-devel in 2014! Something else I don't have to do. :)

  

> | $ ar x traceroute_1%3a2.0.20-2+b1_armel.deb data.tar.xz
> | $ xz --list --verbose --verbose data.tar.xz
> | (...)
> |   Compressed size:47,9 KiB (49.056 B)
> |   Uncompressed size:  130,0 KiB (133.120 B)
> | (...)
> |   Memory needed:  65 MiB
> | (...)

As mentioned on that thread, you might want to use --robot which is
the supported interface for other tools to use.

Thanks,
Guillem

Bug#829100: lintian: [patch] Warn about over-eagerly xz-compressed data.tar.xz

[Jakub Wilk]

* Christoph Biedl , 2016-06-30, 22:56:

So we would only bother people who overrode the default

add: In an unreasonable way.


Right.

However I'm undecided now how to deal with that. Using lintian, 
actually promoted by the developer's reference 7.1.1, is somewhat late 
in the chain. It was way better if dpkg-deb could automatically decide 
upon an optimal compression value. After a quick glance into the 
sources this seems possible but not very nice to do. Afterwards, 
dpkg-deb could discourage setting the compression level manually, and 
lintian could enforce that.


"Optimal" is subjective.

It makes sense for dpkg to use lower compression level than the user 
requested when the tarball is small.


But I don't think dpkg should use anything beyond -6 just because the 
tarball is big. Higher compression levels should be used only when 
explicitly requested by user, because their costs are non-negligible and 
their benefits are greatly variable.


--
Jakub Wilk

Bug#829100: lintian: [patch] Warn about over-eagerly xz-compressed data.tar.xz

[Christoph Biedl]

Jakub Wilk wrote...

> dpkg's current default is -6, which uses 9MiB of memory for decompression.
> Before dpkg 1.15.6, the default was -9, which uses 65 MiB.
> 
> If I understand Christoph's patch correctly, the tag won't be emitted if
> memory needed for decompression is 10 MiB or less.

Correct.

> So we would only bother people who overrode the default

add: In an unreasonable way.

However I'm undecided now how to deal with that. Using lintian,
actually promoted by the developer's reference 7.1.1, is somewhat
late in the chain. It was way better if dpkg-deb could automatically
decide upon an optimal compression value. After a quick glance into
the sources this seems possible but not very nice to do. Afterwards,
dpkg-deb could discourage setting the compression level manually, and
lintian could enforce that. 

At the moment, some 44[1] packages need a review, something that
might lead into a MBF.

Christoph

[1] https://codesearch.debian.net/search?q=-Zxz+-z


signature.asc
Description: Digital signature

Bug#829100: lintian: [patch] Warn about over-eagerly xz-compressed data.tar.xz

[Russ Allbery]

Christoph Biedl  writes:

> The patch attached is just a proof of concept and not ready for
> production yet, especially since data.tar.xz is unpacked (and later
> removed) to the current working directory.

Does dh_builddeb (or dpkg-deb) already pick appropriate options to avoid
this problem?  I think this sort of Lintian warning is great for packages
that explicitly overrode default behavior to use higher compression
unnecessarily, but if they're just calling dh or dh_builddeb with no
special options, I'm worried about Lintian encouraging people to add a
bunch of overrides to fix the problem in the wrong place.

If dh_builddeb or dpkg-deb is already smart about this, there shouldn't be
a concern; if they're not, it might make sense to gate the Lintian tag on
the existence of a dh_builddeb override in debian/rules so that people
don't get Lintian warnings about just using the default behavior
(something that we would want to encourage so that we can fix this
globally in the correct place).

-- 
Russ Allbery (r...@debian.org)