Re: Please add lzip support in the repository

[Maria Bisen]

Adam Borowski wrote:

> If you want a fair comparison:
> -rw-r--r-- 1 kilobyte kilobyte 98826240 Jun 16 20:26 octave-4.2.1.tar
> -rw-r--r-- 1 kilobyte kilobyte 15826565 Jul 7 17:13 octave-4.2.1.tar.lz
> -rw-r--r-- 1 kilobyte kilobyte 15174400 Jul 7 17:13 octave-4.2.1.tar.xz
>
> xz wins by 4.2%, with the same settings.

Those are not the same settings, and you can see the reason why in
this benchmark:

http://www.nongnu.org/lzip/lzip_benchmark.html

""xz -9" uses a dictionary size twice as large as "lzip -9". This makes it
appear as if xz could compress large files a little more than lzip."

If you pass to lzip the same dictionary size used by xz -9, then lzip wins.

-rw-r--r--  1 15167441 Jul  7 22:17 octave-4.2.1.tar.lz

and so the comparison is fair enough : )

Maria Bisen

PS: I'm sorry, I'm aware that the message appears misplaced.

Re: Please add lzip support in the repository

[Russ Allbery]

Adam Borowski  writes:

> Thus, I'd recommend dropping lzip completely.  It's worse and obscure.
> With every distro having standardized on xz, providing lzip tarballs is
> a pure waste of space.

Personally, I don't see why anyone should care which compression formats
upstream provides as long as they also provide the one that a given person
wants to use.  Usually the process of producing the tarballs is automated,
adding a new compression format is thirty minutes (if that) of one-time
work, and as long as they have plenty of disk space to distribute multiple
copies, why not?

The only real downside is that compression formats do fade away again and
you may have to drop the compression format again if the tools aren't
available to keep generating it.  But as long as one is reasonably clear
about the reliably-available compression formats and the ones that are
just provided as "why not" and may disappear again, it's not too hard to
manage expectations there.

-- 
Russ Allbery (r...@debian.org)   

Re: Please add lzip support in the repository

[Adam Borowski]

On Fri, Jul 07, 2017 at 11:01:12AM -0400, Lennart Sorensen wrote:
> On Mon, Jul 03, 2017 at 12:38:59PM +0100, Thomas Pircher wrote:
> > in the example you mentioned upstream have added xz to the set of archives
> > they distribute their source in. Currently[1] the GNU Octave source code is
> > being distributed as .gz, lz and .xz tarballs.
> > 
> > [1] https://ftp.gnu.org/gnu/octave/
> 
> Looking at the timestamps, it appears starting with 4.2.0, only gz and
> lz was provided, and again for 4.2.1 that was the case, and then in
> the middle of June this year (so some 7 months after the 4.2.0 release)
> someone went and added xz archives as well, probably because they used
> to have them, and someone asked to keep having them.
> 
> So they used to be gz and xz only, then went to gz and lz only, and then
> later had xz added back again so they now have 3 types.  Seems good in
> the end.
> 
> No idea what compression options were used, but certainly the lz looks
> a good chunk smaller than the xz for those archives.

That's because lzip was used with max settings, xz with the defaults.

If you want a fair comparison:
-rw-r--r--  1 kilobyte kilobyte 98826240 Jun 16 20:26 octave-4.2.1.tar
-rw-r--r--  1 kilobyte kilobyte 15826565 Jul  7 17:13 octave-4.2.1.tar.lz
-rw-r--r--  1 kilobyte kilobyte 15174400 Jul  7 17:13 octave-4.2.1.tar.xz

xz wins by 4.2%, with the same settings.

Thus, I'd recommend dropping lzip completely.  It's worse and obscure.
With every distro having standardized on xz, providing lzip tarballs is
a pure waste of space.


-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄ A master species delegates.

Re: Please add lzip support in the repository

[Lennart Sorensen]

On Mon, Jul 03, 2017 at 12:38:59PM +0100, Thomas Pircher wrote:
> Hi Maria,
> 
> in the example you mentioned upstream have added xz to the set of archives
> they distribute their source in. Currently[1] the GNU Octave source code is
> being distributed as .gz, lz and .xz tarballs.
> 
> I don't get it; what exactly is the problem when upstream distributes their
> source in multiple formats, including .xz and .lz, among others?
> 
> Thomas
> 
> [1] https://ftp.gnu.org/gnu/octave/

Looking at the timestamps, it appears starting with 4.2.0, only gz and
lz was provided, and again for 4.2.1 that was the case, and then in
the middle of June this year (so some 7 months after the 4.2.0 release)
someone went and added xz archives as well, probably because they used
to have them, and someone asked to keep having them.

So they used to be gz and xz only, then went to gz and lz only, and then
later had xz added back again so they now have 3 types.  Seems good in
the end.

No idea what compression options were used, but certainly the lz looks
a good chunk smaller than the xz for those archives.

-- 
Len Sorensen

Re: Please add lzip support in the repository

[Christoph Biedl]

Matthias Klumpp wrote...

> So, lzip isn't adopted widely, that's certainly not because of Debian
> or any other Linux distribution.

The war is over, the winner is VHS.

Trying to get lzip support in wider usage is somewhat a boot-up
problem: Few people see an advantage in doing this, so it doesn't
happen a lot, so people in Debian have no reason to add it either. And
while Debian might have enough power to force lzip into the market,
again nobody sees the big gain, a lot of work though. Overall, I fail
to see why this should happen. Perhaps, perhaps for upstream tarballs.

Regarding compression methods, neither Debian nor other users are at a
critical juncture right now, not even close. The last one has been
2008-ish when the shortcomings of lzma were quite obvious. In retrospect
it was interesting to learn why the decision was made for xz, still this
would not affect the current situation.

The next time for a change was either if xz became inacceptable to use
for whatever reason, or if another compression method pops up that is
an improvement in as many factors as possible: Compression ratio,
(de-)compression memory and time usage, a few things more. The first
option seems highly unlikely to happen, and I doubt lzip would have a
good standing in the second. In the past, bzip2 was quite an
improvement over gzip, again lzma over bzip2 brought much better
compression ratio. These are numbers that make people switch. Also,
back then saving disk and bandwidth was more important than today, so
people were willing to trade compression time for this.

For me, lzip's only advantage is the better design. However, xz is not
that bad. The issues mentioned do not really affect the way Debian uses
the compressed data. Also I'm not aware of reports where xz caused
trouble *that* serious people strongly discourage from using it. After
almost ten years of xz in the wild, this should have happened. There
are some annoyances in xz, I might have been bitten by one of them as
well, but that's not a justification to look for alternatives.

Regarding the non-technical issues about lzip, Russ already provided
excellent statements last Friday. I have nothing to add.

Christoph


signature.asc
Description: Digital signature

Re: Please add lzip support in the repository

[Marc Haber]

On Mon, 3 Jul 2017 16:25:37 +0200, Maria Bisen 
wrote:
>2017-07-03 15:11 GMT+02:00 Matthias Klumpp :
>> So, lzip isn't adopted widely, that's certainly not because of Debian
>> or any other Linux distribution.
>
>I agree, but I thought that Debian adopting lzip could make lzip more
>widely adopted; and that's why I started this thread. Now I know that
>this is a long process, that you will consider when it would be suitable.

Please notice that you did lzip the thing that Germans call a
Bärendienst by being so persistent. I surely am not the only one who
will cringe at any mention of lzip in the nearer future.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: Please add lzip support in the repository

[Marc Haber]

On Mon, 03 Jul 2017 12:38:59 +0100, Thomas Pircher
 wrote:
>I don't get it; what exactly is the problem when upstream distributes 
>their source in multiple formats, including .xz and .lz, among others?

That the lzip community knows that the lzipped sources will almost
never be decompressed by anybody if there is .xz or .gz available. So
they need the other formats to go away if they want theirs to be used.
They seem to be totally desperate.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: Please add lzip support in the repository

[Maria Bisen]

Hi Matthias,

2017-07-03 15:11 GMT+02:00 Matthias Klumpp :
>
> So, lzip isn't adopted widely, that's certainly not because of Debian
> or any other Linux distribution.
>

I agree, but I thought that Debian adopting lzip could make lzip more
widely adopted; and that's why I started this thread. Now I know that
this is a long process, that you will consider when it would be suitable.

Thanks,
Maria Bisen

Re: Please add lzip support in the repository

[Matthias Klumpp]

2017-07-03 14:42 GMT+02:00 Maria Bisen :
> [...]
> 4- As a result, lzip is almost never used alone (without xz), and Debian can
> justify forever the lack of lzip support
>
> You need to consider all four points to understand the issue.

No, please read again the mails previous developers wrote. Lzip is
considered if it is widely used, *not* if it is widely used *alone* as
the sole format.
So, if a huge number of projects starts to ship xz and lzip tarballs,
or gz and lzip tarballs, this would already provide a metric for
sufficient upstream interest to support lzip as source package format.

But just like other interated over countless of times, this is a very
long process Debian will not do lightly, nobody is stopping upstreams
from providing only lzip sources (in which case we would just
recompress the tarball at the moment) and lzip is already maintained
in Debian so any user who wants to use it, can use it.

So, lzip isn't adopted widely, that's certainly not because of Debian
or any other Linux distribution.

Cheers,
Matthias

Re: Please add lzip support in the repository

[Maria Bisen]

Hi Thomas,

Thomas wrote:

> I don't get it; what exactly is the problem when upstream distributes
their
> source in multiple formats, including .xz and .lz, among others?

Please check again point 1 and 2. See below:

1- Somebody from Debian says: "if a lot of upstream tarballs start to be
natively avaiable in .gz and .lzip format (no .xz), *then* it becomes
interesting
to at least support lzip for source packages"

2- An upstream decides to switch its tarballs from xz to lzip

3- Somebody else, also from Debian, asks the upstream above to bring back
the xz tarball

4- As a result, lzip is almost never used alone (without xz), and Debian can
justify forever the lack of lzip support

You need to consider all four points to understand the issue.

Maria Bisen

Re: Please add lzip support in the repository

[Ian Jackson]

Maria Bisen writes ("Re: Please add lzip support in the repository"):
> Moreover, software errors have already killed people:

Good grief.

This conversation is:
 1. determined advocacy from an external project
 2. going badly
 3. not capable of leading to any productive outcome

listmaster, can you please aske Maria to desist ?

Ian.

Re: Please add lzip support in the repository

[Thomas Pircher]

On 2017-07-03 11:41, Maria Bisen wrote:
3- Somebody else, also from Debian, asks the upstream above to bring 
back

the xz tarball

4- As a result, lzip is almost never used alone (without xz), and 
Debian can

justify forever the lack of lzip support


Hi Maria,

in the example you mentioned upstream have added xz to the set of 
archives they distribute their source in. Currently[1] the GNU Octave 
source code is being distributed as .gz, lz and .xz tarballs.


I don't get it; what exactly is the problem when upstream distributes 
their source in multiple formats, including .xz and .lz, among others?


Thomas

[1] https://ftp.gnu.org/gnu/octave/

Re: Please add lzip support in the repository

[Maria Bisen]

Hi,

Russ Allbery wrote:

>> As an user of Octave who wish to see more lzip adoption, I don't think
>> this to be fair.

> Octave's use of lzip is completely unrelated to Debian asking for xz.
> Providing xz in no way prevents Octave from also providing lzip.  I think
> you are inventing a conflict here where none exists.

I'd like to apologize for not being clear enough about what I was refering
to
when speaking about a conflict of interest. Probably, I haven't explained it
well. I wasn't saying that there was a conflict with Octave. In any case,
what
I think the conflict is goes as follows:

1- Somebody from Debian says: "if a lot of upstream tarballs start to be
natively avaiable in .gz and .lzip format (no .xz), *then* it becomes
interesting
to at least support lzip for source packages"

2- An upstream decides to switch its tarballs from xz to lzip

3- Somebody else, also from Debian, asks the upstream above to bring back
the xz tarball

4- As a result, lzip is almost never used alone (without xz), and Debian can
justify forever the lack of lzip support

Thank you for your advise, but I'd like to make a brief comment.

> It's just a compression formatThe world might be mildly better if more
> people used a better compression format, but no one is going to die.

How could you know? I would not make such an absolute affirmation.
IMO software must be taken seriously. It may end in places you couldn't
imagine: in a hospital's operating theatres, in an air-traffic control
center,
or in the London stock exchange.

Moreover, software errors have already killed people:

https://en.wikipedia.org/wiki/2015_Seville_Airbus_A400M_Atlas_crash

"Airbus Chief Strategy Officer Marwan Lahoud confirmed on 29 May that
incorrectly installed engine control software caused the fatal crash."

Maria Bisen

Re: Please add lzip support in the repository

[Christoph Biedl]

Paul Wise wrote...

> On Thu, Jun 15, 2017 at 9:05 PM, Christoph Biedl wrote:
> 
> > I'm not keen on extending regular expressions like
> >
> > \.(gz|bz2|lzma|xz)$
> >
> > that I have in many places again and again.
> 
> That sort of hard-coding should stop,

Understandable and desirable, but perhaps it's just me: Quite often I
find myself in the situation where I have to poke things on a much
lower level than I'm comfortable with. Because the tools and/or libraries
are just painfully slow or don't even exist at all. There still might be
something in devscripts - where so many things are hidden I wasn't even
surprised to find the Amber Chamber - but at some point it seems easier
to hack something together instead of looking around whether there's
something available for the job.

> if you see it somewhere please
> switch to using apt, either via the apt libraries or via apt-helper
> cat-file.

How would that help me in the situation as above?

(
The correct recommendation was to use
Dpkg::Compression::compression_get_file_extension_regex
)

Christoph


signature.asc
Description: Digital signature

Re: Please add lzip support in the repository

[Russ Allbery]

Maria Bisen  writes:

> Also, I think the issue here it's not just proponents of lzip "coming
> here", but Debian people "going out", in what I reckon can be a conflict
> of interest.

This isn't what "conflict of interest" means.  This is just an interest.
There is no conflict.

Currently, Debian packaging tools support gzip, bzip2, lzma, and xz.  For
upstream releases, we therefore prefer to be able to consume a tarball in
one of those formats, since it makes working with our packaging tools
easier.  We can consider adopting another compression format in our core
tools, but this is a large decision about which we are extremely
conservative.  Adding another compression format will take on the order of
years due to constraints around support for stable releases.

We therefore have an interest in upstream source releases coming in a
format that we can consume directly, although we can recompress if needed
if an upstream doesn't want to provide one of those formats.

This in no way prevents upstreams from *also* providing lzip.  It is
extremely common to release software in multiple compression formats
simultaneously, and has been for literally 30 years.  This is not at all a
novel problem, and has always been solved by *adding* new compression
formats and then waiting years or (more commonly) decades before dropping
older formats.  I was still seeing *.Z distributions well into the 2000s,
after most of the world had long-since switched to gzip.

> As an user of Octave who wish to see more lzip adoption, I don't think
> this to be fair.

Octave's use of lzip is completely unrelated to Debian asking for xz.
Providing xz in no way prevents Octave from also providing lzip.  I think
you are inventing a conflict here where none exists.

Similarly, asking for xz to be made available because we currently support
it is in no way an attack on lzip.  I am confused by how incredibly touchy
the lzip community seems to be.

If you would like to see more upstreams add lzip compression to their
distribution method, just ask for that.  Most upstreams will provide it
alongside their other distribution formats, and then periodically look at
which ones people are actually using and drop the ones no one uses.  This
is often easy to determine from web logs.  Debian's use or lack of use of
lzip in our core packaging formats is irrelevant to this effort.

Debian already packages lzip, so in that sense we already support lzip.

If you would like Debian to adopt lzip in our core packaging formats, some
advice:

1. Calm *way* down about this topic.  It's just a compression format.  It
   isn't world peace or starving children.  The world might be mildly
   better if more people used a better compression format, but no one is
   going to die.  There's plenty of time for a relaxed discussion of the
   advantages and tradeoffs.

2. Be aware that Debian is going to move *extremely slowly* in this area,
   and you're talking about a process of years, not weeks.  This is a very
   core part of our packaging infrastructure, and we're not going to adopt
   *anything* quickly, no matter how great it may be.

3. Try to convince Antonio Diaz Diaz to either radically change his
   approach from the one he tried to use with us in 2015, or to stay
   completely out of the discussion.  His incredibly hostile communication
   style left a lasting memory, and while he may not have burned that
   bridge to the ground, he at least left some significant scorch marks.

-- 
Russ Allbery (r...@debian.org)   

Re: Please add lzip support in the repository

[Maria Bisen]

Hi Russ,

Russ Allbery wrote:

> Debian has never expressed any opinion about lzip outside of our project
> mailing lists.  The only reason why it's even on our radar is that
> proponents of lzip keep *coming here* and trying to push it on us.  Some
> of them are polite about it, and we've had polite conversations as a
> result.

My first message was polite, I think, but it received at least one
aggressive
answer.

Also, I think the issue here it's not just proponents of lzip "coming
here", but
Debian people "going out", in what I reckon can be a conflict of interest.
For
example, in this same thread we can read:

https://lists.debian.org/debian-devel/2017/06/msg00209.html

Henrique de Moraes Holschuh wrote:

> Now, if a lot of upstream tarballs start to be natively avaiable in .gz
> and .lzip format (no .xz), *then* it becomes interesting to at least
> support lzip for source packages.

https://lists.debian.org/debian-devel/2017/06/msg00212.html

Russ Allbery wrote:

> We're very unlikely to adopt lzip as a native upstream tarball
> format until it is in very widespread use elsewhere.

And when Octave switched from xz to lzip, the following message was
received in the Octave list:

http://lists.gnu.org/archive/html/octave-maintainers/2017-06/msg00037.html

Mike Miller wrote:

> Can we bring back the octave-x.y.z.tar.xz source format for future
> official releases?
> ... IMHO including .tar.xz would be a nice improvement in the Debian
> packaging domain.

Doesn't this mean that Debian has a COI? It establishes a criterion to adopt
a format and then tries to influence upstreams so the criterion is not met.

As an user of Octave who wish to see more lzip adoption, I don't think this
to be fair.

Maria Bisen

Re: Please add lzip support in the repository

[Russ Allbery]

Maria Bisen  writes:

> After reading again Guillem Jover's answer it seems to me that the only
> marketing campaign here is Debian against lzip. Even if you don't like
> something, for whatever personal reasons, I don't think it's fine to
> influence thousands of people, and Debian has the capacity to do so.

You seem to be extremely confused about how this conversation keeps
starting.

Debian has never expressed any opinion about lzip outside of our project
mailing lists.  The only reason why it's even on our radar is that
proponents of lzip keep *coming here* and trying to push it on us.  Some
of them are polite about it, and we've had polite conversations as a
result.  Some of them were obnoxious, aggressive, and unproductive.  This
is a problem for us; being able to work together in a productive and
respectful fashion with our upstreams matters just as much as technical
features.

If you want us to stop expressing opinions about lzip, stop coming on our
mailing lists and trying to shove lzip down our throats.  It's that
simple.

If this aggressive marketing of lzip is typical of the lzip community and
the style of interactions we would have to have with the project, that is,
by itself, an *excellent* reason to not use it.

-- 
Russ Allbery (r...@debian.org)   

Re: Please add lzip support in the repository

[Maria Bisen]

Hi,

Sorry for the delay, but I think this needs a clarification.

Ian Jackson wrote:

> For Debian binary and source packages, there is no benefit in ECC
> in the compression layer.
>
> I'm not sure why all of this isn't obvious.
>
> As an aside: I am sceptical of the value of ECC as part of a general
> purpose compression scheme.  That proponents of lzip are pushing this
> as an advantage merely adds to the scepticism which their marketing
> campaign is generating in my mind.

There must be a misunderstanding here. On this thread you have been
criticizing lzip and its "proponents" on the basis of a false claim (that
lzip
contains ECC). It's easy verifiable that lzip format doesn't contain any
ECC:

http://www.nongnu.org/lzip/manual/lzip_manual.html#File-format

After reading again Guillem Jover's answer it seems to me that the only
marketing campaign here is Debian against lzip. Even if you don't like
something, for whatever personal reasons, I don't think it's fine to
influence
thousands of people, and Debian has the capacity to do so.

Maria Bisen

Re: Please add lzip support in the repository

[Ian Jackson]

Henrique de Moraes Holschuh writes ("Re: Please add lzip support in the 
repository"):
> On Fri, 16 Jun 2017, Adrian Bunk wrote:
> > On Thu, Jun 15, 2017 at 08:36:48PM -0300, Henrique de Moraes Holschuh wrote:
> > >...
> > > We pretty much need Debian packages to be 100% correct in the first
> > > place, they are not going to be subject to lossy recovery from
> > > corruption (which is where lzip is supposed to be much better than xz):
> > > we need to replace any that is even slightly corrupt with a fully
> > > correct copy.
> > > 
> > > So, it would make more sense to have a par2 (or create a modern version
> > > of it, actually) ECC layer on top of the compression layer, at which
> > > point we can use one of the already supported compression formats.
> > >...
> > 
> > A digital signature is an ECC layer.
> 
> ECC as in eliptic-curve crypto?  That's useless for repair.
> 
> It should have been obvious by context, especially since I even
> mentioned "par2", but it was ECC as in Error-Correcting Code.
> 
> https://en.wikipedia.org/wiki/Error-correcting_code

Adrian's point is correct.  He meant ECC as in error-correcting code.
Of course most digital signature schemes, including the ones we use,
are not able to correct errors.  But they are very good at detecting
them.

I think corrupted downloads are rare enough (due to ECC applied at
lower layers in networks and storage media) that the additional
complication of ECC would not be valuable.

And, our current practice is to sign the compressed streams.  That
means that a corrupted download would be rejected by our digital
signature mechanism, before any compression-layer ECC would see it.

We don't want to reverse the order, and do the compression after
signature, for a number of reasons.  One very useful benefit of the
digital signature scheme is that it means the decompressor used during
package download (which typically runs with rather too much
privilege) is not exposed to potentially hostile input.  Another is
that the signatures can be verified with standard tools.

For Debian binary and source packages, there is no benefit in ECC
in the compression layer.

I'm not sure why all of this isn't obvious.

As an aside: I am sceptical of the value of ECC as part of a general
purpose compression scheme.  That proponents of lzip are pushing this
as an advantage merely adds to the scepticism which their marketing
campaign is generating in my mind.

Ian.

Re: Please add lzip support in the repository

[Henrique de Moraes Holschuh]

On Fri, 16 Jun 2017, Adrian Bunk wrote:
> On Thu, Jun 15, 2017 at 08:36:48PM -0300, Henrique de Moraes Holschuh wrote:
> >...
> > We pretty much need Debian packages to be 100% correct in the first
> > place, they are not going to be subject to lossy recovery from
> > corruption (which is where lzip is supposed to be much better than xz):
> > we need to replace any that is even slightly corrupt with a fully
> > correct copy.
> > 
> > So, it would make more sense to have a par2 (or create a modern version
> > of it, actually) ECC layer on top of the compression layer, at which
> > point we can use one of the already supported compression formats.
> >...
> 
> A digital signature is an ECC layer.

ECC as in eliptic-curve crypto?  That's useless for repair.

It should have been obvious by context, especially since I even
mentioned "par2", but it was ECC as in Error-Correcting Code.

https://en.wikipedia.org/wiki/Error-correcting_code

-- 
  Henrique Holschuh

Re: Please add lzip support in the repository

[Andreas Metzler]

Adrian Bunk  wrote:
> On Thu, Jun 15, 2017 at 08:36:48PM -0300, Henrique de Moraes Holschuh wrote:
[...]
>> So, it would make more sense to have a par2 (or create a modern version
>> of it, actually) ECC layer on top of the compression layer, at which
>> point we can use one of the already supported compression formats.
>>...

> A digital signature is an ECC layer.

Hello,
I do not think so. A digital signature will only help with error
/detection/ but not *correction*.

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Re: Please add lzip support in the repository

[Adrian Bunk]

On Thu, Jun 15, 2017 at 08:36:48PM -0300, Henrique de Moraes Holschuh wrote:
>...
> We pretty much need Debian packages to be 100% correct in the first
> place, they are not going to be subject to lossy recovery from
> corruption (which is where lzip is supposed to be much better than xz):
> we need to replace any that is even slightly corrupt with a fully
> correct copy.
> 
> So, it would make more sense to have a par2 (or create a modern version
> of it, actually) ECC layer on top of the compression layer, at which
> point we can use one of the already supported compression formats.
>...

A digital signature is an ECC layer.

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: Please add lzip support in the repository

[Jeremy Stanley]

On 2017-06-16 12:42:00 +0200 (+0200), Maria Bisen wrote:
[...]
> When I saw in the gcc thread that there's only one distribution
> not supporting lzip
[...]

Following the GCC discussion you linked, I believe it was actually a
reference to SLES lacking any package of lzip at all:

https://gcc.gnu.org/ml/gcc/2017-06/msg00046.html

-- 
Jeremy Stanley

Re: Please add lzip support in the repository

[Maria Bisen]

Russ Allbery wrote:

> Oh, you're concerned with what upstream tarballs Debian can consume
> without repackaging.
>
> I don't see any reason why this should prevent GCC from releasing tarballs
> compressed with lzip if they want to.  They certainly wouldn't stop
> releasing tarballs in other formats, for a host of reasons, and Debian can
> just use one of the other formats.
>
> In other words, this is a "fake" dependency; there is nothing about
> Debian's tools or formats that prevents GCC from releasing tarballs with
> lzip.

Thanks for the explanation.

When I saw in the gcc thread that there's only one distribution not supporting
lzip, I wanted to know more. Now, thanks to your explanations, I know more
about this topic.

> Debian is the last project that you should wait for to make a decision
> like this.  We're very unlikely to adopt lzip as a native upstream tarball
> format until it is in very widespread use elsewhere.  (That's the pattern
> followed with previous formats except for lzma, and I think our somewhat
> premature adoption of lzma support is now seen as a mistake we shouldn't
> repeat.)  We are *extremely* conservative about source package formats
> because, once we adopt one, we have to support it for nearly forever;
> phasing one out again is quite difficult.

I think lzip is presently quite (maybe not “very”) wide spread. I wish that
when the moment lzip has enough use comes, you will make a decision
and will support lzip.

Thanks for all,

Maria Bisen

Re: Please add lzip support in the repository

[Maria Bisen]

> lzip 1.19 is available just in Debian experimental, because we are in
> final-countdown nearly-absolute freeze: we will release the next Debian
> stable this weekend, with lzip 1.18.
>
> lzip 1.19 should be uploaded to Debian unstable sometime after we
> release, at its debian maintainer discretion.
>
> --
>   Henrique Holschuh

Thanks for the information!

Maria Bisen

Re: Please add lzip support in the repository

[Adrian Bunk]

On Thu, Jun 15, 2017 at 08:30:33PM -0700, Russ Allbery wrote:
>  writes:
> 
> > First of all, thank you for your kind and sympathetic message. I'm
> > referring to the second option you mentioned. We are using gcc, and it
> > seems that a reason to not use lzip in gcc is that Debian doesn't
> > support source tarballs in lzip format.
> 
> Oh, you're concerned with what upstream tarballs Debian can consume
> without repackaging.
> 
> I don't see any reason why this should prevent GCC from releasing tarballs
> compressed with lzip if they want to.  They certainly wouldn't stop
> releasing tarballs in other formats, for a host of reasons, and Debian can
> just use one of the other formats.
> 
> In other words, this is a "fake" dependency; there is nothing about
> Debian's tools or formats that prevents GCC from releasing tarballs with
> lzip.
>...

GCC is actually pretty much the worst example for two reasons:

First, Debian already repackages the upstream tarballs (to .tar.xz)
for reasons unrelated to the compression format (GFDL).

Second, the GCC tarball (the repackaged .tar.xz) is inside the sources 
and manually unpacked during the build.

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: Please add lzip support in the repository

[Russ Allbery]

 writes:

> First of all, thank you for your kind and sympathetic message. I'm
> referring to the second option you mentioned. We are using gcc, and it
> seems that a reason to not use lzip in gcc is that Debian doesn't
> support source tarballs in lzip format.

Oh, you're concerned with what upstream tarballs Debian can consume
without repackaging.

I don't see any reason why this should prevent GCC from releasing tarballs
compressed with lzip if they want to.  They certainly wouldn't stop
releasing tarballs in other formats, for a host of reasons, and Debian can
just use one of the other formats.

In other words, this is a "fake" dependency; there is nothing about
Debian's tools or formats that prevents GCC from releasing tarballs with
lzip.

Debian is the last project that you should wait for to make a decision
like this.  We're very unlikely to adopt lzip as a native upstream tarball
format until it is in very widespread use elsewhere.  (That's the pattern
followed with previous formats except for lzma, and I think our somewhat
premature adoption of lzma support is now seen as a mistake we shouldn't
repeat.)  We are *extremely* conservative about source package formats
because, once we adopt one, we have to support it for nearly forever;
phasing one out again is quite difficult.

-- 
Russ Allbery (r...@debian.org)   

Re: Please add lzip support in the repository

[Henrique de Moraes Holschuh]

On Thu, 15 Jun 2017, mariabi...@gmail.com wrote:
> PS: lzip version available in Debian is 1.16, but the last one is 1.19. Maybe 
> it's time to update! :)

lzip 1.19 is available just in Debian experimental, because we are in
final-countdown nearly-absolute freeze: we will release the next Debian
stable this weekend, with lzip 1.18.

lzip 1.19 should be uploaded to Debian unstable sometime after we
release, at its debian maintainer discretion.

-- 
  Henrique Holschuh

Re: Please add lzip support in the repository

[Henrique de Moraes Holschuh]

On Thu, 15 Jun 2017, Christoph Biedl wrote:
> Also I doubt the reduced disk space and network bandwitdth usage of any
> new kid on the block (there's also zstd) really justifies the work
> needed to implement the support in the many tools that deal with the
> files. I might be convinced otherwise.

I agree with almost everything you said, but the whole point of using
lzip instead of xz would be for a less fragile on-disk format, not
performance (be it speed, memory usage, or compression ratio).

And I don't think we actually benefit any from a less fragile compressed
container format for Debian *packages* (source or binary).

We pretty much need Debian packages to be 100% correct in the first
place, they are not going to be subject to lossy recovery from
corruption (which is where lzip is supposed to be much better than xz):
we need to replace any that is even slightly corrupt with a fully
correct copy.

So, it would make more sense to have a par2 (or create a modern version
of it, actually) ECC layer on top of the compression layer, at which
point we can use one of the already supported compression formats.

And the same is also very likely true for data you hold dear.  par2 ECC
information will allow you to recover from much worse file damage than
lzip's better format ever could.

Now, if a lot of upstream tarballs start to be natively avaiable in .gz
and .lzip format (no .xz), *then* it becomes interesting to at least
support lzip for source packages.

-- 
  Henrique Holschuh

Re: Re: Please add lzip support in the repository

[mariabisen]

Hi,

Russ Allbery wrote:

> > Maria Bisen  writes:

> > It's been drawn to my attention the topic included in this thread:

> > https://gcc.gnu.org/ml/gcc/2017-06/msg00084.html

> > I've got the feeling that the distribution the thread talks about is
> > precisely yours, Debian's. As stated there, giving support to lzip in
> > Debian seems feasable and easy. Could it be possible, then, to add
> > lzip support? : )

> > Besides, it seems that, among researchers, myself included, there are
> > voices demanding a more widespread use of lzip, as is the case of the
> > German doctor mentioned in that same thread.

> > I really think that including lzip support in a really important and
> > well-known distribution as Debian will help users everywhere.

> One possibly important clarification: Debian already supports lzip, in the
> sense that lzip is packaged for Debian and can be installed like any other
> package.  Anyone who wants to use it on their Debian system can do so.  So
> if you'd like to use it for your research, please go ahead!

> There is a separate discussion about replacing the compression used
> internally in Debian packages with lzip, but that discussion is only about
> Debian internals and is irrelevant to (and would go unnoticed by) 99% of
> the people who use Debian.  It has no impact on use of lzip on Debian
> systems for whatever purpose.

First of all, thank you for your kind and sympathetic message. I'm referring to 
the second option you mentioned. We are using gcc, and it seems that a reason 
to not use lzip in gcc is that Debian doesn't support source tarballs in lzip 
format.

Anyway, as I've said, I was just asking if it was possible to have lzip support 
in Debian, and I was hoping for an answer such as your own or the one from 
Christoph Bield, and not having to read a meaningless discussion.

Thanks,

Maria Bisen

PS: lzip version available in Debian is 1.16, but the last one is 1.19. Maybe 
it's time to update! :)

Re: Re: Please add lzip support in the repository

[mariabisen]

Hi Guillem,

Guillem Jover wrote:

> On Thu, 2017-06-15 at 17:22:53 +0500, Andrey Rahmatullin wrote:
> > On Thu, Jun 15, 2017 at 01:55:10PM +0200, Maria Bisen wrote:
> > > It's been drawn to my attention the topic included in this thread:
> > >
> > > https://gcc.gnu.org/ml/gcc/2017-06/msg00084.html
> > >
> > > I've got the feeling that the distribution the thread talks about is
> > > precisely yours, Debian's. As stated there, giving support to lzip in
> > > Debian seems feasable and easy. Could it be possible, then, to add
> > > lzip support? : )
>
> I'm afraid this pretty much is not going to happen, besides being
> there no technical advantage at all (to the contrary), and given the
> multiple previous very obnoxious interactions with upstream and some
> of its supporters (not all of course), filled with FUD and extremely
> biased and invalid claims.
>
> > Please read the thread starting with
> > https://lists.debian.org/debian-devel/2015/06/msg00173.html
>
> That thread started at
> 
> and continued at
> 
> and further at
> .
>
> Please also read the bug reports referenced.

I don't know how to read your strong emphasis in referring me to a discussion, 
somehow filled with personal opinions, where few issues are really answered and 
nothing is really achieved.

A year after that thread the following webpage was published:

http://www.nongnu.org/lzip/xz_inadequate.html

A refreshing and profound reading with confirmable results if you have access 
to formats specifications. The fact that the author of one of them is the one 
telling it doesn't mean the results weren't verifiable.

In any case, I was just asking if it was possible to have lzip support in 
Debian, considering there are people interested in having precisely that (such 
as Thomas Goirand and other people in that thread you advised me to read).

> > It is similar to the thread you've referenced, with the same person (the
> > software author) making the same arguments which are then turned down as
> > invalid.

> Indeed. And at this point, I'm starting to consider adding an entry of
> its own just to cover the lzip situation in the dpkg FAQ…

I wish I'm wrong, but I detect some irony or criticism in your message, and I 
don't think that only because I use lzip I deserve that kind of treatment. 
After carefully reading those threads, I've got the impression you don't seem 
to have an objective technical viewpoint regarding this issue. But please, do 
not involve me in what looks like a personal matter for you.

However, I appreciate a lot Christoph Bield' answer, saying that giving support 
to lzip could be time consuming; that is a reason I can understand, even if 
some people consider that giving that support could be easy.

Maria Bisen.

Re: Please add lzip support in the repository

[Russ Allbery]

Maria Bisen  writes:

> It's been drawn to my attention the topic included in this thread:

> https://gcc.gnu.org/ml/gcc/2017-06/msg00084.html

> I've got the feeling that the distribution the thread talks about is
> precisely yours, Debian's. As stated there, giving support to lzip in
> Debian seems feasable and easy. Could it be possible, then, to add
> lzip support? : )

> Besides, it seems that, among researchers, myself included, there are
> voices demanding a more widespread use of lzip, as is the case of the
> German doctor mentioned in that same thread.

> I really think that including lzip support in a really important and
> well-known distribution as Debian will help users everywhere.

One possibly important clarification: Debian already supports lzip, in the
sense that lzip is packaged for Debian and can be installed like any other
package.  Anyone who wants to use it on their Debian system can do so.  So
if you'd like to use it for your research, please go ahead!

There is a separate discussion about replacing the compression used
internally in Debian packages with lzip, but that discussion is only about
Debian internals and is irrelevant to (and would go unnoticed by) 99% of
the people who use Debian.  It has no impact on use of lzip on Debian
systems for whatever purpose.

-- 
Russ Allbery (r...@debian.org)   

Re: Please add lzip support in the repository

[Ian Jackson]

Stuart Prescott writes ("Re: Please add lzip support in the repository"):
> > What is `apt-helper cat-file' and how does it help ?
> 
> On stretch:
> 
> $ apt-file search apt-helper
> apt: /usr/lib/apt/apt-helper

Ah.  I looked on PATH.  I expect "Front door" programs to be on PATH
nowadays.

> $ /usr/lib/apt/apt-helper download-file 
> http://deb.debian.org/debian/dists/sid/main/binary-amd64/Packages.xz 
> Packages.xz
> Get:1 http://deb.debian.org/debian/dists/sid/main/binary-amd64/Packages.xz 
> [7,547 kB]
> Fetched 7,547 kB in 5s (1,446 kB/s)
> 
> $ /usr/lib/apt/apt-helper cat-file Packages.xz | less

So this is not really a replacement for the impugned compression
regexps because
 * it's not available in stable (some of us care about backportability
and supporting stable users)
 * it involves invoking a command and parsing the output to get
trivial information which should be available as a simple
variable in a scripting language
 * it provides only a more-cooked interface than is probably wanted

A better answer would be the perl function
   Dpkg::Compression::compression_guess_from_filename
which is fairly easy to use and has been available for a long time.
Useable something like this:
  https://browse.dgit.debian.org/dgit.git/tree/dgit#n2163

Or if you just want to strip the compression extension then
  \.([^\.]+)
and call compression_guess_from_filename on $1 and see if it's undef.

See for example
  https://manpages.debian.org/wheezy/libdpkg-perl/Dpkg::Compression.3.en.html

Ideally other languages should have something similar.

Ian.

Re: Please add lzip support in the repository

[Guillem Jover]

Hi!

On Fri, 2017-06-16 at 00:35:37 +1000, Stuart Prescott wrote:
> > What is `apt-helper cat-file' and how does it help ?

> $ apt-file search apt-helper
> apt: /usr/lib/apt/apt-helper

> $ /usr/lib/apt/apt-helper download-file 
> http://deb.debian.org/debian/dists/sid/main/binary-amd64/Packages.xz 
> Packages.xz
> Get:1 http://deb.debian.org/debian/dists/sid/main/binary-amd64/Packages.xz 
> [7,547 kB]
> Fetched 7,547 kB in 5s (1,446 kB/s)
> 
> $ /usr/lib/apt/apt-helper cat-file Packages.xz | less

And it can be combined with «apt-get indextargets» which makes it even
more useful, for example with something like:

 $ apt-get indextargets --format '$(FILENAME)' "Created-by: Packages" \
 | xargs /usr/lib/apt/apt-helper cat-file

Please see apt-get(8), and acquire-additional-files.txt in the apt-doc
package.

Thanks,
Guillem

Re: Please add lzip support in the repository

[Stuart Prescott]

> What is `apt-helper cat-file' and how does it help ?

On stretch:

$ apt-file search apt-helper
apt: /usr/lib/apt/apt-helper

$ /usr/lib/apt/apt-helper
apt 1.4.6 (amd64)
Usage: apt-helper [options] command
   apt-helper [options] cat-file file ...
   apt-helper [options] download-file uri target-path

apt-helper bundles a variety of commands for shell scripts to use
e.g. the same proxy configuration or acquire system as APT would.

Most used commands:
  download-file - download the given uri to the target-path
  srv-lookup - lookup a SRV record (e.g. _http._tcp.ftp.debian.org)
  cat-file - concatenate files, with automatic decompression
  auto-detect-proxy - detect proxy using apt.conf


Configuration options and syntax is detailed in apt.conf(5).
Information about how to configure sources can be found in sources.list(5).
Package and version choices can be expressed via apt_preferences(5).
Security details are available in apt-secure(8).
This APT helper has Super Meep Powers.

$ /usr/lib/apt/apt-helper download-file 
http://deb.debian.org/debian/dists/sid/main/binary-amd64/Packages.xz 
Packages.xz
Get:1 http://deb.debian.org/debian/dists/sid/main/binary-amd64/Packages.xz 
[7,547 kB]
Fetched 7,547 kB in 5s (1,446 kB/s)

$ /usr/lib/apt/apt-helper cat-file Packages.xz | less


(this only looks at the repo and doesn't address package compression which 
Guillem discussed elsewhere)


-- 
Stuart Prescotthttp://www.nanonanonano.net/   stu...@nanonanonano.net
Debian Developer   http://www.debian.org/ stu...@debian.org
GPG fingerprint90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7

Re: Please add lzip support in the repository

[Ian Jackson]

Paul Wise writes ("Re: Please add lzip support in the repository"):
> On Thu, Jun 15, 2017 at 9:05 PM, Christoph Biedl wrote:
> > I'm not keen on extending regular expressions like
> >
> > \.(gz|bz2|lzma|xz)$
> >
> > that I have in many places again and again.
> 
> That sort of hard-coding should stop, if you see it somewhere please
> switch to using apt, either via the apt libraries or via apt-helper
> cat-file.

What is `apt-helper cat-file' and how does it help ?

I don't find it on my stretch system, let alone on jessie.  Presumably
the caller (usually some kind of script) would have to parse its
output.

Rather, if you want to lift this out of individual programs, it should
be provided as a bare regexp fragment in per-language libraries.  That
would reduce the number of different places to edit to a dozen or so.

Ian.

Re: Please add lzip support in the repository

[Guillem Jover]

Hi!

On Thu, 2017-06-15 at 17:22:53 +0500, Andrey Rahmatullin wrote:
> On Thu, Jun 15, 2017 at 01:55:10PM +0200, Maria Bisen wrote:
> > It's been drawn to my attention the topic included in this thread:
> > 
> > https://gcc.gnu.org/ml/gcc/2017-06/msg00084.html
> > 
> > I've got the feeling that the distribution the thread talks about is
> > precisely yours, Debian's. As stated there, giving support to lzip in
> > Debian seems feasable and easy. Could it be possible, then, to add
> > lzip support? : )

I'm afraid this pretty much is not going to happen, besides being
there no technical advantage at all (to the contrary), and given the
multiple previous very obnoxious interactions with upstream and some
of its supporters (not all of course), filled with FUD and extremely
biased and invalid claims.

> Please read the thread starting with
> https://lists.debian.org/debian-devel/2015/06/msg00173.html

That thread started at

and continued at

and further at
.

Please also read the bug reports referenced.

> It is similar to the thread you've referenced, with the same person (the
> software author) making the same arguments which are then turned down as
> invalid.

Indeed. And at this point, I'm starting to consider adding an entry of
its own just to cover the lzip situation in the dpkg FAQ…

Thanks,
Guillem

Re: Please add lzip support in the repository

[Paul Wise]

On Thu, Jun 15, 2017 at 9:05 PM, Christoph Biedl wrote:

> I'm not keen on extending regular expressions like
>
> \.(gz|bz2|lzma|xz)$
>
> that I have in many places again and again.

That sort of hard-coding should stop, if you see it somewhere please
switch to using apt, either via the apt libraries or via apt-helper
cat-file.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Please add lzip support in the repository

[Christoph Biedl]

Maria Bisen wrote...

> I've got the feeling that the distribution the thread talks about is
> precisely yours, Debian's. As stated there, giving support to lzip in
> Debian seems feasable and easy. Could it be possible, then, to add
> lzip support? : )

If I understand correctly, it's about using using the lzip method in the
places where Debian uses compression as part of actually distributing
data, like the Packages index file or inside .deb files - the lzip
application has been around for years.

Without a particular feeling for or against lzip, I just don't like to
idea of adding yet another compression method. The zoo is already quite
huge and I'm not keen on extending regular expressions like

\.(gz|bz2|lzma|xz)$

that I have in many places again and again. So, before adding more,
I'd appreciate a cleanup first, and would try to keep the number of
compression methods that are in use - back to oldstable - at a maximum
of three. The old gzip should always be part of the list since it still
has the widest support. The bz2 and lzma compressions, neither fast nor
efficient, are obviously the first candidates for removal, this has
already happened to some extent.

Also I doubt the reduced disk space and network bandwitdth usage of any
new kid on the block (there's also zstd) really justifies the work
needed to implement the support in the many tools that deal with the
files. I might be convinced otherwise.

Christoph


signature.asc
Description: Digital signature

Re: Please add lzip support in the repository

[Andrey Rahmatullin]

On Thu, Jun 15, 2017 at 01:55:10PM +0200, Maria Bisen wrote:
> It's been drawn to my attention the topic included in this thread:
> 
> https://gcc.gnu.org/ml/gcc/2017-06/msg00084.html
> 
> I've got the feeling that the distribution the thread talks about is
> precisely yours, Debian's. As stated there, giving support to lzip in
> Debian seems feasable and easy. Could it be possible, then, to add
> lzip support? : )
Please read the thread starting with
https://lists.debian.org/debian-devel/2015/06/msg00173.html
It is similar to the thread you've referenced, with the same person (the
software author) making the same arguments which are then turned down as
invalid.

-- 
WBR, wRAR


signature.asc
Description: PGP signature