Re: lzip support

2008-11-29 Thread Jim Meyering
Jan Engelhardt [EMAIL PROTECTED] wrote:
 On Friday 2008-11-28 17:21, Bob Friesenhahn wrote:
 Since LZIP support has appeared apparently out of the blue (no
 prior discussion on this list), and Automake already had LZMA
 support, can someone please explain LZIP vs LZMA and why we now
 have at least two LZMA compressed targets?

 See http://lists.gnu.org/archive/html/lzip-bug/2008-11/msg3.html ,
 I think this should answer it.

But nothing I saw there mentioned the upcoming (and superior)
xz format/tool (aka lzma-utils' unstable branch).  That is what's on
the current head of the master branch of the lzma-utils git tree.

git://ctrl.tukaani.org/lzma-utils.git

xz is the name of the new tool as well as the corresponding suffix.
Lasse Collin says there may well be a beta release this year.

I have been following lzma-utils development closely for some time,
and my impression is that xz obviates lzip.  I would not want to
encourage use of lzip without a convincing argument to the contrary.

As soon as there's a beta xz release (i.e., stable format),
I'll be switching from .lzma to .xz suffixes for all tarballs I create.




Re: lzip support

2008-11-29 Thread Jan Engelhardt

On Saturday 2008-11-29 10:06, Jim Meyering wrote:
Jan Engelhardt [EMAIL PROTECTED] wrote:
 On Friday 2008-11-28 17:21, Bob Friesenhahn wrote:
 Since LZIP support has appeared apparently out of the blue (no
 prior discussion on this list), and Automake already had LZMA
 support, can someone please explain LZIP vs LZMA and why we now
 have at least two LZMA compressed targets?

 See http://lists.gnu.org/archive/html/lzip-bug/2008-11/msg3.html ,
 I think this should answer it.

But nothing I saw there mentioned the upcoming (and superior)
xz format/tool (aka lzma-utils' unstable branch).  That is what's on
the current head of the master branch of the lzma-utils git tree.

git://ctrl.tukaani.org/lzma-utils.git

xz is the name of the new tool as well as the corresponding suffix.
Lasse Collin says there may well be a beta release this year.

Nothing has happened since at least July when I inquired. I mean, it is 
not that hard to add the two features, magic byte string and checksum, 
is it? (IMHO the format should have had these from the beginning even.)

I have been following lzma-utils development closely for some time,
and my impression is that xz obviates lzip.  I would not want to
encourage use of lzip without a convincing argument to the contrary.

As soon as there's a beta xz release (i.e., stable format),
I'll be switching from .lzma to .xz suffixes for all tarballs I create.

lzip is (marked as) stable now, it was enough waiting for lzma.




Re: lzip support

2008-11-29 Thread Jim Meyering
Jan Engelhardt [EMAIL PROTECTED] wrote:
 On Saturday 2008-11-29 10:06, Jim Meyering wrote:
Jan Engelhardt [EMAIL PROTECTED] wrote:
 On Friday 2008-11-28 17:21, Bob Friesenhahn wrote:
 Since LZIP support has appeared apparently out of the blue (no
 prior discussion on this list), and Automake already had LZMA
 support, can someone please explain LZIP vs LZMA and why we now
 have at least two LZMA compressed targets?

 See http://lists.gnu.org/archive/html/lzip-bug/2008-11/msg3.html ,
 I think this should answer it.

But nothing I saw there mentioned the upcoming (and superior)
xz format/tool (aka lzma-utils' unstable branch).  That is what's on
the current head of the master branch of the lzma-utils git tree.

git://ctrl.tukaani.org/lzma-utils.git

xz is the name of the new tool as well as the corresponding suffix.
Lasse Collin says there may well be a beta release this year.

 Nothing has happened since at least July when I inquired. I mean, it is
 not that hard to add the two features, magic byte string and checksum,
 is it? (IMHO the format should have had these from the beginning even.)

Nothing has happened ?
Did you look at the code at all, or ask Lasse?

$ git log --since=2008-07-01 -p|diffstat|tail -1
 416 files changed, 14731 insertions(+), 13348 deletions(-)

I have been following lzma-utils development closely for some time,
and my impression is that xz obviates lzip.  I would not want to
encourage use of lzip without a convincing argument to the contrary.

As soon as there's a beta xz release (i.e., stable format),
I'll be switching from .lzma to .xz suffixes for all tarballs I create.

 lzip is (marked as) stable now, it was enough waiting for lzma.

I see xz as the right format and tool, so prefer not to
encourage the use of any other new tool to do the same job.




Re: lzip support

2008-11-29 Thread Bob Friesenhahn

On Sat, 29 Nov 2008, Jim Meyering wrote:


I have been following lzma-utils development closely for some time,
and my impression is that xz obviates lzip.  I would not want to
encourage use of lzip without a convincing argument to the contrary.

As soon as there's a beta xz release (i.e., stable format),
I'll be switching from .lzma to .xz suffixes for all tarballs I create.


Competition is good and even between open source projects.  However, 
since many free projects depend on Automake, it makes sense for 
Automake to channel the energy into a smaller set of preferred 
formats.  Note that formats may be independent from the tools which 
produce and consume them so that tools may still compete.  If new 
formats are added, the least worthy of the existing supported 
distribution formats should be deprecated and eventually removed. 
This means that if .xz is added that .lzma should be immediately 
deprecated and slated for retirement from Automake.  Do you agree with 
this philosophy?


Bob
==
Bob Friesenhahn
[EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/





Re: lzip support

2008-11-29 Thread Jan Engelhardt

On Saturday 2008-11-29 17:04, Bob Friesenhahn wrote:
 On Sat, 29 Nov 2008, Jim Meyering wrote:

 I have been following lzma-utils development closely for some time,
 and my impression is that xz obviates lzip.  I would not want to
 encourage use of lzip without a convincing argument to the contrary.

 As soon as there's a beta xz release (i.e., stable format),
 I'll be switching from .lzma to .xz suffixes for all tarballs I create.

 Competition is good and even between open source projects.  However, since 
 many
 free projects depend on Automake, it makes sense for Automake to channel the
 energy into a smaller set of preferred formats.  Note that formats may be
 independent from the tools which produce and consume them so that tools may
 still compete.  If new formats are added, the least worthy of the existing
 supported distribution formats should be deprecated and eventually removed.
 This means that if .xz is added that .lzma should be immediately deprecated 
 and
 slated for retirement from Automake.  Do you agree with this philosophy?

As for me, yes.




Re: lzip support

2008-11-29 Thread Jim Meyering
Bob Friesenhahn [EMAIL PROTECTED] wrote:
 On Sat, 29 Nov 2008, Jim Meyering wrote:
 I have been following lzma-utils development closely for some time,
 and my impression is that xz obviates lzip.  I would not want to
 encourage use of lzip without a convincing argument to the contrary.

 As soon as there's a beta xz release (i.e., stable format),
 I'll be switching from .lzma to .xz suffixes for all tarballs I create.

 Competition is good and even between open source projects.  However,
 since many free projects depend on Automake, it makes sense for
 Automake to channel the energy into a smaller set of preferred
 formats.  Note that formats may be independent from the tools which
 produce and consume them so that tools may still compete.  If new
 formats are added, the least worthy of the existing supported
 distribution formats should be deprecated and eventually removed. This
 means that if .xz is added that .lzma should be immediately deprecated
 and slated for retirement from Automake.  Do you agree with this
 philosophy?

Sure: once there is a beta release of xz, lzma can go.

When adding xz support, I considered whether to remove mention of lzma
from NEWS, since it's now slated for removal.  But of course, this is
just my opinion.  Ralf's is the one who matters here ;-)




Re: lzip support

2008-11-29 Thread Ralf Wildenhues
 On Friday 2008-11-28 21:37, Bob Friesenhahn wrote:
 
  It makes sense to me that periodically Automake maintainers make an
  evaluation (and with the blessing of the FSF) intentionally
  deprecate generation of certain archive types as new archive types
  are added. The intention would be to diminish the number of archive
  types, which needlessly clog disk space and consume developer time.

I don't understand.  The default is to create gzip'ed archives only.
That hasn't changed in more than a decade.  And gzip isn't going to go
away as the default format, if only because that is by far the most
portable one.  Package authors can tweak to their liking which sets of
archives they wish to generate, and that includes not generating gzip.

Cheers,
Ralf




Re: lzip support

2008-11-29 Thread Ralf Wildenhues
Hello,

* Jim Meyering wrote on Sat, Nov 29, 2008 at 05:13:04PM CET:
 Bob Friesenhahn [EMAIL PROTECTED] wrote:
  If new formats are added, the least worthy of the existing supported
  distribution formats should be deprecated and eventually removed.
  This means that if .xz is added that .lzma should be immediately
  deprecated and slated for retirement from Automake.  Do you agree
  with this philosophy?
 
 Sure: once there is a beta release of xz, lzma can go.

 When adding xz support, I considered whether to remove mention of lzma
 from NEWS, since it's now slated for removal.

I can agree with the following: once there is a stable xz release, we
immediately mark lzma as deprecated.  Then, in the next major Automake
release, we remove support for lzma.  If xz releases before Automake
1.11, then removal in 1.12 is ok, otherwise in 1.13.  Point releases
shouldn't remove features.

Experience shows that this will amount to a roughly two-year delay,  ;-)
but we decided to put lzma support in 1.10.1 without deprecating it
immediately.

Cheers,
Ralf




Re: lzip support

2008-11-29 Thread Jan Engelhardt

On Saturday 2008-11-29 17:30, Ralf Wildenhues wrote:
 On Friday 2008-11-28 21:37, Bob Friesenhahn wrote:
 
  It makes sense to me that periodically Automake maintainers make an
  evaluation (and with the blessing of the FSF) intentionally
  deprecate generation of certain archive types as new archive types
  are added. The intention would be to diminish the number of archive
  types, which needlessly clog disk space and consume developer time.

I don't understand.  The default is to create gzip'ed archives only.
That hasn't changed in more than a decade.  And gzip isn't going to go
away as the default format, if only because that is by far the most
portable one.  Package authors can tweak to their liking which sets of
archives they wish to generate, and that includes not generating gzip.

Speaking of tweaking, how about adding some flags than can be used with
AUTOMAKE_OPTIONS= so that only dist-* targets of a specific compression
are output. Makes for a smaller Makefile in the end.




Re: lzip support

2008-11-29 Thread Bob Friesenhahn

On Sat, 29 Nov 2008, Ralf Wildenhues wrote:


On Friday 2008-11-28 21:37, Bob Friesenhahn wrote:


It makes sense to me that periodically Automake maintainers make an
evaluation (and with the blessing of the FSF) intentionally
deprecate generation of certain archive types as new archive types
are added. The intention would be to diminish the number of archive
types, which needlessly clog disk space and consume developer time.


I don't understand.  The default is to create gzip'ed archives only.
That hasn't changed in more than a decade.  And gzip isn't going to go


Right.  However, a profusion of formats creates confusion for users 
and burdens archive sites because package authors/maintainers 
naturally feel that they should enable and upload all of the supported 
formats.  Automake is in a unique position to influence the commonly 
used formats.


It is not helpful to the world to have the same stuff packaged up in 
five different archive formats.  In fact, it is quite wasteful.


Bob
==
Bob Friesenhahn
[EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/





Re: lzip support

2008-11-28 Thread Jan Engelhardt

On Friday 2008-11-28 17:21, Bob Friesenhahn wrote:

 Since LZIP support has appeared apparently out of the blue (no
 prior discussion on this list), and Automake already had LZMA
 support, can someone please explain LZIP vs LZMA and why we now
 have at least two LZMA compressed targets?

See http://lists.gnu.org/archive/html/lzip-bug/2008-11/msg3.html ,
I think this should answer it.

 I see that LZIP is GPL licensed and is pretty small, and with just
 one author.

Sometimes, simplicity is the key. And I do not think that having
exactly 1.0 authors makes a project insignificant.

 It also seems that LZIP is not capable of decoding LZMA
 utils output.

In the gzip × bzip2 × lzma matrix, neither can decode another.
So it's not like lzip would be missing a feature others would have.

 If automake now supports 'lzip', why does it not also offer to support 7-Zip',
 'srpm', 'zoo', 'arc', and the many other possible archiving formats so that
 confusion of the user base can become complete?

I would say because 7zip, ZOO and ARC (what's with these 1990s packers?)
do not support UNIX owners nor permissions (required for the beloved +x bit
on scripts.).




Re: lzip support

2008-11-28 Thread Bob Friesenhahn

On Fri, 28 Nov 2008, Jan Engelhardt wrote:


I see that LZIP is GPL licensed and is pretty small, and with just
one author.


Sometimes, simplicity is the key. And I do not think that having
exactly 1.0 authors makes a project insignificant.


Actually I like the 1.0 authors since it makes the copyright issues 
more clear and means that there is someone empowered to update the 
license or defend the copyright if necessary.  In contrast FSF GNU 
projects require that all authors sign a contract with the FSF to 
assign copyrights.  That is a tedious task.


I like simplicity as well.  From my point of view 'gzip' is an ideal 
package other than its compression ratio.



If automake now supports 'lzip', why does it not also offer to support 7-Zip',
'srpm', 'zoo', 'arc', and the many other possible archiving formats so that
confusion of the user base can become complete?


I would say because 7zip, ZOO and ARC (what's with these 1990s packers?)
do not support UNIX owners nor permissions (required for the beloved +x bit
on scripts.).


It was my impression that Automake adopted LZMA utils without fully 
evaluating the impact.  Introducing a new archive format is really 
quite a big deal since it impacts many thousands of open source users 
well into the future.  As it turned out, LZMA utils conflicted with 
another available LZMA utility, which caused some problems for FreeBSD 
and likely other distributions as well.


My own package is now distributing .lzma packages.  This is a big deal 
for it moving forward since changing the package format will break 
something.  OS distributions are only recently becoming used to .lzma 
and have had to update scripts and tools to deal with it.


Due to the preponderance of distribution formats, the actual amount of 
data on ftp sites is dramatically increasing rather than decreasing 
since packages feel that they must produce archives in every possible 
format.  If an archive format was ever offered before, the feeling is 
that it must continue to be offered for the rest of time.


It would be useful if the Automake project would thoroughly research 
all issues and come up with a plan which reduces total world impact. 
We need a Green Solution which avoids wasteful practices which surely 
increase global warming and further tax our dwindling fossil fuel 
supply.  These are all factors which should be considered:


  * Number of files needing to be uploaded to distribution sites, or
mirrored.

  * Individual file size.

  * Utility implementation license and copyrights.

  * Utility portability.

  * Utility performance and reliability.

  * Utility usage complexity.

  * Utility long-term maintenance expectations.

  * Effort to integrate into established packaging and source
distribution systems.

Bob
==
Bob Friesenhahn
[EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/





Re: lzip support

2008-11-28 Thread Jan Engelhardt

On Friday 2008-11-28 19:38, Bob Friesenhahn wrote:
 On Fri, 28 Nov 2008, Jan Engelhardt wrote:

 It was my impression that Automake adopted LZMA utils without fully
 evaluating the impact. My own package is now distributing .lzma
 packages.

It's only great until something better comes up :)

 This is a big deal for it moving forward since changing the package
 format will break something. OS distributions are only recently
 becoming used to .lzma and have had to update scripts and tools to
 deal with it.

All I actually needed was a patch to the package builder (e.g.
rpmbuild) to unpack lzip without having to explicitly call lzip.

 Due to the preponderance of distribution formats, the actual amount
 of data on ftp sites is dramatically increasing rather than
 decreasing since packages feel that they must produce archives in
 every possible format.

Well talk to kernel.org for example. They are the ones still using gz
besides bz2 (and they got one of the biggest packages) even when we
can assume bzip2 is reasonably mature, more than any lzma.

 If an archive format was ever offered before, the feeling is that
 it must continue to be offered for the rest of time.

*sigh* well, everybody is entitled to do his own liking and
if that's providing all formats just because.




Re: lzip support

2008-11-28 Thread Bob Friesenhahn

On Fri, 28 Nov 2008, Jan Engelhardt wrote:


On Friday 2008-11-28 19:38, Bob Friesenhahn wrote:

On Fri, 28 Nov 2008, Jan Engelhardt wrote:

It was my impression that Automake adopted LZMA utils without fully
evaluating the impact. My own package is now distributing .lzma
packages.


It's only great until something better comes up :)


In this case, better means a single author and much less source code. 
But it does not currently seem to mean faster:


freddy:~% ls -l GraphicsMagick-1.3.tar
-r--r--r--   1 bfriesen home 36925440 Nov  9 14:54 GraphicsMagick-1.3.tar
freddy:~% ptime lzip -9 GraphicsMagick-1.3.tar

real 1:19.836
user 1:19.347
sys 0.327
freddy:~% ptime lzip -d GraphicsMagick-1.3.tar.lz

real0.882
user0.818
sys 0.061
freddy:~% ptime lzma -9 GraphicsMagick-1.3.tar

real   55.439
user   54.630
sys 0.640
freddy:~% ptime lzma -d GraphicsMagick-1.3.tar.lzma

real0.688
user0.622
sys 0.064
freddy:~% ptime gzip -9 GraphicsMagick-1.3.tar

real2.970
user2.924
sys 0.039
freddy:~% ptime gzip -d GraphicsMagick-1.3.tar.gz

real0.265
user0.223
sys 0.040
freddy:~% size /usr/local/bin/lzma
116956 + 8820 + 5748 = 131524
freddy:~% size /usr/local/bin/lzip
80304 + 4689 + 4307 = 89300

Compressed file sizes are quite similar.


If an archive format was ever offered before, the feeling is that
it must continue to be offered for the rest of time.


*sigh* well, everybody is entitled to do his own liking and
if that's providing all formats just because.


Currently Automake does not seem to allow disabling gzip support.  It 
makes sense to me that periodically Automake maintainers make an 
evaluation (and with the blessing of the FSF) intentionally deprecate 
generation of certain archive types as new archive types are added. 
The intention would be to diminish the number of archive types, which 
needlessly clog disk space and consume developer time.


Initially there would be a warning, and after a couple of years, the 
less desired archive type would be removed entirely.  At the moment I 
think that it is more desirable for bzip2 to be deprecated than gzip 
since the compression advantage of bzip2 is not that high and it takes 
much more CPU and memory. Zip is quite wasteful, but is perhaps most 
useful for Windows since it does not require 'tar' and there is native 
support in Windows.  It should only be necessary to support one LZMA 
format.


Bob
==
Bob Friesenhahn
[EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/





Re: lzip support

2008-11-28 Thread Jan Engelhardt

On Friday 2008-11-28 21:37, Bob Friesenhahn wrote:

  If an archive format was ever offered before, the feeling is that
  it must continue to be offered for the rest of time.

 *sigh* well, everybody is entitled to do his own liking and
 if that's providing all formats just because.

 Currently Automake does not seem to allow disabling gzip support.

no-dist-gzip? What I was saying: you do not have to run make dist.
You could run make dist-bzip2 instead. Producing one files as a side
effect of build system is one thing, but uploading them to a public
required manual intervention.

 It makes sense to me that periodically Automake maintainers make an
 evaluation (and with the blessing of the FSF) intentionally
 deprecate generation of certain archive types as new archive types
 are added. The intention would be to diminish the number of archive
 types, which needlessly clog disk space and consume developer time.

 Initially there would be a warning, and after a couple of years,

Woha, that's long. I take the freedom to usually do it within two
releases.

 the less desired archive type would be removed entirely. At the
 moment I think that it is more desirable for bzip2 to be deprecated
 than gzip since the compression advantage of bzip2 is not that high
 and it takes much more CPU and memory.

Well, compression always takes time. If you wanted to go for
the best compression-to-time-ratio, you would have to go with
uncompressed as the premium.

 Zip is quite wasteful, but
 is perhaps most useful for Windows since it does not require 'tar'
 and there is native support in Windows. It should only be necessary
 to support one LZMA format.

Now how many Windows users can actually run shell scripts (as
produced by autotools) out of the blue, without having, uh, a shell
(from cygwin or msys). Once they however have such a unix layer, they
also have tar and gzip at least.




Re: lzip support

2008-11-28 Thread Bob Friesenhahn

On Sat, 29 Nov 2008, Jan Engelhardt wrote:


Currently Automake does not seem to allow disabling gzip support.


no-dist-gzip? What I was saying: you do not have to run make dist.


If that works, then I was unaware of it.


Initially there would be a warning, and after a couple of years,


Woha, that's long. I take the freedom to usually do it within two
releases.


Many major OS distributions seem to be on a two-year cycle.  An 
existing current distribution should not require continual maintenance 
to keep it its bits from rotting.



Now how many Windows users can actually run shell scripts (as
produced by autotools) out of the blue, without having, uh, a shell
(from cygwin or msys). Once they however have such a unix layer, they
also have tar and gzip at least.


Even though Automake builds the distribution package, most of my 
Windows users use means other than shell scripts to download, 
extract, and build the package.  The most they know about Unix is 
perhaps how to spell it.


Bob
==
Bob Friesenhahn
[EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/