Christopher Faylor wrote: > You've said what you want to do but you haven't said *why*. What is the > advantage of further complicating a complicated program by adding .xz > support, espcially given the lack of adoption of the lzma format?
Even I haven't adopted the .lzma format for cygwin packages, because I still (feel the need to) support cygwin-1.5, whose setup is a Dead Parrot, and which doesn't include the support for .lzma. So, rather than complicated my life by making 1.5 packages as .tar.bz2, and 1.7 packages as .tar.lzma, I'm still using .tar.bz2 for both -- until 1.7 goes gold and I can drop 1.5 like a bad habit. Then I'll switch to (one of) .lzma|.xz. Yaakov has expressed interest in making the default behavior of cygport use .lzma -- and I explicitly waved him off for the reasons above (plus, I was hoping to skip .lzma and go straight to .xz, but we can't do THAT until setup supports it). Note that there is no rush to try to get this done before 1.7. It should be transparent to end users: eventually setup.exe would support it. Then a few packages might be uploaded to release-2 in .xz form (and hopefully no one even notices!). Then a few more. Finally, a month or two or three later, cygport switches its default. This ITP is just the first step in a methodical (read: slow, no rush) process. The reason to switch from .tar.bz2 to .lzma or .xz are 1) better compression; smaller, faster downloads, etc. 2) faster decompression for equivalent uncompressed size (like .gz, but unlike .bz2 IIRC, .lzma/.xz is an non-symmetric algorithm: compression is significantly harder, so that decompression is significantly easier, faster). Hopefully this means that setup will be able to install packages faster, but I'm not sure if the compute-bound of the compression algorithm dominates, or if the io-bound write-to-the-disk + mess-with-the-ACLs does. But faster decompression certainly can't HURT. Now, I don't think adding support for .xz really complicates setup.exe much at all; by LOCC it probably simplifies it by moving the codec to an external library. Since the xz library (liblzma) supports both the old .lzma format and the newer .xz format -- AND the decoder interface is quite similar to that of the liblzmdadec files I used in setup originally -- I'd rename compress_lzma.h --> compress_xz.h compress_lzma.cc --> compress_xz.cc and slightly modify them to use the liblzma functions that are cognate with the ones I was using from lzma-sdk/LzmaDec.h lzma-sdk/LzmaDec.c lzma-sdk/Types.h There are a few simplifications to the "recognize the file as an .lzma/.xz" procedure, too, by offloading to the library IIRC. Also, I'd remove those lzma-sdk files. Then some Makefile.am changes. ======== Aside: I have another reason as well, but it's not important for the cygwin list. I'd like to investigate adapting cygwin's setup program for use with the MinGW project. Their distribution is soon going to get very unwieldy, what with separate packages for the libiconv DLL, PPL dll, GMP, MPFR, Cloog, etc, not to mention all the "normal" separately-distributed pieces like gfortran, g++, java. Plus, the DLLized runtime libraries will all be distributed in separate packages. They *need* something better -- with decent dependency tracking -- than they have. However, there is a wrinkle: none of their packages can use symlinks, because (at least until Vista, and maybe not even then) native apps can't grok them. Therefore, whenever symlinks would be used on sane systems, mingw packages use copies (I'd have chosen hardlinks, but that wouldn't have worked on non-NTFS disks; I suppose running-from-a-FAT-thumb-drive is a reasonable configuration for mingw to try to support). Now, neither bzip2 nor gz are very efficient when "compressing" multiple copies of large files within a tarball, because their dictionary size is fairly small. However, .lzma/.xz does an outstanding job -- almost as efficient as if the tarball actually included real hardlink references for the copies. So, when I get around to trying to adapt setup for MinGW, I'd like it to support .lzma/.xz. But, the other reasons outlined above that provide benefit to cygwin proper are still valid. ======== End Aside Ok, so that's why I want to support .lzma/.xz. Since we already have .lzma, why do I want to add .xz? Because the .lzma file format itself is a hack. It doesn't support proper identification nor internal integrity checks (this may be less of an issue for cygwin/setup, as we have external integrity checking via md5sum). It has no reasonable identity/header bytes at the front of the file -- the best you can do is "this kinda looks like it might be an .lzma file, let's try to decode it and hope it works". Plus, development of the .lzma format/encoder/decoder software is dead -- .xz IS .lzma2, where all the development, and support, is available. Some projects: http://ftp.gnu.org/pub/gnu/m4/ introduced .lzma tarballs for their source distributions last year; and are now switching to .xz instead of .lzma. So, for all intents and purposes, .lzma is actually a *deprecated* compression format. If we're going to support packages using one of these variants of the LZMA algorithm for compression, it should be the new, non-deprecated, one. -- Chuck
