On Fri, Feb 01, 2008 at 10:08:05PM -0600, Mike Gerdts wrote: > I would like to see something different than tar, because... > > - The logical thing to do with a tar file is to compress it. If you > are working on a system with slow single thread performance, > processing compressed tar (or cpio) archives is extremely slow.
We could, of course, compress each file in the archive individually, which would allow for multiple threads to do the decompression in parallel. Of course, you still have to read through the archivve to find out where things are, unless you tack on a header that contains the table of contents. > >From my recollection of the zap discussion and my own frustration with > single-threaded performance on US-T1 systems, I would like to see: > > - All metadata at the beginning of the file. This is helpful when the > file is delivered via a standard HTTP download (no IPS server) to > determine which byte ranges need to be downloaded. This is one of my big beefs with zip-based formats. There's no reason that a smart protocol couldn't download the TOC first, place it at the end of a holey file, and fill in the rest afterwards. Seems a bit icky, but at least opensource zip tools are very widely available (including the python module), so we wouldn't have to write anything new. > - No barriers in the file format to having a multi-threaded pkg > command that can simultaneously uncompress, checksum, verify > signatures, etc. multiple files from the same package at the same > time. Zip archives compress the files individually, as I understand it, so it's at least capable of better multithreaded operations, but I don't know if there are any zip utilities or libraries that actually do this. (Or 7z, for that matter, since you brought it up.) RAR would be another possibility, but it's not free. 7z seems like it would be ideal (individually compressed files if desired, LGPL), but all the implementations are in C++, and nasty C++ at that. But we could probably work with it if we had to. Danek _______________________________________________ pkg-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
