Re: 7-Zip / Bzip2
Compression algorithms are something that should be handled via an extendle library. And the front-end apps (gzip, bzip2, 7z, etc) should just use that library to do the heavy lifting. Instead of pulling in another app, consider pulling in libarchive and friends from FreeBSD 6+, and then adding 7z support to that. Then you can remove compress, gzip, gunzip, bzip2, bunzip2, 7z, etc, and just put in stub programs that call the library functions. -- Freddie Cash [EMAIL PROTECTED]
Re: 7-Zip / Bzip2
On Sun, 11 May 2008 10:46:38 -0700 Freddie Cash [EMAIL PROTECTED] wrote: Compression algorithms are something that should be handled via an extendle library. And the front-end apps (gzip, bzip2, 7z, etc) should just use that library to do the heavy lifting. Instead of pulling in another app, consider pulling in libarchive and friends from FreeBSD 6+, and then adding 7z support to that. libarchive is in DragonFly. -- C:WIN | Directable Mirror Arrays The computer obeys and wins.| A better way to focus the sun You lose and Bill collects. |licences available see |http://www.sohara.org/
Re: 7-Zip / Bzip2
On Sun, May 11, 2008 at 10:46:38AM -0700, Freddie Cash wrote: Instead of pulling in another app, consider pulling in libarchive and friends from FreeBSD 6+, and then adding 7z support to that. You know that one of the two reasons I wrote the compression_program support in libarchive was 7z's lack of a proper library interface? Joerg
Re: 7-Zip / Bzip2
On Sun, May 11, 2008 at 11:22 AM, Joerg Sonnenberger [EMAIL PROTECTED] wrote: On Sun, May 11, 2008 at 10:46:38AM -0700, Freddie Cash wrote: Instead of pulling in another app, consider pulling in libarchive and friends from FreeBSD 6+, and then adding 7z support to that. You know that one of the two reasons I wrote the compression_program support in libarchive was 7z's lack of a proper library interface? Well, there you go. :) It looks like all the pieces are in place. What more needs to be done? ;) -- Freddie Cash [EMAIL PROTECTED]
Re: 7-Zip / Bzip2
thegraze [EMAIL PROTECTED] wrote: But 7z is GPL! Which is not an impediment per-se. After all it's just a userland tool, not a library or even part of the kernel. Remember that gzip was GPL for a long time, before the NetBSD people started writing a replacement tool. On the other hand, it should be noted that both bzip2 and gzip (at least the one used by the BSDs) are now under BSD licenses. Replacing a BSD-licensed piece of software with a GPL-licensed one is only desirable if the latter is much more useful or has features that are needed, and if there are good reasons to have it in base instead of pkgsrc. So, is 7z useful enough to add a fourth compression tool to the base system? And keep it there forever? (Remember that we also have to keep compress(1) for compatibility, even though it compresses worse and is slower than gzip, so the usefulness is very small.) I've given it a quick test and fed a 1 MB logfile to 7z. It was only marginally better than bz2 ( 1%), but it was noticeably slower. And bz2 is already painfully slow for both compression and decompression. Not everybody has a 3 GHz multicore machine. That's why I still use gzip most of the time -- the compression is a little worse, but it's a *lot* faster. There were even cases when people reported that they weren't able to decompress a bz2 file on a small system (embedded or otherwise), because it required several MB of RAM for decompression. It appears that 7z is even worse. The memory footprint of gunzip is negligible. It should also be mentioned that about every other year another compression tool pops up that claims to be better than all the others. Last year (or the year before) it was paq, before that it was rzip and lrzip, and so on. So this year it is 7z. What will be the next one? So, my personal opinion is: Leave 7z in pkgsrc. Just my 2 cents ... Best regards Oliver -- Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd
Re: 7-Zip / Bzip2
Which is not an impediment per-se. After all it's just a userland tool, not a library or even part of the kernel. Not to nit-pick, but it's LGPL, not GPL, so it's not *so* bad :) So, is 7z useful enough to add a fourth compression tool to the base system? And keep it there forever? (Remember that we also have to keep compress(1) for compatibility, even though it compresses worse and is slower than gzip, so the usefulness is very small.) I would argue that 'compress' should be removed and put in the ports tree. There's always this argument concerning shell scripts that are still using these tools...who's using one? If you're swapping to DragonflyBSD, chances are you're not worried about breaking compatibility a little, and it's not difficult to put it in the docs that it's been removed. I've given it a quick test and fed a 1 MB logfile to 7z. It was only marginally better than bz2 ( 1%), but it was noticeably slower. And bz2 is already painfully slow for both compression and decompression. Not everybody has a bz2 is painfully slow for compression, not decompression. Small files aren't much of a concern for any of this, though. As I mentioned previously, if you use the compression level argument with 7z, you can beat bzip2 by compression AND speed in many cases. I compressed a 4gb image 90mb smaller and 3 minutes faster than bzip2 with 7z's compression level '4'. 3 GHz multicore machine. That's why I still use gzip most of the time -- the compression is a little worse, but it's a *lot* faster. I wasn't suggesting removal of gzip. Just like not everyone has a 3 GHz multicore machine, not everyone is still using a crappy 486 --- they shouldn't be limited by the restrictions your particular machine has. There were even cases when people reported that they weren't able to decompress a bz2 file on a small system (embedded or otherwise), because it required several MB of RAM for decompression. It appears that 7z is even worse. The memory footprint of gunzip is negligible. RAM usage for compression depends on the level of compression you're using. If someone's going to be using DflyBSD for an embedded device with limited resources, they would tailor it accordingly and remove 7zip and bzip2. A great many systems don't need half the kernel modules either, that doesn't mean those should not have been added to DFly. It should also be mentioned that about every other year another compression tool pops up that claims to be better than all the others. Last year (or the year before) it was paq, before that it was rzip and lrzip, and so on. So this year it is 7z. What will be the next one? 7-zip has been around since ~2000. It's not hype. rzip's features, having more flexibility than just 100-900k block sizes in bzip2, should've been available in bzip2 from the start -- unfortunately some of Unix's best strengths are its greatest weaknesses. bzip2 can't simply be altered, otherwise compatibility will be broken. bzip2 would never have been added if gzip could've been modified at whim. bzip2 vs 7z: - 7z supports listing decompressed size of contained file(s) - 7z can compress faster with a better ratio, or a much better ratio but slower - 7z can create volumes. while I realize one can use 'split'...try putting a huge set of volumes back together on windows. 7z has a really decent windows port, tooand ports on many other OSes. Best Regards, Ben Cadieux
Re: 7-Zip / Bzip2
But 7z is GPL! 2008/5/6 Matthew Dillon [EMAIL PROTECTED]: :I just noticed I still have these sitting here so I figured I would post :sizes. 1.12.1 ISO 294M, bz2 108M, 7z 74M, zip 118M, gz 120M. : :Sam Those are impressive numbers. I will keep an eye on it. If 7z is consistently better then bz2 I expect it will become widely adopted. I just think it is a bit early to put it into base when it is so readily accessible via pkgsrc. -Matt Matthew Dillon [EMAIL PROTECTED]
Re: 7-Zip / Bzip2
thegraze wrote: But 7z is GPL! so? your point being? -- Serve - BSD +++ RENT this banner advert +++ASCII Ribbon /\ Work - Mac +++ space for low €€€ NOW!1 +++ Campaign \ / Party Enjoy Relax | http://dragonflybsd.org Against HTML \ Dude 2c 2 the max ! http://golden-apple.biz Mail + News / \
Re: 7-Zip / Bzip2
Samuel J. Greear [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] Matthew Dillon [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] :Hi, : :Posted this to kernel@ by accident, please reply here instead :) : :I just wanted to know if there's any interest for the devs to add :something like p7zip to the base install; even if it's a simple fork :that only supports 7z. While 7zip is about as obnoxiously slow as :bzip2, it usually gets much better compression. : :That's not why I'm suggesting it though - what really gets me is that :bzip2 has no list option. Does that 10 gb bzip2 backup archive :contain 100gb of data, or 200gb? Other than dumping the entire :archive to /dev/null through wc, there's really no way to do it. Gzip :will list files, but its compression ratio is awful. : :I imagine that other OSes are going to be watching Dragonfly very :carefully in the next while as new the features (especially HAMMER) :mature. Maybe adding 7z will get yet another bandwagon going and :there will be support across the board :) : :Best Regards, :Ben Cadieux Well, I think not in base, at least not unless a lot of people are using it. p7zip is readily available via the pkgsrc tree and that's the most reasonable method of accessibility for now. -Matt Matthew Dillon [EMAIL PROTECTED] I do not know if DragonFly is actively tracking libarchive, but it seems to be the ticket for implementing new archiving/compression methods through a common mechanism. If one wanted to see 7z functionality in base implementing a libarchive provider is probably the way to go. I did a quick test at one point on the dfly distribution ISO and as I recall 7z was ~60% the size of bzip2 using standard settings. food+thought, etc. Sam I just noticed I still have these sitting here so I figured I would post sizes. 1.12.1 ISO 294M, bz2 108M, 7z 74M, zip 118M, gz 120M. Sam
Re: 7-Zip / Bzip2
Samuel J. Greear [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] Matthew Dillon [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] :Hi, : :Posted this to kernel@ by accident, please reply here instead :) : :I just wanted to know if there's any interest for the devs to add :something like p7zip to the base install; even if it's a simple fork :that only supports 7z. While 7zip is about as obnoxiously slow as :bzip2, it usually gets much better compression. : :That's not why I'm suggesting it though - what really gets me is that :bzip2 has no list option. Does that 10 gb bzip2 backup archive :contain 100gb of data, or 200gb? Other than dumping the entire :archive to /dev/null through wc, there's really no way to do it. Gzip :will list files, but its compression ratio is awful. : :I imagine that other OSes are going to be watching Dragonfly very :carefully in the next while as new the features (especially HAMMER) :mature. Maybe adding 7z will get yet another bandwagon going and :there will be support across the board :) : :Best Regards, :Ben Cadieux Well, I think not in base, at least not unless a lot of people are using it. p7zip is readily available via the pkgsrc tree and that's the most reasonable method of accessibility for now. -Matt Matthew Dillon [EMAIL PROTECTED] I do not know if DragonFly is actively tracking libarchive, but it seems to be the ticket for implementing new archiving/compression methods through a common mechanism. If one wanted to see 7z functionality in base implementing a libarchive provider is probably the way to go. I did a quick test at one point on the dfly distribution ISO and as I recall 7z was ~60% the size of bzip2 using standard settings. food+thought, etc. Sam I got yelled at on IRC for not providing times, so here: Key: Selected Archiver - Compression Time - Threads - FileSize Source file: dfly-1.12.2_REL.iso - 295,512KB Bzip2 - 0:27 - 4 - 108,742KB Bzip2 - 1:24 - 1 - 108,742KB Decompress: 0:10 GZip - 0:55 - 1 - 118,024KB Decompress: 0:10 Zip - 0:55 - 4 - 118,024KB Zip - 0:55 - 1 - 118,024KB Decompress: 0:11 7Zip - 1:50 - 2 - 73,952KB 7Zip - 3:09 - 1 - 73,952KB Decompress: 0:12 All done via the most unscientific methods available (reporting user time as displayed by the 7Zip user interface), so treat it as such. Tested on an Intel Q6600 (4x2.4GHz) w/ single SATA disk. Default compression settings were used across the board. Everything was tested using the windows 7-Zip program, which integrates all of these algorithms. Sam
Re: 7-Zip / Bzip2
Hello, On Tue, 2008-05-06 at 09:51 -0700, Samuel J. Greear wrote: Samuel J. Greear [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] Matthew Dillon [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] :Hi, : :Posted this to kernel@ by accident, please reply here instead :) : :I just wanted to know if there's any interest for the devs to add :something like p7zip to the base install; even if it's a simple fork :that only supports 7z. While 7zip is about as obnoxiously slow as :bzip2, it usually gets much better compression. : :That's not why I'm suggesting it though - what really gets me is that :bzip2 has no list option. Does that 10 gb bzip2 backup archive :contain 100gb of data, or 200gb? Other than dumping the entire :archive to /dev/null through wc, there's really no way to do it. Gzip :will list files, but its compression ratio is awful. : :I imagine that other OSes are going to be watching Dragonfly very :carefully in the next while as new the features (especially HAMMER) :mature. Maybe adding 7z will get yet another bandwagon going and :there will be support across the board :) : :Best Regards, :Ben Cadieux Well, I think not in base, at least not unless a lot of people are using it. p7zip is readily available via the pkgsrc tree and that's the most reasonable method of accessibility for now. -Matt Matthew Dillon [EMAIL PROTECTED] I do not know if DragonFly is actively tracking libarchive, but it seems to be the ticket for implementing new archiving/compression methods through a common mechanism. If one wanted to see 7z functionality in base implementing a libarchive provider is probably the way to go. I did a quick test at one point on the dfly distribution ISO and as I recall 7z was ~60% the size of bzip2 using standard settings. food+thought, etc. Sam I got yelled at on IRC for not providing times, so here: Key: Selected Archiver - Compression Time - Threads - FileSize Source file: dfly-1.12.2_REL.iso - 295,512KB Bzip2 - 0:27 - 4 - 108,742KB Bzip2 - 1:24 - 1 - 108,742KB Decompress: 0:10 GZip - 0:55 - 1 - 118,024KB Decompress: 0:10 Zip - 0:55 - 4 - 118,024KB Zip - 0:55 - 1 - 118,024KB Decompress: 0:11 7Zip - 1:50 - 2 - 73,952KB 7Zip - 3:09 - 1 - 73,952KB Decompress: 0:12 All done via the most unscientific methods available (reporting user time as displayed by the 7Zip user interface), so treat it as such. Tested on an Intel Q6600 (4x2.4GHz) w/ single SATA disk. Default compression settings were used across the board. Everything was tested using the windows 7-Zip program, which integrates all of these algorithms. Sam The compression time is not an issue for install disks, and decompression extra time of 7z is insignificant, but the best usage for this would be when packaging sources, not the installer CD, whose size will grow slower than the size of CD/DVD/BlueRay... Anyway our CD image could easily fit a mini-CD, so we don't need this, for now. Regards, Cristi. -- Cristi Magherusan, Universitatea Tehnica din Cluj - Napoca Centrul de Comunicatii Pusztai Kalman Tel. 0264/401247 http://cc.utcluj.ro signature.asc Description: This is a digitally signed message part
Re: 7-Zip / Bzip2
:I just noticed I still have these sitting here so I figured I would post :sizes. 1.12.1 ISO 294M, bz2 108M, 7z 74M, zip 118M, gz 120M. : :Sam Those are impressive numbers. I will keep an eye on it. If 7z is consistently better then bz2 I expect it will become widely adopted. I just think it is a bit early to put it into base when it is so readily accessible via pkgsrc. -Matt Matthew Dillon [EMAIL PROTECTED]
Re: 7-Zip / Bzip2
On Tue, 06 May 2008 20:29:53 +0300 Cristi Magherusan [EMAIL PROTECTED] wrote: The compression time is not an issue for install disks, and decompression extra time of 7z is insignificant, but the best usage for this would be when packaging sources, not the installer CD, whose size will grow slower than the size of CD/DVD/BlueRay... Anyway our CD image could easily fit a mini-CD, so we don't need this, for now. There is still the little matter of download time and bandwidth utilisation. Certainly if I saw a 73MB file and a 108MB file with the same contents I know which one I would choose to download (once I'd assured myself of being able to unpack it). -- C:WIN | Directable Mirror Arrays The computer obeys and wins.| A better way to focus the sun You lose and Bill collects. |licences available see |http://www.sohara.org/
Re: 7-Zip / Bzip2
Well, the issue is not so much one of download bandwidth as it is one of accessibility. Everyone has gzip, not everyone has 7z (yet). I would rather not have multiple compression formats on the download site, nor require that people have 7z in order to be able to unpack our release CDs! -Matt Matthew Dillon [EMAIL PROTECTED]
Re: 7-Zip / Bzip2
Well, the issue is not so much one of download bandwidth as it is one of accessibility. Everyone has gzip, not everyone has 7z (yet). I would rather not have multiple compression formats on the download site, nor require that people have 7z in order to be able to unpack our release CDs! This is the 'bandwagon' I was referring to --- it's a chicken and egg problem. I can safely say I would rarely be using gzip or bzip2 if 7z were in the base installs of all Linux/BSD releases. No one uses compress any more because gzip/bzip2 are in the base. If they weren't, you'd still see .Z files everywhere. Best Regards, Ben Cadieux
Re: 7-Zip / Bzip2
For everyone concerned with timing, I tested compressing a 4gb image of a quad boot USB key with a variety of junk on it at compression level 4. It still beats bzip2 by 90 mb and was almost 3 minutes quicker: # time -h bzip2 -k key.img 11m59.71s real 10m39.48s user 5.03s sys # time -h 7z a -mx=4 key.img.7z key.img 8m39.02s real 7m38.57s user 4.95s sys # ls -la | grep key -rw-r- 1 sol wheel 4102029312 Mar 8 11:27 key.img -rw--- 1 root wheel 904489376 May 6 11:38 key.img.7z -rw-r- 1 sol wheel 997632056 Mar 8 11:27 key.img.bz2 Best Regards, Ben Cadieux
Re: 7-Zip / Bzip2
Samuel J. Greear wrote: I just noticed I still have these sitting here so I figured I would post sizes. 1.12.1 ISO 294M, bz2 108M, 7z 74M, zip 118M, gz 120M. lrz 69M It doesn't mean that I care though. In fact, I use gzip most of time - bandwith is quite cheap nowadays. -- Hasso Tepper
Re: 7-Zip / Bzip2
:Hi, : :Posted this to kernel@ by accident, please reply here instead :) : :I just wanted to know if there's any interest for the devs to add :something like p7zip to the base install; even if it's a simple fork :that only supports 7z. While 7zip is about as obnoxiously slow as :bzip2, it usually gets much better compression. : :That's not why I'm suggesting it though - what really gets me is that :bzip2 has no list option. Does that 10 gb bzip2 backup archive :contain 100gb of data, or 200gb? Other than dumping the entire :archive to /dev/null through wc, there's really no way to do it. Gzip :will list files, but its compression ratio is awful. : :I imagine that other OSes are going to be watching Dragonfly very :carefully in the next while as new the features (especially HAMMER) :mature. Maybe adding 7z will get yet another bandwagon going and :there will be support across the board :) : :Best Regards, :Ben Cadieux Well, I think not in base, at least not unless a lot of people are using it. p7zip is readily available via the pkgsrc tree and that's the most reasonable method of accessibility for now. -Matt Matthew Dillon [EMAIL PROTECTED]
Re: 7-Zip / Bzip2
Matthew Dillon [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] :Hi, : :Posted this to kernel@ by accident, please reply here instead :) : :I just wanted to know if there's any interest for the devs to add :something like p7zip to the base install; even if it's a simple fork :that only supports 7z. While 7zip is about as obnoxiously slow as :bzip2, it usually gets much better compression. : :That's not why I'm suggesting it though - what really gets me is that :bzip2 has no list option. Does that 10 gb bzip2 backup archive :contain 100gb of data, or 200gb? Other than dumping the entire :archive to /dev/null through wc, there's really no way to do it. Gzip :will list files, but its compression ratio is awful. : :I imagine that other OSes are going to be watching Dragonfly very :carefully in the next while as new the features (especially HAMMER) :mature. Maybe adding 7z will get yet another bandwagon going and :there will be support across the board :) : :Best Regards, :Ben Cadieux Well, I think not in base, at least not unless a lot of people are using it. p7zip is readily available via the pkgsrc tree and that's the most reasonable method of accessibility for now. -Matt Matthew Dillon [EMAIL PROTECTED] I do not know if DragonFly is actively tracking libarchive, but it seems to be the ticket for implementing new archiving/compression methods through a common mechanism. If one wanted to see 7z functionality in base implementing a libarchive provider is probably the way to go. I did a quick test at one point on the dfly distribution ISO and as I recall 7z was ~60% the size of bzip2 using standard settings. food+thought, etc. Sam
