Re: Suggestion to and how to alow different compression for .debs
Previously sparc porters wrote: For this reason, we should not allow arbitrary compression tools to be used. Let me give another good reason why using a uncompress.sh script is something I will never accept: it means that unpacking a package in an arbitrary location is no longer a guaranteed safe action, since uncompress.sh could do something nasty. Wichert. -- / Generally uninteresting signature - ignore at your convenience \ | [EMAIL PROTECTED]http://www.liacs.nl/~wichert/ | | 1024D/2FA3BC2D 576E 100B 518D 2F16 36B0 2805 3CB8 9250 2FA3 BC2D | pgpe9vy4fXHhA.pgp Description: PGP signature
Re: Suggestion to and how to alow different compression for .debs
On Thu, 28 Oct 1999, Wichert Akkerman wrote: Let me give another good reason why using a uncompress.sh script is something I will never accept: it means that unpacking a package in an arbitrary location is no longer a guaranteed safe action, since uncompress.sh could do something nasty. You might want to check out how dpkg actually unpacks the control.tar.gz, if memory serves me it uses tar without chroot to do it, which means that control.tar.gz could easially contain /bin/sh or something equally nasty.. So it is hardly a guaranteed safe action right now. Of course that is fixable, Goswin's idea isn't. Jason
Re: Suggestion to and how to alow different compression for .debs
There's still problems with using pre-depends to make sure bzip2 is installed. It's not exactly what pre-depends was intended for. It's not going to be pretty if a user tries to remove bzip2 and dselect shoots up the dependency/conflict screen and marks every single package that was bzip2 compressed for removal. It's also specifically warned in the packaging manual that pre-depends should be used sparingly, for a number of reasons. Chris Pimlott (IANADD)
Re: Suggestion to and how to alow different compression for .debs
[I've elected not to Cc: debian-devel] On Tue, Oct 26, 1999 at 08:32:18PM -0400, Chris Pimlott wrote: There's still problems with using pre-depends to make sure bzip2 is installed. If we decide to use bzip2 in this capacity the package should be required and essential. -- Raul
Re: Suggestion to and how to alow different compression for .debs
On Tue, Oct 26, 1999 at 05:06:34PM -0400, Chris Pimlott wrote: On 21 Oct 1999, Goswin Brederlow wrote: Of cause policy should encourage to use bzip2 (or gzip if smaller) and base packages must use tar.gz (or tar.bz2 if bzip2 is in base) so that one can update debian. Any package using a non default compression must predepend on that compressor, but that should be clear for the packaging scripts and maintainer. If there has to be a line added to control, why not instead make a new line such as Archive: tar.bz2 (with Archive: tar.gz assumed if not specified) and not worry about a custom how_to_unpack script? Granted, this method would only support predefined types but would be simpler, and there aren't _that_ many formats people are dying to use - tar.gz and tar.bz2 (and possibly tar.Z) could probably make 99.9% of everyone happy, and stuff like tar.zip, tar.arj, tar.lha etc could be added if anyone really wanted it. Why even have that when the name of the .tar.(bz2|gz) is already in the .dsc? This is how I already have it setup in a test package. There is no need to add all the complexity. Ben
Re: Suggestion to and how to alow different compression for .debs
On Tue, Oct 26, 1999 at 11:23:24PM +0200, Goswin Brederlow wrote: Chris Pimlott [EMAIL PROTECTED] writes: You would need a switch case statement that tests for all possible formats that might be allowed. Having an uncompress.sh the procedure will be identical for all compressors, just pipe it through it. Its far easier to replace the gzip call with an uncompress.sh call than to program a switch case into all scripts. May the Source be with you. Uhh, so how would moving all of this external to dpkg-source make it any easier to program? All of the knowledge still has to be there. Ben
Re: Suggestion to and how to alow different compression for .debs
On Tue, Oct 26, 1999 at 08:32:18PM -0400, Chris Pimlott wrote: There's still problems with using pre-depends to make sure bzip2 is installed. It's not exactly what pre-depends was intended for. It's not going to be pretty if a user tries to remove bzip2 and dselect shoots up the dependency/conflict screen and marks every single package that was bzip2 compressed for removal. It's also specifically warned in the packaging manual that pre-depends should be used sparingly, for a number of reasons. That's what Essential: yes is for. Ben
Re: Suggestion to and how to alow different compression for .debs
Ben Collins [EMAIL PROTECTED] writes: On Tue, Oct 26, 1999 at 11:23:24PM +0200, Goswin Brederlow wrote: Chris Pimlott [EMAIL PROTECTED] writes: You would need a switch case statement that tests for all possible formats that might be allowed. Having an uncompress.sh the procedure will be identical for all compressors, just pipe it through it. Its far easier to replace the gzip call with an uncompress.sh call than to program a switch case into all scripts. May the Source be with you. Uhh, so how would moving all of this external to dpkg-source make it any easier to program? All of the knowledge still has to be there. My idea was to include a uncompress.sh file in the control.tar.gz and pipe the compressed tar file through that to get the uncompressed tar. All the change that would be needed is a simple test of existence (or the dpkg could ensure that there allways is such a script) and then pipe through that. Without that you have to unpack the .deb file, look around for a data.tar.xxx and make a switch/case over xxx. Each compressor would need its own entry there and as soon as the third compressor comes up for debian packages a lot of work is saved with my idea. I´m not suggesting that deb packages should be allowed to use any compression they like, but that dpkg and similar should be changed to handle arbitrary compression instead of just the few allowed once (namely gzip and hopefully soon bzip2). I call for generality here. Just think about the guy who said that 6 characters for a date will be enough. :) Two, three, four or even houndert different compression formats could prove to be one less than needed. My idea would carry the needed knowledge to unpack a package to the package itself and therby to the place that should realy know how to deal with the uncompresson. May the SOurce be with you. Goswin
Re: Suggestion to and how to alow different compression for .debs
On Wed, Oct 27, 1999 at 06:23:44PM +0200, Goswin Brederlow wrote: Without that you have to unpack the .deb file, look around for a data.tar.xxx and make a switch/case over xxx. Each compressor would need its own entry there and as soon as the third compressor comes up for debian packages a lot of work is saved with my idea. That's completely untrue. Try ar t foo.deb and you will get the contents of the .deb. Like I said, I already have this implemented (I also have a bug filed against file for an update to the magic parsing for .debs so it shows the compression type, easy huh :). The only thing needed for dpkg-deb to parse any other compression types is a simple addition of a single line for each one to the struct (this feature is based off of a bug report in the BTS). It's very quick and it works on the fly, not having to worry about an uncompress.sh being pulled from the control.tar.gz and forking to pipe through it. I´m not suggesting that deb packages should be allowed to use any compression they like, but that dpkg and similar should be changed to handle arbitrary compression instead of just the few allowed once (namely gzip and hopefully soon bzip2). I call for generality here. Just think about the guy who said that 6 characters for a date will be enough. :) Two, three, four or even houndert different compression formats could prove to be one less than needed. That's a bad analogy, what about the OS that allowed too many features only to let itself bloat and become unusable (Windows?). My idea would carry the needed knowledge to unpack a package to the package itself and therby to the place that should realy know how to deal with the uncompresson. No, compression types should be discernable by extension, and in the worst case magic on the data header. Why have _all_ of the packages include this file, when at most there will only need to be 3 or 4 different ones? Then we leave ourselves open to problems, not to mention the I'm better than everyone else, so I'll rewrite my script and make it better type of poeple, or the ones who decide to user crazy and rediculous compression types. Ben
Re: Suggestion to and how to alow different compression for .debs
Ben Collins writes: With the current implementation I have, it is a simple matter of adding a line for each (de)compressor wanted. I think we should choose carefully what compressions we allow, as it could lead to problems if we don't. For this reason, we should not allow arbitrary compression tools to be used. I agree. Specifically, machines tight on disk space (but with an NFS mirror, for example), will not want to have to have installed n different decompressors just to upgrade their packages. Matthew -- At least you know where you are with Microsoft. True. I just wish I'd brought a paddle. http://www.debian.org
Re: Suggestion to and how to alow different compression for .debs
Goswin Brederlow wrote: Joey Hess [EMAIL PROTECTED] writes: Goswin Brederlow wrote: Whats complicated about using uncompress.sh instead of gzip and fallback to gzip if not present? Tons of things. What about programs called in uncompress.sh -- are dependancies supposed to be fullfilled then? What happens when the script fails? What if you don't trust debian, but want to unpack a debian package anyway, without running any scripts from it? What about speed? It takes 2 hours to download X sources, 15-30 minutes to bunzip2 them and 24 Hours to compile the stuff (slink X on an MC68060). Do you realy think saving 10 minutes by using gzip2 and adding about 30% of size to the download is faster? I'm talking about _uncompress_speed_. Since BenC already has bzip2 .deb's implementedm in pure C, going through some shell script to make them work instead is obviously going to be slower. What about rar or mathematical/fractal compressors? For some data far better compression can be gained with a specialised packer. You're going overboard. Debian should not use compression methords like rar that are non-free. Nor should it use archive formats, like rar, that do not include unix permissions and ownership information. When a good unix fractal compression format comes along, we can talk. -- see shy jo
Re: Suggestion to and how to alow different compression for .debs
While this is debated I will upgrade hard drives. The 6 gb is not enough to continue mirroring debian and debian-non-US anymore. I am only mirroring i386 and have to usually have to make a delete pass before I can get all the updates. I can come up with a larger drive but I am thinking about fellow (poverty-stricken) debianites. I also think that any reduction in space and bandwidth requirements would be a big help to those wishing to provide a mirror. But congratulations are in order every time I need a larger drive to mirror debian. A veritable candy store of software packages. PW
Re: Suggestion to and how to alow different compression for .debs
Goswin Brederlow wrote: Why not pipe it through uncompress.sh as and if present in the control.tar.gz? Why not change to using the shar archive format for our packages? Because it's overly complicated, and unnecessary. -- see shy jo
Re: Suggestion to and how to alow different compression for .debs
On 21 Oct 1999, Goswin Brederlow wrote: Of cause policy should encourage to use bzip2 (or gzip if smaller) and base packages must use tar.gz (or tar.bz2 if bzip2 is in base) so that one can update debian. Any package using a non default compression must predepend on that compressor, but that should be clear for the packaging scripts and maintainer. If there has to be a line added to control, why not instead make a new line such as Archive: tar.bz2 (with Archive: tar.gz assumed if not specified) and not worry about a custom how_to_unpack script? Granted, this method would only support predefined types but would be simpler, and there aren't _that_ many formats people are dying to use - tar.gz and tar.bz2 (and possibly tar.Z) could probably make 99.9% of everyone happy, and stuff like tar.zip, tar.arj, tar.lha etc could be added if anyone really wanted it. Chris Pimlott (IANADD)
Re: Suggestion to and how to alow different compression for .debs
Chris Pimlott [EMAIL PROTECTED] writes: On 21 Oct 1999, Goswin Brederlow wrote: Of cause policy should encourage to use bzip2 (or gzip if smaller) and base packages must use tar.gz (or tar.bz2 if bzip2 is in base) so that one can update debian. Any package using a non default compression must predepend on that compressor, but that should be clear for the packaging scripts and maintainer. If there has to be a line added to control, why not instead make a new line such as Archive: tar.bz2 (with Archive: tar.gz assumed if not specified) and not worry about a custom how_to_unpack script? Granted, this method would only support predefined types but would be simpler, and there aren't _that_ many formats people are dying to use - tar.gz and tar.bz2 (and possibly tar.Z) could probably make 99.9% of everyone happy, and stuff like tar.zip, tar.arj, tar.lha etc could be added if anyone really wanted it. Chris Pimlott (IANADD) You would need a switch case statement that tests for all possible formats that might be allowed. Having an uncompress.sh the procedure will be identical for all compressors, just pipe it through it. Its far easier to replace the gzip call with an uncompress.sh call than to program a switch case into all scripts. May the Source be with you. Goswin
Re: Suggestion to and how to alow different compression for .debs
Joey Hess [EMAIL PROTECTED] writes: Goswin Brederlow wrote: Why not pipe it through uncompress.sh as and if present in the control.tar.gz? Why not change to using the shar archive format for our packages? Because it's overly complicated, and unnecessary. Whats complicated about using uncompress.sh instead of gzip and fallback to gzip if not present? The gain would be bzip2 compression, which is needed to reduce the size of debian and also any other compressor that might become commonly used or be extremly good at some sort of data. Don´t tell me that saving 3-4 GB of mirror space is unnecessary. May the Source be with you. Goswin
Re: Suggestion to and how to alow different compression for .debs
Goswin Brederlow wrote: Whats complicated about using uncompress.sh instead of gzip and fallback to gzip if not present? Tons of things. What about programs called in uncompress.sh -- are dependancies supposed to be fullfilled then? What happens when the script fails? What if you don't trust debian, but want to unpack a debian package anyway, without running any scripts from it? What about speed? The gain would be bzip2 compression BenC has already implemented this, from what I hear. -- see shy jo
Re: Suggestion to and how to alow different compression for .debs
On Thu, Oct 21, 1999 at 12:35:43AM +0200, Goswin Brederlow wrote: The control.tar.gz stays that way for the moment, but the data.tar.gz might be any format the maintainer seems fit. The control.tar.gz then contains a file called how_to_unpack that will be used to unpack the data file. Normaly it would be a shell script that just contains a call to tar -xz, but tar -xI or anything else would be allowed. This doesn't match how dpkg actually works --- internally it handles the untaring itself, and pipes that to gzip -dc. Piping it to bzip2 -dc based on the file extension (data.tar.bz2) would be a reasonably convenient change, I imagine; changing how it untar's may not be. YMMV, of course. Cheers, aj -- Anthony Towns [EMAIL PROTECTED] http://azure.humbug.org.au/~aj/ I don't speak for anyone save myself. PGP encrypted mail preferred. ``The thing is: trying to be too generic is EVIL. It's stupid, it results in slower code, and it results in more bugs.'' -- Linus Torvalds pgpRa1fq1YhJf.pgp Description: PGP signature
