Re: [gentoo-alt] Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On Sun, 07 Dec 2014, Zac Medico wrote: On 12/07/2014 07:06 PM, Christoph Junghans wrote: I know, I am late to the party, I just wanted to say that in unpacker.eclass I implemented a variant which uses neither deb2tags nor ar on prefix, but just bash's read and head. The code uses head -c which is a GNU extension: $ man 1p head HEAD(1P)POSIX Programmer's ManualHEAD(1P) [...] SYNOPSIS head [−n number] [file...] [...] For example, on OpenBSD and HPUX there is no -c option, or it is used differently. Also Busybox doesn't have it. Are we better off by requiring GNU head instead of GNU ar? Very nice. We should borrow that code and use it to eliminate the need for deb2targz in portage. In PMS, this would have to go into EAPI 6, I suppose? Ulrich pgpfesq0ME4CD.pgp Description: PGP signature
Re: [gentoo-alt] Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
Dnia 2014-12-07, o godz. 19:50:24 Zac Medico zmed...@gentoo.org napisał(a): On 12/07/2014 07:06 PM, Christoph Junghans wrote: I know, I am late to the party, I just wanted to say that in unpacker.eclass I implemented a variant which uses neither deb2tags nor ar on prefix, but just bash's read and head. Very nice. We should borrow that code and use it to eliminate the need for deb2targz in portage. You shouldn't borrow code written by vapier. Ever. We have enough work on our shoulders having to fix it in one location. -- Best regards, Michał Górny pgpFIoLYEHMy3.pgp Description: OpenPGP digital signature
Re: [gentoo-alt] Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/08/2014 12:45 AM, Ulrich Mueller wrote: On Sun, 07 Dec 2014, Zac Medico wrote: On 12/07/2014 07:06 PM, Christoph Junghans wrote: I know, I am late to the party, I just wanted to say that in unpacker.eclass I implemented a variant which uses neither deb2tags nor ar on prefix, but just bash's read and head. The code uses head -c which is a GNU extension: $ man 1p head HEAD(1P)POSIX Programmer's ManualHEAD(1P) [...] SYNOPSIS head [−n number] [file...] [...] For example, on OpenBSD and HPUX there is no -c option, or it is used differently. Also Busybox doesn't have it. Are we better off by requiring GNU head instead of GNU ar? That's a good question. We might use dd instead of head, or port the shell code to python. Very nice. We should borrow that code and use it to eliminate the need for deb2targz in portage. In PMS, this would have to go into EAPI 6, I suppose? Yes, I suppose so, since PMS currently says deb2targz is required when ar is not compatible with GNU ar. However, earlier you said that other package managers currently use ar unconditionally. Also, I have to wonder if any of the other package managers are used on AIX. So, maybe a retroactive change to PMS would be acceptable? Maybe it's not worth the trouble, and we should simply keep the deb2targz usage for AIX. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On Sat, 06 Dec 2014, Zac Medico wrote: The PMS people should be *very* interested in any changes to unpack behavior like this. It supports behavior that will lead to failures for older versions of portage and other package managers. Some remarks: - The upstream deb2targz program supports only data.tar.gz. Only the Gentoo version has been patched up to support other file types. - There is only a single package in the tree with a dependency on app-arch/deb2targz, and this only on AIX [1]. So on most users' systems deb2targz will not be installed, and ar x will be used for unpacking. - Paludis and Pkgcore unconditionally use ar x for unpacking. - PMS says [2]: deb packages (*.deb). Ebuilds must ensure that the deb2targz program is installed on those platforms where the GNU binutils ar program is not available and the installed ar program is incompatible with GNU archives. Otherwise, ebuilds must ensure that GNU binutils is installed. One could conclude that GNU binutils ar should be used on platforms where it is available. deb2targz should be used only where ar is incompatible with the GNU version. So in any case, this is a problem of Portage only. I'd suggest to change the logic in unpack() as follows: if ar is GNU; then ar x ... elif deb2targz installed; then deb2targz ... else ar x ... fi And then the deb2targz case could be patched as proposed. After all, it is only a fallback that should be compatible with ar x extraction. Ulrich [1] http://qa-reports.gentoo.org/output/genrdeps/dindex/app-arch/deb2targz [2] http://dev.gentoo.org/~ulm/pms/5/pms.html#x1-14100011.3.3.13 pgph26utPMmUN.pgp Description: PGP signature
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/07/2014 03:04 AM, Ulrich Mueller wrote: On Sat, 06 Dec 2014, Zac Medico wrote: The PMS people should be *very* interested in any changes to unpack behavior like this. It supports behavior that will lead to failures for older versions of portage and other package managers. Some remarks: - The upstream deb2targz program supports only data.tar.gz. Only the Gentoo version has been patched up to support other file types. - There is only a single package in the tree with a dependency on app-arch/deb2targz, and this only on AIX [1]. So on most users' systems deb2targz will not be installed, and ar x will be used for unpacking. - Paludis and Pkgcore unconditionally use ar x for unpacking. - PMS says [2]: deb packages (*.deb). Ebuilds must ensure that the deb2targz program is installed on those platforms where the GNU binutils ar program is not available and the installed ar program is incompatible with GNU archives. Otherwise, ebuilds must ensure that GNU binutils is installed. One could conclude that GNU binutils ar should be used on platforms where it is available. deb2targz should be used only where ar is incompatible with the GNU version. So in any case, this is a problem of Portage only. I'd suggest to change the logic in unpack() as follows: if ar is GNU; then ar x ... elif deb2targz installed; then deb2targz ... else ar x ... fi And then the deb2targz case could be patched as proposed. After all, it is only a fallback that should be compatible with ar x extraction. Ulrich [1] http://qa-reports.gentoo.org/output/genrdeps/dindex/app-arch/deb2targz [2] http://dev.gentoo.org/~ulm/pms/5/pms.html#x1-14100011.3.3.13 Okay, so we may want to examine this AIX case more closely, because dropping support for AIX would solve our problem with the lack of xz support in deb2targz. Anyway, Brian's patch for xz support with debt2targz appears to be compatible with other package managers as well as AIX, so that seems like a good way to go. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/06/2014 04:52 PM, Brian Dolbec wrote: From 4cb661d994cf8503c4459b8f7da7ee0f739a9826 Mon Sep 17 00:00:00 2001 From: Brian Dolbec dol...@gentoo.org Date: Sat, 6 Dec 2014 14:51:13 -0800 Subject: [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit LGTM, given that other package managers default to 'ar x' which already works with xz files. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On Sun, 07 Dec 2014, Zac Medico wrote: Anyway, Brian's patch for xz support with debt2targz appears to be compatible with other package managers as well as AIX, so that seems like a good way to go. I still think that ar should always be used on platforms where GNU binutils is installed. Currently unpack uses deb2targz when it finds the program, which is not a well defined behaviour. Ulrich pgppXonqP1VjG.pgp Description: PGP signature
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/07/2014 10:05 AM, Ulrich Mueller wrote: On Sun, 07 Dec 2014, Zac Medico wrote: Anyway, Brian's patch for xz support with debt2targz appears to be compatible with other package managers as well as AIX, so that seems like a good way to go. I still think that ar should always be used on platforms where GNU binutils is installed. Currently unpack uses deb2targz when it finds the program, which is not a well defined behaviour. Okay, I guess we can default to ar if [[ $(ar --version 21) == GNU ar* ]] and otherwise fall back to deb2targz. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/07/2014 09:54 AM, Zac Medico wrote: On 12/06/2014 04:52 PM, Brian Dolbec wrote: From 4cb661d994cf8503c4459b8f7da7ee0f739a9826 Mon Sep 17 00:00:00 2001 From: Brian Dolbec dol...@gentoo.org Date: Sat, 6 Dec 2014 14:51:13 -0800 Subject: [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit LGTM, given that other package managers default to 'ar x' which already works with xz files. Based on discussion with Ulrich Mueller I think we should default to ar if [[ $(ar --version 21) == GNU ar* ]] and otherwise fall back to deb2targz. -- Thanks, Zac
Re: [gentoo-alt] Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/07/2014 11:23 AM, Fabian Groffen wrote: On 07-12-2014 10:49:30 -0800, Zac Medico wrote: On 12/07/2014 10:47 AM, Ulrich Mueller wrote: On Sun, 07 Dec 2014, Zac Medico wrote: On 12/07/2014 10:37 AM, Ulrich Mueller wrote: It's sort of trivial, but here is a patch: From c53e7057f94728d6e0c7d16c675702ca831b9a5a Mon Sep 17 00:00:00 2001 From: Ulrich Müller u...@gentoo.org Date: Sun, 7 Dec 2014 19:33:52 +0100 Subject: [PATCH] Prefer GNU ar when unpacking .deb packages. [...] LGTM. So shall I push it? Yes, please do. FYI: % portageq envvar CHOST x86_64-apple-darwin13 % ar --version ar: illegal option -- - usage: ar -d [-TLsv] archive file ... ar -m [-TLsv] archive file ... ar -m [-abiTLsv] position archive file ... ar -p [-TLsv] archive [file ...] ar -q [-cTLsv] archive file ... ar -r [-cuTLsv] archive file ... ar -r [-abciuTLsv] position archive file ... ar -t [-TLsv] archive [file ...] ar -x [-ouTLsv] archive [file ...] % ar x $EPREFIX/usr/portage/distfiles/realpath_1.16_i386.deb % ls control.tar.gz data.tar.gz debian-binary deb2targz should work (but I don't have it installed, so not sure if tested). Fabian It's hard to whitelist it if doesn't support --version, so it seems reasonable to require deb2targz whenever ar is not GNU ar. -- Thanks, Zac
Re: [gentoo-alt] Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On Sun, 07 Dec 2014, Zac Medico wrote: On 12/07/2014 11:23 AM, Fabian Groffen wrote: FYI: % portageq envvar CHOST x86_64-apple-darwin13 % ar --version ar: illegal option -- - usage: ar -d [-TLsv] archive file ... [...] It's hard to whitelist it if doesn't support --version, so it seems reasonable to require deb2targz whenever ar is not GNU ar. unpack will still use ar if deb2targz is not installed. So on Darwin, behaviour should be the same as before. Ulrich pgpPQEfcOLwSJ.pgp Description: PGP signature
Re: [gentoo-alt] Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/07/2014 11:41 AM, Ulrich Mueller wrote: On Sun, 07 Dec 2014, Zac Medico wrote: On 12/07/2014 11:23 AM, Fabian Groffen wrote: FYI: % portageq envvar CHOST x86_64-apple-darwin13 % ar --version ar: illegal option -- - usage: ar -d [-TLsv] archive file ... [...] It's hard to whitelist it if doesn't support --version, so it seems reasonable to require deb2targz whenever ar is not GNU ar. unpack will still use ar if deb2targz is not installed. So on Darwin, behaviour should be the same as before. Ulrich Ah, that's a nice feature. That way, we can just leave it to platforms like AIX to pull in deb2targz when necessary. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/06/2014 04:52 PM, Brian Dolbec wrote:
From 4cb661d994cf8503c4459b8f7da7ee0f739a9826 Mon Sep 17 00:00:00 2001
From: Brian Dolbec dol...@gentoo.org
Date: Sat, 6 Dec 2014 14:51:13 -0800
Subject: [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz
types
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Issue reported by: mva in irc
Unpacking source...
Unpacking 2gis_4.14.11-0trusty1+shv139+r191_amd64.deb to
/var/tmp/portage/app-misc/2gis-4.14.11.139.191/work
deb2targz: converting '2gis_4.14.11-0trusty1+shv139+r191_amd64.deb' ...
deb2targz: skipping section 'debian-binary'
deb2targz: skipping section 'control.tar.gz'
deb2targz: wrote '2gis_4.14.11-0trusty1+shv139+r191_amd64.tar.xz'
mv: cannot stat ‘2gis_4.14.11-0trusty1+shv139+r191_amd64.tar.gz’: No such
file or directory
---
bin/phase-helpers.sh | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
index 6e437da..91f80c9 100644
--- a/bin/phase-helpers.sh
+++ b/bin/phase-helpers.sh
@@ -475,8 +475,9 @@ unpack() {
rm -f $y
fi
if ! mv -f ${y%.deb}.tar.gz
data.tar.gz; then
- __helpers_die $myfail
- return 1
+ if ! mv -f ${y%.deb}.tar.xz
data.tar.xz; then
+ __helpers_die $myfail
+ return 1
fi
else
if ! ar x $srcdir$x; then
The PMS people should be *very* interested in any changes to unpack
behavior like this. It supports behavior that will lead to failures for
older versions of portage and other package managers.
--
Thanks,
Zac
