Re: [gentoo-alt] Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/08/2014 12:45 AM, Ulrich Mueller wrote: >> On Sun, 07 Dec 2014, Zac Medico wrote: > >> On 12/07/2014 07:06 PM, Christoph Junghans wrote: >>> I know, I am late to the party, I just wanted to say that in >>> unpacker.eclass I implemented a variant which uses neither deb2tags >>> nor ar on prefix, but just bash's read and head. > > The code uses "head -c" which is a GNU extension: > >$ man 1p head >HEAD(1P)POSIX Programmer's ManualHEAD(1P) >[...] >SYNOPSIS > head [−n number] [file...] >[...] > > For example, on OpenBSD and HPUX there is no -c option, or it is used > differently. Also Busybox doesn't have it. > > Are we better off by requiring GNU head instead of GNU ar? That's a good question. We might use dd instead of head, or port the shell code to python. >> Very nice. We should borrow that code and use it to eliminate the >> need for deb2targz in portage. > > In PMS, this would have to go into EAPI 6, I suppose? Yes, I suppose so, since PMS currently says deb2targz is required when ar is not compatible with GNU ar. However, earlier you said that other package managers currently use ar unconditionally. Also, I have to wonder if any of the other package managers are used on AIX. So, maybe a retroactive change to PMS would be acceptable? Maybe it's not worth the trouble, and we should simply keep the deb2targz usage for AIX. -- Thanks, Zac
Re: [gentoo-alt] Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
Dnia 2014-12-07, o godz. 19:50:24 Zac Medico napisał(a): > On 12/07/2014 07:06 PM, Christoph Junghans wrote: > > I know, I am late to the party, I just wanted to say that in > > unpacker.eclass I implemented a variant which uses neither deb2tags > > nor ar on prefix, but just bash's read and head. > > Very nice. We should borrow that code and use it to eliminate the need > for deb2targz in portage. You shouldn't borrow code written by vapier. Ever. We have enough work on our shoulders having to fix it in one location. -- Best regards, Michał Górny pgpFIoLYEHMy3.pgp Description: OpenPGP digital signature
Re: [gentoo-alt] Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
> On Sun, 07 Dec 2014, Zac Medico wrote: > On 12/07/2014 07:06 PM, Christoph Junghans wrote: >> I know, I am late to the party, I just wanted to say that in >> unpacker.eclass I implemented a variant which uses neither deb2tags >> nor ar on prefix, but just bash's read and head. The code uses "head -c" which is a GNU extension: $ man 1p head HEAD(1P)POSIX Programmer's ManualHEAD(1P) [...] SYNOPSIS head [−n number] [file...] [...] For example, on OpenBSD and HPUX there is no -c option, or it is used differently. Also Busybox doesn't have it. Are we better off by requiring GNU head instead of GNU ar? > Very nice. We should borrow that code and use it to eliminate the > need for deb2targz in portage. In PMS, this would have to go into EAPI 6, I suppose? Ulrich pgpfesq0ME4CD.pgp Description: PGP signature
Re: [gentoo-alt] Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/07/2014 07:06 PM, Christoph Junghans wrote: > I know, I am late to the party, I just wanted to say that in > unpacker.eclass I implemented a variant which uses neither deb2tags > nor ar on prefix, but just bash's read and head. Very nice. We should borrow that code and use it to eliminate the need for deb2targz in portage. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On Sun, 7 Dec 2014 19:47:21 +0100 Ulrich Mueller wrote: > > > In a separate patch we can also include Brian's deb2targz xz > > handling for AIX, I guess. > > +1 > > Ulrich OK, pushed :) -- Brian Dolbec signature.asc Description: PGP signature
Re: [gentoo-alt] Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/07/2014 11:41 AM, Ulrich Mueller wrote: >> On Sun, 07 Dec 2014, Zac Medico wrote: > >> On 12/07/2014 11:23 AM, Fabian Groffen wrote: >>> FYI: >>> >>> % portageq envvar CHOST >>> x86_64-apple-darwin13 >>> % ar --version >>> ar: illegal option -- - >>> usage: ar -d [-TLsv] archive file ... >>> [...] > >> It's hard to whitelist it if doesn't support --version, so it seems >> reasonable to require deb2targz whenever ar is not GNU ar. > > unpack will still use ar if deb2targz is not installed. So on Darwin, > behaviour should be the same as before. > > Ulrich > Ah, that's a nice feature. That way, we can just leave it to platforms like AIX to pull in deb2targz when necessary. -- Thanks, Zac
Re: [gentoo-alt] Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
> On Sun, 07 Dec 2014, Zac Medico wrote: > On 12/07/2014 11:23 AM, Fabian Groffen wrote: >> FYI: >> >> % portageq envvar CHOST >> x86_64-apple-darwin13 >> % ar --version >> ar: illegal option -- - >> usage: ar -d [-TLsv] archive file ... >> [...] > It's hard to whitelist it if doesn't support --version, so it seems > reasonable to require deb2targz whenever ar is not GNU ar. unpack will still use ar if deb2targz is not installed. So on Darwin, behaviour should be the same as before. Ulrich pgpPQEfcOLwSJ.pgp Description: PGP signature
Re: [gentoo-alt] Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/07/2014 11:23 AM, Fabian Groffen wrote: > On 07-12-2014 10:49:30 -0800, Zac Medico wrote: >> On 12/07/2014 10:47 AM, Ulrich Mueller wrote: On Sun, 07 Dec 2014, Zac Medico wrote: >>> On 12/07/2014 10:37 AM, Ulrich Mueller wrote: > It's sort of trivial, but here is a patch: > > From c53e7057f94728d6e0c7d16c675702ca831b9a5a Mon Sep 17 00:00:00 2001 > From: Ulrich Müller > Date: Sun, 7 Dec 2014 19:33:52 +0100 > Subject: [PATCH] Prefer GNU ar when unpacking .deb packages. > > [...] >>> LGTM. >>> >>> So shall I push it? >> >> Yes, please do. > > FYI: > > % portageq envvar CHOST > x86_64-apple-darwin13 > % ar --version > ar: illegal option -- - > usage: ar -d [-TLsv] archive file ... > ar -m [-TLsv] archive file ... > ar -m [-abiTLsv] position archive file ... > ar -p [-TLsv] archive [file ...] > ar -q [-cTLsv] archive file ... > ar -r [-cuTLsv] archive file ... > ar -r [-abciuTLsv] position archive file ... > ar -t [-TLsv] archive [file ...] > ar -x [-ouTLsv] archive [file ...] > % ar x $EPREFIX/usr/portage/distfiles/realpath_1.16_i386.deb > % ls > control.tar.gz data.tar.gz debian-binary > > deb2targz should work (but I don't have it installed, so not sure if tested). > > Fabian > It's hard to whitelist it if doesn't support --version, so it seems reasonable to require deb2targz whenever ar is not GNU ar. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 07-12-2014 10:49:30 -0800, Zac Medico wrote: > On 12/07/2014 10:47 AM, Ulrich Mueller wrote: > >> On Sun, 07 Dec 2014, Zac Medico wrote: > > > >> On 12/07/2014 10:37 AM, Ulrich Mueller wrote: > >>> It's sort of trivial, but here is a patch: > >>> > >>> From c53e7057f94728d6e0c7d16c675702ca831b9a5a Mon Sep 17 00:00:00 2001 > >>> From: Ulrich Müller > >>> Date: Sun, 7 Dec 2014 19:33:52 +0100 > >>> Subject: [PATCH] Prefer GNU ar when unpacking .deb packages. > >>> > >>> [...] > > > >> LGTM. > > > > So shall I push it? > > Yes, please do. FYI: % portageq envvar CHOST x86_64-apple-darwin13 % ar --version ar: illegal option -- - usage: ar -d [-TLsv] archive file ... ar -m [-TLsv] archive file ... ar -m [-abiTLsv] position archive file ... ar -p [-TLsv] archive [file ...] ar -q [-cTLsv] archive file ... ar -r [-cuTLsv] archive file ... ar -r [-abciuTLsv] position archive file ... ar -t [-TLsv] archive [file ...] ar -x [-ouTLsv] archive [file ...] % ar x $EPREFIX/usr/portage/distfiles/realpath_1.16_i386.deb % ls control.tar.gz data.tar.gz debian-binary deb2targz should work (but I don't have it installed, so not sure if tested). Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/07/2014 10:47 AM, Ulrich Mueller wrote: >> On Sun, 07 Dec 2014, Zac Medico wrote: > >> On 12/07/2014 10:37 AM, Ulrich Mueller wrote: >>> It's sort of trivial, but here is a patch: >>> >>> From c53e7057f94728d6e0c7d16c675702ca831b9a5a Mon Sep 17 00:00:00 2001 >>> From: Ulrich Müller >>> Date: Sun, 7 Dec 2014 19:33:52 +0100 >>> Subject: [PATCH] Prefer GNU ar when unpacking .deb packages. >>> >>> [...] > >> LGTM. > > So shall I push it? Yes, please do. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
> On Sun, 07 Dec 2014, Zac Medico wrote: > On 12/07/2014 10:37 AM, Ulrich Mueller wrote: >> It's sort of trivial, but here is a patch: >> >> From c53e7057f94728d6e0c7d16c675702ca831b9a5a Mon Sep 17 00:00:00 2001 >> From: Ulrich Müller >> Date: Sun, 7 Dec 2014 19:33:52 +0100 >> Subject: [PATCH] Prefer GNU ar when unpacking .deb packages. >> >> [...] > LGTM. So shall I push it? > In a separate patch we can also include Brian's deb2targz xz > handling for AIX, I guess. +1 Ulrich pgpBWrrK3N45J.pgp Description: PGP signature
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/07/2014 10:37 AM, Ulrich Mueller wrote: >> On Sun, 07 Dec 2014, Zac Medico wrote: > >> Okay, I guess we can default to ar if [[ $(ar --version 2>&1) == "GNU >> ar"* ]] and otherwise fall back to deb2targz. > > It's sort of trivial, but here is a patch: > > From c53e7057f94728d6e0c7d16c675702ca831b9a5a Mon Sep 17 00:00:00 2001 > From: =?UTF-8?q?Ulrich=20M=C3=BCller?= > Date: Sun, 7 Dec 2014 19:33:52 +0100 > Subject: [PATCH] Prefer GNU ar when unpacking .deb packages. > > PMS reference: > deb packages (*.deb). Ebuilds must ensure that the deb2targz program > is installed on those platforms where the GNU binutils ar program is > not available and the installed ar program is incompatible with GNU > archives. Otherwise, ebuilds must ensure that GNU binutils is > installed. > --- > bin/phase-helpers.sh | 10 ++ > 1 file changed, 6 insertions(+), 4 deletions(-) > > diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh > index 6e437da..afad2e9 100644 > --- a/bin/phase-helpers.sh > +++ b/bin/phase-helpers.sh > @@ -448,11 +448,13 @@ unpack() { > "with EAPI '${EAPI}'. Instead > use 'deb'." > fi > # Unpacking .deb archives can not always be > done with > - # `ar`. For instance on AIX this doesn't work > out. If > - # we have `deb2targz` installed, prefer it over > `ar` for > - # that reason. We just make sure on AIX > `deb2targz` is > + # `ar`. For instance on AIX this doesn't work > out. > + # If `ar` is not the GNU binutils version and > we have > + # `deb2targz` installed, prefer it over `ar` > for that > + # reason. We just make sure on AIX `deb2targz` > is > # installed. > - if type -P deb2targz > /dev/null; then > + if [[ $(ar --version 2>/dev/null) != "GNU ar"* > ]] && \ > + type -P deb2targz > /dev/null; then > y=${x##*/} > local created_symlink=0 > if [ ! "$srcdir$x" -ef "$y" ] ; then > LGTM. In a separate patch we can also include Brian's deb2targz xz handling for AIX, I guess. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
> On Sun, 07 Dec 2014, Zac Medico wrote: > Okay, I guess we can default to ar if [[ $(ar --version 2>&1) == "GNU > ar"* ]] and otherwise fall back to deb2targz. It's sort of trivial, but here is a patch: >From c53e7057f94728d6e0c7d16c675702ca831b9a5a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ulrich=20M=C3=BCller?= Date: Sun, 7 Dec 2014 19:33:52 +0100 Subject: [PATCH] Prefer GNU ar when unpacking .deb packages. PMS reference: deb packages (*.deb). Ebuilds must ensure that the deb2targz program is installed on those platforms where the GNU binutils ar program is not available and the installed ar program is incompatible with GNU archives. Otherwise, ebuilds must ensure that GNU binutils is installed. --- bin/phase-helpers.sh | 10 ++ 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh index 6e437da..afad2e9 100644 --- a/bin/phase-helpers.sh +++ b/bin/phase-helpers.sh @@ -448,11 +448,13 @@ unpack() { "with EAPI '${EAPI}'. Instead use 'deb'." fi # Unpacking .deb archives can not always be done with - # `ar`. For instance on AIX this doesn't work out. If - # we have `deb2targz` installed, prefer it over `ar` for - # that reason. We just make sure on AIX `deb2targz` is + # `ar`. For instance on AIX this doesn't work out. + # If `ar` is not the GNU binutils version and we have + # `deb2targz` installed, prefer it over `ar` for that + # reason. We just make sure on AIX `deb2targz` is # installed. - if type -P deb2targz > /dev/null; then + if [[ $(ar --version 2>/dev/null) != "GNU ar"* ]] && \ + type -P deb2targz > /dev/null; then y=${x##*/} local created_symlink=0 if [ ! "$srcdir$x" -ef "$y" ] ; then -- 2.2.0
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/07/2014 09:54 AM, Zac Medico wrote: > On 12/06/2014 04:52 PM, Brian Dolbec wrote: >> >> From 4cb661d994cf8503c4459b8f7da7ee0f739a9826 Mon Sep 17 00:00:00 2001 >> From: Brian Dolbec >> Date: Sat, 6 Dec 2014 14:51:13 -0800 >> Subject: [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz >> types >> MIME-Version: 1.0 >> Content-Type: text/plain; charset=UTF-8 >> Content-Transfer-Encoding: 8bit > > LGTM, given that other package managers default to 'ar x' which already > works with xz files. Based on discussion with Ulrich Mueller I think we should default to ar if [[ $(ar --version 2>&1) == "GNU ar"* ]] and otherwise fall back to deb2targz. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/07/2014 10:05 AM, Ulrich Mueller wrote: >> On Sun, 07 Dec 2014, Zac Medico wrote: > >> Anyway, Brian's patch for xz support with debt2targz appears to be >> compatible with other package managers as well as AIX, so that seems >> like a good way to go. > > I still think that ar should always be used on platforms where GNU > binutils is installed. Currently unpack uses deb2targz when it finds > the program, which is not a well defined behaviour. Okay, I guess we can default to ar if [[ $(ar --version 2>&1) == "GNU ar"* ]] and otherwise fall back to deb2targz. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
> On Sun, 07 Dec 2014, Zac Medico wrote: > Anyway, Brian's patch for xz support with debt2targz appears to be > compatible with other package managers as well as AIX, so that seems > like a good way to go. I still think that ar should always be used on platforms where GNU binutils is installed. Currently unpack uses deb2targz when it finds the program, which is not a well defined behaviour. Ulrich pgppXonqP1VjG.pgp Description: PGP signature
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/06/2014 04:52 PM, Brian Dolbec wrote: > > From 4cb661d994cf8503c4459b8f7da7ee0f739a9826 Mon Sep 17 00:00:00 2001 > From: Brian Dolbec > Date: Sat, 6 Dec 2014 14:51:13 -0800 > Subject: [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz > types > MIME-Version: 1.0 > Content-Type: text/plain; charset=UTF-8 > Content-Transfer-Encoding: 8bit LGTM, given that other package managers default to 'ar x' which already works with xz files. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/07/2014 03:04 AM, Ulrich Mueller wrote: >> On Sat, 06 Dec 2014, Zac Medico wrote: > >> The PMS people should be *very* interested in any changes to unpack >> behavior like this. It supports behavior that will lead to failures for >> older versions of portage and other package managers. > > Some remarks: > > - The upstream deb2targz program supports only data.tar.gz. Only the > Gentoo version has been patched up to support other file types. > > - There is only a single package in the tree with a dependency on > app-arch/deb2targz, and this only on AIX [1]. So on most users' > systems deb2targz will not be installed, and "ar x" will be used for > unpacking. > > - Paludis and Pkgcore unconditionally use "ar x" for unpacking. > > - PMS says [2]: > > deb packages (*.deb). Ebuilds must ensure that the deb2targz > program is installed on those platforms where the GNU binutils > ar program is not available and the installed ar program is > incompatible with GNU archives. Otherwise, ebuilds must ensure > that GNU binutils is installed. > > One could conclude that GNU binutils ar should be used on platforms > where it is available. deb2targz should be used only where ar is > incompatible with the GNU version. > > So in any case, this is a problem of Portage only. I'd suggest to > change the logic in unpack() as follows: > > if ; then > ar x ... > elif ; then > deb2targz ... > else > ar x ... > fi > > And then the deb2targz case could be patched as proposed. After all, > it is only a fallback that should be compatible with "ar x" > extraction. > > Ulrich > > [1] http://qa-reports.gentoo.org/output/genrdeps/dindex/app-arch/deb2targz > [2] http://dev.gentoo.org/~ulm/pms/5/pms.html#x1-14100011.3.3.13 Okay, so we may want to examine this AIX case more closely, because dropping support for AIX would solve our problem with the lack of xz support in deb2targz. Anyway, Brian's patch for xz support with debt2targz appears to be compatible with other package managers as well as AIX, so that seems like a good way to go. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
> On Sat, 06 Dec 2014, Zac Medico wrote: > The PMS people should be *very* interested in any changes to unpack > behavior like this. It supports behavior that will lead to failures for > older versions of portage and other package managers. Some remarks: - The upstream deb2targz program supports only data.tar.gz. Only the Gentoo version has been patched up to support other file types. - There is only a single package in the tree with a dependency on app-arch/deb2targz, and this only on AIX [1]. So on most users' systems deb2targz will not be installed, and "ar x" will be used for unpacking. - Paludis and Pkgcore unconditionally use "ar x" for unpacking. - PMS says [2]: deb packages (*.deb). Ebuilds must ensure that the deb2targz program is installed on those platforms where the GNU binutils ar program is not available and the installed ar program is incompatible with GNU archives. Otherwise, ebuilds must ensure that GNU binutils is installed. One could conclude that GNU binutils ar should be used on platforms where it is available. deb2targz should be used only where ar is incompatible with the GNU version. So in any case, this is a problem of Portage only. I'd suggest to change the logic in unpack() as follows: if ; then ar x ... elif ; then deb2targz ... else ar x ... fi And then the deb2targz case could be patched as proposed. After all, it is only a fallback that should be compatible with "ar x" extraction. Ulrich [1] http://qa-reports.gentoo.org/output/genrdeps/dindex/app-arch/deb2targz [2] http://dev.gentoo.org/~ulm/pms/5/pms.html#x1-14100011.3.3.13 pgph26utPMmUN.pgp Description: PGP signature
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 12/06/2014 04:52 PM, Brian Dolbec wrote: > > From 4cb661d994cf8503c4459b8f7da7ee0f739a9826 Mon Sep 17 00:00:00 2001 > From: Brian Dolbec > Date: Sat, 6 Dec 2014 14:51:13 -0800 > Subject: [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz > types > MIME-Version: 1.0 > Content-Type: text/plain; charset=UTF-8 > Content-Transfer-Encoding: 8bit > > Issue reported by: mva in irc > Unpacking source... Unpacking 2gis_4.14.11-0trusty1+shv139+r191_amd64.deb to /var/tmp/portage/app-misc/2gis-4.14.11.139.191/work > deb2targz: converting '2gis_4.14.11-0trusty1+shv139+r191_amd64.deb' ... > deb2targz: skipping section 'debian-binary' > deb2targz: skipping section 'control.tar.gz' > deb2targz: wrote '2gis_4.14.11-0trusty1+shv139+r191_amd64.tar.xz' > mv: cannot stat ‘2gis_4.14.11-0trusty1+shv139+r191_amd64.tar.gz’: No such > file or directory > --- > bin/phase-helpers.sh | 5 +++-- > 1 file changed, 3 insertions(+), 2 deletions(-) > > diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh > index 6e437da..91f80c9 100644 > --- a/bin/phase-helpers.sh > +++ b/bin/phase-helpers.sh > @@ -475,8 +475,9 @@ unpack() { > rm -f "$y" > fi > if ! mv -f "${y%.deb}".tar.gz > data.tar.gz; then > - __helpers_die "$myfail" > - return 1 > + if ! mv -f "${y%.deb}".tar.xz > data.tar.xz; then > + __helpers_die "$myfail" > + return 1 > fi > else > if ! ar x "$srcdir$x"; then > The PMS people should be *very* interested in any changes to unpack behavior like this. It supports behavior that will lead to failures for older versions of portage and other package managers. -- Thanks, Zac