Re: [gentoo-dev] RFC: vcs-snapshot-r1.eclass -- a better eclass for VCS snapshots (and others)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 06/11/2012 04:25 PM, Michał Górny wrote: Committed onto vcs-snapshot.eclass after fixing all in-tree users. Thanks for the update. One suggestion: Have you thought about using bsdtar from app-arch/libarchive instead of GNU ones? You'll need an additional dependency, but it has zip file support and I've verified the options you use work [1]. To be clear, I prefer tar over zip, too, but you can restore do not prefer zip over tar, but you can restore the lost zip functionality (plus others i assume). Michael [1] bsdtar -C /tmp/xxx -x --strip-components 1 -f ~/keithw-mosh-mosh-1.2.1-0-g778b5af.zip - -- Gentoo Dev http://xmw.de/ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iF4EAREIAAYFAk/XP6IACgkQknrdDGLu8JBx4QD/dcCp2zdAAjz0kMEVmmdIgQn3 dqU6wv4vBhipVrlaJRYA/R1LAd8lfPBh3uiUEvJ7MY/m1ExTkDXA64QQ5LC61jOA =CAmK -END PGP SIGNATURE-
Re: [gentoo-dev] RFC: vcs-snapshot-r1.eclass -- a better eclass for VCS snapshots (and others)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 - From the eclass: # XXX: check whether the directory structure inside is # fine? i.e. if the tarball has actually a parent dir. This should work with gnu and bsd tar, and sed+sort+wc from sys-apps/coreutils as well as busybox. [ $(bsdtar -t -f ~/keithw-mosh-mosh-1.2.1-0-g778b5af.zip \ | sed 's:/.*::' | sort -u | wc -l) == 1 ] I'm totally sure if we have to handle leading slashes and ./ in obscure tars/zips but I'd say this tests qualifies for an || ewarn Please check $f for an single parent directory or something like that. My 2 cents - -- Gentoo Dev http://xmw.de/ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iF4EAREIAAYFAk/XQ6IACgkQknrdDGLu8JCXrQD/esMpA3z4jiZIsI3qkiz3zM7X 0I2ck7tG6WHGqTP/HEQA/jCAxkDt2hUXM+govuZaKrNHj5pBNvJIQNUF7VnzYKYr =Lf4O -END PGP SIGNATURE-
Re: [gentoo-dev] RFC: vcs-snapshot-r1.eclass -- a better eclass for VCS snapshots (and others)
On Tue, 12 Jun 2012 15:09:54 +0200 Michael Weber x...@gentoo.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 06/11/2012 04:25 PM, Michał Górny wrote: Committed onto vcs-snapshot.eclass after fixing all in-tree users. Thanks for the update. One suggestion: Have you thought about using bsdtar from app-arch/libarchive instead of GNU ones? You'll need an additional dependency, but it has zip file support and I've verified the options you use work [1]. To be clear, I prefer tar over zip, too, but you can restore do not prefer zip over tar, but you can restore the lost zip functionality (plus others i assume). Thanks for the idea. I will be happy to go that way if someone actually requests zipfile support. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: vcs-snapshot-r1.eclass -- a better eclass for VCS snapshots (and others)
On Tue, 12 Jun 2012 15:26:58 +0200 Michael Weber x...@gentoo.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 - From the eclass: # XXX: check whether the directory structure inside is # fine? i.e. if the tarball has actually a parent dir. This should work with gnu and bsd tar, and sed+sort+wc from sys-apps/coreutils as well as busybox. [ $(bsdtar -t -f ~/keithw-mosh-mosh-1.2.1-0-g778b5af.zip \ | sed 's:/.*::' | sort -u | wc -l) == 1 ] I'm totally sure if we have to handle leading slashes and ./ in obscure tars/zips but I'd say this tests qualifies for an || ewarn Please check $f for an single parent directory or something like that. Well, I was hoping to come up with something which doesn't involve running additional 'tar -t', to be honest. I have to think about it some more time. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: vcs-snapshot-r1.eclass -- a better eclass for VCS snapshots (and others)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 06/12/2012 04:46 PM, Michał Górny wrote: Well, I was hoping to come up with something which doesn't involve running additional 'tar -t', to be honest. I have to think about it some more time. FTR, filelist=$(tar xvC /tmp/xxx --strip-components 1 \ -f ~/keithw-mosh-mosh-1.2.1-0-g778b5af.tar.gz 21) || die [ $(echo $filelist | sed 's:/.*::' | sort -u | wc -l) == 1 ] \ || ewarn ... does work with GNU tar, but not with bsdtar, which displays the stripped filenames, plus an leading x . Michael - -- Gentoo Dev http://xmw.de/ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iF4EAREIAAYFAk/XrOwACgkQknrdDGLu8JCbwwD+InGUaLTki1W6gYwy+O2zAEek peIoy1cjDig3EWqRtacA/RVFf3P5y8MBRKaE7yeyzypTUomvXq6tWqVZSUywnjYL =3Zw6 -END PGP SIGNATURE-
Re: [gentoo-dev] RFC: vcs-snapshot-r1.eclass -- a better eclass for VCS snapshots (and others)
On Thu, 7 Jun 2012 23:49:52 +0200
Michał Górny mgo...@gentoo.org wrote:
The idea with the new eclass is to extract all supported archives
(which means .tar* at the point) into ${WORKDIR} subdirectories
matching their names. In other words, if you've got:
SRC_URI=http://foo/getbar - ${P}.tar.bz2
http://foo/getbaz - ${P}-data.tar.bz2
You'd get the contents in ${WORKDIR}/${P} and ${WORKDIR}/${P}-data
respectively.
Committed onto vcs-snapshot.eclass after fixing all in-tree users.
--
Best regards,
Michał Górny
signature.asc
Description: PGP signature
Re: [gentoo-dev] RFC: vcs-snapshot-r1.eclass -- a better eclass for VCS snapshots (and others)
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 06/07/2012 11:49 PM, Michał Górny wrote:
Hello,
As it was pointed out, vcs-snapshot is unable to handle multiple
tarballs nicely. As fixing this would involve changing eclass API
(and they are packages which are known to break thanks to it), I'm
posting a new eclass for review. Of course, if the community
insists, I could just update the old eclass and fix the ebuilds
broken that way.
The idea with the new eclass is to extract all supported archives
(which means .tar* at the point) into ${WORKDIR} subdirectories
matching their names. In other words, if you've got:
SRC_URI=http://foo/getbar - ${P}.tar.bz2 http://foo/getbaz -
${P}-data.tar.bz2
You'd get the contents in ${WORKDIR}/${P} and ${WORKDIR}/${P}-data
respectively.
A few things to note:
1) the implemention just dumbly uses 'tar --strip-components'. If
it hits a tarball without a single top-level directory, scary
things will happen;
2) only tarballs are handled, other archives are just passed
through to unpack;
3) at some point the eclass could be extended to handle more
formats. So if you've got other archives in your SRC_URI, please
make sure they are named consistently with the parent dir, so that
they would not relocate when support for that particular format is
added.
By the way, the credit for the overall idea goes to hasufell.
What is with zipballs from github for example? They are not supported
by this method afais.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJP0f9oAAoJEFpvPKfnPDWzprAH/ikedjEAZ6Hg95zd+F7XNxLN
XqfpAaE9N5bQ9AXfE+a1IHQRh6doSY6aw10he053SQhHEVJnB6ey0rMvkICRZ+w1
ZvygNSZCP2XJg4weZinIiNSJbQ7HCHFE2DuEkz78rdQ7cFwV5Y0AEGJiFDyMw4no
RPJoNHNyFUk578W14ioog9oDOookpRN6nttvwq/Bqoh8+tCbmxTlowJndfCrw4iS
T/q5UUKWsEEBVd2zz2TE0+eku9kdfBnez/9e2XLMUYfsTBE17/xluaqiRcLQC666
FFe0+MGI8sz7t1h+PjflNXIh8jochI7p4thgHqsWP5Wky04/Av8eD7br5uQ3YLE=
=A5LA
-END PGP SIGNATURE-
Re: [gentoo-dev] RFC: vcs-snapshot-r1.eclass -- a better eclass for VCS snapshots (and others)
On Fri, 08 Jun 2012 15:34:32 +0200 hasufell hasuf...@gentoo.org wrote: What is with zipballs from github for example? They are not supported by this method afais. Is there a need for that? Considering that I can't name any snapshot provider which does offer .zip but doesn't offer .tar*, and the fact that .tar* is the format preferred for Linux and smaller, is there a need for that support? As the doctext states, I'm open to implementing support for new formats but I'd rather wait with that till it's needed for anything. Tarballs are easy to handle, zips are not. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: vcs-snapshot-r1.eclass -- a better eclass for VCS snapshots (and others)
On Thu, 7 Jun 2012 23:49:52 +0200
Michał Górny mgo...@gentoo.org wrote:
Of course, if the community insists, I could just update the old
eclass and fix the ebuilds broken that way.
The following ebuilds would need fixing if -r1 is to replace -r0
in place:
a) naming the tarball ${P}-src.tar.gz:
- dev-java/jna,
- dev-python/ws4py.
They could be easily fixed by either renaming the destination file (it
shouldn't collide with anything, I think) or setting appropriate S=.
b) downloading .zip for no good reason:
- dev-util/ninja (betagarden overlay).
If the eclass is to spread, there are two more packages in gx86 using
zipballs (which would not be supported by -r1).
All the packages mentioned above are from github so they could be
easily changed to grab tarballs instead.
--
Best regards,
Michał Górny
signature.asc
Description: PGP signature
Re: [gentoo-dev] RFC: vcs-snapshot-r1.eclass -- a better eclass for VCS snapshots (and others)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/08/2012 03:55 PM, Michał Górny wrote: On Fri, 08 Jun 2012 15:34:32 +0200 Is there a need for that? I don't know, do you? This reduces the amount of archives the eclass can handle. Unless gentoo decides to drop zip support I don't see a reason to do that in an eclass too just for the sake of code-style. My previous implementation had no trouble with zipballs. So if you suggest a new implementation I would expect that to be better. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJP0gbjAAoJEFpvPKfnPDWzmm4H/0lNb6dH/RNExQw4XWt/1HrX mxrTx2oruI1LkNCRWKX9wJB96v7i89nfJS6qD6tj/ma3CX3EzSx3MbJJfs18SwO/ pLJnROiW3y5tHadrYsoqj0BFN+UkFVHbwalM1hUIBKBchifUqsFeHmcMl9Hiqpva dMf2xOhvj0w4e8feVMbamHjWSMTUO12r8XjNxm7Y4CiQx85DX+0PcUrMV6Nt1sT1 u9sWqCSLdbIQGFlxdykeERrtb4Us0bJ/BopJ0YFCXBvpfoePNoODqp/U+P4We2bC nUiqIimVEYh8jPVx2om36eRLiIuH7gSb4gc1Dljc9TfBEvFb38bMweYW9y2iA+8= =jSxt -END PGP SIGNATURE-
Re: [gentoo-dev] RFC: vcs-snapshot-r1.eclass -- a better eclass for VCS snapshots (and others)
On Fri, 08 Jun 2012 16:06:27 +0200 hasufell hasuf...@gentoo.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/08/2012 03:55 PM, Michał Górny wrote: On Fri, 08 Jun 2012 15:34:32 +0200 Is there a need for that? I don't know, do you? Estimated to three packages, two in gx86, one in betagarden. All of them grabbing zipballs from github, so can be easily changed to download tarballs instead. Do you see a reason why they should use zipballs which are larger and require adding unzip to DEPEND rather than tarballs? The only reason for that I can see is that they copy-pasted the 'download' URI from somewhere where author posted only zipball link. This reduces the amount of archives the eclass can handle. Unless gentoo decides to drop zip support I don't see a reason to do that in an eclass too just for the sake of code-style. It's for a sake of code work amount. And zip support in Gentoo is not obligatory. You need to add unzip to your DEPEND yourself. My previous implementation had no trouble with zipballs. So if you suggest a new implementation I would expect that to be better. Your previous implementation was against the KISS principle. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: vcs-snapshot-r1.eclass -- a better eclass for VCS snapshots (and others)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/08/2012 04:24 PM, Michał Górny wrote: On Fri, 08 Jun 2012 16:06:27 +0200 hasufell hasuf...@gentoo.org wrote: On 06/08/2012 03:55 PM, Michał Górny wrote: On Fri, 08 Jun 2012 15:34:32 +0200 Is there a need for that? I don't know, do you? Estimated to three packages, two in gx86, one in betagarden. All of them grabbing zipballs from github, so can be easily changed to download tarballs instead. Do you see a reason why they should use zipballs which are larger and require adding unzip to DEPEND rather than tarballs? The only reason for that I can see is that they copy-pasted the 'download' URI from somewhere where author posted only zipball link. a) gentoo supports zip and zip is available. Not supporting zip would need a comment or a conditional error. Neither is implemented in the current code. Then you can start telling users what they should use and what not. b) future proof c) less breakage, less confusion This reduces the amount of archives the eclass can handle. Unless gentoo decides to drop zip support I don't see a reason to do that in an eclass too just for the sake of code-style. It's for a sake of code work amount. And zip support in Gentoo is not obligatory. You need to add unzip to your DEPEND yourself. My previous implementation had no trouble with zipballs. So if you suggest a new implementation I would expect that to be better. Your previous implementation was against the KISS principle. Your implementation breaks existing functionality. - -1 for this -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJP0hJyAAoJEFpvPKfnPDWzl6MIALSHrIp5sunNpYZziXnWte2s zx8NZLzi8YjZpymrPDWIHhTPN4t84IxzQW0J9rzbiB5VvjgzYKhHrStDoSGgam5E uPtR5p34W/pkMO5oxiZXkgOYv4+NbnfvCN1qjuexevsf7AN7bmJwB12/IZOdfHhI Dt1SE5EI+Puj9ydHUMFe6ooDlkiM6gpyP6RkhEouXHy6+Ji1CqHjMjzD4NMa67Gs 08IhIsQ3WatOATdRgnNjx/WOSiFio4hGnR8cPNfK5BH2OuuORcMCGoV8psrYGiXm /yRUW6a0msxzFLivyiCqnM+Hwar2oBzspwm2pHD7Pu1v0B8zOSkLW7cdzqiGQrQ= =3aNm -END PGP SIGNATURE-
[gentoo-dev] RFC: vcs-snapshot-r1.eclass -- a better eclass for VCS snapshots (and others)
Hello,
As it was pointed out, vcs-snapshot is unable to handle multiple
tarballs nicely. As fixing this would involve changing eclass API
(and they are packages which are known to break thanks to it), I'm
posting a new eclass for review. Of course, if the community insists,
I could just update the old eclass and fix the ebuilds broken that way.
The idea with the new eclass is to extract all supported archives
(which means .tar* at the point) into ${WORKDIR} subdirectories
matching their names. In other words, if you've got:
SRC_URI=http://foo/getbar - ${P}.tar.bz2
http://foo/getbaz - ${P}-data.tar.bz2
You'd get the contents in ${WORKDIR}/${P} and ${WORKDIR}/${P}-data
respectively.
A few things to note:
1) the implemention just dumbly uses 'tar --strip-components'. If it
hits a tarball without a single top-level directory, scary things will
happen;
2) only tarballs are handled, other archives are just passed through
to unpack;
3) at some point the eclass could be extended to handle more formats.
So if you've got other archives in your SRC_URI, please make sure they
are named consistently with the parent dir, so that they would not
relocate when support for that particular format is added.
By the way, the credit for the overall idea goes to hasufell.
--
Best regards,
Michał Górny
# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
# @ECLASS: vcs-snapshot-r1.eclass
# @MAINTAINER:
# mgo...@gentoo.org
# @BLURB: support eclass for unpacking VCS snapshot tarballs
# @DESCRIPTION:
# This eclass provides a convenience src_unpack() which does unpack all
# the tarballs in SRC_URI to locations matching their (local) names,
# discarding the original parent directory.
#
# The typical use case are VCS snapshots, coming from github, bitbucket
# and similar services. They have hash appended to the directory name
# which makes extracting them a painful experience. But if you just use
# a SRC_URI arrow to rename it (which you're likely have to do anyway),
# vcs-snapshot will just extract it into a matching directory.
#
# Please note that this eclass handles only tarballs (.tar, .tar.gz,
# .tar.bz2 .tar.xz). For any other file format (or suffix) it will
# fall back to regular unpack. Support for additional formats may be
# added at some point so please keep your SRC_URIs clean.
#
# @EXAMPLE:
#
# @CODE
# EAPI=4
# AUTOTOOLS_AUTORECONF=1
# inherit autotools-utils vcs-snapshot-r1
#
# SRC_URI=http://github.com/example/${PN}/tarball/v${PV} - ${P}.tar.gz
# @CODE
#
# and however the tarball was originally named, all files will appear
# in ${WORKDIR}/${P}.
case ${EAPI:-0} in
0|1|2|3|4) ;;
*) die vcs-snapshot-r1.eclass API in EAPI ${EAPI} not yet established.
esac
EXPORT_FUNCTIONS src_unpack
vcs-snapshot-r1_src_unpack() {
local f
for f in ${A}
do
case ${f} in
*.tar|*.tar.gz|*.tar.bz2|*.tar.xz)
local destdir=${WORKDIR}/${f%.tar*}
# XXX: check whether the directory structure
inside is
# fine? i.e. if the tarball has actually a
parent dir.
mkdir ${destdir} || die
tar -C ${destdir} -x --strip-components 1 \
-f ${DISTDIR}/${f} || die
;;
*)
# fall back to the default method
unpack ${f}
;;
esac
done
}
signature.asc
Description: PGP signature
