Bug#535767: Allow for debian/-only repositories
On Sat, Jul 04, 2009 at 10:41:53PM +0200, Joachim Breitner wrote:
Hi zack,
Howdy (but note that devscripts, including debcheckout, is maintained
by way more people than me alone ;-))
One option, which I find appealing, is to only store the contents of the
debian/ directory in the VCS. This would be most useful if there is
some support for it in debcheckout. In human pseudo code, I could
imagine debcheckout following these steps:
* debcheckout, after getting the repository, detects this layout
(absense of a debian/ directory, presence of control and changelog
files in the top directory)
* If detected, it gets an upstream tarball. Possible locations are:
- current working directory, if named as expected (orig.tar.gz)
- the debian archive
- whatever is referenced in ./watch
* It extracts the tarball
* It moves the repository checkout to debian/ inside the checked-out
sources.
What do you think? Does it make sense? Would you accept a patch
implementing that?
First of all, I must confess that I don't see the advantage. Are you
doing all that only to gain the storage of the _content_ of debian/
instead of having a repository that contains debian/ itself? If this
is the case, I noticed that what you are proposing will defeat the
implicit API of many devscripts, which expect a debian/ dir inside
`pwd`. ... and in fact you are asking debcheckout to recreate that
layout: it looks like _very_ twisted to me.
If, on the contrary, what you are trying to achieve is to not store
upstream sources but only debian/ stuff, then for some $VCS you
already have support for that in package building tool. For instance,
svn-buildpackage supports the so called debian only layout which
does exactly that. Using such a tool, the work flow you request is
already available executing a couple of commands:
- debcheckout foo
- uscan --force-download
- # move downloaded stuff in ../tarballs/
And you're done.
2 comments:
1) this is because svn-buildpackage has knowledge of some debian
only layout. Does your $VCS support, with its package building
tool, anything like that? If not, there is probably where to add
the support you need
2) how to actually retrieve the orig tarball is a tricky
business. uscan is a possibility, the get-orig-source of
debian/rules is another. How to find an entry point which does the
right thing is a complex subject by his own, which probably
requires some adjustments in policy too.
Concluding, as I see things now, it doesn't look like a good idea to
me to implement what you asked, ... but I might have misunderstood it
:-)
Cheers.
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
z...@{upsilon.cc,pps.jussieu.fr,debian.org} -- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..| . |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...| ..: | Je dis tu à tous ceux que j'aime
signature.asc
Description: Digital signature
Bug#535767: Allow for debian/-only repositories
Hi,
Am Sonntag, den 05.07.2009, 10:57 +0200 schrieb Stefano Zacchiroli:
On Sat, Jul 04, 2009 at 10:41:53PM +0200, Joachim Breitner wrote:
Hi zack,
Howdy (but note that devscripts, including debcheckout, is maintained
by way more people than me alone ;-))
I checked the manpage for the mention of an author :-)
First of all, I must confess that I don't see the advantage. Are you
doing all that only to gain the storage of the _content_ of debian/
instead of having a repository that contains debian/ itself? If this
is the case, I noticed that what you are proposing will defeat the
implicit API of many devscripts, which expect a debian/ dir inside
`pwd`. ... and in fact you are asking debcheckout to recreate that
layout: it looks like _very_ twisted to me.
On the developer’s machine, I want the regular layout where debian/ is
in $PWD, just that not the whole $PWD is in the VCS (I’m thinking of
darcs), but only debian/. The main goal is to keep the repository small
and slick.
If, on the contrary, what you are trying to achieve is to not store
upstream sources but only debian/ stuff, then for some $VCS you
already have support for that in package building tool. For instance,
svn-buildpackage supports the so called debian only layout which
does exactly that. Using such a tool, the work flow you request is
already available executing a couple of commands:
- debcheckout foo
- uscan --force-download
- # move downloaded stuff in ../tarballs/
And you're done.
I knew I saw something like that once. svn-buildpackage even has code to
to fetch the tarball from the archive or, when not found, via watch.
if($opt_download_tarball) {
if(!defined($origfile)) {
print Trying to download tarball using apt\n;
my $olddir = getcwd();
mychdir $$c{origDir};
my @archive_versions = split(\n, `apt-cache policy $package`);
@archive_versions = grep /^(( \*\*\* | )[0-9])/, @archive_versions;
map {$_ =~ s/^ \*\*\* / /; $_ =~ s/^ ([^ ]+) .*$/$1/; $_}
@archive_versions;
foreach(@archive_versions) {
my $upstream_version = $_;
$upstream_version =~ s/^.*://;
$upstream_version =~ s/(.*)-([^-]+)/$1/;
if($upstream_version eq $upVersion) {
system(apt-get source --tar-only $package=$_);
if(($? 8) == 0 -f $orig) {
$origfile = long_path($orig);
last;
}
}
}
mychdir $olddir;
}
if(-f debian/watch !defined($origfile)) {
print Trying to download tarball using uscan\n;
system(uscan, --destdir, $$c{origDir}, --repack,
--download-version, $upVersion, --force-download);
if(-f $$c{origDir} . /$orig) {
$origfile = long_path($$c{origDir} . /$orig);
}
};
if($opt_download_tarball !defined($origfile)) {
die(Couldn't find a tarball\n);
}
}
Having it in debcheckout would make it possible to use it with other
VCSes as well, i.e. darcs. The code above can probably be used without
much changes.
2 comments:
1) this is because svn-buildpackage has knowledge of some debian
only layout. Does your $VCS support, with its package building
tool, anything like that? If not, there is probably where to add
the support you need
darcs-buildpackage does not have the support. And, since the support
only affects the _retrieving_ of the source, I don’t see much merit in
using darcs-buildpackage at all. Following the unix philosophy of small
tools, I’d use
* debcheckout to get the sources and set them up correctly
* regular dpkg-buildpackage -I to build
* regular darcs (inside ./debian/) to commit, tag and push changes
2) how to actually retrieve the orig tarball is a tricky
business. uscan is a possibility, the get-orig-source of
debian/rules is another. How to find an entry point which does the
right thing is a complex subject by his own, which probably
requires some adjustments in policy too.
Given that there is code in svn-buildpackage, I’d think it’s feasible.
I’d expect that only those maintainers will use such the feature where
it actually works :-)
Concluding, as I see things now, it doesn't look like a good idea to
me to implement what you asked, ... but I might have misunderstood it
:-)
Ok. I guess, at first, I can wrap the functionality in a shell script
for our team (if we even decide on such a repository layout). If it
works out, we can still consider adding it to debcheckout.
Thanks for your comments,
Joachim
--
Joachim nomeata Breitner
Debian Developer
nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata
signature.asc
Description: Dies ist ein digital signierter Nachrichtenteil
Bug#535767: Allow for debian/-only repositories
On Sun, Jul 05, 2009 at 11:35:58AM +0200, Joachim Breitner wrote:
I knew I saw something like that once. svn-buildpackage even has code to
to fetch the tarball from the archive or, when not found, via watch.
Thanks for the code, I did not know that.
Still, duplicating code is always bad. So, if we really want to go
this way, I'd prefer to add support to debcheckout to invoke some kind
of hooks, and have this as a possible hook. How about that?
darcs-buildpackage does not have the support. And, since the support
only affects the _retrieving_ of the source, I don’t see much merit in
using darcs-buildpackage at all. Following the unix philosophy of small
tools, I’d use
I disagree on this. With svn-buildpackage the debian only layout is
used at build time. You never have upstream code unpacked where you
are, but rather elsewhere. That elsewhere is unpacked upon build.
Given that there is code in svn-buildpackage, I’d think it’s
feasible. I’d expect that only those maintainers will use such the
feature where it actually works :-)
It's feasible, the point is whether the implementation is the right
way to go. Before going that way, I'd rather trigger a discussion on
-devel about that. For instance, what if get-orig-source invokes uscan
itself?
Cheers.
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
z...@{upsilon.cc,pps.jussieu.fr,debian.org} -- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..| . |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...| ..: | Je dis tu à tous ceux que j'aime
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#535767: Allow for debian/-only repositories
Hi Zack, Am Sonntag, den 05.07.2009, 15:03 +0200 schrieb Stefano Zacchiroli: On Sun, Jul 05, 2009 at 11:35:58AM +0200, Joachim Breitner wrote: I knew I saw something like that once. svn-buildpackage even has code to to fetch the tarball from the archive or, when not found, via watch. Thanks for the code, I did not know that. Still, duplicating code is always bad. So, if we really want to go this way, I'd prefer to add support to debcheckout to invoke some kind of hooks, and have this as a possible hook. How about that? You mean a hook inside the checked-out-repository? E.g. a file debcheckout-hook that, if present, will always be called by debcheckout? Seems to be risky security-wise. darcs-buildpackage does not have the support. And, since the support only affects the _retrieving_ of the source, I don’t see much merit in using darcs-buildpackage at all. Following the unix philosophy of small tools, I’d use I disagree on this. With svn-buildpackage the debian only layout is used at build time. You never have upstream code unpacked where you are, but rather elsewhere. That elsewhere is unpacked upon build. I only saw that function, I didn’t really follow through how it’s used in svn-buildpackage. A piece of code says more than thousands words, so here is a prototypial wrapper: http://people.debian.org/~nomeata/pkg-haskell-checkout Given that there is code in svn-buildpackage, I’d think it’s feasible. I’d expect that only those maintainers will use such the feature where it actually works :-) It's feasible, the point is whether the implementation is the right way to go. Before going that way, I'd rather trigger a discussion on -devel about that. For instance, what if get-orig-source invokes uscan itself? I haven’t worked with get-orig-source before. Is it a standard? I thought it’s only a thin people use personally. I think I’ll stop bugging you for a while and work with the shell script linked above. If it turns out to be a good way of using a repository, I’ll come back and we can pick up the discussion. Or we talk about this at DebConf :-) Greetings, Joachim -- Joachim nomeata Breitner Debian Developer nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata signature.asc Description: Dies ist ein digital signierter Nachrichtenteil
Bug#535767: Allow for debian/-only repositories
On Sun, Jul 05, 2009 at 05:52:27PM +0200, Joachim Breitner wrote:
Thanks for the code, I did not know that.
Still, duplicating code is always bad. So, if we really want to go
this way, I'd prefer to add support to debcheckout to invoke some kind
of hooks, and have this as a possible hook. How about that?
You mean a hook inside the checked-out-repository? E.g. a file
debcheckout-hook that, if present, will always be called by
debcheckout? Seems to be risky security-wise.
No, I meant a hook on your machine which is executed always and which
will try to checkout upstream sources if they aren't there. Actually,
also the solution of a hook in the checked out repository might do the
trick, but as you observer will need an explicitly enabling flag that
triggers that only on repositories which are known to be safe.
A piece of code says more than thousands words, so here is a prototypial
wrapper:
http://people.debian.org/~nomeata/pkg-haskell-checkout
Urm, I understand the deal, but I'd prefer it to support
get-orig-source, read on.
I haven’t worked with get-orig-source before. Is it a standard? I
thought it’s only a thin people use personally.
Yup, is in policy. It is the target to use if your upstream tarball
needs some kind of mangling (e.g., stripping non-DFSG bits or packing
together multiple upstream tarballs for multi-upstream source
packages). What I'd love from your script is to test if
get-orig-source is defined (possibly invoking make in some dry-run
way, not just parsing debian/rules cause it can include other files)
before trying uscan.
Or we talk about this at DebConf :-)
Deal :-)
Cheers.
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
z...@{upsilon.cc,pps.jussieu.fr,debian.org} -- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..| . |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...| ..: | Je dis tu à tous ceux que j'aime
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#535767: Allow for debian/-only repositories
Hi, Am Sonntag, den 05.07.2009, 18:53 +0200 schrieb Stefano Zacchiroli: I haven’t worked with get-orig-source before. Is it a standard? I thought it’s only a thin people use personally. Yup, is in policy. It is the target to use if your upstream tarball needs some kind of mangling (e.g., stripping non-DFSG bits or packing together multiple upstream tarballs for multi-upstream source packages). What I'd love from your script is to test if get-orig-source is defined (possibly invoking make in some dry-run way, not just parsing debian/rules cause it can include other files) before trying uscan. Of course, ./rules get-orig-source has the same security implications than a hook. (In a sense, it is a hook.) Is it really desirable to use an imperative feature (run this Makefile rule to get the tarball), when there is a declarative way of doing it (the watch file) as well? I’d say as long as someone does not have to modify the tarball, using debian/watch and uscan should be preferred to coding stuff in get-orig-source. Greetings, Joachim -- Joachim nomeata Breitner Debian Developer nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata signature.asc Description: Dies ist ein digital signierter Nachrichtenteil
Bug#535767: Allow for debian/-only repositories
On Sun, Jul 05, 2009 at 08:09:37PM +0200, Joachim Breitner wrote:
Is it really desirable to use an imperative feature (run this Makefile
rule to get the tarball), when there is a declarative way of doing it
(the watch file) as well?
Yes, but that discussion does not belong here. Check policy, use
cases, or related discussions. (And, BTW, a lot of uscan options are
totally non-declarative ...)
I’d say as long as someone does not have to modify the tarball, using
debian/watch and uscan should be preferred to coding stuff in
get-orig-source.
... but well, *modifying* tarball is exactly the main use case of
get-orig-source.
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
z...@{upsilon.cc,pps.jussieu.fr,debian.org} -- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..| . |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...| ..: | Je dis tu à tous ceux que j'aime
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#535767: Allow for debian/-only repositories
Package: devscripts Version: 2.10.52 Severity: wishlist File: /usr/bin/debcheckout -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi zack, while setting up the pkg-haskell group, we had some discussions on the VCS layout. One option, which I find appealing, is to only store the contents of the debian/ directory in the VCS. This would be most useful if there is some support for it in debcheckout. In human pseudo code, I could imagine debcheckout following these steps: * debcheckout, after getting the repository, detects this layout (absense of a debian/ directory, presence of control and changelog files in the top directory) * If detected, it gets an upstream tarball. Possible locations are: - current working directory, if named as expected (orig.tar.gz) - the debian archive - whatever is referenced in ./watch * It extracts the tarball * It moves the repository checkout to debian/ inside the checked-out sources. What do you think? Does it make sense? Would you accept a patch implementing that? Thanks, Joachim - -- Package-specific info: - --- /etc/devscripts.conf --- - --- ~/.devscripts --- DEBCHANGE_RELEASE_HEURISTIC=changelog DEBRELEASE_UPLOADER=dput - -- System Information: Debian Release: squeeze/sid APT prefers unstable APT policy: (500, 'unstable'), (1, 'experimental') Architecture: amd64 (x86_64) Kernel: Linux 2.6.29-2-amd64 (SMP w/2 CPU cores) Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/bash Versions of packages devscripts depends on: ii dpkg-dev 1.15.3 Debian package development tools ii libc6 2.9-18 GNU C Library: Shared libraries ii perl 5.10.0-23 Larry Wall's Practical Extraction Versions of packages devscripts recommends: ii at 3.1.10.2 Delayed job execution and batch pr ii bsd-mailx [mailx] 8.1.2-0.20081101cvs-2 A simple mail user agent ii bzr1.16.1-1 easy to use distributed version co ii curl 7.19.5-1 Get a file from an HTTP, HTTPS or ii cvs1:1.12.13-12 Concurrent Versions System ii dctrl-tools2.13.1Command-line tools to process Debi ii debian-keyring 2009.05.28GnuPG (and obsolete PGP) keys of D pn debian-maintainers none(no description available) ii dput 0.9.4 Debian package upload tool ii equivs 2.0.7-0.1 Circumvent Debian package dependen ii fakeroot 1.12.4Gives a fake root environment ii galeon [www-browse 2.0.7-1 GNOME web browser for advanced use ii git-core 1:1.6.3.3-2 fast, scalable, distributed revisi ii gnupg 1.4.9-4 GNU privacy guard - a free PGP rep ii iceweasel [www-bro 3.0.11-1 lightweight web browser based on M ii libauthen-sasl-per 2.12-1Authen::SASL - SASL Authentication ii libcrypt-ssleay-pe 0.57-1+b1 Support for https protocol in LWP ii libparse-debcontro 2.005-2 Easy OO parsing of Debian control- ii libsoap-lite-perl 0.710.08-2Client and server side SOAP implem ii libterm-size-perl 0.2-4+b1 Perl extension for retrieving term ii libtimedate-perl 1.1600-9 Time and date functions for Perl ii liburi-perl1.37+dfsg-1 Manipulates and accesses URI strin ii libwww-perl5.828-1 WWW client/server library for Perl ii libyaml-syck-perl 1.07-1fast, lightweight YAML loader and ii lintian2.2.12Debian package checker ii lsb-release3.2-22Linux Standard Base version report ii lynx-cur [www-brow 2.8.7pre6-1 Text-mode WWW Browser with NLS sup ii man-db 2.5.5-2 on-line manual pager ii mercurial 1.2.1-3 scalable distributed version contr ii openssh-client [ss 1:5.1p1-6 secure shell client, an rlogin/rsh ii patch 2.5.9-5 Apply a diff file to an original ii patchutils 0.3.1-2 Utilities to work with patches ii strace 4.5.18-1 A system call tracer ii subversion 1.5.6dfsg-1 Advanced version control system ii unzip 6.0-1 De-archiver for .zip files ii w3m [www-browser] 0.5.2-2.1 WWW browsable pager with excellent ii wdiff 0.5-18Compares two files word by word ii wget 1.11.4-2 retrieves files from the web Versions of packages devscripts suggests: ii build-essential 11.4 Informational list of build-essent pn cvs-buildpackage none (no description available) pn devscripts-el none
