subject:"RFC\: DEP\-14\: Recommended layout for Git packaging repositories"

Hi Raphael,
On Thu, Nov 13, 2014 at 09:46:13AM +0100, Raphael Hertzog wrote:
[..snip..]
 Problem 1: the derivatives
 --
 
 So I am a Kali Linux contributor. We use git repos to maintain all our
 packages and we use git-buildpackage. Most of the Kali contributors
 are not long-term Debian contributors, I write documentation so that
 they can contribute to Kali (while basing our work on Debian).
 
 To make this manageable I opted to always use a workflow based on
 git-import-orig. Even when Debian has its own git repo, we start
 from the released source packages because that's the only level of
 uniformity that we can rely on. And it's a pity because if we could
 build on Debian's git repo, it also means that our work would be easier
 to merge for the Debian maintainer. They could add the Kali repo as a
 remote (without fearing any conflict in terms of tags, branch names)
 and just merge or cherry-pick as appropriate.

I don't think Kali is on it's own here. The above workflow was the
first major use case I saw gbp being used at a couple of years ago. We
had contributors with different skill sets that needed to modify
various packages building a (private) downstream. Having a standardized
work flow for modifying and building packages helped a lot.

Being based on stable, changes we contributed back stayed in our tree
for a long time until the next Debian stable release. Other changes
needed to be kept forever. We didn't want to get into every VCS Debian
used (and by far not all packages were in a VCS at that time) and use
the powers of git to track patches, rebase, etc.

Therefore I see a lot of value having a common namespace / branch
layout for Debian as well as downstream branches and tags. This
allows us to easily track the modifications.

Cheers,
 -- Guido


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141118080234.gb7...@bogon.m.sigxcpu.org

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Fri, Nov 14, 2014 at 11:35:45AM +0100, Marco d'Itri wrote:
 On Nov 14, Raphael Hertzog hert...@debian.org wrote:
 
  This is just a proof that storing the patches as real commits is useful.
  And that's the point of the patch-queue tag. Instead of having the patches
  only as real commits in the local repository, they get pushed to the
  public repository too under that tag (instead of appearing only within
  debian/patches/).

 In principle I agree, but in practice this must support perfect 
 roundtrips preserving all comments and metadata in the patches and gbp 
 is still a long way from supporting this.

gbp-pq currently is not meant for a perfect round trip but rather for:
import once and use what git-format-patch puts out from there
on. I agree that this doesn't help much with e.g. NMUs so we should
probably improve on that part.
Cheers,
 -- Guido


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141118080655.gc7...@bogon.m.sigxcpu.org

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Sat, Nov 15, 2014 at 06:35:07PM +, Simon McVittie wrote:
[..snip..]
 with upstream changes - and try packaging them with each of gbp-pq,

Note that nothing withing gbp forces you to use gbp-pq. You can use
e.g. use

--no-unapply-patches
--auto-commit
--single-debian-patch

in your debian/source/local-options and always work with a patched
tree.

 git-dpm and gitpkg. To have the complete set, I think I need one project
 where the upstream tarball is a simple git-archive of the upstream git
 repository, one where the upstream tarball has extra detritus (e.g.
 Autotools) and/or missing files (upstream's .gitignore not being in the
 tarball is also common in Autotools), and one where the Debian
 maintainer needs to filter out a non-free file. Anything else you can
 think of?
 
 S
 
 
 -- 
 To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
 with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
 Archive: https://lists.debian.org/54679cdb.4090...@debian.org
 


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141118072026.ga7...@bogon.m.sigxcpu.org

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-18 Thread Simon McVittie

On 18/11/14 07:20, Guido Günther wrote:
 Note that nothing withing gbp forces you to use gbp-pq.

I know; but the gbp-pq-like repository structure, as produced by git
import-dsc with no special options, is a popular one (pkg-perl and their
 3000 packages, as well as smaller teams like -telepathy, -utopia,
-systemd) so any comparison of repository structures should probably
include it.

I've been trying to be consistent about calling it gbp-pq and not gbp,
because I am aware that git-buildpackage does not specifically require
this structure.

S


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/546b0b7b.4000...@debian.org

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-18 Thread Ron

On Mon, Nov 17, 2014 at 11:39:10PM +, Simon McVittie wrote:
 On 17/11/14 16:24, Ian Jackson wrote:
  I don't think this problem, of a mass of different branch structures,
  is going to go away any time soon.  Simply because people don't seem
  able to agree.

I agree it's unlikely to go away, but I do also think that if the reason
this is the reality we have to work with were actually as simple as a
lack of consensus on how to name them, then it would have already gone
away a long time ago.

git didn't make branching such a powerful and central part of its
paradigm just because chaos is fun.  It's because it accurately
describes the reality of the topology that forms naturally as soon
as you decentralise version control, and allow truly distributed
development, where every clone is a first class entity in its own
view of the set.

  My answer is to create a parallel universe in which the branch
  structure is known.
 
 As far as I can see, in effect, Kali's answer can be described by the
 same sentence, except that Kali's parallel universe doesn't have the
 same known branch structure you do, presumably because Raphael's
 preferences are different; Kali's known branch structure is what
 git-import-dsc does whereas yours is based on what dpkg-source -x
 produces, resulting in different trees in the repository for 3.0 (quilt)
 packages.

Right, and there are lots of other tools that do exactly this too.
git-debimport from gitpkg does the same thing from the history of
existing packages, and Marco wrote and published one that did this
job for the packages he imported to git from other forms too, just
to name two.  The basic structure of all those is almost certainly
congruent, but in addition to the problem that Simon points out,
they're still limited to only dealing with a simple linear history.

Which isn't at all the reality of really working in git for anything
but the most simple and parochial of packages.

As soon as you have backports, or stable updates, or tweaks for some
other distro flavour, or multiple developers working on individual
new features, or WIP branches that aren't ready for release yet but
might contain some useful thing already ...  chaos has laid its egg
in your nest.  And you can either incubate it lovingly, or get it
on your face :)  What you can't do is avoid it.

I think we're still quite some way from having everyone on the same
page about how to embrace that and empower downstream developers
with it.  There's certainly far too many permutations to robotically
name them all (and their relationships to each other) in any kind of
rigorously meaningful way.  But I think that this is the problem
which, if we solve it, will actually get us on the path for some
sort of sustainable collaboration, that doesn't require incompatible
forks as some sort of creole to roughly paste things together.


At the risk of opening another somewhat partisan question here, the
common elephant that does appear to be taking shape in the room here,
for all the currently existing tools, _is_ actually the problems
created by format 3.  Which makes me wonder if addressing those as
a first priority, isn't actually the best thing that we could really
be looking at first here, rather than looking at what complicated
things we might do to sort of work around them in some cases ...


  The maintainers of each package choose whether to use the dgit
  `universe', in which case certain basic assumptions can be relied
  on, or run their own `universe' in which case they can structure
  it however they like.

Is there somewhere I can clone a dgit'ified package repo from,
using the normal git tools, to have a look at it - or do I need
to drink the whole jug of kool-aid to be able to do that?

I am curious about how its 'universe' maps to packages that
aren't strictly linear because there have been backports or
security uploads, or other stable updates, or minor changes
to suit some other distro release etc.


  Cheers,
  Ron



-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141118113715.gw10...@hex.shelbyville.oz

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Fri, Nov 14, 2014 at 03:44:35PM +, Ian Jackson wrote:
 Bernhard R. Link writes (Re: RFC: DEP-14: Recommended layout for Git 
 packaging repositories):
  Raphael Hertzog hert...@debian.org [14 22:26]:
 ...
   When a Git tag needs to refer to a specific version of a Debian package,
   the Debian version needs to be mangled to cope with Git's restrictions.
   The colon (`:`) needs to be replaced with a percent (`%`), and the tilde
   (`~`) needs to be replaced with an underscore (`_`).
  
  Is there a previous case of encoding colons with percent signs?
  If it is inventing a third way instead of using on of the existing ones,
  is sounds like a bad idea.
 
 Raphael, is there precedent for this ?
 
 Current practice seem so be to replace both : and ~ with _.  Unless
 we

This didn't work well so gbp switched to what Raphael documented years
ago (: - %, ~ - _).
Cheers,
 -- Guido


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141118155215.gd7...@bogon.m.sigxcpu.org

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-18 Thread Ian Jackson

Guido Günther writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
repositories):
 On Fri, Nov 14, 2014 at 03:44:35PM +, Ian Jackson wrote:
  Current practice seem so be to replace both : and ~ with _.  Unless
  we
 
 This didn't work well so gbp switched to what Raphael documented years
 ago (: - %, ~ - _).

In fact it appears I noticed this earlier and already fixed dgit to
work this way (but had forgotten...)

I have updated
   https://wiki.debian.org/Punctuation

Are there other gitish tools which are still using _ for both ?

Thanks,
Ian.


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/21611.29374.423255.264...@chiark.greenend.org.uk

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-18 Thread James McCoy

On Tue, Nov 18, 2014 at 04:24:30PM +, Ian Jackson wrote:
 Guido Günther writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
 repositories):
  On Fri, Nov 14, 2014 at 03:44:35PM +, Ian Jackson wrote:
   Current practice seem so be to replace both : and ~ with _.  Unless
   we
  
  This didn't work well so gbp switched to what Raphael documented years
  ago (: - %, ~ - _).
 
 In fact it appears I noticed this earlier and already fixed dgit to
 work this way (but had forgotten...)
 
 I have updated
https://wiki.debian.org/Punctuation
 
 Are there other gitish tools which are still using _ for both ?

Hmm, seems debcommit is converting ~ to . and stripping the epoch
entirely.  I guess at least the former should probably be changed to
follow the common pattern.

Cheers,
-- 
James
GPG Key: 4096R/331BA3DB 2011-12-05 James McCoy james...@debian.org


signature.asc
Description: Digital signature

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-18 Thread Ron


Ian wrote:
 Guido Günther writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
 repositories):
  On Fri, Nov 14, 2014 at 03:44:35PM +, Ian Jackson wrote:
   Current practice seem so be to replace both : and ~ with _.  Unless
   we
  
  This didn't work well so gbp switched to what Raphael documented years
  ago (: - %, ~ - _).

Can you recall and share some details on what didn't work so well
with gbp that this fixed?


 Are there other gitish tools which are still using _ for both ?

The version mangling code in gitpkg converts all constructs that are
illegal for git to _ (and collapses consecutive instances of them to
a single _).

I believe Bernhard explained earlier that git-dpm allows the replacement
character to be configurable (but also offers just a single character
for all replacements).

As I explained in the earlier discussion with Henrique, there are more
things than just : and ~ which are perfectly legal in debian version
strings, but are illegal constructs in git refnames.

To make this even more amusing, somewhere between git 1.7 (on squeeze)
and git 2.1, the rules for what were legal in git refs were tightened
(at least once) to exclude even more things.  I don't know offhand if
this is likely to happen again at any stage.

I'd also be a bit wary of changing existing tools to behave differently
(between different Debian releases that are still supported even) without
a very careful look at what might be affected if previous transformations
were no longer reproduced in the same way, or behaved differently depending
on exactly which chroot you ran them in.


I'd be more interested in a clear statement of what the problems are
with various options, and what we really want to achieve by specifying
anything about this, at this stage.  That would at least give us some
objective measure of what is necessary, and what is actually possible
(and at what cost) to consider.

  Cheers,
  Ron



-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141119062136.ga29...@hex.shelbyville.oz

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-18 Thread gregor herrmann

On Tue, 18 Nov 2014 21:17:15 -0500, James McCoy wrote:

  Guido Günther writes (Re: RFC: DEP-14: Recommended layout for Git 
  packaging repositories):
   This didn't work well so gbp switched to what Raphael documented years
   ago (: - %, ~ - _).
 Hmm, seems debcommit is converting ~ to . and stripping the epoch
 entirely.  I guess at least the former should probably be changed to
 follow the common pattern.

Yes, please.
In the pkg-perl repos we have a mixture of debcommit-created tags and
gbp-created tags, and the different conversion scheme is at least
aesthetically unpleasing.

Cheers,
gregor

-- 
 .''`.  Homepage: http://info.comodo.priv.at/ - OpenPGP key 0xBB3A68018649AA06
 : :' : Debian GNU/Linux user, admin, and developer  -  http://www.debian.org/
 `. `'  Member of VIBE!AT  SPI, fellow of the Free Software Foundation Europe
   `-   BOFH excuse #62:  need to wrap system in aluminum foil to fix problem 


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20141119074535.gg17...@colleen.colgarra.priv.at

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-17 Thread Henrique de Moraes Holschuh

On Mon, 17 Nov 2014, Ron wrote:
  Anything that doesn't store the full debian version (be it a git tag or a
  filename) will have a colision risk.  Consider a package that has versions
  1:2.3.5-1 and 5:2.3.5-1.
 
 Right, that was one of the first cases I mentioned.  When you upload
 5:2.3.5-1, dak will reject it exactly because they will collide. :)

Yes, you're right.  Our archive is braindead re. epochs since basically day
one, and blocking colisions in dak was deemeed the lesser evil when compared
to the fallout of fixing the archive filenames.

 fwiw, I don't consider that a compelling reason alone to not include
 the epoch in tags - but you appear to have just confirmed that not
 many people know this, even when it's explicitly mentioned in a
 thread they just replied to!  So I think *that* is probably worthy
 of a mention if we do recommend including the epoch in tags :)

Ah, sorry about that...

   If the answer is instead something like I want automated tools to be
   able to assume they can perfectly reverse the mangling and assume some
   semantic meaning to the text in the tag, then we're into Broken By
   Design territory and we really need to look more closely at *exactly*
  
  Well, we can have a perfect, reversible transformation between the debian
  version namespace and the debian git-tag version namespace, where debian
  means a vendor that uses dpkg and the deb format.
 
 No.  I don't believe you can.
 
 Mangle this version in a way that's legal to git: 0:1.2:3..4-1~~2..3.lock

True. Getting git to accept .. and a .lock suffix in a tag.. urk.

Point taken.  Yeah, that'd call for %-encoding or something equivalent, and
those always require also the definition of a canonical form.

 Yes, it's a corner case you've probably never seen, but the troublesome
 parts of that are all legal debian versions.

Yes, just like 1:1.2.3-4 versus 5:1.2.3-4, that dak rejects because the
Debian archive doesn't preserve epochs in its storage layer (filenames).

One interesting detail is that at least apt _DOES_ restore the epoch
information in the deb filenames (it uses URL-like %-encoding, though, so
: becomes %3a).

I'd say we should get the epoch-in-tags detail documented in DEP-14.

And the archive restriction re. epochs really should have been documented in
debian policy.  It is not even in the developer's reference... but this is
offtopic for this thread.

   Henrique, would you care to elaborate on your definition of safer?
  
  Sure.  I consider safer a design that loses no information when translating
  between the debian version namespace and debian version git-tag namespace,
  because you won't have permanent colisions among debian versions that differ
  only on the epoch.
 
 Ok, I can agree with that definition, but as I showed above, this
 plan doesn't satisfy it.  You'd need to fully URL encode them or
 similar to have that, and I don't think we should do that.

I agree that a full encoding that requires canonical forms is way too
painful, and that we should not go there.

 You can still semi-meaningfully encode the epoch in a way that
 is a clue to humans, but you *can't* guarantee the version is
 reversible from the necessary mangling.

If we mangle by removing whatever git objects to, yes, it won't be
reversible, at which point we might as well not bother with epochs.

I'd still allow epochs as optional, but that would cause a headache when you
have something like 1:1:4-1 for a debian version... best to recommend that
they be removed when tagging, instead.

So I retract any objections I had against epoch-less tagging.  But I feel
DEP-14 should document this stuff.

 Are there actually already existing tools that rely on reversing this?

I sure hope there aren't any.

 If there aren't we certainly shouldn't be encouraging creating new ones
 with this flaw in the future.

I agree.

-- 
  One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie. -- The Silicon Valley Tarot
  Henrique Holschuh


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141117114929.ga14...@khazad-dum.debian.net

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-17 Thread Ian Jackson

Simon McVittie writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
repositories):
 I agree that the expected contents of the branches are far more
 important than their names. Unfortunately, while acting as the Debian
 expert for Debian derivatives at $day_job, I keep finding that the
 answer to OK, I've cloned a package's git repository, I know what code
 change I want, now do I change the upstream source or drop a patch into
 debian/patches or what? is ... I can't actually answer that until you
 tell me which source package you're working on.

This is indeed a very big problem.  It is why I am working on dgit.

I don't think this problem, of a mass of different branch structures,
is going to go away any time soon.  Simply because people don't seem
able to agree.

My answer is to create a parallel universe in which the branch
structure is known.  The maintainers of each package choose whether to
use the dgit `universe', in which case certain basic assumptions can
be relied on, or run their own `universe' in which case they can
structure it however they like.

Luckily git makes it fairly easy to transport changes from one
universe into another.

Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/21610.8498.518053.339...@chiark.greenend.org.uk

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-17 Thread Ian Jackson

Simon McVittie writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
repositories):
 gbp-pq and git-dpm are the other way round: the tree can be built with
 dpkg-buildpackage, but the cost is that you have to commit in a way that
 isn't the normal git thing (either using a specific tool, or for the
 gbp-pq layout, dropping in pre-prepared patches and hoping they don't
 have conflicts, in the same way you might for svn-buildpackage).

dgit is capable of tacking a git tree that came from dpkg-source, and
which you have committed to in the normal way (eg as part of an NMU),
and turning it into one that dpkg-source can represent.

The latest version I have (unreleased, sadly, as there are a few more
things that needs finishing off) is capable of converting the git
commits individually into quilt patches, provided the extra history
you have added is `linear enough'.

Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/21610.8733.41862.987...@chiark.greenend.org.uk

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-17 Thread Simon McVittie

On 17/11/14 16:24, Ian Jackson wrote:
 I don't think this problem, of a mass of different branch structures,
 is going to go away any time soon.  Simply because people don't seem
 able to agree.
 
 My answer is to create a parallel universe in which the branch
 structure is known.

As far as I can see, in effect, Kali's answer can be described by the
same sentence, except that Kali's parallel universe doesn't have the
same known branch structure you do, presumably because Raphael's
preferences are different; Kali's known branch structure is what
git-import-dsc does whereas yours is based on what dpkg-source -x
produces, resulting in different trees in the repository for 3.0 (quilt)
packages.

As far as I can see from what you've said elsewhere, for source format
3.0 (quilt), you're aiming for the patches applied and also serialized
in debian/patches/ state (matching git-dpm I think?), whereas Kali ends
up with the patches serialized in debian/patches/ but not applied
state (matching gbp-pq).

Is that a fair characterization?

Is your current work-in-progress on teaching dgit better 3.0 (quilt)
handling available anywhere? I'd be interested in trying out the new way
while comparing the same package in different repository layouts.

S


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/546a871e.80...@debian.org

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-17 Thread Ian Jackson

Simon McVittie writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
repositories):
 As far as I can see from what you've said elsewhere, for source format
 3.0 (quilt), you're aiming for the patches applied and also serialized
 in debian/patches/ state (matching git-dpm I think?), whereas Kali ends
 up with the patches serialized in debian/patches/ but not applied
 state (matching gbp-pq).

That is a correct characterisation of dgit's view of history.

But dgit will also cope with some patches applied and also serialised
but some further commits made to git but not serialised in
debian/patches.

 Is your current work-in-progress on teaching dgit better 3.0 (quilt)
 handling available anywhere? I'd be interested in trying out the new way
 while comparing the same package in different repository layouts.

 
http://www.chiark.greenend.org.uk/ucgi/~ianmdlvl/git?p=dgit.git;a=shortlog;h=refs/heads/wip.rebasing
aka
 git://git.chiark.greenend.org.uk/~ianmdlvl/dgit.git#wip.rebasing

But DO NOT USE THAT VERSION TO UPLOAD.  It generates git trees that
existing dgits cannot cope with.

Thanks,
Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/21610.38337.477993.49...@chiark.greenend.org.uk

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Sat, Nov 15, 2014 at 06:15:33PM +, Simon McVittie wrote:
 On 12/11/14 22:07, Ron wrote:
  I am also interested to hear more
  about whatever the confusion was you had with this was when you
  started working with Tollef's systemd repo that you mentioned
  in the previous thread.
 
 Having played with gitpkg some more, I'm reminded that the answer to
 this is that unlike (AIUI) both gbp-pq and git-dpm, it did not meet my
 assumption that the contents of the git tree were in a suitable form to
 run dpkg-buildpackage and have a 3.0 (quilt) Debian package fall out. I
 realise that's partly a property of 3.0 (quilt).

Ah, yes.  That probably is something we could document a bit better.
I guess it got overlooked, partly because there was no format 3 when
gitpkg was first written (and this does work just fine for format 1,
or the single patch style of format 3), and partly because by the
time this was an issue the common workflow was probably not to build
packages in the (potentially dirty) working tree anyway, so it took
someone new coming along to notice it.  This is actually the first
time anyone has mentioned tripping over this to me.

You should be guaranteed to get a functionally correct package if you
do this still, but not necessarily a source package with the /patches
split out individually.  However ...


 For gitpkg, you can commit in the normal git way, but the cost is that
 you have to build in a way that isn't the normal dpkg thing (exporting
 with gitpkg and building the result).

... if you want to build in the local system like this (as opposed
to throwing them off to a buildd chroot), you can enable the dpkg-bp
hook, at which point something like `gitpkg master` will do the
same things as dpkg-bp in the local tree with the added advantage
of also respecting your .gitattributes et al. and ignoring any
dirty state you might have in the working tree.

Which on the one hand, if you know this, is just as easy, if not
easier to do, and gives you much stronger guarantees about what
you're actually building -- but on the other, if you don't, is
something you need to become aware of somehow.

The latter of which is probably somewhat unavoidable for just about
any tool if you're actually planning to push your changes to the
repo and not stuff up later users of the tool - but I agree this
is another thing it's worth thinking more about to make things
easier for more casual users.


Do you have any suggestions for something that might have made this
more immediately obvious to you?  I can think of a few things we
could do, but I really do believe in designing things around actual
user experiences rather than trying to guess blindly about problems
nobody has actually ever had.

With the new tools for exporting a patch series, it would be possible
to export one into the (henceforth) 'dirty' working tree and build
directly in it with dpkg-bp as you tried.  And for people who really
wanted to, it would even be possible to commit that to a branch where
a 'naive' checkout would work as you expected here -- but avoiding
that kind of cruft in the repos was sort of exactly the reason for
writing those tools in the first place :)  It's generated source,
the same as autotools, it doesn't need to be in the VCS unless you
really have some personal reason to want it duplicated there.

It's not clear to me that any of the above is a better general
solution than just getting people on the right track for how
things work best to begin with - but I mention them because all
of those things are possible if you know you can do them.

I can think of some other far more clever magic that we could do
to simply make this work for a naive user that just does git clone
 dpkg-bp -- but I don't think I'll mention it, because this is
the internet, and Poe's Law means someone might go Wow!  That's a
Great Idea, I'm going to do that too and then it would be My Fault :)


I suspect the Best Answer is documenting this somehow and somewhere,
but it's not clear to me yet where the best place for that would have
been to have avoided what happened to you.

Maybe a README.DebianFromGit or something, that's export-ignored
so it doesn't clutter the package but is seen in a cloned repo?
I can see a few ways that could be less than ideal too though.


 gbp-pq and git-dpm are the other way round: the tree can be built with
 dpkg-buildpackage, but the cost is that you have to commit in a way that
 isn't the normal git thing (either using a specific tool, or for the
 gbp-pq layout, dropping in pre-prepared patches and hoping they don't
 have conflicts, in the same way you might for svn-buildpackage).
 
 I think I was also thrown by the fact that gitpkg does not encapsulate
 its configuration in what you commit: if two developers build the same
 tree, the debdiff might well be rather large, because one developer's
 .git/config results in separate git-debcherry patches and the other's
 .git/config results in a single large patch.

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-16 Thread Raphael Hertzog

On Fri, 14 Nov 2014, Thorsten Glaser wrote:
 On Fri, 14 Nov 2014, Ian Jackson wrote:
 
  expect to find version numbers matching ^\d+\~, then anything matching
 
 These are common enough for me to have not only seen,
 but also used (in native packages, of course) them.

Not only in native packages, when upsream doesn't make official releases
(and just makes a git repo available), I invent upstream versions with
0~MMDD.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141116100223.gf25...@home.ouaza.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Sat, Nov 15, 2014 at 06:35:07PM +, Simon McVittie wrote:
 On 13/11/14 14:04, Ron wrote:
  I really do think that the names of the branches are actually going to
  be the least of your worries here, unfortunately.  Even with a naming
  scheme that's widely adopted, things just aren't going to be that sort
  of uniform outside of (a fairly large number of) fairly small subsets.
 
 I agree that the expected contents of the branches are far more
 important than their names. Unfortunately, while acting as the Debian
 expert for Debian derivatives at $day_job, I keep finding that the
 answer to OK, I've cloned a package's git repository, I know what code
 change I want, now do I change the upstream source or drop a patch into
 debian/patches or what? is ... I can't actually answer that until you
 tell me which source package you're working on.

I'm not sure that problem is actually avoidable if a generic set of
instructions is all the person asking that has to work with.  At least
not outside something like pkg-perl where upstream itself is already
very well structured through something like CPAN, and the group that
is maintaining it inside Debian has a very coherent plan (and/or set
of tools).

I mean, if you had to ask *me* that question about one of *my* repos,
I'd *still* have to ask you which source package you were working on :)
Partly because every upstream is different in some small way or another,
making the best way to work with them have some slight variations, and
partly because I think best practice here is still an evolving thing
and so I'm still (after all these years) experimenting with slightly
different things that may (or may not :) work better for a specific
type of problem.

None of them are drastically different from the others, and a bunch of
common best practices have emerged.  But if you have to ask because
you couldn't figure it out for yourself, the sort of answer that you're
probably going to need is going to have some domain specific details
to it.

I do think you might have seeded a useful clue to how we might better
approach this in the other thread though (and sorry for the long answer
there, but I do think background information and details are an
important part of getting everyone on the same page for understanding
the real problems and what sort of solutions might really work here).


 At the moment, I suspect Kali's approach - arbitrarily choosing one of
 the popular approaches, only cloning packaging repositories from Debian
 that happen to match that approach, and restarting a new packaging
 repository for those that do not - is likely to be the only viable
 solution to that. There's always going to be a certain amount of
 re-importing in any case, because some packages in Debian are maintained
 in a non-git VCS or in no VCS at all; but it's easier to inspect history
 if it's possible to clone the existing packaging repository for most
 Debian packages of interest.

I think there are two fairly distinct problems there.  At least if I
reasonably understand what Raphael has described of it so far.

In the case he most recently described we have a relative novice
who just needs to apply a patch from somewhere else, or import a
new upstream tarball - it's not really clear that doing that work
manually from a VCS is the best, or even a good, solution anyway.

In such a case, it's probably much easier to just say Download
the existing package, modify it as required, and upload it.
And then if the project as a whole really wants that in a VCS,
it can just import that new package to its VCS history automatically
from the uploaded one.

Teaching them how to use dpkg is much simpler and less fraught with
unwanted complication than trying to force them into setting up gbp
and getting its use correct too, just to do something that doesn't
really need it or benefit from it at all.


In the case he originally described we want people to be able to
apply changes that can easily be pushed back to Debian, the only
viable answer really is to clone the Debian maintainer's repo,
figure out its structure, and commit your patch in a way that
this can really happen (no different to how you or the Debian
maintainer would push a patch to the upstream maintainer if they
are using git).

While I don't think naming conventions for branches or tags can
really help with that at all, I do believe there may be ways that
we can provide some extra meta data which could help with this.
But before I can really say what extra data might be helpful, or
even decide if I really believe it wouldn't just also be a major
waste^W inefficient use of effort, I'd really like to see some
*actual* questions from the actual people having actual trouble
with that.

The feedback you gave about where you came unstuck with gitpkg
the first time you encountered it was great.  There's some
actually actionable things we can look at good ways to improve
which might avoid that for others in the future.  If we're just

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-16 Thread Henrique de Moraes Holschuh

On Sun, 16 Nov 2014, Ron wrote:
 On Sat, Nov 15, 2014 at 03:49:56PM -0200, Henrique de Moraes Holschuh wrote:
  On Sat, 15 Nov 2014, Raphael Hertzog wrote:
   On Fri, 14 Nov 2014, Ian Jackson wrote:
 What exactly is your use case you feel this is essential for?

I think this discussion is in danger of going round in circles.
I'm going to leave it here and let Raphael get on with it.
   
   I understand Ron's logic and there's certainly value in questioning the
   need for something. But this is always a question of cost/value.
   
   Even if we don't have an immediate need for this property, the problem
   is that we can't always envision all the ways people will want to use
   the repositories (isn't that what you were trying to tell me about
   how upstream can use git?) and I'm pretty sure that dropping the epoch
   will be annoying to someone at some point. And the cost of not dropping
   the epoch is not very high.

...

 If the answer to that is simply it's relatively harmless, has minimal
 cost, some things already do it, and it may be a useful visual cue to
 human users, and that's all, then that's a perfectly good reason.

Agreed.  But there is a strong technical reason, too, on top of some
technical advantages.

So let me state the strong technical reason (which should at least be
documented in DEP-14, because this _is_ something people often gloss over):

Anything that doesn't store the full debian version (be it a git tag or a
filename) will have a colision risk.  Consider a package that has versions
1:2.3.5-1 and 5:2.3.5-1.

 If the answer is instead something like I want automated tools to be
 able to assume they can perfectly reverse the mangling and assume some
 semantic meaning to the text in the tag, then we're into Broken By
 Design territory and we really need to look more closely at *exactly*

Well, we can have a perfect, reversible transformation between the debian
version namespace and the debian git-tag version namespace, where debian
means a vendor that uses dpkg and the deb format.

In that case, a debian version tag really could have one valid semantic
meaning: one can derive the debian version from the tag, and also the
opposite.

What one cannot do is try to use either the debian version tag or the debian
version to infer anything about the upstream version/versio tags.  We agree
on this.

This assumes that DEP-14 will mandate exactly how debian version tags are
formed, and that tools that require DEP-14 semanthics to work correctly will
be upfront about that.

 what it is that people want to do and/or assume, and we probably need
 to look at less fragile solutions that really do satisfy that need.

Trying to handle colisions due to missing epoch information in the git tag
is far more fragile, IMO.

Any tool that needs the full debian version information would either need it
in the tag in a known format, or it would have to parse debian/changelog,
which requires fetching the correct blob from git, and parsing it using
dpkg-parsechangelog.  This is *not* fun to implement correctly, especially
when you don't want to mess with the work tree.

  I fully agree.  Please, lets just keep the epoch which is the safer design
  choice anyway, and get done with it.
 
 Henrique, would you care to elaborate on your definition of safer?

Sure.  I consider safer a design that loses no information when translating
between the debian version namespace and debian version git-tag namespace,
because you won't have permanent colisions among debian versions that differ
only on the epoch.

 They still aren't guaranteed to be so even with this convention,
 because there's no hard guarantee that everyone or everything will
 use them.  And even if they did, I still don't think you've covered
 every combination of things that are illegal in a git refname which
 might need to be mangled here.

I'd expect tools that make use of DEP-14 strongly recommended/mandatory
definitions to require DEP-14-compliant repositories.  I don't think there
is any other sane way to deal with this, other than don't write such a
tool.

This doesn't mean the tool should not be configurable to try its best in
non-DEP-14 mode, nor does it mean the tool shouldn't try to protect the user
and have sanity checks in place.

If we don't mandate anything in DEP-14, this point is moot, of course.  But
IMO we should at least document the debian version tag colision risk when
epochs are not present in the git tags.

 The question of should we recommend people include epochs in their tag
 names is entirely separate to the question of how should tools
 determine the version of a package at a particular ref in the repo.

Sort of.  If we recommend an exact format (doesn't have to be a single one),
tools can depend on it as long as they list that clearly as a prerequisite.

-- 
  One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Sun, Nov 16, 2014 at 02:03:23PM -0200, Henrique de Moraes Holschuh wrote:
 On Sun, 16 Nov 2014, Ron wrote:
  On Sat, Nov 15, 2014 at 03:49:56PM -0200, Henrique de Moraes Holschuh wrote:
   On Sat, 15 Nov 2014, Raphael Hertzog wrote:
On Fri, 14 Nov 2014, Ian Jackson wrote:
  What exactly is your use case you feel this is essential for?
 
 I think this discussion is in danger of going round in circles.
 I'm going to leave it here and let Raphael get on with it.

I understand Ron's logic and there's certainly value in questioning the
need for something. But this is always a question of cost/value.

Even if we don't have an immediate need for this property, the problem
is that we can't always envision all the ways people will want to use
the repositories (isn't that what you were trying to tell me about
how upstream can use git?) and I'm pretty sure that dropping the epoch
will be annoying to someone at some point. And the cost of not dropping
the epoch is not very high.
 
 ...
 
  If the answer to that is simply it's relatively harmless, has minimal
  cost, some things already do it, and it may be a useful visual cue to
  human users, and that's all, then that's a perfectly good reason.
 
 Agreed.  But there is a strong technical reason, too, on top of some
 technical advantages.
 
 So let me state the strong technical reason (which should at least be
 documented in DEP-14, because this _is_ something people often gloss over):
 
 Anything that doesn't store the full debian version (be it a git tag or a
 filename) will have a colision risk.  Consider a package that has versions
 1:2.3.5-1 and 5:2.3.5-1.

Right, that was one of the first cases I mentioned.  When you upload
5:2.3.5-1, dak will reject it exactly because they will collide. :)

Having the epoch in the tag means you won't find that out until you
try to upload it.  Not having it would mean trying that would fail
early and you might investigate why and find that out before you've
overwritten some existing file on your own system or uploaded a
broken package.

fwiw, I don't consider that a compelling reason alone to not include
the epoch in tags - but you appear to have just confirmed that not
many people know this, even when it's explicitly mentioned in a
thread they just replied to!  So I think *that* is probably worthy
of a mention if we do recommend including the epoch in tags :)


  If the answer is instead something like I want automated tools to be
  able to assume they can perfectly reverse the mangling and assume some
  semantic meaning to the text in the tag, then we're into Broken By
  Design territory and we really need to look more closely at *exactly*
 
 Well, we can have a perfect, reversible transformation between the debian
 version namespace and the debian git-tag version namespace, where debian
 means a vendor that uses dpkg and the deb format.

No.  I don't believe you can.

Mangle this version in a way that's legal to git: 0:1.2:3..4-1~~2..3.lock

Now try to explain to me how you can reverse that.

Yes, it's a corner case you've probably never seen, but the troublesome
parts of that are all legal debian versions.


 In that case, a debian version tag really could have one valid semantic
 meaning: one can derive the debian version from the tag, and also the
 opposite.
 
 What one cannot do is try to use either the debian version tag or the debian
 version to infer anything about the upstream version/versio tags.  We agree
 on this.

There's really no reason to need to depend on either.  There are robust
ways to actually get this information correctly if you do need it.
There's no need to invent a new one that only partly works when the
planets are aligned auspiciously.

I do strongly believe we should not be encouraging any tool to be that
kind of fragile.  But that's a separate question to should we include
the epochs in tags anyway, despite they fact they should *never* be
used for this.


 This assumes that DEP-14 will mandate exactly how debian version tags are
 formed, and that tools that require DEP-14 semanthics to work correctly will
 be upfront about that.

Which is itself a fragile assumption.  It can't *mandate* anything,
or go back and change history to date.


  what it is that people want to do and/or assume, and we probably need
  to look at less fragile solutions that really do satisfy that need.
 
 Trying to handle colisions due to missing epoch information in the git tag
 is far more fragile, IMO.
 
 Any tool that needs the full debian version information would either need it
 in the tag in a known format, or it would have to parse debian/changelog,
 which requires fetching the correct blob from git, and parsing it using
 dpkg-parsechangelog.  This is *not* fun to implement correctly, especially
 when you don't want to mess with the work tree.

This is, a) trivial, gitpkg has done it for years.  b) it doesn't need
to mess with the work tree.  c) any

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Sun, Nov 16, 2014 at 07:33:32PM +1030, Ron wrote:
 On Sat, Nov 15, 2014 at 06:15:33PM +, Simon McVittie wrote:
  On 12/11/14 22:07, Ron wrote:
   I am also interested to hear more
   about whatever the confusion was you had with this was when you
   started working with Tollef's systemd repo that you mentioned
   in the previous thread.
  
  Having played with gitpkg some more, I'm reminded that the answer to
  this is that unlike (AIUI) both gbp-pq and git-dpm, it did not meet my
  assumption that the contents of the git tree were in a suitable form to
  run dpkg-buildpackage and have a 3.0 (quilt) Debian package fall out. I
  realise that's partly a property of 3.0 (quilt).

Oh, one other thing I probably should clarify about this, that might
not be immediately evident to others, is that this isn't actually a
problem that gitpkg *requires* you to have.  In the case you hit,
it was purely a function of the workflow that the person who created
that repo chose to use.

You could quite equally use a workflow where you do commit your
patch series (so that the checked out tree is in the same as one
you'd see for the other tools), and gitpkg will quite happily
work with that too (and you don't need to enable any hooks to
do it if you go that way).

If you have any repo where you can run dpkg-bp in the working dir
and get a valid package (regardless of how it was created or what
other tools were used to do that), then gitpkg can export a source
package from it, in its default configuration, with no other hooks.


The big difference is gitpkg also gives you the option to use other
alternative workflows too, on a repo by repo basis, depending on
how the various tradeoffs of doing that balance out for you.


Initially, I was rather coy about overspecifying the workflows that
*I* thought were useful in the documentation, partly because there
was no established best practice and it was quite possible that
other people would invent things even better than what I had been
experimenting with at the time, if given a free hand to do that
and minimal prior brainwashing about how it should work.

That seemed like an opportunity not to waste.  But we may be at
the stage where we could describe a few of the more proven options
in a bit more detail now with less risk to short-circuiting new
and useful innovation.

[this is another one of those things where I'm really too close to
it to easily remember all the things I just take for granted that
aren't necessarily obvious to everyone else (yet), without the
perspective and input of other people :]

  Cheers,
  Ron



-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141117020239.go10...@hex.shelbyville.oz

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-15 Thread Matthias Urlichs

Hi,

Bernhard R. Link:
 I think such a document would do well to not say much about upstream
 branches or imply how they should be managed.
 
It is helpful to base the Debian release on an Upstream release that is
actually tagged by Upstream, provided that the version thus tagged is the
basis for ongoing development (or fixes, for stable branches).

On the other hand, there are upstreams where releasing means make a
release branch, run autotools and/or doxygen, add all these auto-gene-
rated files to the branch, and tag that. IMHO it's good practice to
discourage this.

 git-dpm creates some git branch head for the upstream branch. But I
 realized that I usually remove this git branch manually directly afterwards
 whenever it creates it.

Is there any advantage to doing so? If not: wouldn't it be easier,
long-term, to file a bug requesting to drop this behavior? ;-)

-- 
-- Matthias Urlichs


signature.asc
Description: Digital signature

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-15 Thread Raphael Hertzog

On Fri, 14 Nov 2014, Ian Jackson wrote:
 Richard Hartmann writes (Re: RFC: DEP-14: Recommended layout for Git 
 packaging repositories):
  Can you at least suggest, not require, that the NMUer should also send
  a link to a publicly available branch with the patch(es) which are
  based on the packaging branch's correct tag? That makes pulling in
  changes from in-repo simpler and does not force you to download
  patches, etc.
  Also, it shows that a consistent pathway from a specific point exists.
 
 This is a good suggestion.

But as I said it's unrelated to this DEP and should be part of the
developer's reference.

Unless you want to suggest a specific standardized name for a NMU patch
branch... but this does seem a bit premature given that nobody is
currently doing stuff like that. The few persons that I saw commit their
NMU to git just used the main branch for this (and not some
alternate/dedicated branch).

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141115170223.gd25...@home.ouaza.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-15 Thread Raphael Hertzog

On Fri, 14 Nov 2014, Ian Jackson wrote:
  What exactly is your use case you feel this is essential for?
 
 I think this discussion is in danger of going round in circles.
 I'm going to leave it here and let Raphael get on with it.

I understand Ron's logic and there's certainly value in questioning the
need for something. But this is always a question of cost/value.

Even if we don't have an immediate need for this property, the problem
is that we can't always envision all the ways people will want to use
the repositories (isn't that what you were trying to tell me about
how upstream can use git?) and I'm pretty sure that dropping the epoch
will be annoying to someone at some point. And the cost of not dropping
the epoch is not very high.

YMMV.
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141115171644.ge25...@home.ouaza.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-15 Thread gregor herrmann

On Sat, 15 Nov 2014 18:02:23 +0100, Raphael Hertzog wrote:

 Unless you want to suggest a specific standardized name for a NMU patch
 branch... but this does seem a bit premature given that nobody is
 currently doing stuff like that. The few persons that I saw commit their
 NMU to git just used the main branch for this (and not some
 alternate/dedicated branch).

This is not completely correct; I know at least one DD who uses
separate branches for pushing NMU/DELAYED changes (Axel Beckert):
http://anonscm.debian.org/cgit/collab-maint/debsums.git/
http://anonscm.debian.org/cgit/collab-maint/wicd.git

The branch is called nmu in both cases, if I'm seeing this correctly.


Cheers,
gregor

-- 
 .''`.  Homepage: http://info.comodo.priv.at/ - OpenPGP key 0xBB3A68018649AA06
 : :' : Debian GNU/Linux user, admin, and developer  -  http://www.debian.org/
 `. `'  Member of VIBE!AT  SPI, fellow of the Free Software Foundation Europe
   `-   NP: Dire Straits: Why Worry


signature.asc
Description: Digital Signature

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-15 Thread Henrique de Moraes Holschuh

On Sat, 15 Nov 2014, Raphael Hertzog wrote:
 On Fri, 14 Nov 2014, Ian Jackson wrote:
   What exactly is your use case you feel this is essential for?
  
  I think this discussion is in danger of going round in circles.
  I'm going to leave it here and let Raphael get on with it.
 
 I understand Ron's logic and there's certainly value in questioning the
 need for something. But this is always a question of cost/value.
 
 Even if we don't have an immediate need for this property, the problem
 is that we can't always envision all the ways people will want to use
 the repositories (isn't that what you were trying to tell me about
 how upstream can use git?) and I'm pretty sure that dropping the epoch
 will be annoying to someone at some point. And the cost of not dropping
 the epoch is not very high.

I fully agree.  Please, lets just keep the epoch which is the safer design
choice anyway, and get done with it.

Also, we have a fully reversible, human-friendly mapping to deal with : and
~ already, it has been in use by git-buildpackage for years, it is tested,
and it works.  I suggest we remove the risk of making things worse by
gratuitous overengineering (URL-like %-encoding, etc), and mandate the use
of the simple 1:1 map already ( [:] maps to [%] and [~] maps to [_] ) for
the Debian-compatible git-tag version namespace.

Whether the upstram version string is compatible or not with the Debian
version namespace and the Debian git-tag version namespace is a non-problem:
the upstream version WILL have been reduced to something compatible to be
usable for the package version in debian/changelog already, anyway.

Therefore, there is no space for ambiguity here. % and _ are *illegal* in
the Debian version namespace, so they can be freely used to translate from
the Debian version namespace to the Debian version git-tag namespace.

And anything that compares the upstream version namespace and the Debian
version namespace directly (or via their git-tag namespaces) is Utterly
Broken in the general case anyway, even if it will often just work
(particular case).

-- 
  One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie. -- The Silicon Valley Tarot
  Henrique Holschuh


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141115174956.ga9...@khazad-dum.debian.net

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-15 Thread Simon McVittie

On 12/11/14 22:07, Ron wrote:
 I am also interested to hear more
 about whatever the confusion was you had with this was when you
 started working with Tollef's systemd repo that you mentioned
 in the previous thread.

Having played with gitpkg some more, I'm reminded that the answer to
this is that unlike (AIUI) both gbp-pq and git-dpm, it did not meet my
assumption that the contents of the git tree were in a suitable form to
run dpkg-buildpackage and have a 3.0 (quilt) Debian package fall out. I
realise that's partly a property of 3.0 (quilt).

For gitpkg, you can commit in the normal git way, but the cost is that
you have to build in a way that isn't the normal dpkg thing (exporting
with gitpkg and building the result).

gbp-pq and git-dpm are the other way round: the tree can be built with
dpkg-buildpackage, but the cost is that you have to commit in a way that
isn't the normal git thing (either using a specific tool, or for the
gbp-pq layout, dropping in pre-prepared patches and hoping they don't
have conflicts, in the same way you might for svn-buildpackage).

I think I was also thrown by the fact that gitpkg does not encapsulate
its configuration in what you commit: if two developers build the same
tree, the debdiff might well be rather large, because one developer's
.git/config results in separate git-debcherry patches and the other's
.git/config results in a single large patch.

git-buildpackage reads both debian/gbp.conf and .git/gbp.conf, with the
latter taking precedence. That lets maintainers provide executable
documentation, in debian/gbp.conf, for here is how I intend this repo
to be used, which seems like something that could be rather useful for
gitpkg: for instance, filter patterns for non-DFSG tarball imports can
go in debian/gbp.conf as a way to avoid mistakes.

S


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/54679845.90...@debian.org

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-15 Thread Simon McVittie

On 13/11/14 14:04, Ron wrote:
 I really do think that the names of the branches are actually going to
 be the least of your worries here, unfortunately.  Even with a naming
 scheme that's widely adopted, things just aren't going to be that sort
 of uniform outside of (a fairly large number of) fairly small subsets.

I agree that the expected contents of the branches are far more
important than their names. Unfortunately, while acting as the Debian
expert for Debian derivatives at $day_job, I keep finding that the
answer to OK, I've cloned a package's git repository, I know what code
change I want, now do I change the upstream source or drop a patch into
debian/patches or what? is ... I can't actually answer that until you
tell me which source package you're working on.

At the moment, I suspect Kali's approach - arbitrarily choosing one of
the popular approaches, only cloning packaging repositories from Debian
that happen to match that approach, and restarting a new packaging
repository for those that do not - is likely to be the only viable
solution to that. There's always going to be a certain amount of
re-importing in any case, because some packages in Debian are maintained
in a non-git VCS or in no VCS at all; but it's easier to inspect history
if it's possible to clone the existing packaging repository for most
Debian packages of interest.

One of my projects for the near future is to put together some simple
test-cases for packaging - a set of simple projects with a downstream
patch that gets applied in the next upstream release, a downstream patch
that doesn't get applied upstream, and a downstream patch that conflicts
with upstream changes - and try packaging them with each of gbp-pq,
git-dpm and gitpkg. To have the complete set, I think I need one project
where the upstream tarball is a simple git-archive of the upstream git
repository, one where the upstream tarball has extra detritus (e.g.
Autotools) and/or missing files (upstream's .gitignore not being in the
tarball is also common in Autotools), and one where the Debian
maintainer needs to filter out a non-free file. Anything else you can
think of?

S


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/54679cdb.4090...@debian.org

Differences between git packaging tools (was: RFC: DEP-14: Recommended layout for Git packaging repositories)

2014-11-15 Thread Nikolaus Rath

Simon McVittie s...@debian.org writes:
 Having played with gitpkg some more, I'm reminded that the answer to
 this is that unlike (AIUI) both gbp-pq and git-dpm, it did not meet my
 assumption that the contents of the git tree were in a suitable form to
 run dpkg-buildpackage and have a 3.0 (quilt) Debian package fall out. I
 realise that's partly a property of 3.0 (quilt).

 For gitpkg, you can commit in the normal git way, but the cost is that
 you have to build in a way that isn't the normal dpkg thing (exporting
 with gitpkg and building the result).

 gbp-pq and git-dpm are the other way round: the tree can be built with
 dpkg-buildpackage, but the cost is that you have to commit in a way that
 isn't the normal git thing (either using a specific tool, or for the
 gbp-pq layout, dropping in pre-prepared patches and hoping they don't
 have conflicts, in the same way you might for svn-buildpackage).

This is such a nice summary that I have to quote it again just to make
it more visible.

I've been very confused by the proliferation of git packaging tools, but
this makes it a lot clearer.


Thanks!
-Nikolaus
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/8761egufce.fsf...@vostro.rath.org

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-15 Thread Ron

On Sat, Nov 15, 2014 at 03:49:56PM -0200, Henrique de Moraes Holschuh wrote:
 On Sat, 15 Nov 2014, Raphael Hertzog wrote:
  On Fri, 14 Nov 2014, Ian Jackson wrote:
What exactly is your use case you feel this is essential for?
   
   I think this discussion is in danger of going round in circles.
   I'm going to leave it here and let Raphael get on with it.
  
  I understand Ron's logic and there's certainly value in questioning the
  need for something. But this is always a question of cost/value.
  
  Even if we don't have an immediate need for this property, the problem
  is that we can't always envision all the ways people will want to use
  the repositories (isn't that what you were trying to tell me about
  how upstream can use git?) and I'm pretty sure that dropping the epoch
  will be annoying to someone at some point. And the cost of not dropping
  the epoch is not very high.

Yes, absolutely.  I think this got kind of derailed through failing to
understand (and answer) the very simple question I asked.

When I asked Why include the epoch in tags at all?, that was not some
politically correct euphemism for the statement I object to including
them (which I certainly do not), it was an actual question.

If the answer to that is simply it's relatively harmless, has minimal
cost, some things already do it, and it may be a useful visual cue to
human users, and that's all, then that's a perfectly good reason.

If the answer is instead something like I want automated tools to be
able to assume they can perfectly reverse the mangling and assume some
semantic meaning to the text in the tag, then we're into Broken By
Design territory and we really need to look more closely at *exactly*
what it is that people want to do and/or assume, and we probably need
to look at less fragile solutions that really do satisfy that need.
But we can't suggest things for that without some clear statements of
what exactly are the use cases people have in mind.


I'm never a fan of discarding information unless you're certain you
won't ever need it back.  But information that you can't trust is
always a mixed blessing, and epochs themselves bring their own extra
caveats too.  I was hoping to get people to think about this in a
bit more detail than they'd already appeared to and have a sensible
discussion to enumerate the pros and cons, and possibly any warnings
we ought to explicitly offer about this.  Having people dig trenches
for a simplistic for or against war isn't really going to solve
anything, or advance us to anything better than what we already have.


 I fully agree.  Please, lets just keep the epoch which is the safer design
 choice anyway, and get done with it.

Henrique, would you care to elaborate on your definition of safer?
I already pointed out some ways it's not.  A well reasoned middle
ground might let us offer people some actually useful advice about
this in a way that just bury your head in the sand and pretend
everything is ok would not.


 Also, we have a fully reversible, human-friendly mapping to deal with

Unless you really do something crazy, like URL encoding (which I don't
think we should do, the cost to humans is way higher than the benefit
to machines, and tags *are* for the benefit of humans), the manglings
to the allowed ref names have never been guaranteed to be reversible,
and we have a decade worth of existing practice where they definitely
were not.

They still aren't guaranteed to be so even with this convention,
because there's no hard guarantee that everyone or everything will
use them.  And even if they did, I still don't think you've covered
every combination of things that are illegal in a git refname which
might need to be mangled here.

The really crazy cases might be rare, but just because you've never
hit them doesn't mean they can't or won't ever exist in a way that
means someone one day will need to deal with them.  Why paint those
people into a corner when there are fairly easy ways to not do that?


 Whether the upstram version string is compatible or not with the Debian
 version namespace and the Debian git-tag version namespace is a non-problem:
 the upstream version WILL have been reduced to something compatible to be
 usable for the package version in debian/changelog already, anyway.
 
 Therefore, there is no space for ambiguity here. % and _ are *illegal* in
 the Debian version namespace, so they can be freely used to translate from
 the Debian version namespace to the Debian version git-tag namespace.
 
 And anything that compares the upstream version namespace and the Debian
 version namespace directly (or via their git-tag namespaces) is Utterly
 Broken in the general case anyway, even if it will often just work
 (particular case).

Anything that assumes you can always determine the exact release version
from a tag name is Utterly Broken in the general case too.

I don't think conflating these things is helpful here.


The question of should we recommend people

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-14 Thread Raphael Hertzog

On Fri, 14 Nov 2014, Ron wrote:
  So I am a Kali Linux contributor. We use git repos to maintain all our
  packages and we use git-buildpackage.
 
 I guess the first question there is what were the arguments put forward
 for deciding to 'standardise' on gbp?  If there wasn't one, maybe that's
 an argument you should have (and if there was, maybe it's one to revisit :)

1/ It's the tool I was most familiar with
2/ It was rather well documented
3/ It seemed to be the most popular tool within Debian
4/ It forces the user to follow some conventions and this is good when
   you have newcomers
5/ It did the job

You can try to sell me gitpkg but I'm not sure it will bring this
discussion forward.

 If you have a clearer idea now of the problems you are facing, it might
 make properly evaluating things that avoid those problems easier.

My problems are not so much technical, my problem is having a general
recipe that work well enough for most packages so that I have to be
involved for the few special cases.

Changing for another tool will not improve the situation if that other
tool requires an evaluation of the layout of each repository before being
able to execute the right incantation.

When Kali forks a package it's usually because we need a new upstream
version, or because we want to apply a pre-existing patch. It's not rocket
science and it ought to be doable rather easily, whatever helper tool
Debian uses.

git-buildpackage has not been a major obstacle for us, on the contrary.
Thanks to git-import-dsc we can hack an any package by always following
the same workflow.

But now if I want to go one step further and build on the existing
repositories while still trying to keep the same level of simplicity,
we have to go one step forward. And that's the step that I'm trying
to push for with this DEP.

 Even if everyone did stick to the conventions already discussed
 (which in reality, they won't) there's still far too many degrees of
 (quite necessary) freedom to really approach a just follow these
 three easy steps kind of uniformity.  And even if you got close to
 achieving that for the debian branches, the (again necessary)
 variability between upstream repos is going to be even greater.

I don't want 100% uniformity, I just want as much uniformity as possible
when there's no good reason to diverge from the recommendations.

For the other cases, we can stick to the current model (building on top of
imported sources packages). It's not perfect but it's a lot better 
if most of our forked packages are based on Debian's git repo.

 I think at some point you're going to have to rely on real human
 intelligence to be able to look at a repo and form their own
 understanding of its structure.  Most really aren't all that
 complicated (however much they vary between each other), and in
 the worst case you can always actually ask the 'upstream'
 maintainer to explain anything that is unclear.

Kali contributors are humans with intelligence but I'd rather
not require too much of their intelligence when it comes to trivial
changes like adding a patch or importing a new upstream version.

Also when you're 2-3 persons maintaining 300+ packages, you really
don't want to have to remember what's special about each package.
This is inavoidable for some packages (say linux) but for the vast
majority one should not need to ask himself too many questions.

 git is already going to namespace them so when you add the remotes
 the branch refs will be (for remotes named upstream, debian, kali):

I know all this, but this requires again a deeper knowledge of git's
features whereas git checkout branch will create a new local branch
based on the remote branch of the same name. If you have branch in
multiple remotes, there is ambiguity.


 If it really is the 'real' debian package, you have no problem
 and don't need a kali tag for it anyway.  If it's a kali special,
 then the package already ought to have a x.y.x-1kali2 type version
 anyway.  So this part already naturally avoids conflicts too.

Using namespaces has other possible benefits: think of building Debian
packages out of a git push. The explicit tag prefix avoids building the
Debian version of the package when I want to build the Kali one (I might
be the maintainer on both Debian and Kali, using the same GPG signature).

  And we could also build on work in progress that has not yet been
  released as a source package. Right now, the only packages where we
  build on top of the Debian git repositories are some native packages
  (like debian-installer).
 
 I can't really tell you how kali ought to work, but if you can solve
 it for this case, you should be able to do the same for non-native
 packages too.  And I would recommend that you try to, even if you
 can't get the uniformity that you'd thing of as ideal.
 
 What is the exact difference that makes native packages work for this
 but others be harder?  In the case of gitpkg, this is really just
 the difference between

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-14 Thread Raphael Hertzog

On Thu, 13 Nov 2014, Tobias Frost wrote:
 One point came to my mind: NMUs
 Can we maybe add some words what would be best practice to handle NMUs?

I think the current best practices are fine: the NMUer should send a
debdiff to the BTS. Maybe the patch doesn't apply on top of the
supplementary changes already committed in the git repository. It's not a
big deal, having to rebase patches is part of the job and it's a task
that maintainers should be able to manage.

If NMUers want to go further and apply their changes to the git
repository, fine, but this is a matter of procedure (and could possibly be
documented in the Developer's Reference) but it doesn't have its place
in this DEP IMHO.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141114105637.gb28...@home.ouaza.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-14 Thread Raphael Hertzog

On Thu, 13 Nov 2014, Bernhard R. Link wrote:
 * Raphael Hertzog hert...@debian.org [14 22:26]:
  Helper tools should usually rely on the output of `dpkg-vendor --query 
  vendor`
  to find out the name of the current vendor. The retrieved string must be
  converted to lower case. This allows users to override the current vendor
  by setting `DEB_VENDOR` in their environment (provided that the vendor
  information does exist in `/etc/dpkg/origins/`).
 
 Is using the vendor you use git on a good default for the vendor code
 you are currently working on? In my experience those two are quite
 unrelated.

Do you have a better default to suggest? In any case, having a default
value is certainly better than not having one and forcing everybody to
configure it in some ways.

I try to work in Kali chroots when I do Kali work. It's true that
it's not always the case but if there were real differences right now I
would pay more attention or would ensure to have the proper environment.

  Version mangling
  
  
  When a Git tag needs to refer to a specific version of a Debian package,
  the Debian version needs to be mangled to cope with Git's restrictions.
  The colon (`:`) needs to be replaced with a percent (`%`), and the tilde
  (`~`) needs to be replaced with an underscore (`_`).
 
 Is there a previous case of encoding colons with percent signs?

This is the current convention used by git-buildpackage and I believe the
% has been picked because it's visually close to : (i.e. two dots/two
circles).

 If it is inventing a third way instead of using on of the existing ones,
 is sounds like a bad idea.

Do you suggest to use URL encoding? That's rather heavyweight for two
special cases that can be easily mapped to other characters that are not
used in version strings. And it would convert characters that do not
need any special escaping currently.

1:1.2.3+dfsg-4~3 gives:
suggested scheme: 1%1.2.3+dfsg-4_3 
full url encoded: 1%3A1.2.3%2Bdfsg-4%7E3 

Or what else do you have in mind?

  Packaging branches and tags
  ===
  
  The Git repository listed in debian/control's `Vcs-Git` field should
  usually have its HEAD point to the branch corresponding to the
  distribution where new upstream versions are usually sent. For Debian,
  it will usually be `debian/sid` (or sometimes `debian/experimental`).
 
 I think this should alternatively allow for Vcs pointing to that branch
 instead. It sometimes makes sense to put the debian package development
 branches in the same repository as the upstream branch, where HEAD might
 be reserved for the upstream branch.

Right, fixed by making it conditional on the absence of -b branch:
-NOTE: The Git repository listed in debian/control's `Vcs-Git` field should
+NOTE: If the Git repository listed in debian/control's `Vcs-Git` field does
+not indicate an explicib branch (with the `-b branch` suffix) then it should
 have its HEAD point to the branch where new upstream versions are being

  When releasing a Debian package, the packager should create and push
  a signed tag named `vendor/version`.
 
 I'd advocate s/signed/(possibly signed)/, as I'm not ready to sign such
 a wildcard with any valuable key and getting another very-low-security
 key just to have signed keys to match this proposal sounds like waste.

Changed to (preferably signed).

 Tags are only based on versions are also quite hard to shuffle around.
 I'd strongly suggest adding the name of the source package to those,
 otherwise accessing multiple packages in on repository causes a big
 mess. (git keeps branches in the per-origin remote namespace when
 fetching stuff, tags only have one global namespace, local and remote).

I don't think this makes sense. The common case is a single software per
git repository. If you have multiple software, then the git way is
to use git submodules. If you have an upstream that doesn't follow those
conventions, then it's a good reason to not follow DEP-14 for the tagging
of your Debian releases.

But I don't think that DEP-14 should cater to upstreams are are going
against the spirit of traditional git usage.

Maybe we can document this as a special case where it's a good idea to
not follow DEP-14 ?

I added this:

@@ -255,6 +257,19 @@ of the `debian/changelog` file are handled by 
`dpkg-mergechangelogs` as
 this will make it much easier to merge between different packaging
 branches.
 
+When to not follow the above recommendations
+
+
+Most of the recommendations in this document assume that the upstream
+developers use their Git repository in a traditional way: one software per
+repository and creating tags for released versions.
+
+When the upstream diverge from those conventions, you are entitled
+to use your common sense and adapt those recommendations accordingly.
+For example, if upstream tags contain the software name (e.g.
+`foo-1.0` and `bar-2.0`), you should probably also consider

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-14 Thread Marco d'Itri

On Nov 14, Raphael Hertzog hert...@debian.org wrote:

 This is just a proof that storing the patches as real commits is useful.
 And that's the point of the patch-queue tag. Instead of having the patches
 only as real commits in the local repository, they get pushed to the
 public repository too under that tag (instead of appearing only within
 debian/patches/).
In principle I agree, but in practice this must support perfect 
roundtrips preserving all comments and metadata in the patches and gbp 
is still a long way from supporting this.

-- 
ciao,
Marco


pgpr4JxbyBlTR.pgp
Description: PGP signature

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Fri, Nov 14, 2014 at 11:25:36AM +0100, Raphael Hertzog wrote:
 On Fri, 14 Nov 2014, Ron wrote:
   So I am a Kali Linux contributor. We use git repos to maintain all our
   packages and we use git-buildpackage.
  
  I guess the first question there is what were the arguments put forward
  for deciding to 'standardise' on gbp?  If there wasn't one, maybe that's
  an argument you should have (and if there was, maybe it's one to revisit :)
 
 1/ It's the tool I was most familiar with
 2/ It was rather well documented
 3/ It seemed to be the most popular tool within Debian
 4/ It forces the user to follow some conventions and this is good when
you have newcomers
 5/ It did the job
 
 You can try to sell me gitpkg but I'm not sure it will bring this
 discussion forward.

I'm not trying to sell you anything in particular here.  I'm trying
to share a little of the view from a perspective where most of the
problems you seem to be worried about are in fact either already solved
or not actually problems at all.

If your answer to Doctor it hurts when I bang my head against the wall
is I know, let's change the building codes to mandate walls must be
made of nerf bricks!, there's still going to be a whole world full of
walls that are going to hurt your head.

You can't solve problems that are inherently chaotic by pretending
that they are not.  If you could, we'd have all been replaced by
very small shell scripts years ago.  Ask ubuntu how well that worked
for them ;)


If you don't want to learn about how the other tools that you don't
really know much about yet work, you're going to have a really hard
time trying to unify them in any actually meaningful way.



  If you have a clearer idea now of the problems you are facing, it might
  make properly evaluating things that avoid those problems easier.
 
 My problems are not so much technical, my problem is having a general
 recipe that work well enough for most packages so that I have to be
 involved for the few special cases.

Have you considered starting with a solution to something a bit simpler
first?  Like, say, world peace ;?

What you're asking for there is not only a technical problem, it's a
technically Hard problem.  There are ways you might be able to make
it easier, but a naming convention really isn't one of them.

That might make the people who like to have their socks strictly sorted
by colour feel a little more at home, but it doesn't actually address
what is actually hard about this.


 Changing for another tool will not improve the situation if that other
 tool requires an evaluation of the layout of each repository before being
 able to execute the right incantation.

How can you possibly know what you want to export from it without
looking at it?  You can't even know if the repo is currently in a
coherent state suitable to be exported from if you don't look at
it and exercise some amount of judgement on what you see.


 When Kali forks a package it's usually because we need a new upstream
 version, or because we want to apply a pre-existing patch. It's not rocket
 science and it ought to be doable rather easily, whatever helper tool
 Debian uses.

Well that sounds a lot less like something you'd be pushing back to
Debian.  And not even something you particularly need a (D)VCS for,
or where you'd get a whole lot of value from one.


 git-buildpackage has not been a major obstacle for us, on the contrary.
 Thanks to git-import-dsc we can hack an any package by always following
 the same workflow.
 
 But now if I want to go one step further and build on the existing
 repositories while still trying to keep the same level of simplicity,
 we have to go one step forward. And that's the step that I'm trying
 to push for with this DEP.

Again, I'm not arguing with the goal.  Just trying to point out that
simply formalising the actual source of many of your current problems
isn't really a way to solve them.


  Even if everyone did stick to the conventions already discussed
  (which in reality, they won't) there's still far too many degrees of
  (quite necessary) freedom to really approach a just follow these
  three easy steps kind of uniformity.  And even if you got close to
  achieving that for the debian branches, the (again necessary)
  variability between upstream repos is going to be even greater.
 
 I don't want 100% uniformity, I just want as much uniformity as possible
 when there's no good reason to diverge from the recommendations.

And I'm not particularly arguing against the value of conventions
where conventions are valuable.  But they aren't going to solve
the problem(s) that you seem to have convinced yourself that they
will - and they especially aren't going to solve them if you go
making up new conventions that are different to what everyone else
in the world not using gbp (ie. 99% of it) are using.

Upstream developers aren't going to read and follow your
recommendations.  If you don't adapt to being able to cope with
the

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

Raphael Hertzog wrote:
 On Thu, 13 Nov 2014, Bernhard R. Link wrote:
  * Raphael Hertzog hert...@debian.org [14 22:26]:
 
  Is using the vendor you use git on a good default for the vendor code
  you are currently working on? In my experience those two are quite
  unrelated.
 
 Do you have a better default to suggest? In any case, having a default
 value is certainly better than not having one and forcing everybody to
 configure it in some ways.
 
 I try to work in Kali chroots when I do Kali work. It's true that
 it's not always the case but if there were real differences right now I
 would pay more attention or would ensure to have the proper environment.

gitpkg makes it completely trivial to work from the comfort of a real
debian system while preparing releases for other distros or suites,
and to throw builds of that off to a proper clean build chroot.
(you can happily do that for packages from the same tag too)

That was part of why I asked earlier about why you thought this would
be useful and how you thought it would work.


  Is there a previous case of encoding colons with percent signs?
 
 This is the current convention used by git-buildpackage and I believe the
 % has been picked because it's visually close to : (i.e. two dots/two
 circles).

Why include the epoch in tags at all?

The package filenames don't include them, and if you upload two versions
of a package that only vary by epoch then dak will reject it too.

I'm not even sure what p-t would do to you if you tried that on it.


So there's already lots of good reasons why tags without the epoch will
always be unique, and why it's probably even a good sanity check to
help new contributors avoid making that mistake when preparing a
new package too.

  Ron



-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141114153234.gg10...@hex.shelbyville.oz

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

Ron writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
repositories):
 Why include the epoch in tags at all?

Because we want to be able to tell not just which tag was which but
also what order they are in.

The only reason I excluded epoch from filenames is that it makes tools
like scp and rsync harder to use, and I sometimes wonder if my
decision was wrong.

Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/21606.8729.280787.992...@chiark.greenend.org.uk

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

Bernhard R. Link writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
repositories):
 Raphael Hertzog hert...@debian.org [14 22:26]:
...
  When a Git tag needs to refer to a specific version of a Debian package,
  the Debian version needs to be mangled to cope with Git's restrictions.
  The colon (`:`) needs to be replaced with a percent (`%`), and the tilde
  (`~`) needs to be replaced with an underscore (`_`).
 
 Is there a previous case of encoding colons with percent signs?
 If it is inventing a third way instead of using on of the existing ones,
 is sounds like a bad idea.

Raphael, is there precedent for this ?

Current practice seem so be to replace both : and ~ with _.  Unless we
expect to find version numbers matching ^\d+\~, then anything matching
^\d+_ is unambiguously an epoch.  Any _ elsewhere is unambigously ~.

I'm not opposed to changing this but I wonder whether changing it
would break anything.  Maybe it would be better to write a restriction
in the version number policy forbidding ^\d+\~

Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/21606.9059.344851.480...@chiark.greenend.org.uk

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Fri, Nov 14, 2014 at 03:39:05PM +, Ian Jackson wrote:
 Ron writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
 repositories):
  Why include the epoch in tags at all?
 
 Because we want to be able to tell not just which tag was which but
 also what order they are in.

Right, but this is git, so that information is also already known.

That said I did consider that side of it too, but I'm having a hard
time thinking of an example where you would actually care about
ordering the tags for some task.  Do you have one that comes to mind?


 The only reason I excluded epoch from filenames is that it makes tools
 like scp and rsync harder to use, and I sometimes wonder if my
 decision was wrong.

Yeah, I'm sure we could both come up with a similar list of pros and
cons for that, but it is the reality we have to deal with, and quite
a few people aren't aware that this would result in rejected uploads
(based on the question coming up in #d-d every few months).

And I'm really not sure whether p-t will handle that sanely or
destroy the universe if you try, so it seemed like a convention where
we could build in an extra safety catch to get people thinking when
they try to do something that will ultimately bite them.


The ordering problem seems like one that could be fairly easily solved
in other ways, but that would mostly depend on what reasons people come
up with for actually needing that.  Purely in 'dpkg order' might not
always be the most interesting order for some of those anyway.

  Ron



-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141114160002.gh10...@hex.shelbyville.oz

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-14 Thread Thorsten Glaser

On Fri, 14 Nov 2014, Ian Jackson wrote:

 expect to find version numbers matching ^\d+\~, then anything matching

These are common enough for me to have not only seen,
but also used (in native packages, of course) them.

(But: Yes, epochs should belong into filenames, except
for filesystem naming restrictions… so they belong
everywhere else.)

bye,
//mirabilos
-- 
Sometimes they [people] care too much: pretty printers [and syntax highligh-
ting, d.A.] mechanically produce pretty output that accentuates irrelevant
detail in the program, which is as sensible as putting all the prepositions
in English text in bold font.   -- Rob Pike in Notes on Programming in C


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/alpine.deb.2.11.1411141700190.17...@tglase.lan.tarent.de

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-14 Thread Richard Hartmann

On Fri, Nov 14, 2014 at 11:56 AM, Raphael Hertzog hert...@debian.org wrote:
 On Thu, 13 Nov 2014, Tobias Frost wrote:
 One point came to my mind: NMUs
 Can we maybe add some words what would be best practice to handle NMUs?

 I think the current best practices are fine: the NMUer should send a
 debdiff to the BTS.

Can you at least suggest, not require, that the NMUer should also send
a link to a publicly available branch with the patch(es) which are
based on the packaging branch's correct tag? That makes pulling in
changes from in-repo simpler and does not force you to download
patches, etc.
Also, it shows that a consistent pathway from a specific point exists.

If the NMU'ed maintainer force-pushes over stuff, or if third parties
don't have immediate access to the original branch, this would save a
lot of potential work  trouble.



Richard


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/CAD77+gTLb_va_s=UGJqgTWorY6g_=j8ab6nk0vg_e_ccl8b...@mail.gmail.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-14 Thread Henrique de Moraes Holschuh

On Sat, 15 Nov 2014, Ron wrote:
 On Fri, Nov 14, 2014 at 03:39:05PM +, Ian Jackson wrote:
  Ron writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
  repositories):
   Why include the epoch in tags at all?
  
  Because we want to be able to tell not just which tag was which but
  also what order they are in.
 
 Right, but this is git, so that information is also already known.
 
 That said I did consider that side of it too, but I'm having a hard
 time thinking of an example where you would actually care about
 ordering the tags for some task.  Do you have one that comes to mind?

The BTS needs it: it is the very base of version-aware bug state tracking.
It builds the DAG by combining the DAGs extracted from the Debian changelogs
of each branch of the package, AFAIK.

So, do we envision any sort of hook that would need to do something like
version-aware state tracking?

 The ordering problem seems like one that could be fairly easily solved
 in other ways, but that would mostly depend on what reasons people come
 up with for actually needing that.  Purely in 'dpkg order' might not
 always be the most interesting order for some of those anyway.

Well, the best example I have (the BTS) has no use for dpkg order :-)

-- 
  One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie. -- The Silicon Valley Tarot
  Henrique Holschuh


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141114171329.gg19...@khazad-dum.debian.net

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-14 Thread Henrique de Moraes Holschuh

On Fri, 14 Nov 2014, Richard Hartmann wrote:
 On Fri, Nov 14, 2014 at 11:56 AM, Raphael Hertzog hert...@debian.org wrote:
  On Thu, 13 Nov 2014, Tobias Frost wrote:
  One point came to my mind: NMUs
  Can we maybe add some words what would be best practice to handle NMUs?
 
  I think the current best practices are fine: the NMUer should send a
  debdiff to the BTS.
 
 Can you at least suggest, not require, that the NMUer should also send
 a link to a publicly available branch with the patch(es) which are
 based on the packaging branch's correct tag? That makes pulling in
 changes from in-repo simpler and does not force you to download
 patches, etc.
 Also, it shows that a consistent pathway from a specific point exists.

And what do we do if they differ?  As soon as you ask for two, you need to
ensure both are the same.

I'd be fine with an exclusive or (either one or the other, but not both),
though.  But the BTS patch has a longevity and archival guarantee that no
git tree will have in the general case.

-- 
  One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie. -- The Silicon Valley Tarot
  Henrique Holschuh


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141114171603.gh19...@khazad-dum.debian.net

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Fri, Nov 14, 2014 at 03:13:30PM -0200, Henrique de Moraes Holschuh wrote:
 On Sat, 15 Nov 2014, Ron wrote:
  On Fri, Nov 14, 2014 at 03:39:05PM +, Ian Jackson wrote:
   Ron writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
   repositories):
Why include the epoch in tags at all?
   
   Because we want to be able to tell not just which tag was which but
   also what order they are in.
  
  Right, but this is git, so that information is also already known.
  
  That said I did consider that side of it too, but I'm having a hard
  time thinking of an example where you would actually care about
  ordering the tags for some task.  Do you have one that comes to mind?
 
 The BTS needs it: it is the very base of version-aware bug state tracking.
 It builds the DAG by combining the DAGs extracted from the Debian changelogs
 of each branch of the package, AFAIK.

Right, gitweb, cgit, gitk, etc. are all going to do exactly the same
thing, take them from the DAG of the repo.  They are unlikely to care
about how the tag names would textually sort (and even less likely to
sort them in the order that dpkg would consider them in).

 So, do we envision any sort of hook that would need to do something like
 version-aware state tracking?

If you have something that really needs the actual exact version of
the package, the only reliable way to do that is to look at the
debian/changelog for the ref of interest anyway.  Otherwise you're
still back to the how do unmangle an unrepresentable version
problem.  gitpkg does exactly this when it needs the real package
version.

(and re the other question on this, all the gitpkg tools and hooks
 also mangle all illegal ref characters to '_' as well)


  The ordering problem seems like one that could be fairly easily solved
  in other ways, but that would mostly depend on what reasons people come
  up with for actually needing that.  Purely in 'dpkg order' might not
  always be the most interesting order for some of those anyway.
 
 Well, the best example I have (the BTS) has no use for dpkg order :-)


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141114173518.gi10...@hex.shelbyville.oz

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

Richard Hartmann writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
repositories):
 Can you at least suggest, not require, that the NMUer should also send
 a link to a publicly available branch with the patch(es) which are
 based on the packaging branch's correct tag? That makes pulling in
 changes from in-repo simpler and does not force you to download
 patches, etc.
 Also, it shows that a consistent pathway from a specific point exists.

This is a good suggestion.

Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/21606.15951.265912.610...@chiark.greenend.org.uk

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

Thorsten Glaser writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
repositories):
 On Fri, 14 Nov 2014, Ian Jackson wrote:
  expect to find version numbers matching ^\d+\~, then anything matching
 
 These are common enough for me to have not only seen,
 but also used (in native packages, of course) them.

OK.  I think then that we should probably change our practice for
epochs in tag names.  The first thing would probably be to canvas the
views of the maintainers of existing git packaging tools.

Should I file bugs against all the known git handling tools which
encode this information, inviting them to comment on this proposal ?

(Is there a convenient list of such tools ?  On the wiki perhaps?)

Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/21606.16063.64999.573...@chiark.greenend.org.uk

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

Ron writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
repositories):
 Right, gitweb, cgit, gitk, etc. are all going to do exactly the same
 thing, take them from the DAG of the repo.  They are unlikely to care
 about how the tag names would textually sort (and even less likely to
 sort them in the order that dpkg would consider them in).

Looking at the DAG of the repo doesn't necessarily let you compare
versions in different suites.

Worse, unless I'm mistaken, there are some current workflows which
involve subsequent uploads to the same suite not necessarily being
fast forwards.

I think it is essential that tags can be ordered by their name, rather
than by looking at the commit graph.

Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/21606.16213.932899.73...@chiark.greenend.org.uk

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-14 Thread Matthias Urlichs

Hi,

Henrique de Moraes Holschuh:
  That said I did consider that side of it too, but I'm having a hard
  time thinking of an example where you would actually care about
  ordering the tags for some task.  Do you have one that comes to mind?
 
 The BTS needs it: it is the very base of version-aware bug state tracking.
 It builds the DAG by combining the DAGs extracted from the Debian changelogs
 of each branch of the package, AFAIK.
 
Sure, but tags only *represent* version numbers. The _real_ version number
is at the top of debian/changelog; if you want that, or an ordered list,
you read that file. Or you incrementally update your state by reading
.changes files.

If we ever implement the crazy(?) idea to feed our builders and the BTS
directly from git, it'll presumably do this by reading the debian/changelog
files in it, not by scanning tags which may or may not correspond to an
actual source upload.

-- 
-- Matthias Urlichs


signature.asc
Description: Digital signature

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-14 Thread Matthias Urlichs

Hi,

Ian Jackson:
 Worse, unless I'm mistaken, there are some current workflows which
 involve subsequent uploads to the same suite not necessarily being
 fast forwards.
 
Meh. Anybody whose upload depends on being able to force-push a branch
deserves to lose. (I'm talking about the general case here, not oops, I
just pushed half-baked buggy .)

-- 
-- Matthias Urlichs


signature.asc
Description: Digital signature

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Fri, Nov 14, 2014 at 05:43:49PM +, Ian Jackson wrote:
 Ron writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
 repositories):
  Right, gitweb, cgit, gitk, etc. are all going to do exactly the same
  thing, take them from the DAG of the repo.  They are unlikely to care
  about how the tag names would textually sort (and even less likely to
  sort them in the order that dpkg would consider them in).
 
 Looking at the DAG of the repo doesn't necessarily let you compare
 versions in different suites.

There is more information available to you than just the DAG though.

 Worse, unless I'm mistaken, there are some current workflows which
 involve subsequent uploads to the same suite not necessarily being
 fast forwards.

Right, and like you already noted, there's potential for ambiguity
in tag names anyway once the illegal characters have been mangled.
And then the totally separate question of how upstream tags might
be named and might sort relative to your Debian ones.

Which is basically why I originally said that there isn't a generic
way to solve this for everyone and every use.  It really depends on
exactly why you want them ordered and the collating sequence (or
other constraint) you want them them ordered with.

 I think it is essential that tags can be ordered by their name, rather
 than by looking at the commit graph.

I'm all for letting people do exactly what solves their problems best
in their own repos.  I'm just trying to get a sense of what problems
people can identify that need consideration for making best practice
type recommendations.

What exactly is your use case you feel this is essential for?

  Ron



-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141114180912.gj10...@hex.shelbyville.oz

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-14 Thread Don Armstrong

On Fri, 14 Nov 2014, Henrique de Moraes Holschuh wrote:
 The BTS needs it: it is the very base of version-aware bug state
 tracking. It builds the DAG by combining the DAGs extracted from the
 Debian changelogs of each branch of the package, AFAIK.

Speaking of which, one of the long standing ideas in my head has been to
connect the DAG of a VCS repository with the DAG that the BTS tracks,
and enable people to mark specific bugs as fixed in a specific series of
commits more easily.

If someone is interested in working on doing the plumbing and coding
necessary to make this possible, please contact ow...@bugs.debian.org or
myself (dondelelcaro) in #debbugs on irc.debian.org.
 

-- 
Don Armstrong  http://www.donarmstrong.com

No amount of force can control a free man, a man whose mind is free
[...] You can't conquer a free man; the most you can do is kill him.
 -- Robert Heinlein _Revolt in 2010_ p54


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141114181922.gh4...@rzlab.ucr.edu

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

Ron writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
repositories):
 Right, and like you already noted, there's potential for ambiguity
 in tag names anyway once the illegal characters have been mangled.

Raphael is proposing an unambiguous mangling which deals with this
problem.

 And then the totally separate question of how upstream tags might
 be named and might sort relative to your Debian ones.

Raphael's document obviously can't deal with that.

 Which is basically why I originally said that there isn't a generic
 way to solve this for everyone and every use.  It really depends on
 exactly why you want them ordered and the collating sequence (or
 other constraint) you want them them ordered with.

We are talking about the Debian tags (and those of our downstreams
using similar tools).  Several people have explained _why_ they want
them ordered.  The collating sequence is already defined.

 What exactly is your use case you feel this is essential for?

I think this discussion is in danger of going round in circles.
I'm going to leave it here and let Raphael get on with it.

Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/21606.19219.884974.974...@chiark.greenend.org.uk

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-14 Thread Bernhard R. Link

* Raphael Hertzog hert...@debian.org [141114 12:34]:
 On Thu, 13 Nov 2014, Bernhard R. Link wrote:
  * Raphael Hertzog hert...@debian.org [14 22:26]:
   Helper tools should usually rely on the output of `dpkg-vendor --query 
   vendor`
   to find out the name of the current vendor. The retrieved string must be
   converted to lower case. This allows users to override the current vendor
   by setting `DEB_VENDOR` in their environment (provided that the vendor
   information does exist in `/etc/dpkg/origins/`).
 
  Is using the vendor you use git on a good default for the vendor code
  you are currently working on? In my experience those two are quite
  unrelated.

 Do you have a better default to suggest? In any case, having a default
 value is certainly better than not having one and forcing everybody to
 configure it in some ways.

I'm not so sure about that. Having a broken default can be more annoying
than not having a default at all.

 I try to work in Kali chroots when I do Kali work. It's true that
 it's not always the case but if there were real differences right now I
 would pay more attention or would ensure to have the proper environment.

Well, for test-building stuff and things like that going to the proper
environment is nice, but why having the need to have gitk and new enough
versions of your helper tools around everywhere?

   Version mangling
   
   
   When a Git tag needs to refer to a specific version of a Debian package,
   the Debian version needs to be mangled to cope with Git's restrictions.
   The colon (`:`) needs to be replaced with a percent (`%`), and the tilde
   (`~`) needs to be replaced with an underscore (`_`).
  
  Is there a previous case of encoding colons with percent signs?
 
 This is the current convention used by git-buildpackage and I believe the
 % has been picked because it's visually close to : (i.e. two dots/two
 circles).

  If it is inventing a third way instead of using on of the existing ones,
  is sounds like a bad idea.
 
 Do you suggest to use URL encoding? That's rather heavyweight for two
 special cases that can be easily mapped to other characters that are not
 used in version strings. And it would convert characters that do not
 need any special escaping currently.

No, I was refering to the many ways to encode. I was yet aware of
encoding 2:1-3~4 as
debian2-1-3_4
debian-2_1-3_4
debian-2.1-3.4

I guess using single letters as format identifiers in git-dpm's tag
specification was a bit optimistic...

  Tags are only based on versions are also quite hard to shuffle around.
  I'd strongly suggest adding the name of the source package to those,
  otherwise accessing multiple packages in on repository causes a big
  mess. (git keeps branches in the per-origin remote namespace when
  fetching stuff, tags only have one global namespace, local and remote).

 I don't think this makes sense. The common case is a single software per
 git repository. If you have multiple software, then the git way is
 to use git submodules. If you have an upstream that doesn't follow those
 conventions, then it's a good reason to not follow DEP-14 for the tagging
 of your Debian releases.

While having a repository hosting multiple projects permanently makes in
many cases not that many sense (though I sometimes prefer it for some
small stuff, as it allows to make/update a mobile backup with a single
git command), cherry picking commits from other repositories in git
to my knowledge requires fetching at least part of that history in your
current working repository. When fetching too much by mistake the global
nature of tags can make quite a mess.

   Native packages
   ===
   
   The above conventions mainly cater to the case of non-native packages,
   that is when the upstream developers and the package maintainers are
   not the same set of persons.
   
   When upstream is Debian (or one of its derivative), the upstream vendor
   should not use the usual `vendor/` prefix (but all others vendors should
   do so). The main development branch can be named `master` instead of
   the codename of the target distribution (although you are free to still
   use the codename if you wish so).
  
  Does native here mean a native package or a package developed primarily
  for Debian (which can also use a non-native packaging)? This paragraph
  seems to mix those two concepts quite a bit.
 
 Yes it mentions both concepts. I tried to word it in a way that is clear
 enough though. If you find some parts ambiguous, let me know which one
 or suggest a better wording.

Perhaps s/If upstream is Debian/If the package uses the native packaging format 
and upstream is
Debian/ ?

 I tried to improve this:
 
 @@ -175,9 +177,8 @@ a byte-for-byte copy of the upstream tarballs, this 
 should be done in the
  Native packages
  ===
  
 -The above conventions mainly cater to the case of non-native packages,
 -that is when the upstream developers and the package

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Matthias Urlichs

Hi,

Ian Jackson:
  Thus, please don't try to shoehorn a divided workflow into this DEP.
  Write your own.
 
 I disagree with half of this but agree with the other half.
 I think that the divided workflow should be documented in this DEP.
 
 But I agree that those who like the divided workflow should be the
 ones to write it up.  I see that Simon (for example) is actively
 engaging in the discussion.
 
Thanks; this is a good way to proceed.

-- 
-- Matthias Urlichs


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141113080504.gk23...@smurf.noris.de

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Raphael Hertzog

[ I skip the more detailed discussions on naming conventions to
  concentrate on your higher level questions for now ]

On Thu, 13 Nov 2014, Ron wrote:
 Sure, I understood those were your goals.
 
 What I haven't seen, and what I'm asking for, is an actual detailed
 rationale describing the actual detailed problem(s) that you think
 these goals will be a remedy for.

Problem 1: the derivatives
--

So I am a Kali Linux contributor. We use git repos to maintain all our
packages and we use git-buildpackage. Most of the Kali contributors
are not long-term Debian contributors, I write documentation so that
they can contribute to Kali (while basing our work on Debian).

To make this manageable I opted to always use a workflow based on
git-import-orig. Even when Debian has its own git repo, we start
from the released source packages because that's the only level of
uniformity that we can rely on. And it's a pity because if we could
build on Debian's git repo, it also means that our work would be easier
to merge for the Debian maintainer. They could add the Kali repo as a
remote (without fearing any conflict in terms of tags, branch names)
and just merge or cherry-pick as appropriate.

And we could also build on work in progress that has not yet been
released as a source package. Right now, the only packages where we
build on top of the Debian git repositories are some native packages
(like debian-installer).

I can't afford to document all the possible ways Debian is maintaining
their package but if I can write a documentation that covers the common
case and if I can tell them when you see those branches, you can follow
the instructions below, then we have made some real progress.

Problem 2: interoperability between the tools
-

I am part of the Python Modules team who wants to switch to git but not
all contributors are using the same git helper tools and yet we would like
to all work together on the same repositories without forcing everybody
to use the same helper tool (habits are hard to change).

We can't just let each maintainer use the default layout suggested
by his preferred helper tool and the defaul tagging scheme, we have to
define some common layout for the whole team. Then it matters less if
people are using git-buildpackage or git-dpm or gitpkg.

It might be awkward at times but at least there is some consistency among
the team, and the few problems that will arise will be occasions to
improve the tools.

But we have to define the common layout to use and this discussion should
hopefully solve this too.

Problem 3: making it easier for new contributors
-

While I can appreciate the versatility of gitpkg, new contributors
are looking for guidance and clear instructions. It's difficult to give
those when we have zero common ground on how we manage our git
repositories within our project.

While I don't see us converging on any single helper tool right now,
it's important to start taking steps that brings them closer so that
we can give more useful explanations to newcomers.

and so that they
can get started 

 Likewise, it's not clear to me that tools other than gitpkg are
 actually interchangeable, because they weren't designed to be from
 the outset and rely on magic being committed into the repo to work.
 
 I don't really see how some naming conventions can fix that either.

Naming conventions won't fix that but it's still a pre-requisite
to be able to fix the tools that (unlike gitpkg) voluntarily set (by
default) more constraints on the expected layout of the repository.

 Maybe if you start by detailing the problems, we will be able to
 see some better solutions that actually achieve your real goals
 and result in real improvements to the tools that created them.

Let's see!

  Fine if the other tools do not need anything like that. But who knows,
  maybe you will want to enhance git-debcherry to not only update
  debian/patches/ but also store the corresponding git branch for long-term
  storage. In which case, you will already have a recommended tag name
  for this purpose :-)
 
 Why would you want to do that?

To share them with upstream in a form ready for merge (or more practical
for review/analysis, etc.).

   I am a bit curious about what patches you think you're going to need
   to make to make things comply with this plan?
  
  For gitpkg? I don't know. See above, maybe it's just documentation update.
 
 I meant for the other tools.  You said you thought they were going to
 need patching for this and were planning to help with that.

Well git-buildpackage obviously needs quite some update:
- it complains when you're trying to build as soon as you're not in a
  branch named master (that can be overridden with --git-ignore-branch
  or setting --git-debian-branch)
- it hardcodes the debian/ prefix for tags instead of retrieving it 
dynamically
- it uses the upstream branch

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Tobias Frost



 I think you need to be more explicit about the implications for `3.0
 (quilt)' format packages.  Something like:

If the git tree contains debian/format specifying `3.0 (quilt)',
the git tree must also contain debian/patches/series and all the
patch files contained within it.  Furthermore, the tree should be
in the `patches applied' state.  (This means that every change to
upstream files is represented twice: once in the contents of that
very file in the git tree, and once as a hunk in one of the
debian/patches.  These two representations must be in step.)

 And you should add:

The packaging branch should not contain a `.pc' directory.


Maybe I got the above wrong: you mean patch applied after e.g quilt push -a ?
I think, removing .pc then would be a bad idea as quilt would be confused?

--
tobi


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/4afd0ddf4225fa49db35803d91c5a451.squir...@isengard.geekcommandos.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Tobias Frost

On Tue, 11 Nov 2014, Raphael Hertzog wrote:

First, I'm a fan of trying to ease the workflow for all by having some
standardisation / best-practice recommendation/documentation.
Kudos to the initiattors!

QUESTION: some people have argued to use debian/master as the latest
packaging targets sometimes sid and sometimes experimental. Should we
standardize on this? Or should we explicitly allow this as an alternative?

Given the feedback received, and the case of derivatives that don't have a
stable name for their development releases, I propose to update the
Packaging branches and tags section with the content below. Does
that sound like an improvement?

Packaging branches and tags
===

In general, packaging branches should be named according to the codename
of the target distribution. We differentiate however the case of development
releases from other releases.

For development releases

Packages uploaded to the current development release should be prepared
in a `vendor/master` branch.

However, when multiple development releases are regularly used (for
example `unstable` and `experimental` in Debian), it is also accepted to
name the branches according to the codename or the suite name of the
target distribution (aka `debian/sid` or `debian/unstable`, and
`debian/experimental`). Those branches can be short-lived (i.e. they exist
only until they are merged into `vendor/master` or until the version in
the associated repository is replaced by the version in `vendor/master`)
or they can be permanent (in which case `vendor/master` should not
exist).

NOTE: The Git repository listed in debian/control's `Vcs-Git` field should
have its HEAD point to the branch where new upstream versions are being
packaged (that is one of the branches associated to a development release).
The helper tools that do create those repositories should use a command
like `git symbolic-ref HEAD refs/heads/debian/master` to update HEAD
to point to the desired branch.

For stable releases
---

When a package targets any release that is not one of the usual
development releases (i.e. stable releases or a frozen development
release), it should be prepared in a branch named with the codename of the
target distribution. In the case of Debian, that means for example
`debian/jessie`, `debian/wheezy`, `debian/wheezy-backports`, etc. We
specifically avoid suite names because those tend to evolve over time
(stable becomes oldstable and so on).

Security updates and stable updates are expected to be handled on
the branch of the associated distribution. For example, in the case of
Debian, uploads to `wheezy-security` or `wheezy-proposed-updates`
are prepared on the `debian/wheezy` branch. Shall there be a need to
manage different versions of the packages in both repositories, then
the branches `debian/wheezy-security` and `debian/wheezy-updates`
can be used.

Tagging package releases

When releasing a Debian package, the packager should create and push
a signed tag named `vendor/version`. For example, a Debian maintainer
releasing a package with version 2:1.2~rc1-1 would create a tag named
`debian/2%1.2_rc1-1` whereas an Ubuntu packager releasing a package with
version 1.3-0ubuntu1 would use `ubuntu/1.3-0ubuntu1`. The tags should
point to the exact commit that was used to build the corresponding upload.

One point came to my mind: NMUs
Can we maybe add some words what would be best practice to handle NMUs?
I think just applying them to debian/sid (or whatever we call it) won't work
in every case, as the packaging repository might have already
(not-for-NMUs-suitableable) commits. I saw even new upstream releases sitting
there, with an maintainer (pseudo)MIA*.
So it is not guaranteed that NMUs can be directly applied to debian/sid and
worst case lots of stuff needs to be reverted before one can start on the NMU
and maybe restored later, rinse-repeat on a later NMU. And if the NMU is
cancelled, there will be lots of noise in the commit log for non-existing
releases.
For the maintainer ack'ing the NMU could be then simple as an merge.

So may I propose to have some procedure recommended for NMUs? (or at least for
such case where it is not easy possible to work directly with debian/sid)
Maybe add the intended NMU-version to it. This branch could also be
short-lived.

* responsive in email, but still non acting 6months

--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
https://lists.debian.org/a8868548708f26ab17b0f2eda6b61ade.squir...@isengard.geekcommandos.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Tobias Frost

2014-11-12 10:28 GMT+01:00 Raphael Hertzog hert...@debian.org:
On Wed, 12 Nov 2014, Mathieu Parent wrote:

OK. Makes sense. The unstripped upstream can then live in an
non-namespaced branch if needed (this is not my usual workflow but should be
possible).

On Wed, 12 Nov 2014, Mathieu Parent wrote:
Maybe a short note would be good then? (but I don't know how to write it).

I suggest this:

--- a/web/deps/dep14.mdwn
+++ b/web/deps/dep14.mdwn
@@ -230,6 +230,17 @@ non-patchable data), you can do so but you should then
document
this in `debian/README.source` along with some explanations of the tool to
use to build the package.

+About repacked upstream sources
+---
+
+When the upstream sources needs to be repacked (for example to drop some
+non-DFSG compliant files), then the branches and tags under the
+`upstream/` prefix should actually contain the repacked sources. +
+How the problematic files are pruned does not matter. What matters is that
+what is stored in the `upstream/*` namespace are the sources that package
+maintainers are effectively using.
+
Managing debian/changelog
-

Does this meet your expectation?

Cheers,

Should we add a word or two (some warning/reminder) how to handle removing
non-free content, especially if (Debian thinks) that the removed files are not
distributeable?
To avoid that we actucally distribute them in some branch without recognizing.

--
tobi

--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
https://lists.debian.org/4dafed01424b46f00229d4e894002d66.squir...@isengard.geekcommandos.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Thibaut Paumard

Le 12/11/2014 12:25, Raphael Hertzog a écrit :
 Hi,
 
 On Wed, 12 Nov 2014, Thibaut Paumard wrote:
 I see nothing about whether the debian branch should contained the
 unpacked or the unpacked *and* patched sources, and whether to ship the
 .pc directory.
 
 That's a volunteer choice at this stage. We have different workflows
 that are worth supporting and they differ at this level of details.

Dear Raphaël,

I see that some people feel very strongly that the source should be in
patch-applied state. Some tools require it. I am very strongly of the
opposite opinion (for reasons detailed below). You are right that this
DEP should not mandate one over the other (actually this DEP is a best
practices document, so strictly speaking it doesn't mandate anything).

However, we should perhaps strongly recommend that this choice be
documented in debian/README.sources.

Not that it's very relevant, to each his own, but since I see advocacies
for the patch-applied state. I'm not trying to convince anyone, but here
is why I chose patch-unapplied:

We have a very nice source package format with 3.0 (quilt). In this
format, we simply have the pristine upstream source, with an additional
debian directory that contains all the packaging stuff. Features are
nicely separated as feature patches.

I think the only workflow that newcomers and NMUers should be required
to learn is the one that involves quilt, they should not be expected to
learn (e.g.) dgit in addition. Also, we need to document the
debian/patches à la DEP3. So the way our modifications to the upstream
source are stored should be in debian/patches/, independent on whether
we use the tar.gz archives or svn or git.

If these modifications are stored already in debian/patches,
representing them also in git in the upstream code is a pointless
duplication of the same information. With the patch-unapplied scheme, I
can do:
  git diff upstream master
and check that there is nothing changed above the debian/ directory.

Likewise, I can do
  git diff debian/1.0.0-1 debian/1.0.0-2
and check the changes that will be represented in the source package.

So to me, the patch-unapplied scheme is easier to understand and less
error prone.

Kind regards, Thibaut.



signature.asc
Description: OpenPGP digital signature

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Guido Günther

On Thu, Nov 13, 2014 at 09:46:13AM +0100, Raphael Hertzog wrote:
[..snip..]
 Well git-buildpackage obviously needs quite some update:
 - it complains when you're trying to build as soon as you're not in a
   branch named master (that can be overridden with --git-ignore-branch
   or setting --git-debian-branch)
 - it hardcodes the debian/ prefix for tags instead of retrieving it 
 dynamically
 - it uses the upstream branch by default in git-import-orig while it should 
 use
   upstream/latest if we follow the current version of the DEP

This mostly means changing the defaults to things we already recommend
in the gbp manual in a similar spirit though not using the exact same
branch names yet as well as using the output of

dpkg-vendor --query Vendor

for the tag prefix.
Cheers,
 -- Guido


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141113101016.ga2...@bogon.m.sigxcpu.org

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Ian Jackson

Thibaut Paumard writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
repositories):
 [patches applied vs unapplied]
 
 However, we should perhaps strongly recommend that this choice be
 documented in debian/README.sources.

I think it would be better to document this by the use of different
branch names.  It is, after all, possible to make both patches-applied
and patches-unapplied branches for the same package.  For example, if
a package's maintainer team use patches-unapplied, a dgit user will
still see patches-applied.  A note in README.sources is (a) in the
wrong place because it attaches not to the particular git branch but
to the whole contents and (b) not machine-readable.

 I think the only workflow that newcomers and NMUers should be required
 to learn is the one that involves quilt, they should not be expected to
 learn (e.g.) dgit in addition. [...]

I certainly don't think people should be expected to learn dgit in
addition to other tools.  I am trying to get to the point where
1. they can learn dgit _instead_ of _all_ other tools[1]  2. no-one else
needs to know that this is going on unless they decide to start using
dgit too.

[1] I'm assuming they know git already, and that they're trying to do
NMUs or be a derivative (public or private).

Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/21604.34838.928787.815...@chiark.greenend.org.uk

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Ian Jackson

Tobias Frost writes (Re: RFC: DEP-14: Recommended layout for Git packaging 
repositories):
 [Ian:]
  And you should add:
 The packaging branch should not contain a `.pc' directory.
 
 Maybe I got the above wrong: you mean patch applied after e.g
 quilt push -a ?  I think, removing .pc then would be a bad idea as
 quilt would be confused?

dpkg-source seems to cope all right.  I think that .pc-less
patches-applied branches are not intended to be manipulated with
quilt.

We're trying to document best practice.  Does anyone have .pc-ful
patches-applied branches (not counting dgit, which is going to change) ?

ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/21604.34985.432393.608...@chiark.greenend.org.uk

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Matthias Urlichs

Hi,

Thibaut Paumard:
 We have a very nice source package format with 3.0 (quilt).

IMHO this format is very nice if you have some opaque upstream,
and a debian/ directory that's under your control.

This restriction does not apply to a DVCS like git. Moreover, git already
has built-in mechanisms to do all of this. People who are used to otherwise
working with git may not see any sense in learning another way of doing
essentially the same thing.

 If these modifications are stored already in debian/patches,
 representing them also in git in the upstream code is a pointless
 duplication of the same information.

No. The git representation is a strict superset of the information in
debian/patches, as it includes patch history that can be actually reasoned
about.

 Likewise, I can do
   git diff debian/1.0.0-1 debian/1.0.0-2
 
If this includes a changed patch, the result of this command is a
patch-of-a-patch, which I find to be completely unreadable.

 and check the changes that will be represented in the source package.

If your changes are applied to the git tree instead of being stored in
debian/patches, this command will do the exact same thing (other than
patch-of-a-patch changes).

-- 
-- Matthias Urlichs


signature.asc
Description: Digital signature

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Scott Kitterman

On Thursday, November 13, 2014 01:32:04 Sam Hartman wrote:
 Hi.
 I've read the original proposal and believe it is generally going in the
 right direction.
 
 things I liked:
 
 * didn't pick between dgit/git-dpm/git-pq; documented the common parts
 
 * Seemed to really focus on one clear scope.
 
 * Discouraged overlay packaging.
 I've tried to read the arguments, and I'm unconvinced that should be a
 recommended practice.
 I'd prefer to simply rule it out of scope: this proposal describes how
 to package debian and upstream sources together.  It does not apply to
 other cases and does not forbid or recommend them.
 It doesn't apply to svn.  It doesn't apply to overlay packaging.

If the scope is narrower than Debian as a whole, then the title of the DEP 
needs to change.  Maybe:

Recommended layout for Git packaging repositories which also contain upstream 
sources

Scott K


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/2509218.QQGvL7qbEk@scott-latitude-e6320

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Ron

On Thu, Nov 13, 2014 at 09:46:13AM +0100, Raphael Hertzog wrote:
 [ I skip the more detailed discussions on naming conventions to
   concentrate on your higher level questions for now ]

Agreed, if we solve the tricky problems, that part is mostly just
yak shaving (and if we can't, it's probably mostly irrelevant ...)

 On Thu, 13 Nov 2014, Ron wrote:
  Sure, I understood those were your goals.
  
  What I haven't seen, and what I'm asking for, is an actual detailed
  rationale describing the actual detailed problem(s) that you think
  these goals will be a remedy for.
 
 Problem 1: the derivatives
 --
 
 So I am a Kali Linux contributor. We use git repos to maintain all our
 packages and we use git-buildpackage.

I guess the first question there is what were the arguments put forward
for deciding to 'standardise' on gbp?  If there wasn't one, maybe that's
an argument you should have (and if there was, maybe it's one to revisit :)

If you have a clearer idea now of the problems you are facing, it might
make properly evaluating things that avoid those problems easier.

 Most of the Kali contributors
 are not long-term Debian contributors, I write documentation so that
 they can contribute to Kali (while basing our work on Debian).
 
 To make this manageable I opted to always use a workflow based on
 git-import-orig. Even when Debian has its own git repo, we start
 from the released source packages because that's the only level of
 uniformity that we can rely on. And it's a pity because if we could
 build on Debian's git repo, it also means that our work would be easier
 to merge for the Debian maintainer.

That's not a totally terrible way to kick off a new repo when there
isn't an existing one.  I wrote git-debimport to build a history from
just existing Debian source packages when that's all you have, and
debsnap (in devscripts) originally got written for use with it to
collect the whole set from snapshot when you didn't already have them.
And you can fairly easily splice a repo created with it to a real
upstream one to continue maintenance in a more sensible way from there.

But it is a kind of terrible way to continue maintaining them if there
is a repo.  Unfortunately I think that if you make uniformity the
overarching consideration here, you've basically doomed yourself to
failure from the outset.

Even if everyone did stick to the conventions already discussed
(which in reality, they won't) there's still far too many degrees of
(quite necessary) freedom to really approach a just follow these
three easy steps kind of uniformity.  And even if you got close to
achieving that for the debian branches, the (again necessary)
variability between upstream repos is going to be even greater.

I think at some point you're going to have to rely on real human
intelligence to be able to look at a repo and form their own
understanding of its structure.  Most really aren't all that
complicated (however much they vary between each other), and in
the worst case you can always actually ask the 'upstream'
maintainer to explain anything that is unclear.

That's going to get annoying really fast if someone new asks me
really basic questions something like that every week.  But if you
do this right it's also something that in the worst case someone
should only need to ever ask once, because they can document what
they learned for the next person if there are no 'dedicated'
maintainers for individual packages, and this is a needed thing.

I don't think I've ever had to ask an upstream how does your repo
work though.  So you possibly could also handle this internally
with a few knowledgeable mentors too.


 They could add the Kali repo as a
 remote (without fearing any conflict in terms of tags, branch names)
 and just merge or cherry-pick as appropriate.

All that said, this part however is not a problem at all.  If you've
branched your repo off the 'upstream' one (so you share its history)
then there's never going to be a conflict between branch names.

You can have a Kali repo, cloned from a Debian repo, cloned from
an upstream repo, and *all* of those repos could have their own
separate and distinct branch called master, and still there would
be no conflict.

git is already going to namespace them so when you add the remotes
the branch refs will be (for remotes named upstream, debian, kali):

remotes/upstream/master
remotes/debian/master
remotes/kali/master

What you name those branches if you check them out locally is totally
up to you.  The local names need to be unique, but you can:

 $ git checkout -t debian/master -b debian-master
 $ git checkout -t kali/master -b master

And that will work just fine.


Tag names aren't quite so forgiving.  But realistically, even if
you simply name your tags v$version in all of upstream, debian,
and kali, then if you actually have conflicting names you were
already in deep trouble anyway, because now you have a kali
package with *exactly* the same version as

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Ralf Jung

Hi,

 I think the only workflow that newcomers and NMUers should be required
 to learn is the one that involves quilt, they should not be expected to
 learn (e.g.) dgit in addition. [...]
 
 I certainly don't think people should be expected to learn dgit in
 addition to other tools.  I am trying to get to the point where
 1. they can learn dgit _instead_ of _all_ other tools[1]  2. no-one else
 needs to know that this is going on unless they decide to start using
 dgit too.

I whole-heartedly agree - I still think quilt is obscure, and since I
only maintain few packages with little upstream activity, I doubt I will
ever learn it enough to stop reading the manpage each time I need it. On
the other hand, git is already widely known in the open source and free
software community. So being able to use dgit and nothing else, would
significantly reduce the barrier of entry. Once there is a version that
DMs can use, I'll give it a try :)

Kind regards,
Ralf


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/5464c6df.2090...@ralfj.de

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Marco d'Itri

On Nov 13, Raphael Hertzog hert...@debian.org wrote:

 I am part of the Python Modules team who wants to switch to git but not
 all contributors are using the same git helper tools and yet we would like
 to all work together on the same repositories without forcing everybody
 to use the same helper tool (habits are hard to change).
I do not understand this focus on helper tools: most of my packages have 
a debian/gbp.conf file, but I never use gbp except sometimes for 
importing a new upstream release and everything still works fine.
I just build my packages with debuild as usual.

-- 
ciao,
Marco


pgp7AthHJH1tI.pgp
Description: PGP signature

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Bernhard R. Link

* Raphael Hertzog hert...@debian.org [14 22:26]:
 Helper tools should usually rely on the output of `dpkg-vendor --query vendor`
 to find out the name of the current vendor. The retrieved string must be
 converted to lower case. This allows users to override the current vendor
 by setting `DEB_VENDOR` in their environment (provided that the vendor
 information does exist in `/etc/dpkg/origins/`).

Is using the vendor you use git on a good default for the vendor code
you are currently working on? In my experience those two are quite
unrelated.

 Version mangling
 
 
 When a Git tag needs to refer to a specific version of a Debian package,
 the Debian version needs to be mangled to cope with Git's restrictions.
 The colon (`:`) needs to be replaced with a percent (`%`), and the tilde
 (`~`) needs to be replaced with an underscore (`_`).

Is there a previous case of encoding colons with percent signs?
If it is inventing a third way instead of using on of the existing ones,
is sounds like a bad idea.


 Packaging branches and tags
 ===
 
 The Git repository listed in debian/control's `Vcs-Git` field should
 usually have its HEAD point to the branch corresponding to the
 distribution where new upstream versions are usually sent. For Debian,
 it will usually be `debian/sid` (or sometimes `debian/experimental`).

I think this should alternatively allow for Vcs pointing to that branch
instead. It sometimes makes sense to put the debian package development
branches in the same repository as the upstream branch, where HEAD might
be reserved for the upstream branch.

 When releasing a Debian package, the packager should create and push
 a signed tag named `vendor/version`.

I'd advocate s/signed/(possibly signed)/, as I'm not ready to sign such
a wildcard with any valuable key and getting another very-low-security
key just to have signed keys to match this proposal sounds like waste.


Tags are only based on versions are also quite hard to shuffle around.
I'd strongly suggest adding the name of the source package to those,
otherwise accessing multiple packages in on repository causes a big
mess. (git keeps branches in the per-origin remote namespace when
fetching stuff, tags only have one global namespace, local and remote).

 Managing upstream sources
 =

 Importing upstream release tarballs in Git
 --

 If the Git workflow in use imports the upstream sources from released
 tarballs, this should be done under the upstream namespace. By default,
 the latest upstream version should be imported in the `upstream/latest`
 branch and when packages for multiple upstream versions are maintained
 concurrently, one should create as many upstream branches as required.

 Their name should be based on the major upstream version tracked:
 for example when upstream maintains a stable 1.2 branch and releases
 1.2.x minor releases in that branch, those releases should be imported
 in a `upstream/1.2.x` branch (the .x suffix makes it clear that we are
 referring to a branch and not to the tag corresponding the upsteam 1.2
 release). If the upstream developers use codenames to refer to their
 releases, the upstream branches can be named according to those codenames.

 Helper tools should detect when the upstream version imported is lower
 than the latest version available in `upstream/latest` and should offer
 either to create a new branch (based on the highest version available in
 the history that is still smaller than the imported version) or to pick
 another pre-existing upstream branch.

I'm very sceptical about this part. I'm not sure it is worth
standardizing and it seems quite a lot of work to implement something
like that (which I expect I'll have to override more often than not
if using such an helper).


 Importing upstream releases from Git tags
 -

 When the packaging branches are directly based on the upstream Git
 branches, upstream usually also provide proper Git tags that can be reused
 for official releases.

 If helper tools have to identify the upstream commit corresponding to a
 given upstream release, they should be able to find it out by looking up
 the following Git tags: `upstream/version`, `version` and
 `vversion`.

 The `upstream/version` tag would be created by the package maintainer
 when needed: for example when it does a release based on a Git snapshot or
 when the tag naming scheme used by upstream is not following the above
 rules.

That lacks possibility three: importing tar content on top of upstream
branches. And again adding the source package name in there would be
nice.

 Native packages
 ===
 
 The above conventions mainly cater to the case of non-native packages,
 that is when the upstream developers and the package maintainers are
 not the same set of persons.
 
 When upstream is Debian (or one of its derivative), the upstream vendor

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Richard Hartmann

On Tue, Nov 11, 2014 at 10:51 PM, Henrique de Moraes Holschuh
h...@debian.org wrote:

 Please allow debian/master as an alternative.  It fits with the general git
 usage of master, it fits with the workflow of several packages, where you
 do experimental-unstable, and it is not going to surprise anyone that has
 even a passing knowledge of the usual git conventions.

 [...] I must say I *like* it.  Well done!  I am
 especially happy about the way it respects the usual git usage conventions,
 this will ease its adoption a lot.

Not much to add here. +1


Richard


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/CAD77+gSVO8=lospehnra+0jn5287eb166rb3+iclpu7tmym...@mail.gmail.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-13 Thread Richard Hartmann

On Wed, Nov 12, 2014 at 4:04 AM, Marco d'Itri m...@linux.it wrote:
 Whatever the decision will be on debian/master, I think that master
 should be at the very least an option.

I.e. a debian-only repo? That's what pabs also seems to prefer so it
probably makes sense to support/allow both.

FWIW, I strongly disagree with putting any packaging into master as
this makes upstreaming different packaging branches a tad harder,
but each to their own.



Richard


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/cad77+grzh1dvh1sebb3ppnp0pcn6hsh3h_aojqxyvwcjjny...@mail.gmail.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-12 Thread Jonathan Dowland

On Wed, Nov 12, 2014 at 01:13:39AM +0100, Matthias Urlichs wrote:
 Please discourage the use of pristine-tar. The format is fragile and can
 suffer from bit rot.

I am not personally interested in pristine-tar, but I don't think this is the
right place to take a stance on its use.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141112080255.ga22...@chew.redmars.org

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-12 Thread Jonathan Dowland

On Wed, Nov 12, 2014 at 03:38:59PM +0800, Paul Wise wrote:
 Personally I wouldn't use anything other than debian-only repos, at
 least for those where I have a choice. I also actively avoid
 contributing to packages that don't use such repos.

And I'm the exact opposite.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141112080511.gb22...@chew.redmars.org

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-12 Thread Gergely Nagy

 Jonathan == Jonathan Dowland j...@debian.org writes:

Jonathan On Wed, Nov 12, 2014 at 03:38:59PM +0800, Paul Wise wrote:
 Personally I wouldn't use anything other than debian-only repos, at
 least for those where I have a choice. I also actively avoid
 contributing to packages that don't use such repos.

Jonathan And I'm the exact opposite.

Which is fine - both layouts have their pros and cons and use cases.
There are some cases where I prefer upstream+debian in the same repo,
too. Hence, providing both as options, without a clear recommendation of
one of them, in DEP-14 would benefit us all, in my opinion.

-- 
|8]


signature.asc
Description: PGP signature

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

Hi,

On Tue, 11 Nov 2014, Iustin Pop wrote:
  Packaging branches and tags
  ===
  
  Packaging branches should be named according to the codename of the
  target distribution. In the case of Debian, that means for example
  `debian/sid`, `debian/jessie`, `debian/experimental`,
  `debian/wheezy`, `debian/wheezy-backports`, etc. We specifically avoid
  suite names because those tend to evolve over time (stable becomes
  oldstable and so on).
  
  The Git repository listed in debian/control's `Vcs-Git` field should
  usually have its HEAD point to the branch corresponding to the
  distribution where new upstream versions are usually sent. For Debian,
  it will usually be `debian/sid` (or sometimes `debian/experimental`).
 
 I find this paragraph confusing. With gbp, this is where new Debian
 developments are made, and new upstream versions are sent to
 upstream/xxx. Or do you mean something else here?

Is it clearer if I rewrite it this way ?

« The Git repository listed in debian/control's `Vcs-Git` field should
usually have its HEAD point to the branch where new upstream versions
are being packaged. For Debian, it will usually be `debian/sid` (or
sometimes `debian/experimental`) »

 Interesting. Assuming a normal Debian package that has just a few
 backports (as opposed to every sid release being backported), and which
 imports only upstream tarballs/snapshots (not the whole history), I
 expect that a high proportion of the commits would happen on this
 branch. In which case, why not make it 'master', without debian/ ? Is it
 (only) in order to cleanly support multiple vendors?

Henrique answered to that. The non-prefixed namespace is dedicated
to upstream development.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141112082156.ga27...@home.ouaza.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-12 Thread Matthias Urlichs

Hi,

Jonathan Dowland:
 On Wed, Nov 12, 2014 at 01:13:39AM +0100, Matthias Urlichs wrote:
  Please discourage the use of pristine-tar. The format is fragile and can
  suffer from bit rot.
 
 I am not personally interested in pristine-tar, but I don't think this is the
 right place to take a stance on its use.
 
Then we should either remove the paragraph entirely, or at least mention
the danger of bit rot and that it's unwise to rely on being able to recover
the tarfile (long term).

Jonathan Dowland:
 On Wed, Nov 12, 2014 at 03:38:59PM +0800, Paul Wise wrote:
  Personally I wouldn't use anything other than debian-only repos, at
  least for those where I have a choice. I also actively avoid
  contributing to packages that don't use such repos.
 
 And I'm the exact opposite.

FWIW: Me too. Debian-only is, to me, quite annoying, and a remnant of a
workflow that was once a necessity (with CVS/SVN). Not so with git.
-- 
-- Matthias Urlichs


signature.asc
Description: Digital signature

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Wed, 12 Nov 2014, Matthias Urlichs wrote:
  If the package maintainers use the pristine-tar tool to efficiently store
  a byte-for-byte copy of the upstream tarballs, this should be done in the
  `pristine-tar` branch.
 
 Please discourage the use of pristine-tar. The format is fragile and can
 suffer from bit rot.

The DEP will neither encourage and discourage its use. It only mentions
that if a maintainer is using it, it should store pristine-tar data
in the pristine-tar branch.

FWIW, I do use pristine-tar and I find it really useful even though I know
that some day my old pristine-tar data will be unusable. It's not a big
deal since we have snapshot.debian.org and archive.debian.org for those
cases. The point of the pristine-tar data is to help contributors who
are cooperating right now, it's not to make sure we can recreate the same
old .orig.tar.gz in 10 years from now (even though it would have been nice
if it achieved this).

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141112082833.gb27...@home.ouaza.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-12 Thread Ben Finney

Paul Wise p...@debian.org writes:

 On Wed, Nov 12, 2014 at 3:34 PM, Gergely Nagy wrote:

  I'd like to note that there are very good reasons for a debian-only,
  overlay-style packaging repository too. This section should, in my
  opinion, at least acknowledge that, and briefly mention it as an
  option. I find it a bit sad that it was outright discouraged.

 Personally I wouldn't use anything other than debian-only repos, at
 least for those where I have a choice. I also actively avoid
 contributing to packages that don't use such repos.

Agreed. The debian-packaging-in-separate-repo approach is fully
supported by our build tools, and should be at least an equal citizen in
this standard, IMO.

-- 
 \“The problem with television is that the people must sit and |
  `\keep their eyes glued on a screen: the average American family |
_o__) hasn't time for it.” —_The New York Times_, 1939 |
Ben Finney


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/85h9y46cgg@benfinney.id.au

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Tue, 11 Nov 2014, Henrique de Moraes Holschuh wrote:
 In fact, I was quite non-amused by the initial versions of this idea, but
 reading this latest version, I must say I *like* it.  Well done!  I am
 especially happy about the way it respects the usual git usage conventions,
 this will ease its adoption a lot.

Thank you! I'm pleased to see that I managed to draft something reasonable.

 It does fail to mention security, and stable-proposed branches.  We can just
 leave those to maintainer's discretion, of course.

I believe that those are not really needed in most cases. You rarely have
conflicting updates for -security and -proposed-updates at the same time.
But of course we can say something along this:

--- a/web/deps/dep14.mdwn
+++ b/web/deps/dep14.mdwn
@@ -73,6 +73,14 @@ target distribution. In the case of Debian, that means for 
example
 suite names because those tend to evolve over time (stable becomes
 oldstable and so on).
 
+Security updates and stable updates are expected to be handled on
+the branch of the associated distribution. For example, in the case of
+Debian, uploads to `wheezy-security` or `wheezy-proposed-updates`
+are prepared on the `debian/wheezy` branch. Shall there be a need to
+manage different versions of the packages in both repositories, then
+the branches `debian/wheezy-security` and `debian/wheezy-updates`
+can be used.
+
 The Git repository listed in debian/control's `Vcs-Git` field should
 usually have its HEAD point to the branch corresponding to the
 distribution where new upstream versions are usually sent. For Debian,


How does that sound ?

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141112084624.gc27...@home.ouaza.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Tue, 11 Nov 2014, Barry Warsaw wrote:
 One question: when this gets adopted, will individual maintainers or teams
 have to know which of the git packaging helpers a particular repository is
 using?  IOW, what happens if I were to use gbp-pq on a package that someone
 else had used git-dpm on?  Do we need some kind of convention to detect which
 helper is being used?  I wouldn't want to restrict choice by defining a
 standard Debian-wide (but it's okay within the context of a team).  I just
 want someone who walks up to a git repo to at least easily know which tools to
 use.

I don't know. My long term hope is that in this process we will get to a
situation where:
- either the tools are sufficiently interoperable that we don't have to
  care about this
- or one of tools emerges as standard supporting all the important
  workflows that people are using

 Each vendor uses its own namespace for its packaging related 
 Git branches and tags: `debian/*` for Debian, `ubuntu/*` for Ubuntu, and
 so on.
 
 My question is whether the vendor namespaces are overkill for the majority of
 packages, where there probably will be just one vendor (i.e. Debian).  Should
 DEP 14 allow for a simplified layout such as master, pristine-tar, upstream
 when there is no other vendor involved?  Allowing for master as an alternative
 to debian/sid or debian/master might simplify the common case.

I am rather opposed to this because it because it doesn't separate clearly
the namespaces for the upstream development and the namespace for the
Debian packaging.

Even when upstream doesn't use Git, you never know whether this won't
change at some point and while it's perfectly possible to rename branches
when needed, this operation is not something that is tracked in any way
in the git clones that all users might have (it's really a branch deletion
+ a new branch) so local configuration about which branch is tracked is
now broken and you have no safeguard against non-fast-forward change
at the time you rename the branches, and so on.

I really believe that always using the prefixed namespace is the correct
course of action.

 for the current Ubuntu development series.  If I needed to support older
 releases in either distro, then debian/wheezy or ubuntu/utopic would be good
 branches to use.  (Or IOW, what's the equivalent of debian/sid for Ubuntu?)

I was wondering that as well. For Ubuntu, it probably makes sense to use
ubuntu/master because the latest development release regularly changes and
it's not a good idea to alway update the branch.

So that is one reason more to allow usage of `vendor/master` as a packaging
branch.

 If the Git workflow in use imports the upstream sources from released
 tarballs, this should be done under the upstream namespace. By default,
 the latest upstream version should be imported in the `upstream/latest`
 branch and when packages for multiple upstream versions are maintained
 concurrently, one should create as many upstream branches as required.
 
 Similarly, is 'upstream' okay for the upstream branch?  It's a little simpler
 than upstream/latest, especially if you don't anticipate importing an other
 upstream branches.

And it's also compatible with the current gbp practice. I'm not set of
upstream/latest but given what I said above about renaming branches, I
still believe that it's best to have a default name that does not need to
be renamed when you want to add more upstream branches.

Or maybe we recommend upstream/latest by default but allow upstream
alone in the case when there are no other upstream branches tracked ?

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141112090254.gd27...@home.ouaza.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

[ Ccing the maintainers of git packaging helper tools ]

Hi Scott,

using your mail as an opportunity to explicity notify the respective
package maintainers of this ongoing DEP.

Guido, Bernhard, Ron, if you are not reading debian-devel, I would
like to bring your attention to a discussion that I recently started
about trying to standardize the layout of Git packaging repositories:
https://lists.debian.org/debian-devel/2014/11/msg00444.html
https://lists.debian.org/debian-devel/2014/11/threads.html#00444

This mail is part of this discussion.

On Tue, 11 Nov 2014, Scott Kitterman wrote:
 Who's going to do patches to existing tools (e.g. git-dpm is the one I use 
 and 
 care about) so they comply with this and similarly scripts to convert 
 existing 
 git repos to match this recommendation?
 
 It doesn't make much sense to have an standard unless there's also a plan to 
 implement using it.

My own idea was to involve the respective package maintainers in this
discussion (now done with this mail) and once we have an agreed-upon DEP
then to suggest a sprint (IRL or not) to update the respective tools.

I would be willing to participate to such a sprint and do my share
of the work (although I'm primarily using git-buildpackage and would
likely start with this helper tool).

Guido, Bernhard, Ron, would you be interested to attend such a sprint?

Are there other persons who would be interested to help us implement
the required changes?

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141112091527.ge27...@home.ouaza.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Wed, 12 Nov 2014, Mathieu Parent wrote:
 A paragraph about repacked upstream is needed. A lot of packages are
 currently stripped for minified JS, non-free additions, included RFCs,
 ... What would the upstream/1.x branch be then? Maybe add an
 upstream/1.x+debian branch?

Yeah, that was another open question I had.

There's no clear conventions about how to handle the removal of files from
upstream releases.

It looks like some people are using uscan/mk-origtaz to drop non-DFSG
files from the downloaded .tar.gz (via Files-Excluded in debian/copyright)
and then the upstream branch is directly the DFSG clean version.

But other persons are using dfsg branches where they handle the removal
of the problematic files and they merge those branches. So their process
is:
- update the dfsg branch by merging the upstream release
- merge the dfsg branch in the debian packaging branch

There are probably other workflows.


I'm not sure what's best. But given that the whole upstream namespaces
is a packaging artifact, it would seem natural that those branches contain
the upstream sources as we define them, i.e. with the non-free files already
stripped.

I would thus limit ourselves to stating that and not having any specific
recommendations on how people should get rid of the files.

Does that sound reasonable?

 Also, the vendor/* branches heads should be at a descendent commit of
 the corresponding upstream branch, diffing only by the debian dir.

Does that need to be explicitly stated?

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141112092823.gf27...@home.ouaza.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

Hi Gergely,

On Wed, 12 Nov 2014, Gergely Nagy wrote:
 Raphael When releasing a Debian package, the packager should create and 
 push
 Raphael a signed tag named `vendor/version`. For example, a Debian 
 maintainer
 Raphael releasing a package with version 2:1.2~rc1-1 would create a tag 
 named
 Raphael `debian/2%1.2_rc1-1` whereas an Ubuntu packager releasing a 
 package with
 Raphael version 1.3-0ubuntu1 would use `ubuntu/1.3-0ubuntu1`. The tags 
 should
 Raphael point to the exact commit that was used to build the 
 corresponding upload.
 
 Mmm... I disagree here too. I think following upstream tagging
 conventions would be the way to go here. So if upstream uses
 `package-version` tags, then debian releases would be tagged like
 `debian/foo-2%1.2_rc1-1`, if upstream is `foo-1.2rc1`.

And if upstream uses v1.2.3 we should use debian/v1.2.3-1?

Do you know many upstreams using foo-1.2.3 as tags?

I must say that I don't see the value that this brings. It might be
aestethically more pleasant when browsing the list of tags but for
everything else, it's just more painful. We lose the ability to
easily identify the commit of a given Debian release (instead
of a simple lookup operation, you have to scan all existing tags, and even
then if you have no clear mapping with the Debian version, you can't be
sure that you will find the correct tag).

Thus I don't think that this is a good idea.

 Raphael It is also important so that contributors are able to use the 
 tool of their
 Raphael choice to update the debian/patches quilt series: multiple 
 helper tools
 Raphael need the upstream sources in Git to manage this patch series as 
 a Git
 Raphael branch.
 
 I'd like to note that there are very good reasons for a debian-only,
 overlay-style packaging repository too. This section should, in my
 opinion, at least acknowledge that, and briefly mention it as an option.
 I find it a bit sad that it was outright discouraged.

I'm open to that, but IMO the only case where there are very good reasons
are the case where the upstream data is really huge and not easily
patchable anyway (i.e. the case of openarena-data that Simon Mc Vittie
described in https://lists.debian.org/debian-devel/2014/08/msg00582.html).

Are there other reasons that you consider good enough to impose the above
penalties on other possible contributors (i.e. making it impossible to use
gbp pq or similar tools to update debian/patches)?

 For the record, I used to hate that style, and was an advocate for
 storing upstream sources in the repo too. Then I started maintaining ~6
 packaging branches: three upstream versions, two packaging branch
 variants of each. The overlay style proved to be far superior in this
 case.

Can you elaborate on how it was superior?

 In short, I'd suggest having the DEP document both layouts, and
 recommend using one of them, while also recommending to document it in
 debian/README.source.

I have put this for now:

@@ -195,6 +203,12 @@ choice to update the debian/patches quilt series: multiple 
helper tools
 need the upstream sources in Git to manage this patch series as a Git
 branch.
 
+When you have good reasons to only store the `debian` packaging directory
+(for example when the uptream sources are really huge and contains mostly
+non-patchable data), you can do so but you should then document
+this in `debian/README.source` along with some explanations of the tool
+to use to build the package.
+
 Managing debian/changelog
 -


Does that sound better?

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141112094730.gg27...@home.ouaza.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-12 Thread Mattia Rizzolo

On Wed, Nov 12, 2014 at 10:02 AM, Raphael Hertzog hert...@debian.org wrote:
 for the current Ubuntu development series.  If I needed to support older
 releases in either distro, then debian/wheezy or ubuntu/utopic would be good
 branches to use.  (Or IOW, what's the equivalent of debian/sid for Ubuntu?)

 I was wondering that as well. For Ubuntu, it probably makes sense to use
 ubuntu/master because the latest development release regularly changes and
 it's not a good idea to alway update the branch.

Ubuntu has a fake devel suite (actually it's just a symlink to the
current development release, iirc:
http://archive.ubuntu.com/ubuntu/dists/devel/Release )

Maybe they can usa ubuntu/devel but also ubuntu/master sounds reasonable.

-- 
regards,
Mattia Rizzolo

GPG Key: 4096R/B9444540 http://goo.gl/I8TMB
more about me: http://mapreri.org
Launchpad User: https://launchpad.net/~mapreri
Ubuntu Wiki page: https://wiki.ubuntu.com/MattiaRizzolo


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/CAHKYmetF8uUHJ5d5+PDecmUzW0XStiMx-C=ywfwsepnyr-p...@mail.gmail.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-12 Thread Mathieu Parent

2014-11-12 10:28 GMT+01:00 Raphael Hertzog hert...@debian.org:
On Wed, 12 Nov 2014, Mathieu Parent wrote:
A paragraph about repacked upstream is needed. A lot of packages are
currently stripped for minified JS, non-free additions, included RFCs,
... What would the upstream/1.x branch be then? Maybe add an
upstream/1.x+debian branch?

Yeah, that was another open question I had.

There's no clear conventions about how to handle the removal of files from
upstream releases.

It looks like some people are using uscan/mk-origtaz to drop non-DFSG
files from the downloaded .tar.gz (via Files-Excluded in debian/copyright)
and then the upstream branch is directly the DFSG clean version.

But other persons are using dfsg branches where they handle the removal
of the problematic files and they merge those branches. So their process
is:
- update the dfsg branch by merging the upstream release
- merge the dfsg branch in the debian packaging branch

There are probably other workflows.

I'm not sure what's best. But given that the whole upstream namespaces
is a packaging artifact, it would seem natural that those branches contain
the upstream sources as we define them, i.e. with the non-free files already
stripped.

OK. Makes sense. The unstripped upstream can then live in an
non-namespaced branch if needed (this is not my usual workflow but
should be possible).

Maybe a short note would be good then? (but I don't know how to write it).

I would thus limit ourselves to stating that and not having any specific
recommendations on how people should get rid of the files.

Does that sound reasonable?

Yes.

Also, the vendor/* branches heads should be at a descendent commit of
the corresponding upstream branch, diffing only by the debian dir.

Does that need to be explicitly stated?

The What to store in the Git repository first paragraph is enough.

Regards

--
Mathieu

--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
https://lists.debian.org/cafx5sbzqyj_73b9nufvxegvwnbz+xtn-dnevwyzb8fzapxg...@mail.gmail.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-12 Thread Guillem Jover

Hi!

On Wed, 2014-11-12 at 15:38:59 +0800, Paul Wise wrote:
 On Wed, Nov 12, 2014 at 3:34 PM, Gergely Nagy wrote:
  I'd like to note that there are very good reasons for a debian-only,
  overlay-style packaging repository too. This section should, in my
  opinion, at least acknowledge that, and briefly mention it as an option.
  I find it a bit sad that it was outright discouraged.
 
 Personally I wouldn't use anything other than debian-only repos, at
 least for those where I have a choice. I also actively avoid
 contributing to packages that don't use such repos.

Exactly. In addition I also only tend to «git clone» native packages,
for anything else I just simply «apt-get source» and can be pretty sure
what I will get, because the alternatives are just annoying.

As a maintainer and upstream, I also find crawling the packaging repos
for many of the RPM based distros, Gentoo and BSDs port collections for
example, actually way more pleasant and clear than many of the Debian
packaging repos with packaging bits mixed with upstream code TBH, because
they only have the build recipes and possibly patches, so I don't need
to know their tools or path layouts to be able to find the packaging
bits, because they are just obviously there, in front of you, alone.

Not to mention the “unholy” practice of having to store autogenerated
stuff shipped only in release tarballs in a git repo! :)

I also fantasize sometimes of a day where the whole distribution would
be stored on VCSs (per package) with a debian-only layout, so that I
could have a local copy for the whole archive, taking only a couple of
GiB at most, instead of the monstrosity of a complete exploded sources
archive (according to http://sources.debian.net/stats/, that's currently
205 GiB, w/o including git history, which could easily double that
size); and don't tell me space is cheap, because that does not take
into account the downloading, nor the huge amount of wasted space if
you only want the packaging bits.

Even though it might be more convenient for the maintainer, in general
I find the mixed up repos to be a disservice to the rest of the world.

Regards,
Guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/2014111212.ga12...@gaara.hadrons.org

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-12 Thread Thibaut Paumard

Le 11/11/2014 22:26, Raphael Hertzog a écrit :
 Hello,
 
 following the initial discussion we had in August
 (https://lists.debian.org/debian-devel/2014/08/thrd2.html#00499), I have
 written a first draft of the Debian Enhancement Proposal that I suggested.
 It's now online at http://dep.debian.net/deps/dep14 and also attached
 below so that you can easily reply and comment.
 
 I have left one question where I have had conflicting feedback
 and I'm not sure what's best. Thus I will welcome a larger set of
 opinions on this specific question (search for QUESTION in the
 text).
 
 Are there things that are missing?


Hi,

I see nothing about whether the debian branch should contained the
unpacked or the unpacked *and* patched sources, and whether to ship the
.pc directory.

I personally ship the unpatched sources and don't ship the .pc directory.

Kind regards, Thibaut.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/5463418a.6060...@debian.org

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

On Tue, 11 Nov 2014, Raphael Hertzog wrote:
   QUESTION: some people have argued to use debian/master as the latest
   packaging targets sometimes sid and sometimes experimental. Should we
   standardize on this? Or should we explicitly allow this as an alternative?

Given the feedback received, and the case of derivatives that don't have a
stable name for their development releases, I propose to update the
Packaging branches and tags section with the content below. Does
that sound like an improvement?

Packaging branches and tags
===

In general, packaging branches should be named according to the codename
of the target distribution. We differentiate however the case of development
releases from other releases.

For development releases


Packages uploaded to the current development release should be prepared
in a `vendor/master` branch.

However, when multiple development releases are regularly used (for
example `unstable` and `experimental` in Debian), it is also accepted to
name the branches according to the codename or the suite name of the
target distribution (aka `debian/sid` or `debian/unstable`, and
`debian/experimental`). Those branches can be short-lived (i.e. they exist
only until they are merged into `vendor/master` or until the version in
the associated repository is replaced by the version in `vendor/master`)
or they can be permanent (in which case `vendor/master` should not
exist).

NOTE: The Git repository listed in debian/control's `Vcs-Git` field should
have its HEAD point to the branch where new upstream versions are being
packaged (that is one of the branches associated to a development release).
The helper tools that do create those repositories should use a command
like `git symbolic-ref HEAD refs/heads/debian/master` to update HEAD
to point to the desired branch.

For stable releases
---

When a package targets any release that is not one of the usual
development releases (i.e. stable releases or a frozen development
release), it should be prepared in a branch named with the codename of the
target distribution. In the case of Debian, that means for example
`debian/jessie`, `debian/wheezy`, `debian/wheezy-backports`, etc.  We
specifically avoid suite names because those tend to evolve over time
(stable becomes oldstable and so on).

Security updates and stable updates are expected to be handled on
the branch of the associated distribution. For example, in the case of
Debian, uploads to `wheezy-security` or `wheezy-proposed-updates`
are prepared on the `debian/wheezy` branch. Shall there be a need to
manage different versions of the packages in both repositories, then
the branches `debian/wheezy-security` and `debian/wheezy-updates`
can be used.

Tagging package releases


When releasing a Debian package, the packager should create and push
a signed tag named `vendor/version`. For example, a Debian maintainer
releasing a package with version 2:1.2~rc1-1 would create a tag named
`debian/2%1.2_rc1-1` whereas an Ubuntu packager releasing a package with
version 1.3-0ubuntu1 would use `ubuntu/1.3-0ubuntu1`. The tags should
point to the exact commit that was used to build the corresponding upload.

-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141112112021.gk27...@home.ouaza.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

Hi,

On Wed, 12 Nov 2014, Thibaut Paumard wrote:
 I see nothing about whether the debian branch should contained the
 unpacked or the unpacked *and* patched sources, and whether to ship the
 .pc directory.
 
 I personally ship the unpatched sources and don't ship the .pc directory.

That's a volunteer choice at this stage. We have different workflows
that are worth supporting and they differ at this level of details.

The only thing that the DEP recommends is that the cloned tree be usable
with `dpkg-buildpackage -b -us -uc`. I think it's enough at this point.

But if we can come up with a consensus that we should say something
about this, I'm happy to update the document.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141112112537.gl27...@home.ouaza.com

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-12 Thread Ron


Hi Raphael,

On Wed, Nov 12, 2014 at 10:15:27AM +0100, Raphael Hertzog wrote:
 Hi Scott,
 
 using your mail as an opportunity to explicity notify the respective
 package maintainers of this ongoing DEP.
 
 Guido, Bernhard, Ron, if you are not reading debian-devel, I would
 like to bring your attention to a discussion that I recently started
 about trying to standardize the layout of Git packaging repositories:
 https://lists.debian.org/debian-devel/2014/11/msg00444.html
 https://lists.debian.org/debian-devel/2014/11/threads.html#00444
 
 This mail is part of this discussion.

Thanks for the heads up on this, I wasn't aware of this before now.
Please keep me in the CC for anything people would like me to see
and/or reply to.

 On Tue, 11 Nov 2014, Scott Kitterman wrote:
  Who's going to do patches to existing tools (e.g. git-dpm is the one I use 
  and 
  care about) so they comply with this and similarly scripts to convert 
  existing 
  git repos to match this recommendation?
  
  It doesn't make much sense to have an standard unless there's also a plan 
  to 
  implement using it.
 
 My own idea was to involve the respective package maintainers in this
 discussion (now done with this mail) and once we have an agreed-upon DEP
 then to suggest a sprint (IRL or not) to update the respective tools.

So I've got a bunch of reading to do to fully catch up on everything
you've proposed here so far, but after a quick first pass, here's a
few first impression comments:

I think you probably need to be careful of overspecifying this.  One
of the beautiful things about git is that if you understand how the
DAG works, it very easily lends itself to creating different ways of
working that are particularly suited to the needs of a particular
project.  So while there are some rough guidelines as to some best
practices for certain situations, it invariably works best when the
workflow is shaped to suit the project and its developers, not when
the project and the developers are rigidly jammed into a framework
they need to contort themselves into.

Likewise, in the same When in Rome vein as submitting patches
upstream, exactly how a repository that is branched off the upstream
one will look, depends at least in part on how upstream has chosen to
work, and we definitely can't start telling them there are now rules
for how they need to manage their repo.


The design of gitpkg was based very much on this simple reality.

Rather than duplicating how cvs-buildpackage worked, or how svn-bp
evolved from it, gitpkg has a very simple rule:

 - It doesn't care what your repository looks like or how you
   choose to work in it, or what you name your tags and branches.

It's designed to be able to work with essentially any repository
that you find anywhere, without requiring you to first mash it into
some particular form, or put tool specific files into it.

Provided you can give it a treeish that contains the debian patched
source you want to export, and (optionally) a treeish that contains
the orig source, it can always make you a source package of that.

The normal workflow is simply: work in git exactly how you normally
would, whether you're making upstream changes or debian patches.
Export a source package with `gitpkg $debtag $upstreamtag`.

That in a nutshell is it.  All of it.  There aren't even any command
line options for you to forget to type, or to screw up in a way that
would make a different source package to the last time you called it.

There are hooks you can use if you want p-t tarballs, or quilt patches,
or to automatically build the package once it's exported (with cowpoke
or dpkg-bp etc.), but they are all optional extras that each local user
running gitpkg can choose for themselves, if they want them.

The ability to export a valid source package does not depend on knowing
some secret incantation or configuration the repo was made to depend on.
All you need to know is the treeish you want to export.

This means you can use it to export any version of a source package
from a repo that existed long before you'd ever heard of gitpkg.
Without needing to go back and rewrite or patch its history.  You can
use it to export packages from a repo where someone used some other
tool previously.  You can use it to branch your packaging off any
upstream repo no matter how it has been used or arranged.  You can
use it to export packages that have been imported to git from
snapshot.d.o to build a full history from the pre-git epoch (with
git-debimport or any other tool you choose to do that with).


The good news from all that though, is that it would seem unlikely
that gitpkg itself would need any changes to be able to cope with
any repo design you could come up with that wasn't completely insane :)

The bad news perhaps, is I'm less convinced about how keen people
already using gitpkg will be to arbitrarily restructure their entire
existing repos to follow this.  At least not without some much clearer
rationale about what

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

2014-11-12 Thread Simon McVittie

On 12/11/14 05:54, Mathieu Parent wrote:
 Also, the vendor/* branches heads should be at a descendent commit of
 the corresponding upstream branch, diffing only by the debian dir.

This is only true for workflows similar to the one normally used with
gbp-pq, in which desired patches are serialized into debian/patches/ by
an out-of-band step (e.g. gbp-pq export, or quilt), and the upstream
tree is unpatched.

In particular, it is not true for git-dpm, dgit, or (as far as I can
tell from Ron's description elsewhere in this thread) gitpkg; with those
workflows, upstream files in the Debian branch *do* have their desired
changes.

Is it the intention of this DEP to mandate the gbp-pq-like repo
structure, which basically forbids use of tools whose design does not
match that? Or is the intention to set some conventions that can be true
regardless of whether you are using a more gbp-pq-like or more
git-dpm-like workflow, in the knowledge that that necessarily makes
those conventions less strict?

(I use gbp-pq for non-native packages myself, but trying out git-dpm is
on my to-do list.)

Concrete example: suppose you maintain an implementation of hello
world, with a Debian patch changing hello.c to say puts (hello,
Debian) instead of puts (hello, world).

In the gbp-pq world, after git checkout debian/sid, hello.c would
contain hello, world, but there would be a patch in debian/patches/ to
change it from hello, world to hello, Debian.

In git-dpm, after git checkout debian/sid, hello.c would contain
hello, Debian, *and* there would be a patch in debian/patches/ to
change it from hello, world to hello, Debian. AIUI, dgit also works
best in this arrangement (or might even require it?)

I'm not so sure about gitpkg - I *think* git checkout debian/sid in a
gitpkg repository would result in hello.c containing hello, Debian,
and no patches in debian/patches. But I could be wrong. (Ron?)

S


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/54635b81.8020...@debian.org

Re: RFC: DEP-14: Recommended layout for Git packaging repositories

Hi Ron,

On Wed, 12 Nov 2014, Ron wrote:
 I think you probably need to be careful of overspecifying this.

Definitely. That's precisely why I don't want to dwelve (too much)
into details of the various workflows and why I try to restrict
the DEP at simple naming conventions for branches and tags that
the various tools might need.

 The normal workflow is simply: work in git exactly how you normally
 would, whether you're making upstream changes or debian patches.
 Export a source package with `gitpkg $debtag $upstreamtag`.
[...]
 The good news from all that though, is that it would seem unlikely
 that gitpkg itself would need any changes to be able to cope with
 any repo design you could come up with that wasn't completely insane :)

Great, then it might be that gitpkg doesn't need any code update
and that only its documentation should be updated to recommend
using the default names when creating $debtag (and $upstreamtag in
some cases).

 The bad news perhaps, is I'm less convinced about how keen people
 already using gitpkg will be to arbitrarily restructure their entire
 existing repos to follow this.  At least not without some much clearer
 rationale about what compelling benefits they'd get in return for all
 that work and disruption and rewriting of history.

I don't necessarily want everybody to update their current repositories
(altought it would be nice). I want at least that when new contributors
create a new git packaging repository, they have some good advice on
how to structure their repository and the helper tools that they
pick work nicely in that layout.

 Maybe I missed something somewhere (I should have been sound asleep
 a few hours ago, so that's quite possible), but I see lots of
 explanation of *what* you want to do, but not the killer reasoning
 about *why* you want to do it, or what concrete things you think
 will be gained from it.

I think I explained my goals in the introduction of the DEP. Making
it easier for Debian and its derivatives to build upon their respective
Git repositories (possibly on a semi-automated basis). And making it
easier to switch between various git packaging helper tools because they
would share (by default) the same basic conventions. And making it easier
to have the upstream git branches in the packaging repository too.

 More specifically, you have things like:
 
  Helper tools should usually rely on the output of dpkg-vendor
   --query vendor to find out the name of the current vendor.
   The retrieved string must be converted to lower case. This
   allows users to override the current vendor by setting
   DEB_VENDOR in their environment
 
 This seems like a git-bp specific hack to me, that has the assumption
 and trained habits of using that tool so deeply embedded in it that
 doesn't bother to explain why you would use this, when you would use
 this, or even what you would expect it to do.

Given that the DEP recommends to create tags like vendor/version it
seems like a good idea to explain how to retrieve vendor, no?

It's not really related to git-bp (except for the fact that git-bp
started this convention of prefixing the tags with debian/). gitpkg
might not take the job of creating that tag, in which case that
particular advice is obviously irrelevant and it's up the package
maintainer to follow that naming convention when he manually creates the
said tag.

 The section on Patch queue tags also seems to be written from the
 perspective of only knowing one tool and one workflow to use it.
 With the work that David Bremner did on git-debcherry, gitpkg can
 fully automatically extract a quilt patch series for a source package
 simply from completely normal commits to your repo, in a form that
 makes them trivial for upstream to cherry pick, and makes them
 automatically vanish when you merge the upstream release that includes
 them.

Yes, the pach queue section is heavily inspired by gbp pq but again it's
nothing mandatory. It says something like if the helper tools produces
something that looks like a patch queue branch and if it wants to store it
for later consumption, it should use that tag name.

Fine if the other tools do not need anything like that. But who knows,
maybe you will want to enhance git-debcherry to not only update
debian/patches/ but also store the corresponding git branch for long-term
storage. In which case, you will already have a recommended tag name
for this purpose :-)

  I would be willing to participate to such a sprint and do my share
  of the work (although I'm primarily using git-buildpackage and would
  likely start with this helper tool).
 
 I am a bit curious about what patches you think you're going to need
 to make to make things comply with this plan?

For gitpkg? I don't know. See above, maybe it's just documentation update.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/


-- 
To

Re: RFC: DEP-14: Recommended layout for Git packaging repositories