Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Shengjing Zhu]

On Tue, Jul 10, 2018 at 1:27 AM Sean Whitton  wrote:
>  `git  merge` is smarter than you think :)

Thanks for the hint! Reading again the manual, and `git merge
--strategy=ours` really keeps the history clean. Previously when I
tried to attempt this work flow, I'm not able to make the history
clean. Usually I want to remove vendor directory(actually they're not
non-free, like the go vendor directory), and these files are often
modified, making normal merge hard...

-- 
Best regards,
Shengjing Zhu

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Sean Whitton]

Hello Shengjing,

On Sat, Jul 07 2018, Shengjing Zhu wrote:

> I'm not against this. My initial proposal is to keep the upstream git
> history. Let me describe my initial proposal a little more.
>
> Branch "upstream":
> This branch is same as upstream. Even there're files listed in
> Files-Excluded. Let's leave these files along.
> Reasons:
> I think if you create a file remove commit every time, the history becomes:
>
> * a2cc547 (upstream) remove-nonfree
> *   fb0eef5 Merge commit '70d7cda' into upstream
> |\
> * | 8206887 remove-nonfree
> | | * 6111635 upstream-commit-f
> | |/
> | * 70d7cda upstream-commit-e
> | * b0218d2 upstream-commit-d
> |/
> * 04a0cee upstream-commit-c
> * c4f272f upstream-commit-b
> * d24bf26 upstream-commit-a
>
> This history becomes a mess, which I don't think it's "gitish"

To my mind, this history represents exactly what is going on, and that
is not messy.

But in any case, I think you're mistaken about the remove-nonfree
commits.  You will only need one of these commits when new files get
added to Files-Excluded.  If you `git merge` an upstream tag into your
DFSG-free branch, files previously deleted will not reappear.  `git
merge` is smarter than you think :)

See dgit-maint-debrebase(7) section "HANDLING DFSG-NON-FREE MATERIAL".

> So I want the the upstream branch *only* contains upstream-commit-*
>
> Let's talk about `git blame`. if you blame the free files, you only
> see upstream commits, that's not influenced.
> If you blame on nonfree files, you will see lots of repeated remove
> commits, do you really need theme? You already know these files are
> remove from Files-Excluded field.
>
> About `git bisect`, that's interesting if the bug is caused by
> remove-commits(which means caused by nonfree files). I don't think
> this will happens so often.

I can't really make sense of this.  You seem to be describing using `git
blame` and `git bisect` on upstream's git history.  That has nothing to
do with Debian.

> However this proposal becomes unaccepted when Ian think the dgit user
> case. Actually I haven't try dgit yet and I don't know if it really
> doesn't work.

I didn't confirm Ian's analysis, but whether or not dgit works is
something that can usually be determined from the description of the
workflow.  You don't need to actually try it.

> Because maybe dgit users can't generate the dfsg tarball, after they
> clone from VCS-Git address.  However my proposal it to let gbp be able
> to generate the dfsg tarballs even the nonfree files are in upstream
> branch. And you can still commit these dfsg tarballs into pristine-tar
> branch or other places.

Generaly comment: if you're using a dgit workflow tarballs are an
implementation detail.

-- 
Sean Whitton

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Shengjing Zhu]

Hi Sean,

On Sat, Jul 7, 2018 at 3:55 PM Sean Whitton  wrote
> > I'm suspect if you really need upstream git log in the scenery you
> > described. What you really care is the packaging history.  [...]
> > However, if you really care dgit user. I may ask, what you want the
> > dfsg branch looks like? Do you prefer full upstream log, along with
> > some `git rm` commits; or like one import commit per release version?
> > If it's latter, maybe it can be easily regenerated with all orig
> > tarballs.
>
> It is useful to provide the full upstream git history to users because
> they can use tools like `git bisect` to figure out how to fix bugs.  The
> packaging history alone is much more coarse grained.

I'm not against this. My initial proposal is to keep the upstream git
history. Let me describe my initial proposal a little more.

Branch "upstream":
This branch is same as upstream. Even there're files listed in
Files-Excluded. Let's leave these files along.
Reasons:
I think if you create a file remove commit every time, the history becomes:

* a2cc547 (upstream) remove-nonfree
*   fb0eef5 Merge commit '70d7cda' into upstream
|\
* | 8206887 remove-nonfree
| | * 6111635 upstream-commit-f
| |/
| * 70d7cda upstream-commit-e
| * b0218d2 upstream-commit-d
|/
* 04a0cee upstream-commit-c
* c4f272f upstream-commit-b
* d24bf26 upstream-commit-a

This history becomes a mess, which I don't think it's "gitish"

So I want the the upstream branch *only* contains upstream-commit-*

Let's talk about `git blame`. if you blame the free files, you only
see upstream commits, that's not influenced.
If you blame on nonfree files, you will see lots of repeated remove
commits, do you really need theme? You already know these files are
remove from Files-Excluded field.

About `git bisect`, that's interesting if the bug is caused by
remove-commits(which means caused by nonfree files). I don't think
this will happens so often.

However this proposal becomes unaccepted when Ian think the dgit user
case. Actually I haven't try dgit yet and I don't know if it really
doesn't work. Because maybe dgit users can't generate the dfsg
tarball, after they clone from VCS-Git address.
However my proposal it to let gbp be able to generate the dfsg
tarballs even the nonfree files are in upstream branch. And you can
still commit these dfsg tarballs into pristine-tar branch or other
places.

-- 
Best regards,
Shengjing Zhu

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Ian Jackson]

Sean Whitton writes ("Re: Bug#812721: gbp could filter out Files-Excluded: 
entries when committing to the pristine-tar branch"):
> It is useful to provide the full upstream git history to users because
> they can use tools like `git bisect` to figure out how to fix bugs.  The
> packaging history alone is much more coarse grained.

Also, tools like "git blame somefile.c" etc. will work properly.  A
user who is trying to fix bugs may want the history for precisely the
same reason as the Debian upstream and maintainers want it.

Personally, I think we do our users a huge disservice when we treat
the git history as something "internal to Debian" or "for the
maintainer's use".

Ian.

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Sean Whitton]

Hello Shengjing,

On Thu, Jul 05 2018, Shengjing Zhu wrote:

> I'm suspect if you really need upstream git log in the scenery you
> described. What you really care is the packaging history.  [...]
> However, if you really care dgit user. I may ask, what you want the
> dfsg branch looks like? Do you prefer full upstream log, along with
> some `git rm` commits; or like one import commit per release version?
> If it's latter, maybe it can be easily regenerated with all orig
> tarballs.

It is useful to provide the full upstream git history to users because
they can use tools like `git bisect` to figure out how to fix bugs.  The
packaging history alone is much more coarse grained.

If bandwidth is really an issue, `git clone` has a --shallow option (not
directly supported by dgit, but you can `git clone` from the dgit-repos
server).

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Sean Whitton]

Hello,

On Fri, Jul 06 2018, Martín Ferrari wrote:

> On 05/07/18 14:19, Guido Günther wrote:
>
>>> So IMHO, the upstream branch should only contain upstream commits.
>>> Then question comes to how we create the dfsg orig tarball.
>>
>> Just create a commit with the files filtered out and tag it
>> approriately (with the original commit as parent). gbp
>> export-orig/buildpackage will then be able to find it to create the
>> tarball.
>
> This is what I have been doing in my packages, and what has been
> agreed to become the standard for the go team. Using one branch for
> the pristine upstream history, and another one that keeps the
> repackaging differences. Gbp buildpackage works well with this
> workflow, as you only need to add the appropriate tags. The missing
> part will be the automation to update these branches, ideally using
> files-exluded as the source of truth.

This would be an application of what I requested in this bug to a
traditional gbp workflow, indeed.  The reference to pristine-tar in the
title of this bug is a bit misleading, as it's not really about that.

Eventually, however, I would like to extend or wrap this functionality
to work with upstream tags, such that a branches tracking the unfiltered
and filtered upstream source are not needed.

The idea is that the upstream source is immutable from Debian's point of
view, and tags reflect and track that better than branches do.

The workflow would be something like this:

% git fetch upstream
% # some gbp command which takes the tag 1.2.3 and produces the filtered
% # tag 1.2.3+dfsg
% git merge 1.2.3+dfsg
% git deborig

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Martín Ferrari]

On 05/07/18 14:19, Guido Günther wrote:

>> So IMHO, the upstream branch should only contain upstream commits.
>> Then question comes to how we create the dfsg orig tarball.
> 
> Just create a commit with the files filtered out and tag it approriately
> (with the original commit as parent). gbp export-orig/buildpackage will
> then be able to find it to create the tarball.

This is what I have been doing in my packages, and what has been agreed
to become the standard for the go team. Using one branch for the
pristine upstream history, and another one that keeps the repackaging
differences. Gbp buildpackage works well with this workflow, as you only
need to add the appropriate tags. The missing part will be the
automation to update these branches, ideally using files-exluded as the
source of truth.

-- 
Martín Ferrari (Tincho)

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Guido Günther]

Hi,
(should we trim the cc: list a bit)?
On Thu, Jul 05, 2018 at 06:05:10PM +0800, Shengjing Zhu wrote:
> Here's my thoughts about the workflow with Files-Excluded.
> 
> 1. Upstream git history is imported *as is* in upstream branch. If we
> want to maintain a upstream branch without some files, then we already
> lost/mess up with the upstream history. Then why not use uscan git mode to
> import the tarball, which results a single import commit.
> 
> So IMHO, the upstream branch should only contain upstream commits.
> Then question comes to how we create the dfsg orig tarball.

Just create a commit with the files filtered out and tag it approriately
(with the original commit as parent). gbp export-orig/buildpackage will
then be able to find it to create the tarball.
Cheers,
 -- Guido

> 
> 2. gbp used `git-archive` to generate the tarball, if the repo only
> contains upstream git tree.
> 
> Like what we proposed in pkg-go team, we don't even want pristine-tar
> branch. we only want upstream git log. Most upstreams maintain go
> libraries with git.
> 
> I write a POC patch[1] to gbp to teach `gbp export-orig` to read
> d/copyright and filter out files in Files-Excluded when running
> `git-archive`.
> 
> So if you run `gbp export-orig --exclude-with-copyright`, you will get a
> orig tarball which excludes files in Files-Excluded.
> 
> [1] https://github.com/agx/git-buildpackage/pull/56
> 
> -- 
> Regards,
> Shengjing Zhu

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Shengjing Zhu]

On Thu, Jul 5, 2018 at 7:51 PM Ian Jackson
 wrote:
> My objective here is to publish to the Debian user the source code of
> the package they are using.
>
> If upstream are using git, then the user should get a git branch which
> contains the upstream history.
>

I'm suspect if you really need upstream git log in the scenery you
described. What you really care is the packaging history.

> But obviously the user who asks for the source code for their Debian
> package must not get the nonfree files.

They still get nonfree files. It contains in the history. If the
nonfree files are large blob, users still waste traffics to clone the
repo.
That's why I'm saying " It's git, you're cheating youself to remove something."


I think you may misunderstand why I refer uscan here.
When this discussion started long ago, uscan can't import upstream git
HEAD version.
But now it supports. Why uscan is related here, because it supports
Files-Excluded, and is integrated with gbp.
You can use `gbp import-orig --uscan`, to import upstream git HEAD
version(eg, 0.0~git20180705.abcdefg), with nonfree files excluded.
This results a familiar repository, and is compatible with dgit.

I'm using this work flow several times. It works fine.

But the only disadvantage is, I lost the upstream git log. For
example, I want to cherry-pick something.

So my perspective is from the maintainer, not the user. So I start
thinking keeping upstream log, and only generated dfsg tarball.
There's no need to waste space to keep another dfsg branch, with some
chaos git commits.
Users are still able to dget dsc; or if they like, gbp clone, and gbp
buildpackage.

However, if you really care dgit user. I may ask, what you want the
dfsg branch looks like? Do you prefer full upstream log, along with
some `git rm` commits; or like one import commit per release version?
If it's latter, maybe it can be easily regenerated with all orig
tarballs.--
Best regards,
Shengjing Zhu

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Ian Jackson]

Shengjing Zhu writes ("Re: Bug#812721: gbp could filter out Files-Excluded: 
entries when committing to the pristine-tar branch"):
> > What I really care about is that this means that users cannot get a
> > sensible git history for packages done this way.
> 
> Maybe from this point, uscan is enough now?

No.

Sorry to derail this but I feel I really have to explain this.

The main point of dgit is to make it possible for a user to get the
source code for their packages, in a directly useable form, as a git
branch. [1]

If necessary, dgit will convert the .dsc from the archive into git.
This gives the user the right files.  But it gives them the wrong
historyu.

It is much better if the maintainer has used `dgit push' to do the
upload.  Then the user will get a useful history.

[1] The Vcs-Git .dsc header, and salsa, are not suitable for this,
because what you get from salsa might be, well, any bizarre thing.
Patches-unapplied, or packaging-only, or even some strange thing like
the Haskell team have, with the metadata for all their packages in a
single git tree.


My objective here is to publish to the Debian user the source code of
the package they are using.

If upstream are using git, then the user should get a git branch which
contains the upstream history.

But obviously the user who asks for the source code for their Debian
package must not get the nonfree files.

Clearly, the only way that this can be done is for there to be git
commit(s) which delete the files.

AIUI that is what Sean is asking for in the original message in this
bug report: specificaly, that gbp should be able to make those
remove-nonfree-files commits and publish the result with an
appropriate git tag.


> I haven't been a user of dgit before. So if I understand correctly, it
> suggests do a `git rm` commit after every release.

This is too manual, which is why Sean is asking for gbp to do it.

> That sounds weird to me. It's git, you're cheating youself to remove
> something.

I don't understand this comment.

> > > I write a POC patch[1] to gbp to teach `gbp export-orig` to read
> > > d/copyright and filter out files in Files-Excluded when running
> > > `git-archive`.
> >
> > Please don't do this.  This workflow is not compatible with dgit.
> > That means the maintainer who uses that, cannot use dgit.
> 
> Yes, it's only compatible with gbp user.

dgit and gbp work very well together.  IMO all gbp users should be
using dgit.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Shengjing Zhu]

On Thu, Jul 5, 2018 at 6:19 PM Ian Jackson
 wrote:
> If you are using a gitish workflow, I don't think the Files-Excluded
> should affect only the .orig.  Rather, there should be a
> dfsg-laundered git tree or branch too.
>
> For examples of how we-the-dgit-maintainers suggest users do these
> things, have a look at these workflow docs:
>
>   
> https://manpages.debian.org/stretch/dgit/dgit-maint-merge.7.en.html#HANDLING_DFSG-NON-FREE_MATERIAL
>
>   
> http://www.chiark.greenend.org.uk/ucgi/~ianmdlvl/git-manpage/dgit.git/dgit-maint-debrebase.7
>   [search in page for "HANDLING DFSG-NON-FREE MATERIAL"]
>

I haven't been a user of dgit before. So if I understand correctly, it
suggests do a `git rm` commit after every release.
That sounds weird  to me. It's git, you're cheating youself to remove something.
And there's no clean upstream log, which I think it's no better than
uscan git mode.
With uscan git mode, you can track upstream version, and import
lastest master version.

> > I write a POC patch[1] to gbp to teach `gbp export-orig` to read
> > d/copyright and filter out files in Files-Excluded when running
> > `git-archive`.
>
> Please don't do this.  This workflow is not compatible with dgit.
> That means the maintainer who uses that, cannot use dgit.
>

Yes, it's only compatible with gbp user.

> What I really care about is that this means that users cannot get a
> sensible git history for packages done this way.

Maybe from this point, uscan is enough now?


-- 
Best regards,
Shengjing Zhu

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Ian Jackson]

Shengjing Zhu writes ("Re: Bug#812721: gbp could filter out Files-Excluded: 
entries when committing to the pristine-tar branch"):
> Here's my thoughts about the workflow with Files-Excluded.
> 
> 1. Upstream git history is imported *as is* in upstream branch. If we
> want to maintain a upstream branch without some files, then we already
> lost/mess up with the upstream history. Then why not use uscan git mode to
> import the tarball, which results a single import commit.
> 
> So IMHO, the upstream branch should only contain upstream commits.
> Then question comes to how we create the dfsg orig tarball.
> 
> 2. gbp used `git-archive` to generate the tarball, if the repo only
> contains upstream git tree.

If you are using a gitish workflow, I don't think the Files-Excluded
should affect only the .orig.  Rather, there should be a
dfsg-laundered git tree or branch too.

For examples of how we-the-dgit-maintainers suggest users do these
things, have a look at these workflow docs:

  
https://manpages.debian.org/stretch/dgit/dgit-maint-merge.7.en.html#HANDLING_DFSG-NON-FREE_MATERIAL

  
http://www.chiark.greenend.org.uk/ucgi/~ianmdlvl/git-manpage/dgit.git/dgit-maint-debrebase.7
  [search in page for "HANDLING DFSG-NON-FREE MATERIAL"]

> I write a POC patch[1] to gbp to teach `gbp export-orig` to read
> d/copyright and filter out files in Files-Excluded when running
> `git-archive`.

Please don't do this.  This workflow is not compatible with dgit.
That means the maintainer who uses that, cannot use dgit.

What I really care about is that this means that users cannot get a
sensible git history for packages done this way.

Ian.

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Shengjing Zhu]

Here's my thoughts about the workflow with Files-Excluded.

1. Upstream git history is imported *as is* in upstream branch. If we
want to maintain a upstream branch without some files, then we already
lost/mess up with the upstream history. Then why not use uscan git mode to
import the tarball, which results a single import commit.

So IMHO, the upstream branch should only contain upstream commits.
Then question comes to how we create the dfsg orig tarball.

2. gbp used `git-archive` to generate the tarball, if the repo only
contains upstream git tree.

Like what we proposed in pkg-go team, we don't even want pristine-tar
branch. we only want upstream git log. Most upstreams maintain go
libraries with git.

I write a POC patch[1] to gbp to teach `gbp export-orig` to read
d/copyright and filter out files in Files-Excluded when running
`git-archive`.

So if you run `gbp export-orig --exclude-with-copyright`, you will get a
orig tarball which excludes files in Files-Excluded.

[1] https://github.com/agx/git-buildpackage/pull/56

-- 
Regards,
Shengjing Zhu


signature.asc
Description: PGP signature

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Guido Günther]

Hi,
On Fri, May 25, 2018 at 03:20:59AM -0700, Sean Whitton wrote:
> Hello,
> 
> On Thu, May 24 2018, Guido Günther wrote:
> 
> > There's no such logic there yet. FWIW I've added gbp-import-ref
> >
> >
> > https://github.com/agx/git-buildpackage/commit/1592882c38fcbb107332d9a0a63b8da4e0bf8a48
> >
> > so we can now look into adding the proper filtering and logic for
> > creating the tag name automatically.
> 
> It occurs to me that this might be usable for workflows where the
> orig.tar for the upload is generated with `git deborig` and the upstream
> tarball is not used.
> 
> Currently, such a workflow[1] involves starting a temporary branch at
> the upstream release tag, manually filtering, committing, tagging with
> the +dfsg version number, deleting the temporary branch and merging the
> +dfsg tag to master.  It would be great if we could instead just type
> `gbp import-ref v1.0.0`.

That's basically the plan.
 -- Guido

> 
> [1] dgit-maint-merge(7) and (unreleased at time of writing)
> dgit-maint-debrebase(7)
> 
> -- 
> Sean Whitton
> 

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Sean Whitton]

Hello,

On Thu, May 24 2018, Guido Günther wrote:

> There's no such logic there yet. FWIW I've added gbp-import-ref
>
>
> https://github.com/agx/git-buildpackage/commit/1592882c38fcbb107332d9a0a63b8da4e0bf8a48
>
> so we can now look into adding the proper filtering and logic for
> creating the tag name automatically.

It occurs to me that this might be usable for workflows where the
orig.tar for the upload is generated with `git deborig` and the upstream
tarball is not used.

Currently, such a workflow[1] involves starting a temporary branch at
the upstream release tag, manually filtering, committing, tagging with
the +dfsg version number, deleting the temporary branch and merging the
+dfsg tag to master.  It would be great if we could instead just type
`gbp import-ref v1.0.0`.

[1] dgit-maint-merge(7) and (unreleased at time of writing)
dgit-maint-debrebase(7)

-- 
Sean Whitton

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Guido Günther]

Hi,
On Fri, Jan 26, 2018 at 10:42:26PM +0100, Michael Stapelberg wrote:
>I looked into implementing the discussed gbp import-git subcommand. Find
>attached preliminary patches. In my tests, the new subcommand tags the
>upstream branch and imports it correctly, but perhaps I missed something.
>Filtering is not yet considered at all. Feedback/review welcome (or, feel
>free to just run with them if that’s easier. I’m not fast with Python.)
>One thing came up during the implementation to which I’d like your
>feedback: if the upstream branch isn’t by chance already tagged with a
>proper version number (e.g. v1.2), how do we get a suitable version number
>for the git snapshot we’re about to package? We have such logic in
>dh-make-golang: 
> [1]https://github.com/Debian/dh-make-golang/blob/5a5180dce36c9c878b7551640ff018a6dfabf0dd/version.go#L19
>— as you can see it’s rather complicated. I couldn’t find anything
>comparable in git-buildpackage yet. Did I miss it? If no, could such logic
>be added?

There's no such logic there yet. FWIW I've added gbp-import-ref

   
https://github.com/agx/git-buildpackage/commit/1592882c38fcbb107332d9a0a63b8da4e0bf8a48

so we can now look into adding the proper filtering and logic for
creating the tag name automatically.

Cheers,
 -- Guido

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Guido Günther]

Hi,
On Fri, Jan 26, 2018 at 10:42:26PM +0100, Michael Stapelberg wrote:
> I looked into implementing the discussed gbp import-git subcommand. Find
> attached preliminary patches. In my tests, the new subcommand tags the
> upstream branch and imports it correctly, but perhaps I missed something.
> Filtering is not yet considered at all. Feedback/review welcome (or, feel
> free to just run with them if that’s easier. I’m not fast with Python.)

Thanks! We need to get rid of the redundancies between import-orig and
import-git and add the actual filtering but it's a start. I'll try to
find some time to work on this - it's not only useful for pkg-go.
Cheers,
 -- Guido

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Raphael Hertzog]

On Thu, 25 Jan 2018, Michael Stapelberg wrote:
> > or (since upstream is somewhat redundant):
> 
> I don’t feel that “upstream” is redundant. I think the contents of a branch
> should be obvious from the name.

Ack.

> The “pristine” namespace could easily be confused with “pristine-tar”, so
> I’d prefer avoiding that name altogether if possible.

Ack, as well.

> > this would also allow to retain upstream/ with the original meaning for
> 
> Wait, now I’m confused. Isn’t “upstream” the same as “upstream/latest”?
> What “original meaning” are you referring to? :)

I'm also confused.

On Fri, 26 Jan 2018, Michael Stapelberg wrote:
> One thing came up during the implementation to which I’d like your
> feedback: if the upstream branch isn’t by chance already tagged with a
> proper version number (e.g. v1.2), how do we get a suitable version number
> for the git snapshot we’re about to package? We have such logic in
> dh-make-golang:
> https://github.com/Debian/dh-make-golang/blob/5a5180dce36c9c878b7551640ff018a6dfabf0dd/version.go#L19
> — as you can see it’s rather complicated. I couldn’t find anything
> comparable in git-buildpackage yet. Did I miss it? If no, could such logic
> be added?

I don't know what's available in git-buildpackage but if it's not, then I
think it's OK to add something like this.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Guido Günther]

Hi,
On Thu, Jan 25, 2018 at 01:04:32PM +0100, Michael Stapelberg wrote:
> On Thu, Jan 25, 2018 at 12:46 PM, Guido Günther  wrote:
[..snip..]
> > To add to the bikeshed: I like path like separation in git:
> >
> > upstream/filtered/latest
> >
> > or (since upstream is somewhat redundant):
> >
> 
> I don’t feel that “upstream” is redundant. I think the contents of a branch
> should be obvious from the name.
> 
> 
> >
> > filtered/latest
> > pristine/latest
> >
> 
> The “pristine” namespace could easily be confused with “pristine-tar”, so
> I’d prefer avoiding that name altogether if possible.
> 
> 
> >
> > this would also allow to retain upstream/ with the original meaning for
> >
> 
> Wait, now I’m confused. Isn’t “upstream” the same as “upstream/latest”?
> What “original meaning” are you referring to? :)

Current default in gbp is "upstream" for the upstream branch. Switching
the default to anything starting with upstream/ causes problems for
people using the current default since the name "upstream" is already
used. While this can be fixed up locally by gbp it also requires to move
the the branch "upstream" out of the way on the _remote_ side since
otherwise pushing anything into the "upstream/" namespace will fail
otherwise.

Cheers,
 -- Guido

> 
> 
> > existing projects and would make the switch of defaults easier since
> > gbp's current default upstream does not conflict with upstream/latest.
> >
> > Cheers,
> >  -- Guido
> >
> 
> 
> 
> -- 
> Best regards,
> Michael

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Michael Stapelberg]

I looked into implementing the discussed gbp import-git subcommand. Find
attached preliminary patches. In my tests, the new subcommand tags the
upstream branch and imports it correctly, but perhaps I missed something.
Filtering is not yet considered at all. Feedback/review welcome (or, feel
free to just run with them if that’s easier. I’m not fast with Python.)

One thing came up during the implementation to which I’d like your
feedback: if the upstream branch isn’t by chance already tagged with a
proper version number (e.g. v1.2), how do we get a suitable version number
for the git snapshot we’re about to package? We have such logic in
dh-make-golang:
https://github.com/Debian/dh-make-golang/blob/5a5180dce36c9c878b7551640ff018a6dfabf0dd/version.go#L19
— as you can see it’s rather complicated. I couldn’t find anything
comparable in git-buildpackage yet. Did I miss it? If no, could such logic
be added?

Thanks,

On Thu, Jan 25, 2018 at 1:04 PM, Michael Stapelberg 
wrote:

>
>
> On Thu, Jan 25, 2018 at 12:46 PM, Guido Günther  wrote:
>
>> Hi,
>> On Thu, Jan 25, 2018 at 10:25:09AM +0100, Raphael Hertzog wrote:
>> > > • branch “upstream/latest” tracks an upstream branch, e.g. remote
>> > > “upstream”, branch “master”.
>> > >
>> > > • branch “upstream/latest-filtered” contains a filtered version of
>> that
>> > > branch, whose commits would be tagged e.g. upstream/1.0+dfsg1 so that
>> > > gbp-buildpackage picks the correct commit.
>> > >
>> > > I think that complies with DEP14 as good as possible, and is still
>> > > reasonably clear for casual users. Thoughts?
>> >
>> > I would use "upstream-filtered/latest" for the latter. You have clean
>> > namespace separation. But this still changes the assumption about
>> > upstream/latest (non-filtered now) so it needs a broader discussion IMO.
>>
>> To add to the bikeshed: I like path like separation in git:
>>
>> upstream/filtered/latest
>>
>> or (since upstream is somewhat redundant):
>>
>
> I don’t feel that “upstream” is redundant. I think the contents of a
> branch should be obvious from the name.
>
>
>>
>> filtered/latest
>> pristine/latest
>>
>
> The “pristine” namespace could easily be confused with “pristine-tar”, so
> I’d prefer avoiding that name altogether if possible.
>
>
>>
>> this would also allow to retain upstream/ with the original meaning for
>>
>
> Wait, now I’m confused. Isn’t “upstream” the same as “upstream/latest”?
> What “original meaning” are you referring to? :)
>
>
>> existing projects and would make the switch of defaults easier since
>> gbp's current default upstream does not conflict with upstream/latest.
>>
>> Cheers,
>>  -- Guido
>>
>
>
>
> --
> Best regards,
> Michael
>



-- 
Best regards,
Michael
From 6c59a6b196382f846b48f1a4d650fa6937d360dc Mon Sep 17 00:00:00 2001
From: Michael Stapelberg 
Date: Fri, 26 Jan 2018 22:36:17 +0100
Subject: [PATCH 1/2] Copy import_orig.py to import_git.py

---
 gbp/scripts/import_git.py | 669 ++
 1 file changed, 669 insertions(+)
 create mode 100644 gbp/scripts/import_git.py

diff --git a/gbp/scripts/import_git.py b/gbp/scripts/import_git.py
new file mode 100644
index 000..97b6d54
--- /dev/null
+++ b/gbp/scripts/import_git.py
@@ -0,0 +1,669 @@
+# vim: set fileencoding=utf-8 :
+#
+# (C) 2006, 2007, 2009, 2011, 2015, 2016 Guido Guenther 
+#This program is free software; you can redistribute it and/or modify
+#it under the terms of the GNU General Public License as published by
+#the Free Software Foundation; either version 2 of the License, or
+#(at your option) any later version.
+#
+#This program is distributed in the hope that it will be useful,
+#but WITHOUT ANY WARRANTY; without even the implied warranty of
+#MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#GNU General Public License for more details.
+#
+#You should have received a copy of the GNU General Public License
+#along with this program; if not, please see
+#
+#
+"""Import a new upstream version into a Git repository"""
+
+import os
+import shutil
+import sys
+import tempfile
+import time
+import gbp.command_wrappers as gbpc
+from gbp.deb import (DebianPkgPolicy, parse_changelog_repo)
+from gbp.deb.format import DebianSourceFormat
+from gbp.deb.upstreamsource import DebianUpstreamSource, unpack_component_tarball
+from gbp.deb.uscan import (Uscan, UscanError)
+from gbp.deb.changelog import ChangeLog, NoChangeLogError
+from gbp.deb.git import (GitRepositoryError, DebianGitRepository)
+from gbp.config import GbpOptionParserDebian, GbpOptionGroup, no_upstream_branch_msg
+from gbp.errors import GbpError
+from gbp.format import format_str
+from gbp.git.vfs import GitVfs
+import gbp.log
+from gbp.scripts.common import ExitCodes, is_download, get_component_tarballs
+from gbp.scripts.common.import_orig import 

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Michael Stapelberg]

On Thu, Jan 25, 2018 at 12:46 PM, Guido Günther  wrote:

> Hi,
> On Thu, Jan 25, 2018 at 10:25:09AM +0100, Raphael Hertzog wrote:
> > > • branch “upstream/latest” tracks an upstream branch, e.g. remote
> > > “upstream”, branch “master”.
> > >
> > > • branch “upstream/latest-filtered” contains a filtered version of that
> > > branch, whose commits would be tagged e.g. upstream/1.0+dfsg1 so that
> > > gbp-buildpackage picks the correct commit.
> > >
> > > I think that complies with DEP14 as good as possible, and is still
> > > reasonably clear for casual users. Thoughts?
> >
> > I would use "upstream-filtered/latest" for the latter. You have clean
> > namespace separation. But this still changes the assumption about
> > upstream/latest (non-filtered now) so it needs a broader discussion IMO.
>
> To add to the bikeshed: I like path like separation in git:
>
> upstream/filtered/latest
>
> or (since upstream is somewhat redundant):
>

I don’t feel that “upstream” is redundant. I think the contents of a branch
should be obvious from the name.


>
> filtered/latest
> pristine/latest
>

The “pristine” namespace could easily be confused with “pristine-tar”, so
I’d prefer avoiding that name altogether if possible.


>
> this would also allow to retain upstream/ with the original meaning for
>

Wait, now I’m confused. Isn’t “upstream” the same as “upstream/latest”?
What “original meaning” are you referring to? :)


> existing projects and would make the switch of defaults easier since
> gbp's current default upstream does not conflict with upstream/latest.
>
> Cheers,
>  -- Guido
>



-- 
Best regards,
Michael

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Guido Günther]

Hi,
On Thu, Jan 25, 2018 at 10:25:09AM +0100, Raphael Hertzog wrote:
> > • branch “upstream/latest” tracks an upstream branch, e.g. remote
> > “upstream”, branch “master”.
> > 
> > • branch “upstream/latest-filtered” contains a filtered version of that
> > branch, whose commits would be tagged e.g. upstream/1.0+dfsg1 so that
> > gbp-buildpackage picks the correct commit.
> > 
> > I think that complies with DEP14 as good as possible, and is still
> > reasonably clear for casual users. Thoughts?
> 
> I would use "upstream-filtered/latest" for the latter. You have clean
> namespace separation. But this still changes the assumption about
> upstream/latest (non-filtered now) so it needs a broader discussion IMO.

To add to the bikeshed: I like path like separation in git:

upstream/filtered/latest

or (since upstream is somewhat redundant):

filtered/latest
pristine/latest

this would also allow to retain upstream/ with the original meaning for
existing projects and would make the switch of defaults easier since
gbp's current default upstream does not conflict with upstream/latest.

Cheers,
 -- Guido

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Raphael Hertzog]

Hi,

On Thu, 25 Jan 2018, Michael Stapelberg wrote:
> > I think that if you want to change this, then we should discuss it on
> > debian-devel.
> >
> > If everything comes down to the fact that you don't find a good name, I'm
> > happy to throw a few suggestions:
> > What about "pristine-upstream" or "original-upstream" or "upstream.orig"
> > or "uptream.real" for the non-filtered upstream branch ?
> >
> 
> None of these fit with DEP14, unless I’m misunderstanding the following
> paragraph?
> 
> If the Git workflow in use imports the upstream sources from released
> tarballs, this should be done under the "upstream" namespace. By default,
> the latest upstream version should be imported in the upstream/latest
> branch and when packages for multiple upstream versions are maintained
> concurrently, one should create as many upstream branches as required.
> 
> So, are you suggesting upstream/pristine?

Not really. I was just saying that if you want to have upstream/* host
non-filtered sources, then we should have a broader discussion on
debian-devel because it's not compatible with the current version
of the DEP.

On the other hand, if you wanted to do this only because you could not
find a better name (outside of upstream/*) for the unfiltered version,
then I made a few suggestions of names that could be acceptable IMO.
 
> Thinking about it, my suggestions would be:
> 
> • branch “upstream/latest” tracks an upstream branch, e.g. remote
> “upstream”, branch “master”.
> 
> • branch “upstream/latest-filtered” contains a filtered version of that
> branch, whose commits would be tagged e.g. upstream/1.0+dfsg1 so that
> gbp-buildpackage picks the correct commit.
> 
> I think that complies with DEP14 as good as possible, and is still
> reasonably clear for casual users. Thoughts?

I would use "upstream-filtered/latest" for the latter. You have clean
namespace separation. But this still changes the assumption about
upstream/latest (non-filtered now) so it needs a broader discussion IMO.

I'm not opposed to it.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Michael Stapelberg]

On Mon, Jan 8, 2018 at 3:04 PM, Raphael Hertzog  wrote:

> Hi,
>
> On Sat, 06 Jan 2018, Michael Stapelberg wrote:
> > > > If yes, then I don’t particularly like the name “nondfsg/latest”, as
> > > > it is a double-negative, but describes a very common case. Why not
> > > > keep calling it “upstream”, or “upstream/latest” if symmetry is
> > > > desired?
> > >
> > > Isn't that what we're using for the filtered sources. The longer I
> think
> > > about it I think that
> > >
> > > https://honk.sigxcpu.org/projects/git-buildpackage/
> > > manual-html/gbp.branch.naming.html
> > >
> > > fits better than DEP14. Maybe we should modify DEP14 to say that
> > > upstream/ keeps the unfiltered source and dfsgclean/ the filtered one?
> >
> > Raphael, any thoughts?
>
> I think that if you want to change this, then we should discuss it on
> debian-devel.
>
> If everything comes down to the fact that you don't find a good name, I'm
> happy to throw a few suggestions:
> What about "pristine-upstream" or "original-upstream" or "upstream.orig"
> or "uptream.real" for the non-filtered upstream branch ?
>

None of these fit with DEP14, unless I’m misunderstanding the following
paragraph?

If the Git workflow in use imports the upstream sources from released
tarballs, this should be done under the "upstream" namespace. By default,
the latest upstream version should be imported in the upstream/latest
branch and when packages for multiple upstream versions are maintained
concurrently, one should create as many upstream branches as required.

So, are you suggesting upstream/pristine?

Thinking about it, my suggestions would be:

• branch “upstream/latest” tracks an upstream branch, e.g. remote
“upstream”, branch “master”.

• branch “upstream/latest-filtered” contains a filtered version of that
branch, whose commits would be tagged e.g. upstream/1.0+dfsg1 so that
gbp-buildpackage picks the correct commit.

I think that complies with DEP14 as good as possible, and is still
reasonably clear for casual users. Thoughts?


>
> Cheers,
> --
> Raphaël Hertzog ◈ Debian Developer
>
> Support Debian LTS: https://www.freexian.com/services/debian-lts.html
> Learn to master Debian: https://debian-handbook.info/get/
>



-- 
Best regards,
Michael

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Raphael Hertzog]

Hi,

On Sat, 06 Jan 2018, Michael Stapelberg wrote:
> > > If yes, then I don’t particularly like the name “nondfsg/latest”, as
> > > it is a double-negative, but describes a very common case. Why not
> > > keep calling it “upstream”, or “upstream/latest” if symmetry is
> > > desired?
> >
> > Isn't that what we're using for the filtered sources. The longer I think
> > about it I think that
> >
> > https://honk.sigxcpu.org/projects/git-buildpackage/
> > manual-html/gbp.branch.naming.html
> >
> > fits better than DEP14. Maybe we should modify DEP14 to say that
> > upstream/ keeps the unfiltered source and dfsgclean/ the filtered one?
> 
> Raphael, any thoughts?

I think that if you want to change this, then we should discuss it on
debian-devel.

If everything comes down to the fact that you don't find a good name, I'm
happy to throw a few suggestions:
What about "pristine-upstream" or "original-upstream" or "upstream.orig"
or "uptream.real" for the non-filtered upstream branch ?

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Michael Stapelberg]

[+cc raphael (listed as DEP-14 driver)]

Sorry for the long delay.

On Wed, Nov 8, 2017 at 9:00 AM, Guido Günther  wrote:

> Hi,
> On Tue, Nov 07, 2017 at 10:07:54PM +0100, Michael Stapelberg wrote:
> > Thanks for your reply. Answers inline:
> >
> > On Mon, Nov 6, 2017 at 8:59 AM, Guido Günther  wrote:
> > > Hi,
> > > On Tue, Oct 31, 2017 at 05:24:05PM +0100, Michael Stapelberg wrote:
> > >> Hi Guido,
> > >>
> > >> The pkg-go team is currently discussing changes to its workflow, and
> > >> we’d be interested in resolving this feature request.
> > >
> > > Can you provide a pointer to the discussion?
> >
> > Have a look at https://lists.alioth.debian.org/pipermail/pkg-go-
> maintainers/Week-of-Mon-20171016/015809.html
> >
> > >
> > >>
> > >> Guido Günther  writes:
> > >> > I would rather do this with a dfsg-clean branch. You delete once and
> > >> > then use git tools from there on.
> > >>
> > >> Searching for how dfsg-clean branches should be named, I found
> > >> https://honk.sigxcpu.org/projects/git-buildpackage/
> manual-html/gbp.branch.naming.html,
> > >> which recommends “dfsg/latest”.
> > >>
> > >> However, my reading of section “About repacked upstream sources” of
> > >> http://dep.debian.net/deps/dep14/ directly contradicts the above
> advice:
> > >> DEP14 says upstream/* should contain the repackaged files.
> > >>
> > >> How do we reconcile this apparent contradiction?
> > >
> > > Since gbp makes no assumptions on this I'm happy to update the docs.
> How
> > > would we call the non-filtered branch then "nondfsg/latest"?  When we
> > > base our packaging on upstream git we'll likely use upstream's branch
> > > name but in case of tarballs we should provide a good recommendation.
> >
> > Just to make sure we’re talking about the same thing: the branch
> > you’re asking for naming recommendations is currently called
> > “upstream”, yes?
>
> Yes, I'm talking about the branch or rather namespace that has the
> pristine, unfiltered upstream sources. We probably don't need such a
> name if upstream uses git but in the case of tarballs when one wants to
> keep the unfiltered source.
>
> > If yes, then I don’t particularly like the name “nondfsg/latest”, as
> > it is a double-negative, but describes a very common case. Why not
> > keep calling it “upstream”, or “upstream/latest” if symmetry is
> > desired?
>
> Isn't that what we're using for the filtered sources. The longer I think
> about it I think that
>
> https://honk.sigxcpu.org/projects/git-buildpackage/
> manual-html/gbp.branch.naming.html
>
> fits better than DEP14. Maybe we should modify DEP14 to say that
> upstream/ keeps the unfiltered source and dfsgclean/ the filtered one?
>

Raphael, any thoughts?


>
> >
> > >
> > >> >> It would be great if gbp could produce the 1.2.3+dfsg tag itself by
> > >> >> reading debian/copyright and excluding the Files-Excluded: files.
> > >> >
> > >> > If somebody comes up with a clean patch I'm happy to merge that.
> > >>
> > >> Which part of gbp specifically should be patched here? AFAICT, there
> is
> > >> no command which pulls a new version from upstream yet. Should one be
> > >> added? What should it be called?
> > >
> > > My first reaction was to teach gbp import-orig to have a
> > >
> > > gbp import-orig "git-ref"
> > >
> > > mode that would do the right thing but I now think having
> > >
> > > gbp update "git-ref"
> > >
> > > that
> > >
> > > - does the excluding and tagging if necessary
> > > - merges to the debian branch
> > >
> > > is better. We need to make sure that gbp import-orig's filtering (using
> > > the --filter command line or filter= gbp.conf option) stays in sync
> with
> > > what we do so we don't have on tool using --filter= and the other one
> > > parsing debian/changelog.
> >
> > You’re saying gbp import-orig and gbp update should both support the
> > same filter option, in additon to d/copyright, yes?
>
> Yes.
>
> >
> > >
> > > If somebody comes up with a better name than "update" that's all fine.
> >
> > “update” is a rather generic term. Given that the underlying git
> > operation is “git pull”, how about “gbp pull-upstream”?
>
> See
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=812721#25
>
> alternatively "gbp import-ref". pull-upstream is wired since we want the
> upstream namespace to contain the already filtered sources.
> Cheers,
>  -- Guido
>



-- 
Best regards,
Michael

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Guido Günther]

Hi,
On Tue, Nov 07, 2017 at 10:07:54PM +0100, Michael Stapelberg wrote:
> Thanks for your reply. Answers inline:
> 
> On Mon, Nov 6, 2017 at 8:59 AM, Guido Günther  wrote:
> > Hi,
> > On Tue, Oct 31, 2017 at 05:24:05PM +0100, Michael Stapelberg wrote:
> >> Hi Guido,
> >>
> >> The pkg-go team is currently discussing changes to its workflow, and
> >> we’d be interested in resolving this feature request.
> >
> > Can you provide a pointer to the discussion?
> 
> Have a look at 
> https://lists.alioth.debian.org/pipermail/pkg-go-maintainers/Week-of-Mon-20171016/015809.html
> 
> >
> >>
> >> Guido Günther  writes:
> >> > I would rather do this with a dfsg-clean branch. You delete once and
> >> > then use git tools from there on.
> >>
> >> Searching for how dfsg-clean branches should be named, I found
> >> https://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.branch.naming.html,
> >> which recommends “dfsg/latest”.
> >>
> >> However, my reading of section “About repacked upstream sources” of
> >> http://dep.debian.net/deps/dep14/ directly contradicts the above advice:
> >> DEP14 says upstream/* should contain the repackaged files.
> >>
> >> How do we reconcile this apparent contradiction?
> >
> > Since gbp makes no assumptions on this I'm happy to update the docs. How
> > would we call the non-filtered branch then "nondfsg/latest"?  When we
> > base our packaging on upstream git we'll likely use upstream's branch
> > name but in case of tarballs we should provide a good recommendation.
> 
> Just to make sure we’re talking about the same thing: the branch
> you’re asking for naming recommendations is currently called
> “upstream”, yes?

Yes, I'm talking about the branch or rather namespace that has the
pristine, unfiltered upstream sources. We probably don't need such a
name if upstream uses git but in the case of tarballs when one wants to
keep the unfiltered source.

> If yes, then I don’t particularly like the name “nondfsg/latest”, as
> it is a double-negative, but describes a very common case. Why not
> keep calling it “upstream”, or “upstream/latest” if symmetry is
> desired?

Isn't that what we're using for the filtered sources. The longer I think
about it I think that


https://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.branch.naming.html

fits better than DEP14. Maybe we should modify DEP14 to say that
upstream/ keeps the unfiltered source and dfsgclean/ the filtered one?

> 
> >
> >> >> It would be great if gbp could produce the 1.2.3+dfsg tag itself by
> >> >> reading debian/copyright and excluding the Files-Excluded: files.
> >> >
> >> > If somebody comes up with a clean patch I'm happy to merge that.
> >>
> >> Which part of gbp specifically should be patched here? AFAICT, there is
> >> no command which pulls a new version from upstream yet. Should one be
> >> added? What should it be called?
> >
> > My first reaction was to teach gbp import-orig to have a
> >
> > gbp import-orig "git-ref"
> >
> > mode that would do the right thing but I now think having
> >
> > gbp update "git-ref"
> >
> > that
> >
> > - does the excluding and tagging if necessary
> > - merges to the debian branch
> >
> > is better. We need to make sure that gbp import-orig's filtering (using
> > the --filter command line or filter= gbp.conf option) stays in sync with
> > what we do so we don't have on tool using --filter= and the other one
> > parsing debian/changelog.
> 
> You’re saying gbp import-orig and gbp update should both support the
> same filter option, in additon to d/copyright, yes?

Yes.

> 
> >
> > If somebody comes up with a better name than "update" that's all fine.
> 
> “update” is a rather generic term. Given that the underlying git
> operation is “git pull”, how about “gbp pull-upstream”?

See

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=812721#25  

alternatively "gbp import-ref". pull-upstream is wired since we want the
upstream namespace to contain the already filtered sources.
Cheers,
 -- Guido

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Michael Stapelberg]

Thanks for your reply. Answers inline:

On Mon, Nov 6, 2017 at 8:59 AM, Guido Günther  wrote:
> Hi,
> On Tue, Oct 31, 2017 at 05:24:05PM +0100, Michael Stapelberg wrote:
>> Hi Guido,
>>
>> The pkg-go team is currently discussing changes to its workflow, and
>> we’d be interested in resolving this feature request.
>
> Can you provide a pointer to the discussion?

Have a look at 
https://lists.alioth.debian.org/pipermail/pkg-go-maintainers/Week-of-Mon-20171016/015809.html

>
>>
>> Guido Günther  writes:
>> > I would rather do this with a dfsg-clean branch. You delete once and
>> > then use git tools from there on.
>>
>> Searching for how dfsg-clean branches should be named, I found
>> https://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.branch.naming.html,
>> which recommends “dfsg/latest”.
>>
>> However, my reading of section “About repacked upstream sources” of
>> http://dep.debian.net/deps/dep14/ directly contradicts the above advice:
>> DEP14 says upstream/* should contain the repackaged files.
>>
>> How do we reconcile this apparent contradiction?
>
> Since gbp makes no assumptions on this I'm happy to update the docs. How
> would we call the non-filtered branch then "nondfsg/latest"?  When we
> base our packaging on upstream git we'll likely use upstream's branch
> name but in case of tarballs we should provide a good recommendation.

Just to make sure we’re talking about the same thing: the branch
you’re asking for naming recommendations is currently called
“upstream”, yes?

If yes, then I don’t particularly like the name “nondfsg/latest”, as
it is a double-negative, but describes a very common case. Why not
keep calling it “upstream”, or “upstream/latest” if symmetry is
desired?

>
>> >> It would be great if gbp could produce the 1.2.3+dfsg tag itself by
>> >> reading debian/copyright and excluding the Files-Excluded: files.
>> >
>> > If somebody comes up with a clean patch I'm happy to merge that.
>>
>> Which part of gbp specifically should be patched here? AFAICT, there is
>> no command which pulls a new version from upstream yet. Should one be
>> added? What should it be called?
>
> My first reaction was to teach gbp import-orig to have a
>
> gbp import-orig "git-ref"
>
> mode that would do the right thing but I now think having
>
> gbp update "git-ref"
>
> that
>
> - does the excluding and tagging if necessary
> - merges to the debian branch
>
> is better. We need to make sure that gbp import-orig's filtering (using
> the --filter command line or filter= gbp.conf option) stays in sync with
> what we do so we don't have on tool using --filter= and the other one
> parsing debian/changelog.

You’re saying gbp import-orig and gbp update should both support the
same filter option, in additon to d/copyright, yes?

>
> If somebody comes up with a better name than "update" that's all fine.

“update” is a rather generic term. Given that the underlying git
operation is “git pull”, how about “gbp pull-upstream”?


-- 
Best regards,
Michael

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Guido Günther]

Hi,
On Mon, Nov 06, 2017 at 12:15:51PM +0100, Guido Günther wrote:
> Hi,
> On Mon, Nov 06, 2017 at 07:13:07PM +0800, Shengjing Zhu wrote:
> > On Mon, 6 Nov 2017 08:59:41 +0100 Guido =?iso-8859-1?Q?G=FCnther?=
> >  wrote:
> > > Hi,
> > > My first reaction was to teach gbp import-orig to have a
> > >
> > > gbp import-orig "git-ref"
> > >
> > > mode that would do the right thing but I now think having
> > >
> > > gbp update "git-ref"
> > >
> > > that
> > >
> > > - does the excluding and tagging if necessary
> > > - merges to the debian branch
> > >
> > > is better. We need to make sure that gbp import-orig's filtering (using
> > > the --filter command line or filter= gbp.conf option) stays in sync with
> > > what we do so we don't have on tool using --filter= and the other one
> > > parsing debian/changelog.
> > >
> > > If somebody comes up with a better name than "update" that's all fine.
> > > Cheers,
> > 
> > I would suggest `gbp import-git`, which is consistent with `import-*`
> 
> Yes, much better.

Maybe "git import-ref" is even better but that's just bikeshedding.
 -- Guido

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Guido Günther]

Hi,
On Mon, Nov 06, 2017 at 07:13:07PM +0800, Shengjing Zhu wrote:
> On Mon, 6 Nov 2017 08:59:41 +0100 Guido =?iso-8859-1?Q?G=FCnther?=
>  wrote:
> > Hi,
> > My first reaction was to teach gbp import-orig to have a
> >
> > gbp import-orig "git-ref"
> >
> > mode that would do the right thing but I now think having
> >
> > gbp update "git-ref"
> >
> > that
> >
> > - does the excluding and tagging if necessary
> > - merges to the debian branch
> >
> > is better. We need to make sure that gbp import-orig's filtering (using
> > the --filter command line or filter= gbp.conf option) stays in sync with
> > what we do so we don't have on tool using --filter= and the other one
> > parsing debian/changelog.
> >
> > If somebody comes up with a better name than "update" that's all fine.
> > Cheers,
> 
> I would suggest `gbp import-git`, which is consistent with `import-*`

Yes, much better.

> 
> also the "git-ref" can be parsed as an option, so we can store it in gbp.conf.
> 
> And I suggest the `--filter` option can also be remained, when
> importing from upstream git.

Yes, that's what I meant to say above.

> Besides, adding an option like `--filter-with-copyright` to indicate
> filtering files from "Files-Excluded" in d/copyright file.

That sounds good to me.

Cheers
 -- Guido

> 
> 
> --
> Shengjing Zhu
> 

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Shengjing Zhu]

On Mon, 6 Nov 2017 08:59:41 +0100 Guido =?iso-8859-1?Q?G=FCnther?=
 wrote:
> Hi,
> My first reaction was to teach gbp import-orig to have a
>
> gbp import-orig "git-ref"
>
> mode that would do the right thing but I now think having
>
> gbp update "git-ref"
>
> that
>
> - does the excluding and tagging if necessary
> - merges to the debian branch
>
> is better. We need to make sure that gbp import-orig's filtering (using
> the --filter command line or filter= gbp.conf option) stays in sync with
> what we do so we don't have on tool using --filter= and the other one
> parsing debian/changelog.
>
> If somebody comes up with a better name than "update" that's all fine.
> Cheers,

I would suggest `gbp import-git`, which is consistent with `import-*`

also the "git-ref" can be parsed as an option, so we can store it in gbp.conf.

And I suggest the `--filter` option can also be remained, when
importing from upstream git.
Besides, adding an option like `--filter-with-copyright` to indicate
filtering files from "Files-Excluded" in d/copyright file.


--
Shengjing Zhu

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Guido Günther]

Hi,
On Tue, Oct 31, 2017 at 05:24:05PM +0100, Michael Stapelberg wrote:
> Hi Guido,
> 
> The pkg-go team is currently discussing changes to its workflow, and
> we’d be interested in resolving this feature request.

Can you provide a pointer to the discussion?

> 
> Guido Günther  writes:
> > I would rather do this with a dfsg-clean branch. You delete once and
> > then use git tools from there on.
> 
> Searching for how dfsg-clean branches should be named, I found
> https://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.branch.naming.html,
> which recommends “dfsg/latest”.
> 
> However, my reading of section “About repacked upstream sources” of
> http://dep.debian.net/deps/dep14/ directly contradicts the above advice:
> DEP14 says upstream/* should contain the repackaged files.
>
> How do we reconcile this apparent contradiction?

Since gbp makes no assumptions on this I'm happy to update the docs. How
would we call the non-filtered branch then "nondfsg/latest"?  When we
base our packaging on upstream git we'll likely use upstream's branch
name but in case of tarballs we should provide a good recommendation.

> >> It would be great if gbp could produce the 1.2.3+dfsg tag itself by
> >> reading debian/copyright and excluding the Files-Excluded: files.
> >
> > If somebody comes up with a clean patch I'm happy to merge that.
> 
> Which part of gbp specifically should be patched here? AFAICT, there is
> no command which pulls a new version from upstream yet. Should one be
> added? What should it be called?

My first reaction was to teach gbp import-orig to have a

gbp import-orig "git-ref"

mode that would do the right thing but I now think having

gbp update "git-ref"

that

- does the excluding and tagging if necessary
- merges to the debian branch

is better. We need to make sure that gbp import-orig's filtering (using
the --filter command line or filter= gbp.conf option) stays in sync with
what we do so we don't have on tool using --filter= and the other one
parsing debian/changelog.

If somebody comes up with a better name than "update" that's all fine.
Cheers,
 -- Guido

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Michael Stapelberg]

Hi Guido,

The pkg-go team is currently discussing changes to its workflow, and
we’d be interested in resolving this feature request.

Guido Günther  writes:
> I would rather do this with a dfsg-clean branch. You delete once and
> then use git tools from there on.

Searching for how dfsg-clean branches should be named, I found
https://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.branch.naming.html,
which recommends “dfsg/latest”.

However, my reading of section “About repacked upstream sources” of
http://dep.debian.net/deps/dep14/ directly contradicts the above advice:
DEP14 says upstream/* should contain the repackaged files.

How do we reconcile this apparent contradiction?

>> It would be great if gbp could produce the 1.2.3+dfsg tag itself by
>> reading debian/copyright and excluding the Files-Excluded: files.
>
> If somebody comes up with a clean patch I'm happy to merge that.

Which part of gbp specifically should be patched here? AFAICT, there is
no command which pulls a new version from upstream yet. Should one be
added? What should it be called?

-- 
Best regards,
Michael

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Sean Whitton]

Package: git-buildpackage
Version: 0.7.1
Severity: wishlist

Dear Maintainer,

When uscan is used to generate an upstream tarball, it filters out files
listed in the Files-Excluded: section of the debian/copyright file and
appends +dfsg to the tarball's version.  This can then be imported to
the packaging repository with gbp import-orig.

Nowadays, many upstreams do not ship tarballs.  When preparing the
Debian packaging for such a project, it is convenient to simply clone
upstream's git repository and add a branch to that for Debian
packaging.

Suppose that upstream tags a release 1.2.3 in the git repository.  The
Debian package maintainer must then manually delete the files listed in
Files-Excluded: and then tag a new version 1.2.3+dfsg.  gbp can then
create a tarball of this tag and commit it to the pristine-tar branch.

It would be great if gbp could produce the 1.2.3+dfsg tag itself by
reading debian/copyright and excluding the Files-Excluded: files.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#812721: gbp could filter out Files-Excluded: entries when committing to the pristine-tar branch

[Guido Günther]

Hi,
On Mon, Jan 25, 2016 at 09:02:50PM -0700, Sean Whitton wrote:
> Package: git-buildpackage
> Version: 0.7.1
> Severity: wishlist
> 
> Dear Maintainer,
> 
> When uscan is used to generate an upstream tarball, it filters out files
> listed in the Files-Excluded: section of the debian/copyright file and
> appends +dfsg to the tarball's version.  This can then be imported to
> the packaging repository with gbp import-orig.
> 
> Nowadays, many upstreams do not ship tarballs.  When preparing the
> Debian packaging for such a project, it is convenient to simply clone
> upstream's git repository and add a branch to that for Debian
> packaging.
> 
> Suppose that upstream tags a release 1.2.3 in the git repository.  The
> Debian package maintainer must then manually delete the files listed in
> Files-Excluded: and then tag a new version 1.2.3+dfsg.  gbp can then
> create a tarball of this tag and commit it to the pristine-tar branch.

I would rather do this with a dfsg-clean branch. You delete once and
then use git tools from there on.

> It would be great if gbp could produce the 1.2.3+dfsg tag itself by
> reading debian/copyright and excluding the Files-Excluded: files.

If somebody comes up with a clean patch I'm happy to merge that.

Cheers,
 -- Guido