Re: PAPT git migration

[Daniel Stender]

On 31.05.2017 20:58, Stefano Rivera wrote:
> It's been a while, but I found some time and energy at PyCon, to work
> our SVN->Git migration.
> 
> I started with our DPMT migration script, pulled out the git-dpm bits,
> and fixed the obvious issues I saw.
> 
> Here's what the migration currently looks like [0].
> 
> [0]: https://anonscm.debian.org/cgit/users/stefanor/python-apps/
> 
> This is after several rounds of migration, and fixes to both the SVN
> repo, and the migration script. Below are the notes I made during the
> rounds of migration. I haven't re-reviewed every package, after fixing
> it, only a couple of spot checks. The first round was ta revision 13997,
> and the final one (published at [0]) was at revision 14089.
> 
> And this is the migration script:
> https://anonscm.debian.org/cgit/users/stefanor/dpmt-migration.git/
> 
> Does that look reasonable? Should we do this?
> 
> SR

+1

pybtex has been moved to Debian Science.

BTW, the fix of #833469 from svn-buildpackage (copy .asc file together with 
tar) is
needed to avoid problems with the error message 
"orig-tarball-missing-upstream-signature"
from Lintian 2.5.52.

DS

-- 
4096R/DF5182C8 (sten...@debian.org)
http://www.danielstender.com/

Re: PAPT git migration

[Jordan Justen]

On 2017-05-31 11:58:15, Stefano Rivera wrote:
> It's been a while, but I found some time and energy at PyCon, to work
> our SVN->Git migration.
> 
> I started with our DPMT migration script, pulled out the git-dpm bits,
> and fixed the obvious issues I saw.
> 
> Here's what the migration currently looks like [0].
> 
> [0]: https://anonscm.debian.org/cgit/users/stefanor/python-apps/
> 
> This is after several rounds of migration, and fixes to both the SVN
> repo, and the migration script. Below are the notes I made during the
> rounds of migration. I haven't re-reviewed every package, after fixing
> it, only a couple of spot checks. The first round was ta revision 13997,
> and the final one (published at [0]) was at revision 14089.
> 
> And this is the migration script:
> https://anonscm.debian.org/cgit/users/stefanor/dpmt-migration.git/

Have you tried using --use-log-author? This pulls the author from the
log message (Signed-off-by).

Could you add me to identity-map:

jljusten-guest = Jordan Justen 

> Does that look reasonable?

The results look good for the 'alot' package.

> Should we do this?

Yes! :)

-Jordan


signature.asc
Description: signature

Re: PAPT git migration

[Mattia Rizzolo]

On Fri, Jun 02, 2017 at 04:23:20PM +0200, Raphael Hertzog wrote:
> Hi,
> 
> On Thu, 01 Jun 2017, Brian May wrote:
> > On 2017-06-01 07:32, Barry Warsaw wrote:
> > 
> > > Let's assume not (which is fine with me; i.e. why adopt git-dpm for PAPT 
> > > when
> > > we know we just want to get rid of it later?).  Then i tried to import a 
> > > new
> > > upstream as described here https://wiki.debian.org/Python/GitPackagingPQ
> > 
> > Another problem I have noticed with that document - but not had time to
> > come up with a solution yet. 
> > 
> > There are several places where "git push --all" (or variant) is
> > recommended. But this will also push the gbp pq branch(es), which should
> > never get pushed.
> 
> "git push origin : --tags" is what I use in general, it pushes branches
> available on both sides and tags.


I have recently discovered the config push.followTags=true, which pushes
all tags which reference objects that are being pushed.
That should save the '--tags' flag :)

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: PAPT git migration

[Simon McVittie]

On Thu, 01 Jun 2017 at 15:06:07 +1000, Brian May wrote:
> So to me it looks like the required changes are:
>  
> * Rename Author field to From. Ensure it is first field.

It doesn't *have* to be the first, but if it isn't, gbp pq export will
re-order it.

> * Add Date field. Set to what?

The date the change was made, by whatever definition seems most appropriate
(conveniently, the format is the same as in debian/changelog). This would
be used as the author date if the patch gets sent upstream (to a git user).

Strictly speaking this isn't mandatory, but if it's missing, gbp pq import
will assume the current date/time.

> The Applied-upstream field looks nice to have, but maybe not essential.

Nothing except From and Subject is mandatory. I only mentioned the
other fields to illustrate that gbp pq wants to see them in a
pseudo-header at the end (next to Signed-off-by if you use that),
rather than in the email header.

S

Re: PAPT git migration

[Brian May]

On 2017-06-01 09:16, Simon McVittie wrote:

> but this style (which is a DEP3 invention, and not used outside Debian and its
> derivatives) is not:
> 
> Author: ...
> Description: First line of description
> More description
> more description
> yet more description
> Bug-Debian: ...

Might be good to know how many of our packages have patches that use
this format too.

Re: PAPT git migration

[Brian May]

On 2017-06-01 09:16, Simon McVittie wrote:

> From: ...
> Date: ...
> Subject: First line of description
> 
> More description
> more description
> yet more description
> 
> Bug-Debian: ...
> Applied-upstream: ...
> More-DEP3-fields-in-pseudo-header: ...
> ---
> optional diffstat here
> 
> diff --git ...
> ...
> 
> but this style (which is a DEP3 invention, and not used outside Debian and its
> derivatives) is not:
> 
> Author: ...
> Description: First line of description
> More description
> more description
> yet more description
> Bug-Debian: ...

So to me it looks like the required changes are: 

* Rename Author field to From. Ensure it is first field. 
* Add Date field. Set to what? 
* Rename Description to Subject. 
* Ensure blank line before description. 
* Remove space at start of every line in description. 
* Ensure blank line after description. 

The Applied-upstream field looks nice to have, but maybe not essential. 

Is this correct? Did I miss anything? 

A tool to automate some, if not all of this, would be good.

Re: PAPT git migration

[Stefano Rivera]

Hi Scott (2017.06.01_05:01:00_+0200)
> As long as we have one team solution for this that is documented I
> don't care what it is as long as it works. I don't think we should
> have two approaches.  Let's pick one and use it.  Team members should
> not be surprised when they look at a team git repo.

I don't think anyone is advocating two approaches. But we don't need to
enforce the use of pq. We have git repos that are compatible with the
use of pq, and you can use it to manage your quilt series, if you wish.
The serialized quilt series is what we have in the shared git repo.

The upside of not doing a once-off migration to pq-friendly patch
metadata is that the git migration will cause less churn on source
packages. That patch metadata churn can be delayed until the next
upstream release is imported.

The downside is that it's up to the maintainer to notice and re-format
their patch metadata, the first time they want to use pq, after the
migration.

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

Re: PAPT git migration

[Scott Kitterman]

On May 31, 2017 7:16:39 PM EDT, Simon McVittie  wrote:
>On Thu, 01 Jun 2017 at 00:16:45 +0200, Stefano Rivera wrote:
>> Hi Barry (2017.05.31_23:32:20_+0200)
>> > $ gbp pq export
>> > - This doesn't work until you at least do a first pq import, but
>now I see the
>> >   d/p/changlog-docs patch gets changed in ways that lose
>information:
>> 
>> Sounds like a limitation of pq import. I'm suprised it doesn't
>support
>> DEP3.
>
>DEP3 describes some metadata fields, and a family of incompatible
>formats
>(which do not necessarily seem to be designed for machine-readability)
>that
>use those metadata fields.
>
>gbp pq import consumes patches in `git format-patch` format. One of the
>possible formats in the DEP3 family (with RFC2822-style
>From/Date/Subject,
>unindented long description, and all non-email fields in a trailing
>pseudo-header similar to common practice for Signed-off-by) is
>compatible
>with that:
>
>From: ...
>Date: ...
>Subject: First line of description
>
>More description
>more description
>yet more description
>
>Bug-Debian: ...
>Applied-upstream: ...
>More-DEP3-fields-in-pseudo-header: ...
>---
> optional diffstat here
>
>diff --git ...
>...
>
>but this style (which is a DEP3 invention, and not used outside Debian
>and its
>derivatives) is not:
>
>Author: ...
>Description: First line of description
> More description
> more description
> yet more description
>Bug-Debian: ...
>
>Converting the latter into the former seems like a valid gbp pq feature
>request, but might not be practically feasible (detecting that style
>mechanically so that it can be parsed is probably not trivial).
>
>For packages that are maintained in git, either upstream or downstream,
>preferring the former format makes a lot of sense IMO. Anything that
>relies
>on round-tripping patches through git like gbp pq does is going to want
>git-format-patch-compatible patches.
>
>> So, our options are:
>> 1. fix pq
>> 2. modify all the patches to a format that pq understands
>> 3. leave this to the maintainer to resolve (I think we expect all pq
>use
>>to be entirely local, so pq use isn't something we're imposing on
>>anyone)
>
>I think it would make sense to leave the patches as-is during initial
>conversion, and expect maintainers who are interested in using gbp pq
>to resolve this when they import and re-export the patch series.
>Hopefully
>a lot of current patches will become unnecessary with newer upstream
>software versions, so preferring git format-patch style for new patches
>might be a good "90%" solution.
>
>S

As long as we have one team solution for this that is documented I don't care 
what it is as long as it works. I don't think we should have two approaches.  
Let's pick one and use it.  Team members should not be surprised when they look 
at a team git repo.

Scott K

Re: PAPT git migration

[Brian May]

On 2017-06-01 07:32, Barry Warsaw wrote:

> Let's assume not (which is fine with me; i.e. why adopt git-dpm for PAPT when
> we know we just want to get rid of it later?).  Then i tried to import a new
> upstream as described here https://wiki.debian.org/Python/GitPackagingPQ

Another problem I have noticed with that document - but not had time to
come up with a solution yet. 

There are several places where "git push --all" (or variant) is
recommended. But this will also push the gbp pq branch(es), which should
never get pushed.

Re: PAPT git migration

[Simon McVittie]

On Thu, 01 Jun 2017 at 00:16:45 +0200, Stefano Rivera wrote:
> Hi Barry (2017.05.31_23:32:20_+0200)
> > $ gbp pq export
> > - This doesn't work until you at least do a first pq import, but now I see 
> > the
> >   d/p/changlog-docs patch gets changed in ways that lose information:
> 
> Sounds like a limitation of pq import. I'm suprised it doesn't support
> DEP3.

DEP3 describes some metadata fields, and a family of incompatible formats
(which do not necessarily seem to be designed for machine-readability) that
use those metadata fields.

gbp pq import consumes patches in `git format-patch` format. One of the
possible formats in the DEP3 family (with RFC2822-style From/Date/Subject,
unindented long description, and all non-email fields in a trailing
pseudo-header similar to common practice for Signed-off-by) is compatible
with that:

From: ...
Date: ...
Subject: First line of description

More description
more description
yet more description

Bug-Debian: ...
Applied-upstream: ...
More-DEP3-fields-in-pseudo-header: ...
---
 optional diffstat here

diff --git ...
...

but this style (which is a DEP3 invention, and not used outside Debian and its
derivatives) is not:

Author: ...
Description: First line of description
 More description
 more description
 yet more description
Bug-Debian: ...

Converting the latter into the former seems like a valid gbp pq feature
request, but might not be practically feasible (detecting that style
mechanically so that it can be parsed is probably not trivial).

For packages that are maintained in git, either upstream or downstream,
preferring the former format makes a lot of sense IMO. Anything that relies
on round-tripping patches through git like gbp pq does is going to want
git-format-patch-compatible patches.

> So, our options are:
> 1. fix pq
> 2. modify all the patches to a format that pq understands
> 3. leave this to the maintainer to resolve (I think we expect all pq use
>to be entirely local, so pq use isn't something we're imposing on
>anyone)

I think it would make sense to leave the patches as-is during initial
conversion, and expect maintainers who are interested in using gbp pq
to resolve this when they import and re-export the patch series. Hopefully
a lot of current patches will become unnecessary with newer upstream
software versions, so preferring git format-patch style for new patches
might be a good "90%" solution.

S

Re: PAPT git migration

[Mattia Rizzolo]

On Thu, Jun 01, 2017 at 12:16:45AM +0200, Stefano Rivera wrote:
> > $ gbp pq export
> > - This doesn't work until you at least do a first pq import, but now I see 
> > the
> >   d/p/changlog-docs patch gets changed in ways that lose information:
> 
> Sounds like a limitation of pq import. I'm suprised it doesn't support
> DEP3. That was something that dpm got right (once we figured out the
> right parameters).

Yes it's a gbp pq bug: https://bugs.debian.org/785274
The support improved over time in the last years, but it's still not
great, as that bug documents.

> 3. leave this to the maintainer to resolve (I think we expect all pq use
>to be entirely local, so pq use isn't something we're imposing on
>anyone)

I would totally welcome this.  Really with gbp the use of pq is not
mandatory at all, as after all the output are always quit patches.  Sure
if somebody uses pq and somebody else doesn't it would end up with some
diff noise due to the import/export, but imho nothing worth concerning
about.

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: PAPT git migration

[Stefano Rivera]

Hi W. (2017.05.31_22:54:10_+0200)
> > * trac-batchmodify: OK
> > * trac-git: missing a tag [DONE]
> 
> Those two are only in oldstable.
> Not sure whether it is worth to migrate them at all.

Yeah, let's ditch them entirely. I picked up on some deleted packages
during my review, when the history was weird. But didn't systematically
look for them.

> > * trac-privateticketsplugin: missing some tags [DONE]
> 
> This git repository should probably renamed to
> trac-privatetickets.

Added rules for all of this in
https://anonscm.debian.org/cgit/users/stefanor/dpmt-migration.git/commit/?id=b746cddd2af2f6b6b24e74866ab8bccd45f1aedc

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

Re: PAPT git migration

[Stefano Rivera]

Hi Barry (2017.05.31_23:32:20_+0200)
> I did a spot check on twine.  I don't see a debian/.git-dpm file on master so
> git-dpm won't work.  Is it supposed to?
> Let's assume not (which is fine with me; i.e. why adopt git-dpm for PAPT when
> we know we just want to get rid of it later?).

Yep, no git-dpm. For those reasons.

> upstream as described here https://wiki.debian.org/Python/GitPackagingPQ
> 
> $ gbp pq export
> - This doesn't work until you at least do a first pq import, but now I see the
>   d/p/changlog-docs patch gets changed in ways that lose information:

Sounds like a limitation of pq import. I'm suprised it doesn't support
DEP3. That was something that dpm got right (once we figured out the
right parameters).

So, our options are:
1. fix pq
2. modify all the patches to a format that pq understands
3. leave this to the maintainer to resolve (I think we expect all pq use
   to be entirely local, so pq use isn't something we're imposing on
   anyone)

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

Re: PAPT git migration

[Barry Warsaw]

On May 31, 2017, at 08:58 PM, Stefano Rivera wrote:

>It's been a while, but I found some time and energy at PyCon, to work
>our SVN->Git migration.

Thanks for all your great work on this Stefano!

>Here's what the migration currently looks like [0].
>[0]: https://anonscm.debian.org/cgit/users/stefanor/python-apps/

I did a spot check on twine.  I don't see a debian/.git-dpm file on master so
git-dpm won't work.  Is it supposed to?

Let's assume not (which is fine with me; i.e. why adopt git-dpm for PAPT when
we know we just want to get rid of it later?).  Then i tried to import a new
upstream as described here https://wiki.debian.org/Python/GitPackagingPQ

$ gbp pq export
- This doesn't work until you at least do a first pq import, but now I see the
  d/p/changlog-docs patch gets changed in ways that lose information:

modified   debian/patches/changelog-docs
@@ -1,14 +1,17 @@
-Description: link changelog.rst from index.rst
- The generated documentation is otherwise useless so let's just link the
- changelog for now. Upstream is aware of this issue and will fix it with the
- next release.
-Bug: https://github.com/dstufft/twine/issues/46
-Forwarded: yes
-Last-Update: 2014-04-20
+From: Python Applications Packaging Team
+ 
+Date: Wed, 31 May 2017 17:21:14 -0400
+Subject: changelog-docs
 
+---
+ docs/index.rst | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/docs/index.rst b/docs/index.rst
+index 69d1847..94c5ffb 100644
 --- a/docs/index.rst
 +++ b/docs/index.rst
-@@ -11,6 +11,7 @@
+@@ -11,6 +11,7 @@ Contents:
  .. toctree::
 :maxdepth: 2

Other than that, `gbp import-orig --pristine-tar --uscan` seems to work well,
and the resulting repo produces a good package.

Cheers,
-Barry

Re: PAPT git migration

[W. Martin Borgert]

Many thanks, Stefano, for your work on this!

On 2017-05-31 20:58, Stefano Rivera wrote:
> * trac-batchmodify: OK
> * trac-git: missing a tag [DONE]

Those two are only in oldstable.
Not sure whether it is worth to migrate them at all.

> * trac-privateticketsplugin: missing some tags [DONE]

This git repository should probably renamed to
trac-privatetickets.

Cheers

PAPT git migration

[Stefano Rivera]

It's been a while, but I found some time and energy at PyCon, to work
our SVN->Git migration.

I started with our DPMT migration script, pulled out the git-dpm bits,
and fixed the obvious issues I saw.

Here's what the migration currently looks like [0].

[0]: https://anonscm.debian.org/cgit/users/stefanor/python-apps/

This is after several rounds of migration, and fixes to both the SVN
repo, and the migration script. Below are the notes I made during the
rounds of migration. I haven't re-reviewed every package, after fixing
it, only a couple of spot checks. The first round was ta revision 13997,
and the final one (published at [0]) was at revision 14089.

And this is the migration script:
https://anonscm.debian.org/cgit/users/stefanor/dpmt-migration.git/

Does that look reasonable? Should we do this?

SR

== Package notes ==

* alienfeed: SVN-only
* alot: needs a tag [DONE]
* andvare: Never uploaded. [DELETED FROM SVN]
* apachedex: LGTM
* archivemail: LGTM (missing a recent NMU)
* archmage: needs a tag [DONE]
* atheist: LGTM (missing a recent NMU)
* autokey: LGTM
* awscli: OK
* babiloo: OK
* backupchecker: needs a tag [DONE]
* beets: OK
* belier: needs a tag [DONE]
* bjsonrpc: OK
* bleachbit: OK
* bluewho: OK
* brebis: most recent upload isn't is SVN, but who cares
* bundlewrap: debian tags are missing debian revision [DONE]
* canto: OK
* catfish: OK
* cherrytree: pristine-tar failed to import an orig-tarball [FIXED]
* cloud-init: moved to cloud team [DELETED FROM SVN]
* clustershell: needs a tag [DONE]
* clvault: Never uploaded. [DELETED FROM SVN]
* convertall: OK
* crudini: OK
* cycle: tag names are incorrect [DONE]
* cython: needs a tag [DONE]
* db2twitter: OK
* debomatic: OK
* decibel-audio-player: orphaned [DELETED FROM SVN]
* diamond: OK
* didjvu: OK
* discus: orphaned [DELETED FROM SVN]
* disper: OK
* djvusmooth: OK 
* doconce: OK
* dodgy: OK
* dot2tex: OK
* dreampie: OK
* driconf: OK (but no history)
* drobo-utils: OK
* drpython: new maintainer [DELETED FROM SVN]
* emesene: OK
* emma: OK
* episoder: OK
* eric: OK
* etm: OK
* etm-qt: OK 
* fookebox: OK
* freealchemist: OK
* freevial: needs a tag [DONE]
* frescobaldi: OK
* fusion-icon: OK
* gaupol: OK
* gedit-classbrowser3g: never uploaded [DELETED FROM SVN]
* gedit-projects: never uploaded [DELETED FROM SVN]
* gespeaker: OK
* gextractwinicons: OK
* ghp-import: OK
* git-bzr-ng: RMed [DELETED FROM SVN]
* gitinspector: OK
* glipper: OK
* gmail-notify: OK
* gmobilemedia: OK
* go2: OK
* google-sitemapgen: OK
* gpxviewer: OK
* gtg: OK
* gtimelog: left PAPT [DELETED FROM SVN]
* gui-ufw: OK
* gwakeonlan: OK
* harvestman: OK
* hellanzb: RMed [DELETED FROM SVN]
* hg-git: OK (although no human maintainers)
* hotssh: OK
* hovercraft: OK
* httpbin: OK
* ibid: OK
* indywiki: OK
* irker: OK
* isort: OK
* itstool: needs a tag [DONE]
* kabikaboo: OK
* kapidox: Left PAPT [DELETED FROM SVN]
* khal: recent uploads weren't committed to svn
* kupfer: needs a tag [DONE]
* kyklop: never uploaded
* lazygal: OK
* legit: OK
* lightyears: most recent upload wasn't committed to SVN
* lottanzb: OK
* lunch: OK
* marave: never uploaded [DELETED FROM SVN]
* mayavi2: OK
* menulibre: OK
* mercurial: OK
* mimms: OK (recent NMU hasn't been committed yet)
* mitter: never uploaded [DELETED FROM SVN]
* moap: OK (recent NMU hasn't been committed yet)
* mpdris: OK
* mugshot: OK
* mypaint: OK
* nagstamon: OK
* nfoview: OK
* ocrfeeder: missing tags [DONE]
* ocrodjvu: OK
* oggconvert: OK
* openerp6: SVN-only (never been uploaded)
* openstv: OK
* openteacher: OK
* osc: Moved to RPM team [DELETED FROM SVN]
* osc-source_validator: RMed [DELETED FROM SVN]
* pdfposter: missing tag [DONE]
* pdfshuffler: OK
* pelican: OK
* pep257: RMed [DELETED FROM SVN]
* petit: OK
* phantom: never uploaded [DELETED FROM SVN]
* phatch: recent svn history is a bit detached, but otherwise OK
* phenny: OK
* photo-uploader: orphaned [DELETED FROM SVN]
* picard: OK
* pkpgcounter: OK
* plainbox-provider-checkbox: OK
* plainbox-provider-piglit: never uploaded
* plainbox-provider-resource-generic: OK
* planet-venus: OK
* policyd-rate-limit: backports got sorta mangled, but OK
* pondus: OK
* preprocess: OK
* prospector: OK
* ptex2tex: OK
* puddletag: OK
* pyaimt: OK
* pybackpack: OK
* pybik: OK
* pyblosxom: left PAPT [DELETED FROM SVN]
* pybtex: OK
* pycarddav: OK
* pychecker: OK
* pydocstyle: OK
* pyflakes: OK
* pygccxml: left PAPT [DELETED FROM SVN]
* py++: Never uploaded [DELETED FROM SVN]
* pyicqt: OK
* pyinfra: OK
* pykaraoke: OK
* pyksoko: Never uploaded [DELETED FROM SVN]
* pylint: OK
* pymilter-milters: OK
* pynagram: OK
* pyneighborhood: Orphaned [DELETED FROM SVN]
* pyomo: most recent upload wasn't in SVN, but otherwise, OK
* pypar2: OK
* pype: missing tag [DONE]
* pypolicyd-spf: OK
* pyrit: OK
* pyspread: missing tag [DONE] most recent release wasn't committed to SVN
* pytagsfs: missing tag [DONE]
* python-afl: OK
* pythoncad: OK
* python-clamav: in DPMT, not PAPT
* python-setproctitle: in DPMT, not PAPT
*